Localization microscopy offers resolutions down to a single nanometer but currently requires additional dedicated hardware or fiducial markers to reduce resolution loss from the drift of the sample. Drift estimation without fiducial markers is typically implemented using redundant cross correlation (RCC). We show that RCC has sub-optimal precision and bias, which leaves room for improvement. Here, we minimize a bound on the entropy of the obtained localizations to efficiently compute a precise drift estimate. Within practical compute-time constraints, simulations show a 5x improvement in drift estimation precision over the widely used RCC algorithm. The algorithm operates directly on fluorophore localizations and is tested on simulated and experimental datasets in 2D and 3D. An open source implementation is provided, implemented in Python and C++, and can utilize a GPU if available.