Shuttling spins with high fidelity is a key requirement to scale up semiconducting quantum computers, enabling qubit entanglement over large distances and favoring the integration of control electronics on-chip. To decouple the spin from the unavoidable charge noise, state-of-the-art spin shuttlers try to minimize the inhomogeneity of the Zeeman field. However, this decoupling is challenging in otherwise promising quantum computing platforms such as hole spin qubits in silicon and germanium, characterized by a large spin-orbit interaction and an electrically tunable qubit frequency. In this work, we show that, surprisingly, the large inhomogeneity of the Zeeman field stabilizes the coherence of a moving spin state, thus also enabling high-fidelity shuttling in these systems. We relate this enhancement in fidelity to the deterministic dynamics of the spin that filters out the dominant low-frequency contributions of the charge noise. By simulating several different scenarios and noise sources, we show that this is a robust phenomenon generally occurring at large field inhomogeneity. By appropriately adjusting the motion of the quantum dot, we also design realistic protocols enabling faster and more coherent spin shuttling. Our findings are generally applicable to a wide range of setups and could pave the way toward large-scale quantum processors.