Open microcavities offer great potential for the exploration and utilization of efficient spin-photon interfaces with Purcell-enhanced quantum emitters thanks to their large spectral and spatial tunability combined with high versatility of sample integration. However, a major challenge for this platform is the sensitivity to cavity length fluctuations in the cryogenic environment, which leads to cavity resonance frequency variations and thereby a lowered averaged Purcell enhancement. This work presents a closed-cycle cryogenic fiber-based microcavity setup, which is in particular designed for a low passive vibration level, while still providing large tunability and flexibility in fiber and sample integration, and high photon collection efficiency from the cavity mode. At temperatures below 10 K, a stability level of around 25 pm is reproducibly achieved in different setup configurations, including the extension with microwave control for manipulating the spin of cavity-coupled quantum emitters, enabling a bright photonic interface with optically active qubits.