We reveal the impact of magnetic ordering on stacking fault energy (SFE) and its influence on the deformation mechanisms and mechanical properties in a class of nonequiatomic quinary Mn-containing compositional complex alloys or high entropy alloys (HEAs). By combining ab initio simulation and experimental validation, we demonstrate magnetic ordering as an important factor in the activation and transition of deformation modes from planar dislocation slip to TWIP (twinning-induced plasticity) and/or TRIP (transformation-induced plasticity). A wide compositional space of Cr20MnxFeyCo20Niz(x+y+z=60, at. %) was probed by density-functional theory calculations to search for potential alloys displaying the TWIP/TRIP effects. Three selected promising HEA compositions with varying Mn concentrations were metallurgically synthesized, processed, and probed for microstructure, deformation mechanism, and mechanical property evaluation. The differences in the deformation modes of the probed HEAs are interpreted in terms of the computed SFEs and their dependence on the predicted magnetic state, as revealed by ab initio calculations and validated by explicit magnetic measurements. It is found that the Mn content plays a key role in the stabilization of antiferromagnetic configurations which strongly impact the SFEs and eventually lead to the prevalent deformation behavior.