Efficient management of water resources is increasingly critical in the face of growing challenges such as climate change and population growth. This research paper introduces RL4Water, an adaptable framework for simulating water management systems using multi-objective reinforcement learning (MORL). Adhering to the Gymnasium API standard, RL4Water ensures seamless integration with existing MORL algorithms. The framework includes diverse facility classes to accurately model the physical components of water networks. Its generalizability is enhanced by allowing users to modify both the physical properties of these components and the key features of the MORL simulations. RL4Water's capabilities are demonstrated through two case studies: simulations of the Nile River and the Susquehanna River, validating its accuracy and flexibility in managing both large, distributed water systems and centralized systems with complex reservoirs. By bridging the gap between water management and reinforcement learning, RL4Water offers a unified platform for developing and researching water management simulations.