This research investigates the application of different deep reinforcement learning methods for optimizing traffic light control in multi-modal urban traffic environments using the SUMO traffic simulator. Urban traffic congestion, with its significant economic, environmental, and social impacts, necessitates more sophisticated control strategies that can adapt to varying traffic conditions. Traditional traffic control systems, like fixed-time and adaptive methods, are often insufficient in handling the complexity of multi-modal traffic, which includes various traffic modes such as passenger cars and buses. Deep reinforcement learning, with its ability to dynamically optimize traffic light control without requiring prior knowledge of traffic patterns, is a promising method to improve traffic efficiency and achieve transit priority in multi-modal traffic. The research aims to address the limitations of the existing relevant research by employing several deep reinforcement learning algorithms, particularly multi-agent deep reinforcement learning methods, to coordinate multiple traffic lights in SUMO simulation. Research experiments are conducted in three different cases, which are set in road networks of different sizes respectively, and fixed traffic light control and max-pressure traffic light control are implemented for comparison. The applied deep reinforcement methods are evaluated in terms of training process and model evaluation. And the research results demonstrate that deep reinforcement learning methods, especially multi-agent deep reinforcement learning methods, can significantly enhance traffic flow efficiency and achieve transit priority in complex urban settings in multi modal simulation.

Connected and Automated Vehicles (CAVs) have the poten- tial to revolutionize transportation systems, but their integration with human-driven vehicles (HDVs) in mixed traffic environments presents significant challenges, particularly in complex scenarios such as on-ramp merging. This paper addresses the challenge of on-ramp merging for CAVs in mixed traffic environments, proposing a novel approach called QMIX-QLambdaM. We formulate the problem as a Centralized Train- ing with Decentralized Execution (CTDE) Cooperative Multi-Agent Re- inforcement Learning (MARL) task, capable of handling dynamic sce- narios with both CAVs and HDVs. QMIX-QLambdaM enhances the QMIX algorithm by incorporating Q(λ) returns for improved value es- timation and an action masking mechanism for safer action selection. Our comprehensive experiments demonstrate that QMIX-QLambdaM consistently outperforms state-of-the-art algorithms, including QMIX, MAA2C, and COMA, across various performance metrics related to traf- fic efficiency and safety. The proposed method exhibits superior adapt- ability across different traffic densities, maintaining high performance in terms of safety, efficiency, and overall rewards. Furthermore, case stud- ies illustrate QMIX-QLambdaM’s ability to generate effective strate- gic control for both main-lane and merging-lane vehicles, showcasing smoother driving behavior and better collision avoidance compared to baseline methods. The learning curve comparison also reveals QMIX- QLambdaM’s advantage in credit assignment compared to other CTDE baselines for the formulated problem. The code are available at https: //github.com/ayton-zhang/MARL_qmix_merging.