Distributed model predictive control (DMPC) has become a focus in the energy management of shipboard power systems due to its capabilities for privacy preservation, robustness, and distributing computing burdens to local processors. DMPC determines control actions in a distributed manner based on the predictions of system statuses. However, the performance of DMPC is affected by inaccurate predictions resulting from uncertain parameters in nominal prediction models. Particularly, these inaccuracies in predicting propulsion loads and solar panel generation powers can lead to power imbalances when implementing the control actions determined by DMPC. To address this challenge, this paper proposed a novel reinforcement learning compensated DMPC (RL-C-DMPC) to distributively compensate for the control actions determined by DMPC baseline control, thereby rectifying the power imbalances caused by uncertain parameters in nominal prediction models. A value-decomposition-network-based training and distributed testing mechanism is designed for our proposed RL-C-DMPC. Furthermore, a method for range selection of compensation rate is specifically proposed for the energy management of shipboard power systems. To validate the effectiveness of our proposed RL-C-DMPC, we conduct a comprehensive case study utilizing real-life voyage data and historical solar power generation data in the area of the voyage to build the environment for training and testing. By comparing power imbalances between DMPC and RL-C-DMPC, our results indicate significant reductions in power imbalances so that frequency stability can be better ensured. Furthermore, via the case study, we also evaluate the communication robustness of RL-C-DMPC.@en

The railway timetable rescheduling problem is regarded as an efficient way to handle disturbances. Typically, it is tackled using a mixed integer linear programming (MILP) formulation. In this paper, an algorithm that combines both reinforcement learning and optimization is proposed to solve the railway timetable rescheduling problem. Specifically, a value-based reinforcement learning algorithm is implemented to determine the independent integer variables of the MILP problem. Then, the values of all the integer variables can be derived from these independent integer variables. With the solution for the integer variables, the MILP problem can be transformed into a linear programming problem, which can be solved efficiently. The simulation results show that the proposed method can reduce passenger delays compared with the baseline, while also reducing the solution time.

@en

Parameterized model predictive control (PMPC) is one of the many approaches that have been developed to alleviate the high computational requirement of model predictive control (MPC), and it has been shown to significantly reduce the computational complexity while providing comparable control performance with conventional MPC. However, PMPC methods still require a sufficiently accurate model to guarantee the control performance. To deal with model mismatches caused by the changing environment and by disturbances, this paper first proposes a novel framework that uses reinforcement learning (RL) to adapt all components of the PMPC scheme in an online way. More specifically, the novel framework integrates various strategies to adjust different components of PMPC (e.g., objective function, state-feedback control function, optimization settings, and system model), which results in a synthesis framework for RL-based adaptive PMPC. We show that existing adaptive (P)MPC approaches can also be embedded in this synthesis framework. The resulting combined RL-PMPC framework provides a solution for an efficient MPC approach that can deal with model mismatches. A case study is performed in which the framework is applied to freeway traffic control. Simulation results show that for the given case study the RL-based adaptive PMPC approach reduces computational complexity by 98% on average compared to conventional MPC while achieving better control performance than the other controllers, in the presence of model mismatches and disturbances.

@en

Model predictive control (MPC) and deep reinforcement learning (DRL) have been developed extensively as two independent techniques for traffic management. Although the features of MPC and DRL complement each other very well, few of the current studies consider combining these two methods for application in the field of freeway traffic control. This paper proposes a novel framework for integrating MPC and DRL methods for freeway traffic control that is different from existing MPC-(D)RL methods. Specifically, the proposed framework adopts a hierarchical structure, where a high-level efficient MPC component works at a low frequency to provide a baseline control input, while the DRL component works at a high frequency to modify online the output generated by MPC. The control framework, therefore, needs only limited online computational resources and is able to handle uncertainties and external disturbances after proper learning with enough training data. The proposed framework is implemented on a benchmark freeway network in order to coordinate ramp metering and variable speed limits, and the performance is compared with standard MPC and DRL approaches. The simulation results show that the proposed framework outperforms standalone MPC and DRL methods in terms of total time spent (TTS) and constraint satisfaction, despite model uncertainties and external disturbances.@en

While Model Predictive Control (MPC) is a promising approach for network-wide control of urban traffic, the computational complexity of the, often nonlinear, online optimization procedure is too high for real-time implementations. In order to make MPC computationally efficient, this paper introduces a parameterized MPC (PMPC) approach for urban traffic networks that uses Grammatical Evolution to construct continuous parameterized control laws using an effective simulation-based training framework. Furthermore, a projection-based method is proposed to remove the nonlinear constraints that are imposed on the parameters of the parameterized control laws and to guarantee the feasibility of the solution of the MPC optimization problem. The performance and computational efficiency of the constructed parameterized control laws are compared to those of a conventional MPC controller in an extensive simulation-based case study. The results show that the parameterized control laws, which are automatically constructed using Grammatical Evolution, decrease the computational complexity of the online optimization problem by more than 80% with a decrease in performance by less than 10%.

@en

Traffic control is essential to reduce congestion in both urban and freeway traffic networks. These control measures include ramp metering and variable speed limits for freeways, and traffic signal control for urban traffic. However, current traffic control methods are either too simple to respond to complex traffic environment, or too sophisticated for real-life implementation. In this paper, we propose an adaptive parameterized control method for traffic management by using reinforcement learning algorithms. This method takes advantage of the simple structure of parameterized state-feedback controllers for traffic; meanwhile, a reinforcement learning agent is employed to adjust the parameters of the controllers on-line to react to the varying environment. Therefore, the proposed method requires limited real-time computational efforts, and is adaptive to external disturbances. Furthermore, the reinforcement learning agent can coordinate multiple local traffic controllers when adjusting their parameters. The method is validated by a numerical case study on a freeway network. Results show that the proposed method outperforms conventional controllers when the system is exposed to a changing environment.@en

We propose a novel method to improve the convergence performance of model predictive control (MPC) for setpoint tracking, by introducing sub-references within a multilevel MPC structure. In some cases, MPC is implemented with a short prediction horizon due to limited on-line computation capacity, which could lead to deteriorated dynamic performance. The introduced multi-level optimization method can generate proper sub-references for the MPC setpoint tracking problem, and efficiently improve the dynamic performance. In the higher level a specific performance criterion is taken as the objective, while explicit MPC is utilized in the lower level to represent the control input. The generated sub-references are then used in MPC for the real system with prediction horizon restrictions. Setpoint-tracking MPC for linear systems is used to illustrate the approach throughout the paper. Numerical simulations show that MPC with sub-references significantly improves the convergence performance compared with regular MPC with the same prediction horizon. Thus, it can be concluded that MPC with sub-references has a high potential to tackle more complicated control problems with limited computation capacity.@en

Model predictive control (MPC) has been widely used for traffic management, such as for minimizing the total time spent or the total emissions of vehicles. When long-term green urban mobility is considered including e.g. a constraint on the total yearly emissions, the optimization horizon of the MPC problem is significantly larger than the control sampling time, and thus the number of the variables that should be optimized per control time step becomes very large. For systems with dynamics that involve nonlinear, non-convex, and non-smooth functions, including urban traffic networks, this results in optimization problems that are computationally intractable in real time. In this paper, we propose a novel bi-level temporal distribution of such complex MPC optimization problems, and we develop two mathematically linked short-term and long-term MPC formulations with small and large control sampling times that will be solved together instead of the original complex optimization problem. The resulting bi-level control architecture is used to solve the two MPC formulations online for real-time control of urban traffic networks with the objective of long-term green mobility. In order to assess the performance of the bi-level control architecture, we perform a case study where a rough version of the model of the urban traffic flow, S-model, is used by the long-term MPC level to estimate the states of the urban traffic networks, and a detailed version of the model is used by the short-term MPC level. The results of the simulations prove the effectiveness (with respect to the objective of control, as well as computational efficiency) of the proposed bi-level MPC approach, compared to state-of-the-art control approaches.@en

This thesis focuses on management and control of traffic networks, including urban networks and freeway networks, in which we aim to reduce traffic congestion by minimizing the total time spent of all the vehicles in the network, and also consider green mobility by minimizing the total emissions produced by the vehicles. In this thesis, we have addressed the challenges of model predictive control (MPC) for traffic management in terms of computational complexity and model mismatches by developing several novel MPC-based control frameworks for urban and freeway traffic networks. More specifically, several multi-level and learning-based MPC control frameworks are proposed. First, a novel bi-level temporally-distributed MPC framework is proposed to deal with the green urban mobility issue that usually involves long-term (e.g., one year) emission constraints, and is thus computationally intractable due to the large window of the problem. Second, we employ a grammatical evolution method to generate parameterized control laws for parameterized MPC (PMPC) with application to urban traffic signal control. Third, we develop a novel combined MPC- deep reinforcement learning (DRL) multi-level control framework, in which the MPC module provides a basic control performance at a lower frequency based on a prediction model, and the DRL module works at a higher frequency to compensate for the model mismatches and external disturbances through learning. Forth, we propose a synthesis framework of reinforcement learning (RL)-based adaptive PMPC. In this framework, all components of the PMPC scheme, such as the cost function, the prediction model, the control law, the constraint set, and the terminal set, can be parameterized and adjusted by a high-level RL agent.@en

In general, the performance of model-based controllers cannot be guaranteed under model uncertainties or disturbances, while learning-based controllers require an extensively sufficient training process to perform well. These issues especially hold for large-scale nonlinear systems such as urban traffic networks. In this paper, a new framework is proposed by combining model predictive control (MPC) and reinforcement learning (RL) to provide desired performance for urban traffic networks even during the learning process, despite model uncertainties and disturbances. MPC and RL complement each other very well, since MPC provides a sub-optimal and constraint-satisfying control input while RL provides adaptive control laws and can handle uncertainties and disturbances. The resulting combined framework is applied for traffic signal control (TSC) of an urban traffic network. A case study is carried out to compare the performance of the proposed framework and other baseline controllers. Results show that the proposed combined framework outperforms conventional control methods under system uncertainties, in terms of reducing traffic congestion.

@en