Many recent robot learning problems, real and simulated, were addressed using deep reinforcement learning. The developed policies can deal with high-dimensional, continuous state and action spaces, and can also incorporate machine-generated or human demonstration data. A great number of them depend on state-action value estimates, especially the ones in the actor-critic framework. Deriving unbiased estimates for these values is still an open research question, mostly since the connection between accurate value estimates and system performance is not yet well-understood. This thesis work has three main research contributions. Firstly, it analyzes the connection between value estimates and performance for the TD3 algorithm. Secondly, it derives theoretical bounds for the true value function when dealing with environments where a reward is only given for successful completion of a task (sparse/binary reward). Lastly, a deliberate underestimation objective is added to the TD3 algorithm together with the theoretical bounds to improve system performance when using human demonstration data that only covers a specific part of the state and action space. All the algorithms are tested and evaluated using simulated robot manipulation tasks in the robosuite environment, where the robot is first trained on the demonstration data and then can gather more experiences in the simulation. Results show that the deliberate underestimation together with the value bounds enable the robot to learn from human demonstration, which was not possible for the standard TD3. Additionally, applying just the value bounds speeds up the learning process when using machine-generated datasets.

The increased diffusion of Renewable Energy Sources (RES) into energy distribution systems gives rise to a number of issues that the Distribution Network Operators (DNOs) need to face. The dependency of RES on weather, along with their intermittent and non-dispatchable nature, urges us to develop frameworks that can guarantee a stable network, despite the power fluctuations.
In this thesis, we develop a Data-Driven Optimal Power Flow (OPF) formulation in order to achieve voltage regulation against operational problems that may occur because of the high PV penetration into energy distribution systems. The unpredictable PV generation deteriorates the system’s reliability and introduces instability. Thus, we develop appropriate formulations to define a limit on exports from residential PV owners to the grid. For this purpose, we employ Distribunally Robust Chance Constraint Programming (DRCCP), as a method that can handle constraints that depend on the uncertain PV generation and
residential demand. We capture the distributional uncertainties with an ambiguity set and we utilize the Wasserstein metric to parameterize the range of this set.
We divide the thesis into two case studies. In the first case study, we perform the DRCCP optimization with only few recorded data. We evaluate the results of our algorithm in a dataset of 500 days of recorded data and we achieve to reduce significantly the overvoltage instances. We tune the DRCCP in a way that we achieve at least 95% satisfaction of constraints that emerge from the DRCCP, avoiding overconservative solutions and high generation cuts that
could harm the PV owners.
In the second case study, we intend to exploit plenty of historical data, but the computational burden hinders us from using them in the DRCCP. Therefore, we employ Wasserstein Barycenters, and we utilize the Wasserstein distance in order to cluster the data. With Wasserstein Barycenters we reduce significantly the running time and we provide an efficient and robust output, for at least 95% satisfaction of the model’s constraints.

Hydrofoil crafts with fully submerged foils can provide fast and economical waterway transport. However, their operation requires reliable onboard control systems to ensure the safety and comfort of their passengers, especially in rough sea conditions. This thesis project is focused on the dynamical modelling and the design of motion control systems for an experimental scale hydrofoil craft that is available at TU Delft, namely the Hydrofoil Education and Research Platform (HEARP).

The development of the dynamical model of HEARP is done by taking inspiration from the dynamics of marine crafts and aircraft and relying on different assumptions to obtain a simple and low-order model. The resulting model is a linearized state-space model with three degrees of freedom, namely heave, roll, and pitch, and includes the influence of regular waves. Because of inaccurate available data for the mass properties of HEARP, variations of the system parameters due to nonlinearities, and changes in the operating conditions, different uncertainties are assigned to most system parameters.

The use of multivariable feedback control methods for the motion control of hydrofoil crafts is limited, so this work is focused on exploiting such methods to improve the performance and robustness of such systems. The representation of the perturbed system using real parametric uncertainties is proved to be computationally expensive for the control design. Thus, the perturbed system is approximated by complex (dynamic) perturbations. A signal-based H-infinity optimal controller is designed using the nominal system, and a mu-synthesis optimal robust controller is designed using the approximated perturbed system.

The performance and robustness of the proposed controllers are evaluated in both frequency and time domains through simulations. From the results, it is concluded that both controllers offer high-performance system responses for both reference tracking and disturbance rejection of incident waves. Furthermore, by comparing the two controllers, it is observed that the mu-synthesis controller shows superior robustness for the modelled uncertainty. In contrast, the H-infinity controller has a slightly better performance when considering the perturbed systems with the real parametric uncertainty. The results of this thesis project can be used in the future to experimentally validate the accuracy of the proposed dynamical model and the performance of the designed controllers.

There is growing interest to control cyber-physical systems under complex specifications while retaining formal performance guarantees. In this thesis we present a framework for formal control of uncertain systems under complex specifications. We consider dynamical systems with random disturbances, whose probability distribution is unknown. When it comes to the specifications, we focus on those given as syntactically co-safe linear temporal logic (scLTL) formulas. Such formulas resemble natural language, and allow us to reason over complex behaviours of the system.

We follow an abstraction-based approach: we abstract the original system to a finite-state Markov model, in which the state discretization error as well as the distributional ambiguity, are embedded as uncertainties. To do so we make use of tools from optimal transport and ambiguity sets of probability distributions. After that, we synthesize a strategy for the abstraction and obtain probabilistic guarantees that the abstraction satisfies the specifications. Finally, we correctly refine the strategy to one that the original system can follow, and prove that the guarantees we obtained for the abstraction also hold for the original system.

We propose two approaches to obtain the abstraction. First, we propose a data-driven approach to abstract the system into an interval Markov decision process (IMDP) when samples from the unknown distribution is available. Then we use already existing algorithms to synthesize a strategy for the IMDP. Secondly, we propose an approach to abstract the original system into a robust Markov decision process (robust MDP). This second approach is applicable to more general uncertainty models besides the data-driven one, and reduces conservatism of the abstraction. Furthermore, we propose an algorithm to synthesize robust strategies for robust MDPs, which also renders the guarantees that the abstraction satisfies the specifications. Finally, we demonstrate the usefulness of our proposed approaches through several case studies that involve both linear and nonlinear systems.

Vehicles are becoming increasingly complex and the resource-demanding due to the developments aimed at driving autonomy. Viewing an automobile as a cyber-physical system (CPS), this work applies the advances of controls applied to CPS for improving resource-consumption. One such idea is that of event-based control. Theoretical developments in event-based control show promising results, but they require practical corroboration. Combining these aspects, an event-triggered vehicle stability controller is designed and tested on a real-time hardware-in-loop setup.