Many recent robot learning problems, real and simulated, were addressed using deep reinforcement learning. The developed policies can deal with high-dimensional, continuous state and action spaces, and can also incorporate machine-generated or human demonstration data. A great number of them depend on state-action value estimates, especially the ones in the actor-critic framework. Deriving unbiased estimates for these values is still an open research question, mostly since the connection between accurate value estimates and system performance is not yet well-understood. This thesis work has three main research contributions. Firstly, it analyzes the connection between value estimates and performance for the TD3 algorithm. Secondly, it derives theoretical bounds for the true value function when dealing with environments where a reward is only given for successful completion of a task (sparse/binary reward). Lastly, a deliberate underestimation objective is added to the TD3 algorithm together with the theoretical bounds to improve system performance when using human demonstration data that only covers a specific part of the state and action space. All the algorithms are tested and evaluated using simulated robot manipulation tasks in the robosuite environment, where the robot is first trained on the demonstration data and then can gather more experiences in the simulation. Results show that the deliberate underestimation together with the value bounds enable the robot to learn from human demonstration, which was not possible for the standard TD3. Additionally, applying just the value bounds speeds up the learning process when using machine-generated datasets.

Event-Triggered Control (ETC) is a control method where the controller is only updated when necessary. The control inputs are kept fixed until a state-dependent event triggers their re-computation. The triggering condition is designed to guarantee the stability and desired performance of the control system. This prevents the excessive use of scarce communication and energy resources. To fully take advantage of these savings when dealing with a (physically) distributed system, it is vital to decentralise the triggering condition. A decentralised triggering condition can be checked locally at the nodes of the system using the locally available states, eliminating the need to send updated sensor measurements to a central location constantly. However, the decentralisation of the triggering condition needs to be optimised for the state of the system to fully take advantage of the possible resource savings. This thesis provides a framework to optimally decentralise quadratic triggering conditions for Periodic Event-Triggered Control (PETC) systems in the absence and presence of bounded disturbances. In addition, a method of constructing a region-based map of such optimisations is provided, making it feasible to implement the optimisation approach on larger systems without needing more powerful hardware.

This research aims at quantifying the uncertainty in the predictions of tensor network constrained kernel machines, focusing on the Canonical Polyadic Decomposition (CPD) constrained kernel machine. Constraining the parameters in the kernel machine optimization problem to be a CPD results in a linear computational complexity in the number of features, whereas the original problem suffers heavily from the curse of dimensionality as the number of parameters scale exponentially. By employing a product feature map with polynomial features, the original data input is transformed to a higher-dimensional space.

Three different methods are investigated for quantifying the uncertainty of the predictions of the CPD constrained kernel machine. Firstly, the delta method is proposed which is a frequentist approach that linearizes a nonlinear parametric model around the estimated model. By estimating the covariance of the model parameters, the delta method can estimate the uncertainty in the model predictions based on the estimated parameter uncertainties. The delta method is compared to two other methods that are able to reflect the prediction uncertainty: the Bayesian method and Single Bayesian Core (SBC) method. The Bayesian method treats the parameters in the factor matrices of the CPD as probability distributions rather than single values and the SBC method incorporates both frequentist and Bayesian aspects. A comparison between the three different methods is performed based on an assessment on the correctness and informativeness of the uncertainty measures of prediction intervals and confidence Intervals.

It was found by regression and classification experiments that all three methods can provide valuable uncertainty quantification measures in terms of correctness and informativeness for the CPD constrained kernel machine. However, the Bayesian method provides in general more conservative uncertainty intervals compared to the delta and SBC method. A major drawback of the Bayesian method is its lack of scalability as the size of the mean and covariance, constructed by the unscented transform in the Bayesian method, scale exponentially. Furthermore, the delta and SBC method produce high quality uncertainty intervals and the methods provide remarkably similar uncertainty quantification on the prediction error variance.

The human brain, with its intricate web of billions of neurons and trillions of synaptic connections, is a remarkable organ responsible for performing complex cognitive processes. While brain imaging techniques like fMRI and EEG provide insights into neural activity, there is no broadly accepted mathematical framework for the collaborative activity of neuronal populations and their communication. In the interdisciplinary fields of neuroscience and computational modelling, this research introduces a data-driven mathematical framework for modelling the brain's cortical network, derived from EEG data, while incorporating an exogenous input.

This framework captures brain dynamics, forming a state-space model using subspace identification, and evaluates statistical dependencies among brain sources, known as functional connectivity. Employed on a dataset mimicking a sensorimotor task, it proves effective in characterizing brain dynamics, with the PO-MOESP method outperforming N4SID. Additionally, the framework is competent to assess functional connectivity between brain sources, resulting in a network connectivity diagram that offers valuable insights into the statistical relationships between distinct brain regions. Altogether, it is concluded the designed algorithm can determine the intricate interactions within the brain during a simple passively performed task stimulating the brain. Consequently, this achievement forms a robust foundation for advancing Brain-Computer Interfaces (BCIs) and enhancing the diagnosis of neurological disorders.

There is growing interest to control cyber-physical systems under complex specifications while retaining formal performance guarantees. In this thesis we present a framework for formal control of uncertain systems under complex specifications. We consider dynamical systems with random disturbances, whose probability distribution is unknown. When it comes to the specifications, we focus on those given as syntactically co-safe linear temporal logic (scLTL) formulas. Such formulas resemble natural language, and allow us to reason over complex behaviours of the system.

We follow an abstraction-based approach: we abstract the original system to a finite-state Markov model, in which the state discretization error as well as the distributional ambiguity, are embedded as uncertainties. To do so we make use of tools from optimal transport and ambiguity sets of probability distributions. After that, we synthesize a strategy for the abstraction and obtain probabilistic guarantees that the abstraction satisfies the specifications. Finally, we correctly refine the strategy to one that the original system can follow, and prove that the guarantees we obtained for the abstraction also hold for the original system.

We propose two approaches to obtain the abstraction. First, we propose a data-driven approach to abstract the system into an interval Markov decision process (IMDP) when samples from the unknown distribution is available. Then we use already existing algorithms to synthesize a strategy for the IMDP. Secondly, we propose an approach to abstract the original system into a robust Markov decision process (robust MDP). This second approach is applicable to more general uncertainty models besides the data-driven one, and reduces conservatism of the abstraction. Furthermore, we propose an algorithm to synthesize robust strategies for robust MDPs, which also renders the guarantees that the abstraction satisfies the specifications. Finally, we demonstrate the usefulness of our proposed approaches through several case studies that involve both linear and nonlinear systems.

Hydrofoil crafts with fully submerged foils can provide fast and economical waterway transport. However, their operation requires reliable onboard control systems to ensure the safety and comfort of their passengers, especially in rough sea conditions. This thesis project is focused on the dynamical modelling and the design of motion control systems for an experimental scale hydrofoil craft that is available at TU Delft, namely the Hydrofoil Education and Research Platform (HEARP).

The development of the dynamical model of HEARP is done by taking inspiration from the dynamics of marine crafts and aircraft and relying on different assumptions to obtain a simple and low-order model. The resulting model is a linearized state-space model with three degrees of freedom, namely heave, roll, and pitch, and includes the influence of regular waves. Because of inaccurate available data for the mass properties of HEARP, variations of the system parameters due to nonlinearities, and changes in the operating conditions, different uncertainties are assigned to most system parameters.

The use of multivariable feedback control methods for the motion control of hydrofoil crafts is limited, so this work is focused on exploiting such methods to improve the performance and robustness of such systems. The representation of the perturbed system using real parametric uncertainties is proved to be computationally expensive for the control design. Thus, the perturbed system is approximated by complex (dynamic) perturbations. A signal-based H-infinity optimal controller is designed using the nominal system, and a mu-synthesis optimal robust controller is designed using the approximated perturbed system.

The performance and robustness of the proposed controllers are evaluated in both frequency and time domains through simulations. From the results, it is concluded that both controllers offer high-performance system responses for both reference tracking and disturbance rejection of incident waves. Furthermore, by comparing the two controllers, it is observed that the mu-synthesis controller shows superior robustness for the modelled uncertainty. In contrast, the H-infinity controller has a slightly better performance when considering the perturbed systems with the real parametric uncertainty. The results of this thesis project can be used in the future to experimentally validate the accuracy of the proposed dynamical model and the performance of the designed controllers.

The high-tech industry continuously pushes the boundaries of controller performance to achieve faster and more precise machines. Currently, linear control is the standard in the industry. These controllers suffer from the waterbed effect and Bode's phase/gain relation, which impose inherent limitations on the precision and robustness of the system. Reset control is a popular strategy to get around these limitations and improve performance. The damping in reset control systems is not only determined by the phase margin of the system but is also dependent on the exact controller element sequence. Currently, finding the optimal controller sequence is done through simulation of the step response. However, this fails to provide insight into the underlying cause of the additional damping achieved by specific controller configurations. This thesis proposes an analytical approach to analyze the damping of the transient response reset control systems. The analytical analysis provides a better understanding of sequence-dependent damping and assists in controller design. First, the analytical expressions of the step response and states of the system are derived, which are used to define the system's energy. The step response and energy equations are used to characterize the damping in a reset control system. To show the value of the proposed method, the damping in a reset control system is assessed as an illustrative example. It is found that when a lead is in front of a reset element, the reset controller can provide more damping because it reduces the oscillatory content in the step response.

To extract energy from wind, both Horizontal Axis Wind Turbine (HAWT)s and Vertical Axis Wind Turbine (VAWT)s are used. Different from the HAWT, the axis of rotation of the VAWT is perpendicular to the ground. This vertical design offers some unique advantages for the VAWT. For example, the generator of the VAWT is located on the ground, hence the maintenance costs are lower compared to the HAWT. Furthermore, the VAWT can be upscaled easier than the HAWT to harvest more energy from a single turbine. These benefits demonstrate the potential of VAWTs for offshore applications. However, the VAWT’s blade suffers from periodic aerodynamic loads from the periodically varying Angle of Attack (AOA), and unsteady aerodynamic phenomena, which harm the durability and hence the economic feasibility of the VAWT.

This thesis aims at resolving the blade loads mitigation problem of the VAWT. For wind turbines, pitching the blade is a common way to reduce the loads on the blade. There have been studies done that focus on the power maximization for the VAWT. However, research on the pitch control system for load reduction is lacking. To tackle the blade loads mitigation problem, a data-driven approach called Subspace Predictive Control (SPC) is used to control the individual pitch actuation of a two-bladed 1.5 m H-Darrieus VAWT.

This closed-loop control strategy consists of two parts, an online identification block and an optimal control law based on the identified system. The online identification method applied is called Recursive Predictor-Based Subspace Identification (RPBSID). It can estimate the system parameters online by using the input and output data collected from the VAWT system. Then, two different types of controllers are designed to reduce the blade loads. In the first approach, the constrained Model Predictive Control (MPC) is used. The optimal control input is calculated by solving a finite horizon constrained optimization problem. The individual pitch control action can be automatically adapted based on the online identified model. In the second approach, a Linear-Quadratic Regulator (LQR) is used as a comparison to the first approach. The LQR solves an infinite horizon unconstrained optimization problem to obtain the optimal state-feedback law. The control gain can be calculated analytically.

Both approaches are first tested on a Simulink model, which is based on the interpolation of lookup tables computed by the Actuator Cylinder (AC) model and blade element theory. Under a stepwise wind condition, the SPC approach shows the potential to reduce the normal load on each blade. It is also found that the MPC is more robust and easier to implement, compared to the LQR. Then, the MPC based SPC is applied to a mid-fidelity simulation tool called Qblade to acquire more realistic results. The controlled pitch trajectory shows good load reduction results under both uniform and turbulent wind conditions. The work presented in this thesis provides a control strategy to reduce the blade loads on VAWTs following a data-driven manner and demonstrates the capability of MPC based SPC in VAWTs’ application.

Research in passive Heating, Ventilation, and Air Conditioning (HVAC) systems has gained traction over the last few years. Although passive HVAC is not a new concept, advances in environment sensing, control methods, and hardware have made it a more viable method. Some difficulties still exist, such as optimal sensor placement and optimal control strategies. Sensor selection is an important aspect of HVAC design. The system can become difficult to control with incorrect placement of sensors, resulting in higher energy consumption, lower comfort levels, or poor air quality.
There are essentially three methods to determine the optimal sensor location: model-driven, data-driven, and simulation-driven. The model-driven methods use mathematical models to maximize the observability of the system but are mostly used for simplified simulated rooms. Data-driven methods often use clustering algorithms, or maximize metrics such as entropy or mutual information. These methods focus on estimating the indoor air temperature distribution. Simulation-driven methods use simulations to determine the airflow or temperature
fields, often with CFD. These are used to find local hot spots or locations for fast detection of contaminants. No research was found that used sensor data of additional building components besides of the indoor air temperature.
In this work, the sensors are selected based on model prediction accuracy and the overall control performance to determine the effect of addition state measurements. A model is constructed to simulate the building, together with an MPC and an extended Kalman filter for state estimation. These are combined to run the optimization and determine the control performance. The sensor set average of each measured state is considered the true temperature. For all possible sensor combinations, the error of the combination average w.r.t. the true temperature is assumed Gaussian. The fitted Gaussian error distributions are then used as measurement noise in the model. The building and control response is simulated
with the measurement error over multiple days. Two algorithms are implemented to find the optimal sensor set: a predictive method and greedy method. The results are compared to each other and both methods showed that the indoor air temperature measurements have the largest effect on performance. Measuring additional states only resulted in a small increase in performance.

The increased diffusion of Renewable Energy Sources (RES) into energy distribution systems gives rise to a number of issues that the Distribution Network Operators (DNOs) need to face. The dependency of RES on weather, along with their intermittent and non-dispatchable nature, urges us to develop frameworks that can guarantee a stable network, despite the power fluctuations.
In this thesis, we develop a Data-Driven Optimal Power Flow (OPF) formulation in order to achieve voltage regulation against operational problems that may occur because of the high PV penetration into energy distribution systems. The unpredictable PV generation deteriorates the system’s reliability and introduces instability. Thus, we develop appropriate formulations to define a limit on exports from residential PV owners to the grid. For this purpose, we employ Distribunally Robust Chance Constraint Programming (DRCCP), as a method that can handle constraints that depend on the uncertain PV generation and
residential demand. We capture the distributional uncertainties with an ambiguity set and we utilize the Wasserstein metric to parameterize the range of this set.
We divide the thesis into two case studies. In the first case study, we perform the DRCCP optimization with only few recorded data. We evaluate the results of our algorithm in a dataset of 500 days of recorded data and we achieve to reduce significantly the overvoltage instances. We tune the DRCCP in a way that we achieve at least 95% satisfaction of constraints that emerge from the DRCCP, avoiding overconservative solutions and high generation cuts that
could harm the PV owners.
In the second case study, we intend to exploit plenty of historical data, but the computational burden hinders us from using them in the DRCCP. Therefore, we employ Wasserstein Barycenters, and we utilize the Wasserstein distance in order to cluster the data. With Wasserstein Barycenters we reduce significantly the running time and we provide an efficient and robust output, for at least 95% satisfaction of the model’s constraints.

With the use of simulation models, predicting and optimising the correct dynamic behaviour and parameters of a propulsion system of a ship can be performed cheap and safe. However, capturing the right dynamic behaviour is very difficult. Besides, building simulation models and determining the correct parameters is a time consuming process. With system identification, in- and output data of a controlled test are used to identify the parameters of the created grey box model structure, which reflects the underlying physical laws. A so called "fingerprint" is generated that imitates the behaviour of the system. The first attempt of system identification of a full-scale propulsion system showed promising results but asked for further research. This thesis further investigates if system identification is a suitable method to obtain the dynamic model- behaviour and parameters of a full-scale propulsion system in a short time, with the use of controlled tests.

To meet the demanding requirements on the environmental impact of aircraft, radically new aircraft concepts need to be developed. Within the NOVAIR project, Royal Netherlands Aerospace Centre (NLR) tests these new concepts on a Scaled flight demonstrator (SFD). Using an SFD allows for testing of dynamic and flight physical behavior of new aircraft concepts, which is difficult with more theoretical methods. Furthermore, by using an SFD, the risks associated with full-scale testing in terms of cost and time are minimized. One of the new concepts developed for the SFD is Distributed electric propulsion (DEP). Here, the two jet engines are replaced by six electric propellers. As these can be used actively for control, this results in an over-actuated system. These propellers interact with the aerodynamics of the wing resulting in Propulsion airframe interaction (PAI) effects. Although using the PAI effects for control has the potential to improve capabilities, efficiency and robustness of the aircraft, research into controllers using these effects actively is limited.

This thesis, therefore, presents a new control method including control allocation for the DEP-SFD aircraft, based on the nonlinear Incremental nonlinear dynamic inversion (INDI) controller. INDI enables controlling the nonlinear dynamics of the DEP aircraft over the complete flight envelope with one controller. By feeding back real-time sensor measurements, robustness to modeling errors and external disturbances is increased. Using the Incremental nonlinear control allocation (INCA) method, the full control authority of the DEP can be used. This technique enables taking into account nonlinear allocation relations and control effector interactions, while solving the control allocation real-time, which is key in actively using the PAI effects for control. The INCA method is used for two performance improvements: tracking performance and propeller power efficiency. To compensate for actuator dynamics, this thesis implements an Model predictive control (MPC) controller which results in improved tracking and higher efficiency. The performance of the controller was analyzed in simulation, where a reference square signal input on the roll angle was applied, while minimizing the sideslip angle and maintaining a steady altitude and velocity. The INCA controller with MPC is compared to a conventional INDI controller, showing a significant decrease in rise time from 2.46 s to 0.703 s with minimal tracking error. Furthermore, the effective bandwidth of the system was increased from 0.186 Hz to 0.663 Hz and the power consumption reduced by 6.3%. Modeling uncertainties, external disturbances and a propeller fault were introduced to verify the robustness of the controller. Finally, the reference altitude and velocity were varied, demonstrating controller performance over a large part of the flight envelope.

Uncertainty can be defined as imperfect or unknown information arising in a stochastic environment. Due to the very limited knowledge, it is difficult to propagate and quantify various uncertainties affecting the system to its next step. As a result, it has been a challenge to consider multiple uncertainties affecting the system in various fields, such as Fault Detection and Diagnosis. Fault detection has been an essential part of a large industrial and manufacturing system to take a proper corrective measure into account in a case of unexpected behavior. However, determining a robust threshold bound for fault detection is a big challenge. Several uncertainties affect the system (such as parametric uncertainties, experimental uncertainties, process noise, measurement noise, etc.). Ignoring the effects of various uncertainties (i.e., Deterministic Bound) can lead to a false alarm.

Therefore, to design a robust threshold probabilistic-based technique is used where all unknown parameters are taken into account. However, the major problem lies in propagating these unknown parameters into the next time step with their limited knowledge. A novel Message Passage Bilinear Uncertainty Propagation (MPBUP) algorithm is being proposed, which is used to quantify and propagate various uncertainties affecting the dynamical system into the next time step. The main aim of the algorithm is to quantify and propagate various uncertain parameters affecting the system at each time step. Uncertainty is propagated in terms of mean and covariance at each algorithm iteration to find the overall effect. Therefore the primary research is to design and validate the proposed algorithm and to check the algorithm can be used to determine a probabilistic threshold.

In the report, a detailed explanation of the algorithm for a trivial example is presented. The algorithm is then developed and implemented in MATLAB. Next, validation of the output generated through the algorithm is performed using Monte Carlo simulation. Finally, various analyses based on the MC simulation are discussed to support the results generated through the algorithm. An innovative approach is also discussed to extend the polynomial system algorithm as the proposed algorithm is limited to the bilinear system.

Further, a detailed explanation is given on applying the algorithm on a general state-space model in terms of mathematics involved, which is further extended on applying the algorithm on a real-time application such as the Four Tank System. After successfully implementing the algorithm on the Four Tank System, the algorithm is used to propagate uncertainty into the dynamical system to determine the robust threshold. A robust threshold is found considering the effect of various uncertainty. The threshold found using the proposed algorithm is dynamic as it evolves based upon the state dynamics and satisfies the required condition. As a result presented algorithm satisfy all the requirement and can also be used in other applications. The algorithm was analyzed based upon various criteria to conclude this thesis, and a comparative study is conducted.

The scheduling algorithm of the printer is an important factor that affects printing efficiency. For current printers, paper scheduling often follows the first-in-first-out principle, so it is often not optimal. The printer system is a type of semi-cyclic discrete-event system with synchronization but no concurrency. The system contains a set of operations that vary over a limited sequence of operations, which can be further modelled as a Switching Max-Plus-Linear (SMPL) system through max-plus-linear algebra, and can be switched between different modes of operation. Since max-plus algebra has a significant analogy with conventional algebra, some properties in conventional algebra can be applied to such kind of system, making the system easier to achieve optimal control.

In this report, a printing scheduler modelled by the max-plus algebra is presented. We review and summarize the basic knowledge of the Max-Plus-Linear (MPL) system from the existing literature in the first half of the report. We first introduce the basic properties of max-plus algebra and SMPL systems. Then we turn to the stochastic case, and consider the two types of stochastic uncertainty that may be included in the SMPL system, namely stochastic parametric uncertainty and stochastic mode switching uncertainty. Next, we review a Model Predictive Control (MPC) approach that can achieve optimal scheduling of systems containing two random uncertainties.

In the second half of the report, based on the content summarized in the first half, we make a further derivation of the printer scheduling system modelled by SMPL approach. We first present the modelling framework of the printer. Three working modes, namely duplex mode, idle mode and simplex mode, are modelled separately and merged into a compact form. Then a scheduler that takes the feeding and processing time of each sheet of paper as design variables is introduced. It can find the global optimal schedule of different types of paper by solving the Mixed-Integer Linear Programming (MILP) problem. After that, we discuss cases involving switching, and consider switching between two sizes of paper and two working modes. Three possible intermediate switching modes are designed. Finally, we take noise/interference into consideration, discuss the impact of noise on scheduling, and the changes that the previously proposed methods may require to achieve optimal scheduling in the presence of noise.

Vehicles are becoming increasingly complex and the resource-demanding due to the developments aimed at driving autonomy. Viewing an automobile as a cyber-physical system (CPS), this work applies the advances of controls applied to CPS for improving resource-consumption. One such idea is that of event-based control. Theoretical developments in event-based control show promising results, but they require practical corroboration. Combining these aspects, an event-triggered vehicle stability controller is designed and tested on a real-time hardware-in-loop setup.