This thesis presents a comprehensive exploration of unwieldy object delivery using mobile robots, focusing on the challenges and advancements in Navigation Among Movable Objects (NAMO). The research addresses critical issues in robotic manipulation, particularly nonprehensile techniques such as pushing, which are essential for handling objects that are difficult to grasp.
The study is structured around four key research questions guiding the investigation into effective pushing manipulation strategies:
1. How to perform pushing manipulation with limited or inaccurate state estimation: We employ the concept of “stable pushing,” ensuring the robot can always “catch” the object during delivery. The stable pushing control problem is simplified as an optimization problem with a concise linear constraint. Experiments show that this approach outperforms reactive pushing strategies, reducing the robot’s traveled distance by 23.8% and time by 77.4%.
2. How to improve pushing manipulation when accurate state estimation is available: We enhance the stable pushing approach to a more maneuverable “free pushing” method, allowing contact point changes to improve manipulation mobility. This approach achieves an average success rate of 83% with an accuracy of 0.085m when pushing to selected goals, demonstrating improved agility and efficiency compared to stable pushing.
3. How to achieve efficient, real-time global trajectory optimization for contact-rich pushing: By investigating the differential flatness property of the pushing system, we simplify the pushing planning problem, significantly reducing computational complexity. This transformation allows for a simpler contact-implicit planning task that is easy to design, fast to solve, and robust to uncertainties.
4. How to perform robust state estimation through sensor fusion: We integrate data from multiple sensors and improve the robustness of the classic Kalman Filter using a deep reinforcement learning (DRL) algorithm. This novel DRL-based orientation estimation method guarantees bounded estimation errors without the need for hyperparameter tuning. Experiments demonstrate its superior performance compared to conventional methods, particularly in challenging scenarios with inaccurate initial state estimates, imprecise filter gains, and non-Gaussian noise environments.
The key contributions of this thesis include:
• A stable pushing approach that simplifies the optimization problem for nonholonomic mobile bases.
• A maneuverable free pushing method that enhances agility while maintaining contact.
• A reactive manipulation strategy leveraging differential flatness for efficient trajectory planning.
• A reinforcement learning-based approach to improve state estimation accuracy.

In conclusion, this research significantly advances the capabilities of mobile robots in handling unwieldy objects, bridging the gap between theoretical navigation planning and practical applications in complex environments. The findings pave the way for future research and broader applications of mobile robots in various domains, including logistics, search and rescue, and autonomous inspections.@en

Simulation environments are useful for a wide range of applications and their functionalities continue to improve every year. The aim of this thesis project was to create a simulation environment with high levels of realism and assess its capabilities through the use case of generating distributed drone traffic rules.

This thesis project was executed at the company Almende in cooperation with its sub-company DoBots. The distributed drone traffic rule generation use case was a contribution to the ADACORSA project which Almende takes part in. The goals of the thesis were the following. A simulation environment incorporating elements of realism was created using the software tools Almende has experience with. The capabilities of this environment were tested through the said use case. According to company requirements, this process was realised using an incremental complexity approach, and it utilized machine learning techniques. It was expected that the incremental complexity approach would help speed up the learning process of more complex scenarios, while machine learning would provide the benefit of smart automation and impartiality to solutions. For this process, a model was also devised for creating such rules, incorporating the incremental complexity approach and a standard machine learning technique. Finally, the benefits of the incremental complexity approach were also assessed.

The four building blocks of the implemented simulation environment were Gazebo, PX4, ROS2 and the iris drone model. The code was built upon the OpenAI ROS software package, with extensive modifications. The main work done during the implementation consisted of multiple parts. The original software package was cleaned of unnecessary elements, the PX4 software and the iris drone model were integrated into the environment and the simulation speed was increased. The whole setup was migrated from ROS to ROS2. The use of multiple drones in the same environment was also solved. An implementation of the deep Q-learning algorithm, which was selected as the learning method, was added. Different scenarios were created, as well as some logging and visualization tools. Additionally, the system was set up to run in docker containers and was highly parameterized for future use.

Different elementary traffic scenarios were implemented in the simulation and then executed. Based on these the following points were determined. It was concluded that the created simulation environment was well-fitted for running scenarios for traffic rule generation. It was also suitable for executing the scenarios using the incremental complexity approach, although its usability also depended on how the neural networks of the scenarios were set up. The environment was also feasible to work with machine learning from an engineering viewpoint, however, it did not perform well for its scientific research. This was due to not having statistically well-supported results, which was caused by the long time it took to execute a scenario. The benefits of the incremental complexity approach were also examined. For the simple case tackled in this project, it was found that the incremental complexity approach provided no significant advantages with regard to learning speed.

Machine learning can be effectively applied in control loops to robustly make optimal control decisions. There is increasing interest in using spiking neural networks (SNNs) as the apparatus for machine learning in control engineering, because SNNs can potentially offer high energy efficiency and new SNN-enabling neuromorphic hardwares are being rapidly developed. A defining character of control problems is that environmental reactions and delayed rewards must be considered. While reinforcement learning (RL) provides the fundamental mechanisms to address such problems, realizing these mechanisms in SNN learning has been underexplored. Previously, schemes of spike timing dependent plasticity (STDP) learning modulated by factors of temporal difference (TD-STDP) or reward (R-STDP) have been proposed for RL with SNN. Here we designed and implemented an SNN controller to explore and compare these two schemes by considering Cart-Pole balancing as a representative example. While the TD-based learning rules are very general ones, the resulted model exhibits rather slow convergence, producing noisy and imperfect results even after prolonged training. We show that by integrating the understanding of the dynamics of the environment into the reward function of R-STDP, a robust SNN-based controller can be learnt much more efficiently than by TD-STDP. The work of this master thesis project has also been published as a paper in Electronics, Vol. 12.

To operate in open world environments a symbolic Artificial Intelligence (AI) to be able to adapt and incorporate new objects and relations in its Knowledge Base (KB). Symbolic AI use the objects and relations in their KB to navigate the world and create plans. These KB are filled with knowledge in advance, so they can not add new knowledge when encountering novel situations. This thesis presents Heuristics-Based Causal Discovery (HBCD), a method that identifies and labels causal relations and transforms those causal relations into logic statements, which can be inserted into a KB autonomously.

HBCD can operate in a partially-observable environment by performing actions and observing the effects of its actions. The actions are chosen by heuristics, which are modelled after human strategies for causal discovery. The discovered causal relations are tested for the properties necessity and sufficiency. These properties provide information on the completeness of the result, whether there are any missing causal relations. HBCD uses this information to adjust its search space by moving variables in and out of it during the search. If there are no more missing causal relations HBCD stops the search. The method was tested in two simulated environments and the results are promising.

High level decision making in Autonomous Driving (AD) is a challenging task due to the presence of multiple actors and complex driving interactions. Multi-Agent Reinforcement Learning (MARL) has been proposed to learn multiple driving policies concurrently to solve AD tasks. In the literature, multi-agent algorithms have been shown to outperform single-agent algorithms and rule-based algorithms. Also several techniques have been employed to facilitate convergence in policy learning such as parameter sharing and local reward design. Further, functional safety in AD has been addressed with techniques such as unsafe action-masking. However, there is a gap in the literature on the study of the effects of entropy regularization and on policies learned with closed-loop stability guarantee to solve AD tasks in MARL. In this thesis, research gaps are addressed in entropy regularization and Lyapunov stability constrained policy objectives applied to Autonomous Driving in MARL. Specifically, it is demonstrated on the lane-keeping task with 2 agents that entropy regularization improves training stability. It was also shown that in stochastic multi-agent algorithms on the lane-keeping task, a Lyapunov constrained policy objective performs better in average episode returns, success rate and collision rate than a policy objective without a Lyapunov constraint with low measurement noise perturbation. However, an algorithm with a stochastic actor performs worse than that with a deterministic actor in stability and lane center proximity on the lane-keeping task.

Introduction - Grasping unknown objects is an important ability for robots in logistic environments. While humans have an excellent understanding of how to grasp objects because of their visual perception and understanding of the 3D world, robotic grasping is still a challenge. Due to the fast-growing development of deep learning methods, it is now possible to train deep neural networks on this grasp task.

Objective - This thesis proposes a bin-picking pipeline that uses deep learning to take care of the perception and estimation task. The pipeline can predict grasps for known and unknown objects with a two-fingered pinch-gripper in real-world environments in a single object and multi-object scenes.

Method - A grasp annotation tool has been developed to generate a wide variety of grasps in the training data that are antipodal and collision-free. Together with annotated objects, the generated grasps are used to train Grasp-RCNN. The developed Grasp-RCNN combines an object- and a grasp-detection network to predict objects masks and grasps, and a decision algorithm that picks the best-estimated grasp based on a grasp score.

Results - Robotic experiments demonstrate that the proposed method allows a robot gripper to grasp both known and unknown objects in single-object and multi-object scenes with a total success-rate of 89.7% and 81.0% with average process-times of 616 ms and 739 ms per scene respectively. In a bin-picking scene a success-rate of 87.5% with a process-time of 1235 ms is achieved.

Conclusion - These results indicate that the proposed Grasp-RCNN is able to grasp known and unknown objects with an accuracy that is comparable to the state-of-the-art. For production purposes, the speed of the network still can be improved.

Pose estimation provides accurate position and orientation information of the intelligent agents in real time. The accuracy of the estimation directly affects the performance of sequential tasks such as mapping, motion planning, and control. EKF (Extended Kalman Filter) is a standard theory for nonlinear pose estimation by modeling state uncertainty to Gaussian distribution. However, EKF has requirements for proper initial estimate and system noise to obtain bounded optimal estimate. Meanwhile, model nonlinearity and non-gaussian noise modeling affect the performance of EKF significantly in practical applications. In this thesis, we focus on improving the performance of nonlinear pose estimation by reinforcement learning. By formulating an EKF measurement update as a Markov Decision Process (MDP), reinforcement learning agents can be trained to learn the estimator gain through data samples and executed as the online estimator for pose estimation tasks.

Based on the above idea, we propose a novel reinforcement learning-compensated EKF estimator (RLC-EKF), where the RL agent serves as a second-time measurement update that subsides the residual error from the standard EKF estimate. The estimator is developed and testified on two specific pose estimation scenarios. Firstly, as a continuous work from the previous study, a framework for 3 DOF orientation estimation using inertial sensor and magnetometer is replicated. Then, the framework is extended by different RL algorithms training and multi-scale robustness validation. Besides, we implement the estimator on a feature-based 2D plane localization framework. The proposed framework shows the feasibility of the underlying algorithm on a localization task with a known map. As a result, the RLC-EKF estimator gives superior performance and convincible robustness compared to classical methods in severe conditions such as varying initial states, degree of noise intensities, and model covariance.

An end-to-end framework is developed to discover physical laws directly from videos, which can help facilitate the study on robust prediction, system stability analysis and gain the physical insight of a dynamic process. In this work, a video information extraction module is proposed to detect and collect the pixel position of moving objects, which would be further transformed into physical states we care about. A physical law discovery module is developed to learn closed-form expressions based on the extracted physical information. The video information extraction module takes advantage of contour detection and Hough transformation to extract position information. The physical law discovery module includes a deep neural network-like hierarchical structure Mathematical Operation Network (MathONet) which is consisted of basic mathematical operations. We develop a sparse Bayesian learning algorithm to learn both the topology and parameters of dynamic systems. Several simulated videos were generated to illustrate Newton’s law of motion, the state space of a Duffing oscillator, and the pendulum motion equation. By demonstrating on these examples, our method can discover the corresponding governing function without requiring much prior information.

Simultaneous Localisation and Mapping (SLAM) provide a novel solution for the robots to localise and navigate an unknown environment. Initial SLAM research focused mainly on the indoor environment, assuming the background to be primarily static. In contrast, the real world has dynamic interactions that restrict the implementation of SLAM to limited scenarios. This brings a higher requirement to deal with the moving objects in dynamic environments for robust SLAM performance.

Semantic understanding of the environment helps in filtering out the influence of dynamic objects in the vicinity. An instance segmentation based on two-stage neural architecture is used for this purpose, which is hard to operate in real-time navigation. In this project, the benefits of single stage neural architecture are studied in terms of speed and accuracy for improving the efficiency of dynamic features removal in the application of SLAM.

Although Instance segmentation architecture helps to identify the potentially dynamic object by learning from the dataset, it cannot differentiate moving objects from non-moving objects in the dynamic class. Hence, all the features corresponding to the predicted dynamic class are removed even when the objects remain stationary, affecting the quality of SLAM performance. A two-stream encoder-decoder architecture is developed to segment the moving masks using RGB and optical flow input, improving feature tracking without affecting robustness. The feasibility of encoding dynamic information to enhance quality semantic mapping is also studied.

System identification is a mature field in physical sciences and an emerging field in social sciences, with a vast range of applications. Nevertheless, it remains of great focus in academia. The main challenge is the efficient use of data to generate good model fits. System identification involves multi-disciplinary techniques from statistical, mathematical and computational sciences. The typical approaches for dynamic system identification include fuzzy models, non-linear auto regressive models, state-space models, subspace identification models and many others. In this thesis, artificial neural networks are evaluated, among these, as black-box methods known to be capable of universal approximation. With no essential prior information, the identification problem exhibits more difficult challenges. These include the complexity of the resulting models, choice of regressors, and uncertainty quantification. Specifically in this thesis, a Sparse Bayesian Learning approach is proposed, as a solution to these challenges. A practical iterative Bayesian procedure is derived and set to identify six benchmark datasets of three non-linear mechanical processes: Cascaded Tanks, Coupled Electric Drives, Bouc-Wen hysteresis model as well as of three linear mechanical processes: Heat Exchanger, Glass Tube Manufacturing and Hair Dryer.

Semantic Segmentation of medical images are used to improve diagnosis and treatment. In recent years, the application of machine learning methods are increasingly used. However, the design of these models is difficult and time-consuming. In this thesis, we investigated the automation of this process using an Automated Machine Learning (AutoML) algorithm called Bayesian Optimization with HyperBand (BOHB). We found several hyperparameters significantly influencing the performance and confirmed that BOHB is applicable to these kind of problems. However, we did not find a significant difference between BOHB and Random Search on a small search, stressing the need of matching the search space to the search strategy. Finally, our approach was only slightly worse than the current state-of-the-art, which shows that AutoML has potential in this field.

Glioma is a kind of slow-growing brain tumor which may result in severe seizures. Currently a major tool used to detect and diagnose the glioma is MRI scan. To better analyze the medical image, segmentation is usually conducted as a basic step for further processing, which partitions an integrate image into multiple physically meaningful regions by annotating objects and boundaries. Deep learning based segmentation methods have attracted significant interest due to their high efficiency and strong generalization ability. With the increasing demands of high-quality segmentation of bio-tissues in medical region, plenty of innovative approaches were proposed to expand the boundary of segmentation capability of deep learning models by taking the spatial or temporal constraints of bio-structure into consideration. Although longitudinal segmentation in 2D natural image sequences has made a lot of success, the potential of deep learning network in segmenting a series of chronological 3DMRI images in terms of improving consistency remains unclear. This thesis aims to investigate whether deep learning models are able to increase segmentation accuracy as well as consistency in longitudinal 3D images, specifically focusing on introducing Recurrent Neural Network(RNN) to 3D Convolutional Neural Network(CNN) for 4D segmentation. In addition to the implementation of several U-Net variants as CNN backbone, three types of longitudinal connection strategies are proposed. A hierarchical workflow is followed to create the optimal version of longitudinal network based on combining multiple CNN variants and connection strategies. The evaluation of the 4D network shows that segmentation accuracy of the longitudinal model is limited by its CNN backbone and temporal information can partially improve the segmentation consistency with regard to maintaining the highest proportion of normal tissue unchangeable over time.

Fully automated vehicles have the potential to increase road safety and improve traffic flow by taking the human element out of the driving loop. They can also provide mobility to people who are unable to operate a conventional vehicle. Safe automated vehicles must be able to respond in emergency situations or drive on slippery roads in bad weather conditions. Therefore it is crucial to have a safe and robust control strategy that can use the full handling capabilities of the vehicle.

This thesis presents how safe reinforcement learning can be used to design a steering policy that can drive an automated vehicle at the limit of friction.

The steering policies are trained using the Lyapunov Safe Actor-Critic (LSAC) algorithm. LSAC is a combination of the Soft Actor-Critic (SAC) algorithm and a Lyapunov stability analysis to solve constrained control problems.

The performance of LSAC is tested in a vehicle simulator against SAC and Model Predictive Control (MPC) in a series of tests that include changing lanes at different speeds, recovering from a destabilizing collision, and driving on a race track at the limit of friction.

The experiments show that LSAC outperforms MPC and SAC control strategies in terms of safety and vehicle stability. LSAC can recover from larger disturbances than MPC and SAC. A control strategy is presented that will keep the vehicle stable when driving at the limit of friction but can use the maneuverability of an unstable vehicle when is it necessary to avoid dangerous situations. Additionally, a policy is presented that can find the fastest way around a race track while staying within the track limits.

Due to a high contribution of human error in fatal traffic accidents, efforts in research and industry for automating vehicles steadily increased the last 3 decades. To reduce accidents with other road users, automated recognition and classification of road users is crucial. The most accurate models are large convolutional neural networks, however they require expensive and powerful hardware to be implemented. The hardware requirement hinders wide embedded use in education and industry, slowing down the potential increase of safety on the road. In pursue of reduced inference time on GPUs, research has focused to reduce the computational demand of neural networks. However the accelerations on microcomputers are not reported, potentially leading to different results.
To contribute to increased insight of the applicability of previous research on microcomputers, this thesis compares accelerations on a microcomputer and GPU. This is pursued by using the pretrained neural network "Squeezedet", trained on the "KITTI" road user dataset. The neural network is pruned in 3 different ways: by reducing the number of filters, by reducing the size of filter kernels along a layer and by reducing the amount of layers. The accelerations
are measured on a Raspberry Pi 3b+ and a Tesla K80 GPU. The process of embedding the neural network on the Raspberry Pi is described in great detail, to promote further research and educational use. Acceleration of the network is up to 12 % better on a Raspberry Pi than a GPU, when pruning filters from the network. When pruning full layers, or decreasing filter size the accelerations are up to 4% worse than on a GPU. This means when working with a microcomputer, the choice of pruning method can not be based on reported accelerations on GPUs. Therefore the recommendation is to do more research on accelerating neural networks for microcomputers. This will lead to wider use in industry and education and eventually safer roads.

Reinforcement Learning (RL) is a learning paradigm where an agent learns a task by trial and error. The agent needs to explore its environment and by simultaneously receiving rewards it learns what is appropriate behaviour.
Even though it has roots in machine learning, RL is essentially different from other machine learning methods. In contrast to others, RL agent has to generate its own data to learn from. In this thesis, we aim to train an RL agent to fly a quadcopter to track any target position (way-point) in three dimensional space. Where conventional control strategies for quadcopters involve a separate attitude and position controller and most RL solutions focus on one of the two controllers, our goal is to design a low level RL controller capable of computing motor commands directly from sensor input, therefore replacing both attitude and position controller with one RL policy. The policies we develop utilize the algorithm ’Twin Delayed Deep Deterministic Policy Gradient’ (TD3) for learning. TD3 is a variant of the Deep Deterministic Policy Gradient (DDPG) algorithm.
The policy for attitude control trained for 3500 episodes 3, around 6.1e5 time steps. The learned policy is able to stabilize the attitude of the quadcopter (in simulation) with a success rate of 94 %. For position control, two policies are generated with two different types of dense reward. The resulting type 1 policy has high fluctuations in motor commands and therefore oscillating attitude and position values. In none of the evaluation trajectories a steady state value is reached. The type 2 produces a working policy after a shorter training time of 1200 episodes, 1.1푒6 time steps. For all tested trajectories, this policy achieves steady state for almost each way-point. This thesis proves that TD3 can be used for low-level quadcopter control, replacing both inner and outer loops of the quadcopter control. Using the dense reward function and applying negative reward on position control only results in a stable policy that can track way-points all throughout the 3D space. Future work requires the twostep parameter estimation to be tested on a real life quadcopter, as well as enrolling the policy onto a real life quadcopter.

Human pose estimation, a challenging computer vision task of estimating various human body joints' locations, has a wide range of applications such as pedestrian tracking for autonomous cars, baby monitoring, video surveillance, human action recognition, virtual reality, gaming, gait analysis, etc. A majority of the research on the development of models for the task of human pose estimation has been focused on improving the accuracy of the task which also increases the complexity of the models. These models demand devices with high computational power to be deployed for real-world applications. Even though a lot of research has been focused on estimating the human pose from monocular images taken from cameras, the complexity of the models makes them impossible to be implemented on edge devices and embedded devices like mobile phones that have built-in cameras. This reduces the scope of applications where human pose estimation can be used. To address the issue, the research focuses on improving the performance of a baseline human pose estimation architecture by reducing the model size(number of parameters) and thereby its inference time without a significant loss in the accuracy. To improve the performance of the model, a structured Bayesian compression algorithm is used and the network is compressed by engineering the model based on the uncertainty of the parameters. The results show that the Bayesian compression method reduces the model size by around 65 percent with only a very little drop in the model accuracy. Also, the comparison of the inference time of the original baseline and the compressed model in an android device shows that the inference time is reduced by around 50 percent because of the reduction in the number of operations in the compressed model architecture.

Aircraft inspections after unexpected incidents, like lightning strikes, currently require a timeconsuming and costly inspection process, due to the small size of the lightning strike damages. Mainblades Inspections is working on an automated, drone-based solution, that scans the aircraft hull with a high-resolution camera. The objective of this project is to assess the
feasibility of using a deep Convolutional Neural Network (CNN) for (semi-)automated damage detection, with the goal of achieving a high recall (low False Negative Rate (FNR)) on the small damages. The problem is framed as a classification problem on limited and imbalanced data. However, it is not pre-defined if single-label or multi-label classification should be used,
and both approaches are investigated.
The main contribution of this work is to show experimentally how common deep CNN architectures and Deep Learning practices can be used to train classifiers that recognize damages in a specialized domain, with application-specific metrics. We present methods for synthesizing, pre-processing and re-sampling of the necessary dataset. It is shown that pre-trained, parameter-efficient CNN architectures that implement skip-connections, complemented by
global max-pooling before the final layer, are well suited for that dataset. The Xception architecture has been chosen as backbone for the classifier due to its high recall and fast convergence. To mitigate the detrimental influence of imbalanced training data, training data re-sampling that equalizes the class distribution is implemented. It has a positive effect recall, especially
when applied to multi-label classification. When using re-sampling and data augmentation, the performance of multi-label and single-label classification can be brought to the same level. However, the best achieved FNR is 5.4%, with a softmax classifier, combining all regularization methods.
Finally, we investigated how regularization can be used to increase generalization capability with limited training data. Data augmentation is the most effective regularization method, even though its full potential has not been explored yet. Dropout benefits single-label classification
but not multi-label classification. L2-regularization has a moderate positive effect on both. Naively combining the regularization techniques without an exhaustive grid search or automated search on average does not yield any additional gains and shows the limit of manual hyper-parameter tuning.

This study aims at a possible solution to predict the fuel consumption of heavy duty diesel trucks, particularly, the tractor semitrailer for their long haul operations using various machine learning techniques. It intends to provide a possible alternative to simulation or physics based models, which often are very complicated. The stringent laws on emission control set by the Paris Agreement and the fact that heavy duty trucks contribute to almost 27% of CO2 emissions from road transport and their dependence on diesel for operations (in long haul) makes it the need of the hour to first, have an estimate on the emissions being produced and second, to develop technologies to reduce those emissions.

This study focuses specifically on the first part i.e., estimating the amount of fuel consumed by heavy duty trucks in the European Union and thereby determine the emissions being produced. The main objective is to examine whether an approach of machine learning could be a viable option to predict fuel consumption. This thesis is part of the AEROFLEX project and was done in collaboration with TNO, which provided all the data-sets required for the study.

The idea was to explore the regime of machine learning for one time step ahead prediction of fuel consumption. Furthermore, this study also focused on the development of another model by not using any variables affected by the driver as input into the training model. This exclusion was necessary to make sure the model remained adaptive to new routes and new trucks, especially because large scale on-road testing of the newly developed trucks is impossible and also because this way would help predict the fuel consumed by a truck without the necessity of it driving on a road. The study concludes with a comparison with an existing simulation model at TNO and provide an alternative machine learning solution. It also provides a comparison between different machine learning techniques and suggest the most accurate one.

It was found that machine learning could potentially be used to predict the amount of fuel consumed by a long haul heavy duty truck driving on a motorway. It was also found that engine torque was the variable that affected the fuel consumption of the truck the most. Furthermore, Neural Network was the most potent algorithm among all the other learning techniques for both the models developed in this study with it performing better than the simulation tool by a factor of approximately 3.8 in the model where the driver/drive influenced inputs were not considered in the training data-set. The results obtained from this work at a sampling frequency of 10 Hz. (i.e., 0.1 seconds) are comparable to the ones reported by other sources at a sampling rate of 0.016 Hz. (i.e., 1 minute) or 0.0016 Hz. (i.e., 10 minutes). This goes on to say that the machine learning algorithms are also potent at much higher sampling frequencies.

Model-free reinforcement learning has proved to be successful in many tasks such as robotic manipulator, video games, and even stock trading. However, as the dynamics of the environment is unmodelled, it is fundamentally difficult to ensure the learned policy to be absolutely reliable and its performance is guaranteed. In this thesis, we borrow the concept of stability and Lyapunov analysis in control theory to design a policy with stability guarantee and assure the guaranteed behaviors of the agent. A novel sample-based approach is proposed for analyzing the stability of a learning control system, and on the basis of the theoretical result, we establish a practical model-free learning framework with provable stability, safety and performance guarantees.
% Specifically, a novel locally constrained method is proposed to solve the safety constrained problems with lower conservatism. In our solution, a Lyapunov function is searched automatically to guarantee the closed-loop system stability, which also guides the simultaneous learning (covering both the policy and value-based learning methods). Our approach is evaluated on a series of discrete and continuous control benchmarks and largely outperforms the state-of-the-art results concerning unconstrained and constrained problems. It is also shown that the algorithm has the ability of recovery to equilibrium under perturbation using the policy with stability guarantee. (Anonymous code is available to reproduce the experimental esults\footnote{\url{https://github.com/RLControlTheoreticGuarantee/Guarantee_Learning_Control}}.) Since sometimes the constraint is hard to define, we introduce a novel method to learn a constraint by representing the bad cases or situations as a distribution, and the constraint is the Wasserstein distance between the distribution.

One-Shot Neural Architecture Search (NAS) is a promising method to significantly reduce search time without any separate training. It can be treated as a Network Compression problem on the architecture parameters from an overparameterized network. However, there are two issues associated with most one-shot NAS methods. First, dependencies between a node and its predecessors and successors are often disregarded which result in improper treatment over zero operations. Second, architecture parameters pruning based on their magnitude is questionable. In this thesis, classic Bayesian learning approach is applied to alleviate these two issues. Unlike other NAS methods, we train the over-parameterized network for only one epoch before update network architecture. Impressively, this enabled us to find the optimal architecture in both proxy and proxyless tasks on CIFAR-10 within only 0.2 GPU days using a single GPU. As a byproduct, our approach can be transferred directly to convolutional neural networks compression by enforcing structural sparsity that is able to achieve extremely sparse networks without accuracy deterioration.