Reinforcement learning (RL) is a powerful tool where the agents – or “robots” can learn from the environment based on their actions. Reinforcement learning approaches were found successful in combining predicting stock returns and portfolio allocation. Diversification is a critical element for achieving high portfolio returns with a lower level of risk. This work explores the application of reinforcement learning and multi-agent reinforcement learning (MARL) in portfolio management, emphasizing the applicability and increased portfolio performance via strategy diversification. A key contribution of this work is the development and evaluation of a MARL environment incorporating novel diversity measures, correlation, and total variation distance. The findings reveal that while fine-tuned single-agent RL models can demonstrate strong performance, roughly tuned MARL models with diverse agents reflecting a "portfolio of portfolios" paradigm show improved action diversification and portfolio performances. The work also highlights the critical role of long-term robustness testing, algorithm- and problem-specific hyperparameter optimization, and the challenges of adapting MARL methods to financial contexts. This work contributes to the RL research in portfolio management by exploring the use of MARL in portfolio management and discussing the limitations and future work directions.

The rapidly growing volume of parcel shipments is straining transportation and logistics sectors, highlighting the need for innovative solutions to optimize packing and loading processes. The online bin packing problem (BPP), an NP-hard computational problem, finds practical applications in numerous sectors, including modern packaging and intelligent logistics. This study proposes a novel reinforcement learning (RL) approach to tackle the online 3D-BPP emphasizing applicability and versatility. The key innovation is the representation of the packing scene as a graph, enabling effective encoding of task-specific high-level features. This graph-based structure serves as the foundation for an RL agent designed to learn an optimal packing strategy through dynamic interaction with the environment. The proposed approach uniquely operates within the continuous domain, enhancing generalization across diverse packing tasks. Experimental evaluations in both simulated environments and a real-world setting demonstrate that the solution achieves state-of-the-art performance across multiple complex three-dimensional packing scenarios.

This work introduces a novel methodology for the development of interpretable reduced-order dynamic models specifically tailored for jumping quadruped robots. Leveraging Symbolic Regression combined with autoencoder neural networks, the framework autonomously derives symbolic equations from data and fundamental physics principles capturing the complex dynamics of jumping actions with high fidelity. This approach significantly reduces model complexity while enhancing interpretability, facilitating deeper insights for legged robotic applications. The efficacy and accuracy of the proposed models are validated through comprehensive experimental studies, marking a substantial advancement in the design of agile and efficient legged robots. This research demonstrates the outperformance of a learned 2D model compared to existing template models such as the ASLIP. Also, an analysis of the dimensionality of the learned model is conducted showing the overarching tradeoff between accuracy and complexity. The method is validated on different simulated quadrupeds and an actual hardware robot.

With the current transition towards renewable and high-tech solutions, the world is becoming increasingly complex. Consequently, the challenges faced by firefighters also intensify. For that reason, firefighting robots are rising in popularity despite being far from perfect. An important area of improvement is the perception capabilities of those robots, given the fact that firefighting robots suffer from occluded camera views in environments filled with smoke. To overcome this challenge a LiDAR sensor may be used but experiments in this work show that even those point clouds are adversely affected by smoke. Consequently, this work presents a method for real-time reconstruction of ground surfaces in occluded environments filled with smoke. The developed method functions in ROS Noetic and merges segmented ground points, when available, with ground surfaces which are reconstructed based on information from segmented wall points. In this way, the method works even without the presence of ground points. To achieve this, a combination of established techniques from scientific literature, along with newly developed techniques were implemented. Doing so gives the robot’s operator an improved representation of the ground surface within environments filled with smoke. Ultimately the developed method may allow for autonomous navigation based on LiDAR data within environments filled with smoke. This research shows that a method consisting of techniques which tackle the independent sub-challenges arising from the use of LiDAR in indoor environments filled with smoke can effectively reconstruct the ground surfaces within those environments. Furthermore, the developed method has the potential to do so in a real-time manner.

The generation of a 3D map of an unseen environment, obtained through solving the SLAM problem, is a popular topic currently in the robotics domain. The Lunar Rover Mini (LRM) at the German Aerospace Center solves this problem using a RGB-D camera system, which is favourable in space applications due to its lightweight characteristics and energy-efficiency. Performing SLAM based on camera images is based on visual odometry: the science of estimating the rover’s trajectory trough a sequence of images. However, the dependency on a
single sensor to perform mapping and navigation poses a threat to the reliability of the system. To increase the reliability and robustness of the SLAM algorithm, an inertial measurement unit (IMU) is incorporated in the robot hardware.
This thesis describes the design for a visual-inertial SLAM algorithm that incorporates both visual and inertial measurements to solve the SLAM problem through performing tightly coupled sensor fusion, which estimates and corrects for IMU biases. The solution is based on a non-linear factor graph, which is a graphical model to represent the relationships between the
rover’s measurements and the unknown variables which are optimised for. This is done using the open-source GTSAM framework. Using experimental data, the robustness of the novel visual-inertial SLAM algorithm is demonstrated by simulating specific sensor failures. Moreover, the novel algorithm shows its capability to incorporate a degree of certainty regarding specific areas of the generated map, closely resembling how a human being would generate a
map of an unknown area.
An additional use case for tightly coupled sensor fusion is the increased accuracy of the estimated trajectory. Assuming Gaussian noise models for both measurement models, averaging the two can yield a higher accuracy than either of the two sensors could have obtained by itself. This hypothesis was tested in another experiment. As the main mechanism behind bias estimation is reducing the error between visual and inertial measurements, bias estimation is quickly affected by this drifting visual odometry, which in its turn deteriorates the accuracy of the visual-inertial odometry module. This observation proves that the bias estimation is not correlated to the underlying physical process, but is rather just a numerical value in the optimisation reducing the residual error. It raises the question whether this strategy of tightly coupled sensor fusion can actually be used to increase the accuracy of a visual odometry algorithm.

Monitoring expeditions in endangered habitats are currently performed by human experts. However, this approach has several disadvantages, including the limited amount of experts, cost-intensive expeditions, and the dangers that are posed by exploring dangerous terrains. Therefore, one can look into using quadrupedal legged robots that would collaborate with the human operators, which would be able to assist them by performing extra measurements in these dangerous habitats. An open problem in quadrupedal legged locomotion is robust periodic forward jumping, a.k.a. pronking, specifically for quadrupedal robots that have flexible joints placed in series. In this paper, we therefore propose a novel framework that generates an energy efficient pronking motion for a quadrupedal series elastic actuated legged robot. This periodic pronking motion is generated using a reduced order model and the serial elasticity of the joints is taken into account using an template-anchor approach.
To minimize the trajectory error we use Functional Iterative Learning Control (fILC) as feedforward control in parallel with a proportional-derivative feedback controller. The advantage of using fILC is that the elasticity of the quadrupedal robot is preserved and that the controller is able to learn the pronking motion in a small number of iterations. This framework is validated on an eight degrees of freedom series elastic actuated robot. In- and outdoor experiments show that this framework is able to work in unknown terrains.

In recent years, soft robots have become a focal point of research due to their ability to mimic natural movements and adapt to unstructured settings. However, their inherent flexibility poses significant challenges, particularly in the areas of modelling and control. While data-driven methods can model soft robot behavior without explicit physical models, they require extensive data and lack interpretability. On the other hand, physics-based low-dimensional models have relied heavily on expert knowledge and intuition, sometimes leading to models that are either too simple and inaccurate, or excessively high-dimensional.

This thesis introduces an end-to-end methodology for automatically identifying low-dimensional kinematic and dynamic models of planar continuum soft robots using image data. Based on the Piecewise Constant Strain (PCS) parametrization, the proposed approach determines an efficient segmentation for the soft robot to approximate its configuration. Afterward, a model identification strategy is employed to obtain a dynamic model that contains only the most essential strains. This model is formulated in the standard Euler-Lagrange framework, facilitating the integration with conventional model-based control schemes. The methodology is validated through simulations involving various planar soft manipulators and in the presence of noise, demonstrating its capability to generate accurate and computationally efficient models. This work provides a fast and practical tool to help the modelling and control of continuum soft robots, highlighting the potential for future applications in more complex actuation systems and real-world soft robots.

Soft robots are characterized by compliant elements that introduce heightened kinematic complexity compared to their rigid counterparts. Such systems, with infinite degrees of freedom, are inherently underactuated, making precise real-time shape regulation a challenging task. Model-based controllers, utilizing tractable reduced-order modelling methods, have emerged as promising solutions. However, practical implementations of these methods often rely on fully-actuated approximations, overlooking the underactuated nature of these continuum structures. In this study, we aim to experimentally validate model-based controllers that explicitly account for underactuation, surpassing the theoretical feasibility demonstrated in simulation. These controllers incorporate gravity cancellation and compliance compensation using the dynamic model of the robot to achieve superior real-time shape regulation compared to conventional PD/PID controllers. To facilitate this experimental validation, we have built a multi-segment soft robot research platform that includes a passively actuated segment, allowing for the utilization of both actuated and unactuated degrees of freedom in the control feedback loop. Through rigorous experimentation, we provide comprehensive evidence of the efficacy of this class of model-based controllers in controlling unconventionally actuated robotic systems. Consequently, our work bridges the gap between theory and practice, resulting in a practical real-time shape regulation framework that is adaptable to a vast variety of soft robotic systems.

In this work, we propose a method of processing patient input on discomfort level during robot shoulder physiotherapy into discomfort maps. These maps represent the patient's discomfort distribution throughout the range of motion of the shoulder, interpretable by both physiotherapists and robots. This method consists of three parts: the patient can input discomfort with a linear push-button; a collaborative robot arm is used to track the motion of the patient's shoulder; and audiovisual feedback of inputted discomfort is given to the patient and the therapist.
The method was validated in human factors experiments simulating shoulder physiotherapy sessions, where the subject is tasked with recreating a reference discomfort map through an auditory reference signal that emulates this discomfort. Here the robot also acts as the physiotherapist, moving the subject's shoulder. The signal is a beeping sound, whose rate scales with the discomfort intensity at the measured pose in the reference discomfort map.
We performed experiments with a total of 10 participants, demonstrating the viability of our method during patient-robot interaction. The results we collected also highlighted the presence of a time delay between the discomfort signal and the user input, and its effect on discomfort maps.

The traumatic loss of a hand is a horrific experience usually followed by significant psychological, functional and rehabilitation challenges. Even though much progress has been made in the past decades, the prosthetic challenge of restoring the human hand functionality is still far from being achieved. Autonomous prosthetic hands showed promising results and wide potential benefits; benefits that must be still explored. Here, we hypothesized that a combination of a radar sensor and a low-resolution time-of-flight camera can provide sufficient spatial and temporal information for deep learning algorithms to detect object shapes and materials in static and dynamic scenarios. To test this hypothesis, we analysed HANDdata, a recent human-object interaction dataset with a particular focus on reach-to-grasp actions, via both common and novel deep learning algorithms. The offline analyses reported here showed a great potential for both static and dynamic object characteristics recognition meant for autonomous grasping. Results seem to suggest modern, low-power radar as a potential key technology for next-generation intelligent and autonomous prostheses.

A substantial portion of workplace-related injuries stem from sprains, strains, and muscle tears, with a significant proportion occurring in the upper body, particularly the
shoulder. To address these issues, there is a growing interest in utilizing supportive devices like exoskeletons that often face challenges such as bulkiness, high costs, and parasitic forces, which have hindered their widespread adoption. Exosuits, an alternative to exoskeletons, offer potential solutions by primarily employing fabrics and leveraging the human body to transmit forces, eliminating the need for cumbersome and expensive external frames. A user study involving a passive-adaptive exosuit equipped with a controllable pretension spring showed a reduction in muscle effort while being limited to controlling only the system’s equilibrium position. It was theorized that the addition of a variable stiffness mechanism with controllable pretension and stiffness could eliminate observed issues and further increase muscle effort reduction, forming the need for
a variable stiffness mechanism. The vast majority of existing variable stiffness solutions are developed for use in joints as opposed to being designed for a linear motion and are thus not ideal for application in an exosuit. Additionally, the fusion of variable stiffness mechanisms and exosuits has not been studied extensively, with no studies using a variable stiffness mechanism capable of controlling both the stiffness and equilibrium position in an exosuit. This work presents the results of the user study, the development and testing of a variable linear stiffness mechanism for a linear motion capable of controlling the stiffness and equilibrium position, and the integration of said actuator in an
exosuit.

Quadrupedal robots possess the ability to move freely in the world and perform a variety of actions that would be unsafe or impractical for humans to perform. In the SNOW project, a quadrupedal robot is tasked with aiding firefighters in rescue missions during house fires by locating humans and assessing their health. Pushing away obstacles that cannot be circumvented otherwise is one of the many capabilities a quadrupedal robot should possess to be of most use in such missions. We develop a proof of concept by solving two problems sequentially: which stance to take on prior to the push and how to perform the push.

The process of stance selection starts with generating a certain amount of stances. Stances are generated starting from a preselected stance appropriate to the goal location and deviating from the 12 joint angles with a normal distribution. All generated stances are ran through a number of filters, which rely on solving for the forward and inverse kinematics of the robot. These filters check if the initial position is sensible and balanced and if the projected final position is close to the goal and balanced. The inverse kinematics are solved using Adaptive-Network-Based Fuzzy Inference Systems (ANFIS), which results in accurate estimations within a time frame that can be used in real-time applications. The final stance is selected by comparing the total displacement of all joint angles per stance, where the lowest total displacement is considered optimal.

The push is controlled by a nonlinear model predictive controller. We strive for a stable push, where the contact between the pusher and the object sticks, by keeping the movement of the end-effector within the motion cone. The motion cone denotes all twists the object can have without slipping at the contact with the pusher and is constructed using the limit surface to model the interaction between the object and the support surface and the generalized friction cone to model the interaction between the pusher and the object. We find that the motion cone predicts stick and slip with an accuracy slightly higher than 80%. Our controller steers accurately to all goals that lie within the motion cone and moves the object with a twist on the edge of the motion cone if the goal location lies outside of the motion cone. The robot remains balanced throughout the pushing motion in the vast majority of cases, but is more at risk of tipping over when pushing heavier objects.

Intelligent manufacturing has become increasingly important in the food packaging industry due to the growing demand for enhanced productivity and flexibility while minimizing waste and lead times. This work explores the integration of such manufacturing in automated secondary robotic food packaging solutions that transfer food products into containers using pick-and-place robots. A major problem in these solutions is varying product supply caused by prior machinery. As a result productivity drops drastically when conveyor belt speeds are not optimally controlled. Conventional heuristic-based engineered approaches are used to address this issue but are inadequate, leading to noncompliance with industry's requirements. Reinforcement learning, on the other hand, has the potential of solving this problem by learning quick and predictive decision-making behavior based on experience. However, the lack of research in reinforcement learning for complex industrial robotic problems, limits its adoption in industry. Therefore, this work aims to investigate the feasibility of reinforcement learning in the robotic packaging industry. We propose a reinforcement learning framework, with policy inference in a highly complex control scheme, designed to optimize the conveyor belt speed of the secondary robotic packaging solution using real-world product supply data. The framework exceeds the 99.8 percent performance requirement and maintains quality at the required 100 percent when tested on real-world data. Compared to the current heuristic-based solution, our proposed framework improves productivity, has smoother control and reduces code execution time.

Soft robots have the potential to accelerate robotiza- tion in areas that are complex and impractical for hard robots. The use of soft materials results in a safe and flexible design that is unattainable for hard robots. However, this attribute results in the need for new control approaches and strategies. Hybrid controllers are a relative unexplored type of controllers that consist of a model-based controller part and a learning part to correct the model-based controller. A hybrid controller benefit by the unrequired need for accurate system identification. Simultaneously, the learning effort is reduced by the preliminary work of the model-based component. In this project, a model-based feedforward controller is pro- posed and compared with a hybrid controller consisting of the same model-based controller enhanced with a Gaussian process to reduce the end-point error in the bending angle. The controllers are tested using a crafted 2-segment pneumatic silicone soft robot, following a circular trajectory with different radii. The results of this new control strategy highlights the poten- tial benefits of adding a learning approach to a model-based controller to reduce model errors. Using a relative small dataset preserves a computational usable Gaussian process. The small dataset remains effective by reducing the range of the training data.

Effectively controlling and exploiting the natural dynamics of Articulated Soft Robots for energy-efficient motions remains challenging. In literature, the problem is often split in two; in energy-efficient motion planning and structure-preserving control, where the focus is on one, and the other is largely disregarded. This work aims to unify these two using a motion planning and control strategy based on trajectory optimization and functional Iterative Learning Control. Using a reduced-order model, the planner generates an energy-efficient reference trajectory by minimizing the Cost of Transport. The controller then iteratively learns the feedforward control signal such that the full-order system tracks the reference, without altering its stiffness characteristics. We show that our strategy results in energy-efficient tracking of the reference. We also show how functional Iterative Learning Control can be used in a continuous approach to learn a stable forward pronking gait. We give experimental validation of this approach through experiments on the compliant quadruped E-Go. We show that the pronking gait can be learned on hardware in minutes and that it is robust to various types of terrain.

In robotic manipulation of deformable objects, the continuum nature of the object state leads to prohibitively high degrees of freedom when traditional modelling techniques are applied, leading to much research focusing on approaches that avoid an explicit object model. However, without such a model the opportunities for long horizon planning over the object shape or the application of optimal control methods are limited. In this work, we take a step to address this gap for the category of deformable linear objects by developing an accurate, low-dimensional manipulator-object model using the Affine Curvature strain parametrisation. We derive the model kinematics and dynamics, and detail a procedure for experimentally determining the unknown material parameters, using heavy electrical cables as a realistic test case. We then prove the viability of the model by experimentally demonstrating accurate feedforward control of the object in steady state, allowing for a significantly expanded reachable workspace and control of the endpoint orientation.

This paper presents a novel approach to regional forecasting of SARS-Cov-2 infections one week ahead, which involves developing a municipality level COVID-19 dataset of the Netherlands and using a spatio-temporal graph neural network (GNN) to predict the number of infections. The developed model captures the spread of infectious diseases within municipalities over time using Gated Recurrent Units (GRUs) and the spatial interactions between municipalities using GATv2 layers. To the best of our knowledge, this model is the first to incorporate sewage data, the stringency index, and commuting information into GNN-based infection prediction.
In experiments on the developed real-world dataset, we demonstrate that the model outperforms simple baselines and purely spatial or temporal models for the COVID-19 wild type, alpha, and delta variants. In combination with an average R2 of 0.795 for forecasting infections and of 0.899 for predicting the associated trend of these variants, we conclude that the model is well suited for predicting the spread of infectious diseases with similar disease dynamics in real world applications. To increase prediction performance and to improve the generalizability of the model for infectious diseases with more complex disease
dynamics, we recommend using additional (synthetic) data or expanding the regional forecasting scale in future work.

During the transfer of calli in plant regeneration, the repetitive work of pick and placement of calli in new agar is still done by human operators. This process can be automated to eliminate labour-intensive work. The thesis focuses on the development of a robotic gripper to automate the transfer of calli. To be able to grip calli, first its physical properties have to be investigated. With different weights, the maximum allowable force before calli damage was determined.
Subsequently a methodological design process was performed, which consisted of formulating system requirements and conceptual design analysis with the use of the Kesselring method. In order to finalize the design, the two highest scoring gripper concepts were chosen to be further explored. Both concepts were transformed in a 3D design and printed to validate its effectiveness for gripping calli. Of those two concepts, the best performing gripper was further optimized and used as prototype for additional validation. To validate this new gripper prototype, calli of different shapes and sizes were transferred to a new petri dish. To correctly grip the calli, a script was created to adequately detect and determine the size of the calli. The plastic prototype had high slack and inaccuracies due its the material. Therefore, the prototype was made of aluminium, with the use of Computer Numerical Control (CNC). Experiments revealed that the aluminium prototype had a success rate of 98% when gripping calli. In conclusion, the designed aluminium gripper was able to grip and release the calli without damage and has therefore great potential to automate calli transfer and thereby improve the quality and quantity of plant regeneration.

Fruit and vegetable production is increasing worldwide, and farmers currently face a tough challenge in finding enough agricultural workers. Automation of the labour-intensive task of crop harvesting could help fill in this gap between supply and demand. However, picking soft fruits is challenging, as they can be easily damaged if not carefully handled. This paper focuses on designing and evaluating a novel robotic gripper for gently harvesting blackberries, which is nicknamed the Twisting-Tube gripper. The gripper consists of a fabric tube which closes in an enveloping manner due to a motorized twisting action of the ends of the tube. A custom tensile-testing bench was used to test blackberry detachment, release, and damage rates for three types of tubes: radially elastic, fully elastic, and compressible. These tubes were manufactured out of different combinations of foam padding, spandex and food-safe cotton cheesecloth and compared to a handpicked control. The results showed that compressible, thicker cheesecloth outperformed radially elastic or fully elastic or extremely compressible tubes with 82% successful detachment and 95% successful release rates. Furthermore, the Twisting-Tube gripper with thick cheesecloth discoloured at least one drupelet of 19% of the blackberries after 48 hours, which is less than the 33% measured in the handpicked control group, but still more than the 10% of the unhandled berries with discolouration. Moreover, the gripper with thick cheesecloth caused leakage to at least one drupelet for 29% of the successfully harvested blackberries compared to 13% of the hand-harvested control. It can be concluded that the presented Twisting-Tube gripper prototype shows promising results for harvesting small soft fruits.