Neural Cellular Automata (NCA) have recently been proposed as neuromorphic robot controllers. Despite their promising behavioural characteristics and small parameter counts, training NCA for control tasks has proven difficult and unstable. It is so tricky that curriculum-like multi-stage training programs must be used for simple control tasks. This thesis posits how criticality theory, an amalgam of statistical mechanics and neuroscience, presents a compelling case for pre-training NCA into a critical state. Mainly the increase in inter-cellular com-munication distance and maximization of available information. This thesis presents a novel NCA update function architecture loosely based on neuroscience and two novel interchangeable pre-training methods, one implicit and one explicit, based on criticality theory aimed at improving training performance. The new architecture and pre-training methods are tested on the Cart-Pole environment and trained with Double Deep Q-Learning (DDQN) and neuro-evolution. The new architecture improves the training speed and general performance of the NCA, whilst the two pre-training steps greatly stabilise the control task training when using DDQN. The explicit methods resulted in faster agent training than the implicit methods, but the pre-training step was prone to failure, whereas the implicit methods always succeeded. Neuro-evolution was more efficient than DDQN when training iterations and helped explain the dynamics that make NCA challenging to train. The architecture and pre-training steps were further tested on the LunarLander problem, a more complex control task. The neuro-evolution method succeeded in training but did not present excellent results, whilst DDQN failed outright to train.

In this master thesis, the development of an innovative gripper specifically designed for grasping the delicate peduncles of vine tomato trusses is presented. This research addresses a critical challenge in agricultural automation: efficiently and safely manipulating high-value crops without causing damage. The thesis begins with a comprehensive review of the current state of agricultural automation, particularly focusing on the post-harvest handling of vine tomatoes.
Building on identified limitations in existing robotic grippers and automated systems, a novel gripper design is proposed. This design uniquely approaches grasping the peduncle of vine tomato trusses. Traditionally, trusses have been grasped by the peduncle with standard pinching grippers for pick and place operations. The proposed gripper in this paper grasps the truss with a hook around the peduncle, often optimally at the centre of mass, increasing the success rate of grasping, stability, and avoiding damage to the truss. Furthermore, it has a higher tolerance for detection errors, allowing for inaccurate positioning of the robotic system. A hook-gripper can successfully grasp a wider range of tomato varieties than a pinch gripper.
The hook, consisting of two fingers, allows for more stable lifting of the truss and can handle peduncles in hard-to-grasp positions, such as in a crate filled with tomatoes. In addition to increased reach capabilities, the hook-gripper also enables manipulations like dragging and pushing.
The study involved an iterative design process, prototyping, and rigorous testing of the gripper. Key features include a hook mechanism for secure grasping, enhanced mobility for reaching into cramped spaces like packed crates, and a delicate touch to prevent bruising or damaging the fruit. The research also integrates the gripper with advanced detection systems for precise and effective operation within automated setups.
Results from extensive testing demonstrate that the newly designed gripper not only improves the success rate of grasping and manipulating vine tomato trusses but also significantly reduces the risk of damage compared to conventional pinch grippers. Individual trusses can be grasped with great success.
Testing for the different positions showed an increased range of grasping position that resulted in a successful grip. In practical experiments, the gripper performed well, lifting trusses with ease. Other test results show that the position of the peduncle is of great importance for the success rate. Test results indicated an 80% success rate in grasping trusses positioned near the edges of the crate. Replaying waypoint with "learning from demonstration" improves the grasping of the trusses compared to existing detection possibilities, and that emptying a crate is a challenging task.
The specialized hook-gripper offers insights into a practical solution for picking and placing vine tomatoes using the peduncle. This thesis contributes to the field of agricultural robotics and facilitates a step towards future innovations in the automation of high-value crop handling. The study delves into hook-gripper design and actuation, crucial for optimal performance in robotic manipulation systems.

Physical human-robot cooperation (pHRC) has the potential to combine human and robot strengths in a team that can achieve more than a human and a robot working on the task separately. However, how much of the potential can be realized depends on the quality of cooperation, in which awarenes of the partner’s intention and preferences plays an important role. Preferences tend to be highly personal, and additionally depend on the cooperation partner and the cooperation itself. They can be hard to define in terms a robot would understand, and may change over time. This thesis focuses on learning ‘useful models’ from observed behavior, to let our robot adapt its behavior to better match its human partner’s preferences, and thus improve the cooperation.
The aim is to capture personalized approximate models of human preferences –how a person likes to do something– from very few interactive observations, providing only small amounts of imprecise data, such that the robot can use the model to improve each user’s comfort. First, we learn a model to predict and optimize the human ergonomics in a pHRC task, such that our robot can ropose a plan, for both the human and itself, to solve the task in a way that is more ergonomic for its human partner. However, people do not necessarily prefer to act ergonomically, nor do we want to impose on them what a robot thinks best. Therefore, next, we apply inverse reinforcement learning (IRL), to capture less restrictive preference models: 1) path and velocity preferences for motion planning, and 2) on a higher level of abstraction, which (grasp or motion) action to initiate for proactive physical support. For learning to take the correct action in cooperation, we developed the disagreement-aware variable impedance (DAVI) controller to smoothly transition between providing active guidance and allowing the human to demonstrate alternative behavior.....@en

State-of-the-art object grasping with 7-DOF robotic manipulators requires joint configuration planning methods in order to provide position control of the end-effector. These motion planners are able to calculate a motion plan to execute a safe grasp, while taking environmental constraints into account. In human-robot interaction, a well known problem is that humans are uneasy with the arm motion the robot executes, because the motion plan lacks parametrization of variables which would account for the impression of legibility. In this study we develop a method which allows for teleoperated learning from a single demonstration that is perceived more legible by humans. The operator uses the Geomagic Touch haptic device to demonstrate a movement of the robot’s end-effector. Modeling a motion path from a single teleoperated demonstration is achieved using Dynamic Movement Primitives. The effectiveness of the teleoperated LfD module has been demonstrated both in simulation and on a TIAGo robot in a variety of poses. An experiment is conducted in which a state-of-the-art motion planner was compared to the proposed LfD method and the ability of human participants to predict the goal object of the robot. Using the teleoperated LfD method, the ability to predict the goal objects increases significantly and the human is more confident in making the prediction (P = 0.0102 and P < 0.001, respectively). This means that with the learning method a more legible grasp was generated than with the state-of-the-art motion planner.

For autonomous navigation of mobile robots in an unknown environment, mapping the robot's environment, and localizing its relative position in the environment is of utmost importance. However, in a known and controlled environment, being able to localize its position at a given time instant is sufficient for the robot to navigate autonomously. Therefore, to solve the problem of pose estimation of a mobile greenhouse robot, Visual Odometry (VO) was the best-suited approach. This thesis focuses on the pose estimation of a greenhouse robot using images from an onboard camera only for its motion on the guided rails. A review of the literature revealed that most of the current state-of-the art VO techniques were developed for autonomous navigation in an urban environment or an indoor space. Even though there were publicly available benchmark datasets for VO evaluation such as KITTI and TUM datasets, there was no VO dataset available for a greenhouse environment. A key challenge in collecting visual datasets in a greenhouse was obtaining the ground truth poses associated with all the images. Therefore, an experimental setup was developed to collect VO datasets along with ground truth using store-bought table plants, tomatoes, a mobile camera, and a stereo camera. The experimental design was focused to emulate the greenhouse environment as much as possible. The camera's motion during the experiment was also chosen to emulate the robot's motion on the guided rails. Subsequently, multiple pose estimation algorithms were run on the collected datasets for a comparison study.

Over the last three decades, labor shortages and increased labor costs in greenhouses have driven investments in the development of agricultural robots. Priva has been developing a robot to automate the repetitive task of deleafing tomato plants. The main challenges for commercializing the robot are in terms of cost-efficiency and cut-quality. High success rates in the detection of leaves and subsequent cutting action are required, which are limited by occlusion and the unstructured, dynamic greenhouse environment. As a consequence of manipulator constraints, detected leaves can not always be cut from the current robot position. In addition, detected leaves with an unfavorable approach angle are skipped as cut quality can not be guaranteed. Conversely, many leaves are not detected at positions from where they can be cut due to detection limitations. By combining detections from different viewpoints, the detection rate can be increased significantly. Moreover, fusing multiple detections of the same object is known to improve detection accuracy. In this thesis, object trackers are investigated, aiming to increase the number of successfully cut leaves by tracking leaves over different robot positions and fusing multiple detections of the same leaf.

Twisted and coiled polymer muscles (TCPMs) show promise to function as artificial muscles, because of their lightweight, low cost, large contraction, and respectively low hysteresis. A TCPM contracts when it is heated and extends when it is cooled. Different modeling and controlling techniques have been implemented. \cite{VanDerWeijde_2019} implemented a self-sensing model that does not need large apparatus for measurements of force and deflection. The goal of this thesis is to design a force controller that works with this model. Parameter estimation of the self-sensing model is done. The fit of the model is not high enough for control. A first order black-box model is estimated and used instead. A P and PI controller is simulated and tested on the setup. The force oscillates around the reference value. This is because the actual model is of order 2. A D-action needs to be added to dampen the oscillations. The integral action reduces the max to min and vice versa input behavior. The model parameter differs for each TCPM. The controller parameters have to be adjusted for each TCPM. This is impractical in large-scale applications. Further research can be done into using model-free controllers.

This MSc thesis presents the development of a viewpoint optimization framework to face the problem of detecting occluded fruits in autonomous harvesting. A Deep Reinforcement Learning (DRL) algorithm is developed in order to train a robotic manipulator to navigate to occlusion-free viewpoints of the tomato-target. Two Convolutional Neural Networks (CNN), You Only Look Once (YOLO) version 3 and Mask Regional Convolutional Neural Network (Mask R-CNN), are trained and evaluated in order to obtain visual information from the tomatoes. The two trained CNN achieve high detection accuracy and surpass other fruit detection methods. The instance segmentation Mask R-CNN result and an image processing algorithm are applied in an occlusion modelling method. The vision-based reward formulation for the DRL algorithm is closely related to the occlusion modelling metric. The DRL algorithm is trained and evaluated in a simulation environment by mounting a camera on a robotic arm. The use of different reward scheme and exploration strategy during DRL training shows their effect on training performance. The DRL training performance is assessed by illustrating the maximum visible tomato-target fraction per episode, the steps needed per episode, the trajectory of the robot's end-effector, and the initial and final tomato-target viewpoint in each episode. The evaluation results show a satisfactory performance, which depends on the reward and exploration strategy, as in the majority of the cases the robot can fully see the tomato-target after a few steps.

This thesis is a contribution to the research on Active Inference for Robotics. Active Inference is an intricate, intriguing theory from neuroscience, a field in which it has already gained a greater following and popularity. This theory, based on the underlying Free Energy Principle, provides a unified account of perception, action and learning in the biological brain. It has great explanatory power of the function of the biological brain and furthermore it is mathematically well-defined. This property makes the theory suitable for a translation to robotics, in which it can also provide a unified account of action and perception. This is not only elegant, but potentially very powerful too. The research for Active Inference in robotics is young, but the current research already shows that Active Inference indeed has great potential for robotics control. Literature on Active Inference is narrow and complex, and provides a lot of concepts to work with in a translation to robotics control. Once such concept are the generalised coordinates of motion, which are the instantaneous derivatives of a dynamic variable. The incorporation of generalised coordinates, especially in combination with the assumption that the noise encountered in a dynamic environment is coloured, has great potential to be beneficial for both action and perception when it comes to robot control in real environments. Generalised coordinates provide a reference frame for the gradient descent that is applied to provide the action and perception laws, which in a dynamic setting has to `hit a moving target'. Furthermore, in combination with coloured noise the generalised coordinates are advantageous for dealing with such noise. In this thesis, detailed research is provided with regards to the application of generalised coordinates in Active Inference for robotics. Current research for robotics in which Active Inference has been applied doesn't exploit the full potential of generalised coordinates. Therefore, this research aims to explore the constructs necessary to apply generalised coordinates of motion in an on-line Active Inference control loop of an LTI State Space system. A detailed derivation of the generalised precision, which relates generalised coordinates and coloured noise, is provided. A method for obtaining generalised output by means of finite differences is proposed, that constructs generalised coordinates from the on-line data in scenarios in which the environment does not provide the required generalised coordinates naturally. The method is implemented in the simulation of a one degree of freedom SISO LTI State Space scenario which highlights the potential but also the difficulties still faced when applying Active Inference for on-line robotics control. Besides the detailed derivations of some aspects of Active Inference for robotics, open problems are identified and suggested for future research that can potentially yield methods to apply Active Inference in robotics at full capacity, providing a true biologically plausible robot control method.

This work addresses the problem of exploration and coverage using visual inputs. Exploration and coverage is a fundamental problem in mobile robotics, the goal of which is to explore an unknown environment in order to gain vital information. Some of the diverse scenarios and applications in which exploratory robots can have a significant impact include search and rescue missions, environmental monitoring, and space exploration. Specifically, we focus on the aspect of finding areas of interest, also referred to as targets, in the environment. In this thesis, we propose a deep reinforcement learning based approach relying solely on visual observations. In particular, our method builds upon the off-policy, actor-critic, Importance Weighted Actor-Learner Architectures (IMPALA) framework by including a set of novel auxiliary tasks, i.e. Pose Estimation and Local Map Prediction. These auxiliary tasks are inspired by Simultaneous Localization and Mapping (SLAM) approaches to exploratory robotics problems. The intuition is to assist internal representation learning and build locale specific knowledge by teaching the agent to predict its position and orientation, as well as transfer the visual information to information about the its local proximity. Experiments conducted in the DeepMind Lab simulation environment show improved performance over the base IMPALA agent and demonstrate the effectiveness of these auxiliary tasks. Furthermore, we investigate the performance of the agent, trained through various stages of curriculum, compared to a human controlled agent. The trained agent is shown to outperform the human in the majority of tested scenarios.

Deep Reinforcement Learning (DRL) enables us to design controllers for complex tasks with a deep learning approach. It allows us to design controllers that are otherwise cumbersome to design with conventional control methodologies. Often, an objective for RL is binary in nature. However, exploring in environments with sparse rewards is a problem in RL, and finding positive reward becomes exponentially more difficult with increased environment complexity. For this project, our objective is to design an RL based controller for the landing of a quadcopter on inclined surfaces. Landing is defined as reaching these inclined surfaces with reasonable speed, such that no damage is done to either the quadcopter or the surface to land on upon impact. We aim to use a binary reward for this task. We use methods to aid exploration in sparse reward environments, namely Hindsight Experience Replay (HER), and non-optimized demonstrations. HER can resample goals from the demonstrator data and the policy rollouts. The resampling of goals is done by considering a portion of the visited states during policy rollouts as the intended goals. The demonstrations are non-optimized in the sense that the demonstrations do not follow the same objective as ours. We consider demonstrations valid if these demonstrations are obtained from arbitrary stable policies. Our results show that the RL system does generalize to other goals when using HER and demonstrations. The demonstrations are not imitated as were to happen in pure imitation learning. HER, on the other hand, enabled us to receive reward in our complex environment, while also allowing us to experience multiple goals in one policy rollout. We found that lack of HER and demonstrations were not able to overcome the problems of exploration in sparse reward environments. We found that landing a quadcopter on inclined surfaces using an RL controller is feasible. Our trajectories clearly showed a swinging motion which in theory should be a valid control strategy for this problem. This swinging motion results in dead spots with the quadcopter being in a state with a minimal translational and rotational velocities under a relatively large angle. Further research is needed to increase the accuracy and robustness of our RL based controller.

There are many stages that involve humans handling food objects in the processing chains from farms to stores. For some of these tasks it is desirable to look for a robotic solution to either assist the human or even take over that task, e.g. if it is physically demanding, imposes contamination risks or because of economical considerations. Moreover, recently the COVID-19 pandemic revealed even more vulnerabilities in our food processing chains, when seasonal labourers where blocked at borders and some food processing sites turned out to be "corona hotspots" with the result that whole food processing chains were disturbed. A step towards solving some of these problems is studying robotic grasping, since it is a crucial skill for many manipulation tasks. This work focuses on force closure grasps for pick-and-place tasks. Since humans grasp novel objects effortlessly, a human inspired approach is proposed that combines visual and tactile sensory information and learns from its mistakes thanks to reinforcement learning. Visual features obtained from an RGB-D camera in combination with pressure information from sensors on the finger tips of the robotic hand are used to set the desired grasp force adaptively while holding the object. A novel algorithm is proposed for learning to grasp with minimal force: LIFT (Learning of Initial Force and Tuning). The novel grasp approach is evaluated in a Gazebo simulation environment and on a real-world robotic setup. The approach results in successful grasping in both simulation and on the real-world setup in tens or hundreds of learning interactions, depending on the size of the state-action space. Success rates of 96.3 % and 96.7 % are obtained in simulation and on the real-world setup, respectively. The results from the experiments indicate that the approach is successful in grasping with minimal force.

This thesis is motivated by evolutionary robot systems where robot bodies and brains evolve simultaneously. In such a robot system, `birth' must be followed by `infant learning' by a learning method that works for various morphologies evolution may produce. Here we address the task of directed locomotion in modular robots with controllers based on Central Pattern Generators. We present a bio-inspired adaptive feedback mechanism that uses a forward model and an inverse model that can be learned on-the-fly. We compare two versions (a simple and a sophisticated one) of this concept to a traditional (open-loop) controller using Bayesian Optimization as a learning algorithm.
The experimental results show that the sophisticated version outperforms the simple one and the traditional controller. It leads to improvement in performance and more robust controllers that cope better with noise.

In this thesis, a geometry-based grasping method for vine-tomato is proposed. The method utilizes a geometric model of the robotic hand and truss to determine an optimal grasping location on the truss stem. This allows grasping a truss without requiring delicate contact sensors or complex mechanical models. A computer vision pipeline is developed to identify required geometric features of the tomatoes and truss stem. To validate the proposed grasping method, real-world experiments were conducted using an RGB-D camera and a robotic manipulator. By combining plastic and real stems with artificial tomatoes two types of trusses were constructed. A success rate of 92% was obtained on the former type and 83% on the latter. The newly developed method is shown to be capable of grasping vine tomato. For real-world implementation future studies are needed, several recommendations are given.

Machine Learning Control is a control paradigm that applies Artificial Intelligence methods to control problems. Within this domain, the field of Reinforcement Learning (RL) is particularly promising, since it provides a framework in which a control policy does not have to be programmed explicitly, but can be learned by an intelligent controller directly from real-world data, allowing to control systems that are either arduous or even impossible to model analytically. However, in spite of such considerable potential, the RL paradigm poses a number of challenges that effectively hinder its applications in the real-world and in industry. It is therefore critical that research in this field is advanced until RL-based controllers can be practically demonstrated to be real-world feasible and reliable. This thesis report presents the attempts made at applying control strategies based on Reinforcement Learning to solve a precise positioning task with a physical experimental setup. The setup at hand is a magnetic manipulator (magman) characterized by a high degree of nonlinearity. The controller uses the spatially continuous magnetic field generated by four actuators to displace a steel ball, constrained to move in one dimension, towards a reference position. Two different implementations of the Q-learning algorithm (Sutton, Barto, et al., 1998) were deployed. In spite of the good results obtained in a simplified simulated environment, both implementations failed on the experimental setup. The negative outcome of these experiments is mainly due to the fact that, since the task at hand is an accurate positioning task, the reward obtained by the learner while interacting with the environment is too sparse for it to be able to learn a stabilizing control policy. Other factors have presumably contributed to the controllers’ failure, such as the circumstance that the agent does not have access to the full system state information and a sub-optimal tuning of the algorithms’ hyper-parameters. Besides model-free RL, the Value Iteration model-based method was successfully applied both in simulations and with the experimental setup. The present findings suggest that, in order to solve the magman task with model-free RL, more sophisticated algorithms need to be deployed, such as for example an agent that can naturally deal with continuous state and action spaces, as the DDPG algorithm (Lillicrap et al., 2015), with exploration carried out in the parameter-space rather than in the control action space (Plappert et al., 2017), in addition to a more optimal exploitation of the information extracted from the environment, for example using Hindsight Experience Replay (Andrychowicz et al., 2017).

Deep Learning performance dependents on the application and methodology. Neural Networks with convolutional layers have been a great success in multiple tasks trained under Supervised Learning algorithms. For higher dimensional problems, the selection of a deep network architecture can significantly improve the accuracy of the network, however for low dimensional problems this might not be true. Shallow Neural Networks have successfully matched the performance of Deep Neural Networks in multiple tasks in the past and have been shown to be expressive enough to represent low dimensional continuous control problems. Through the thesis, the performance and expressiveness of Shallow and Deep networks is compared for low-dimensional continuous control tasks. The thesis begins by comparing the two network architectures in a Supervised Learning algorithm and progresses towards state-of-
the-art Reinforcement Learning algorithms. The thesis provides an empirical approach
towards comparison of neural networks and makes conclusions that can support the selection of a network architecture for continuous control applications using Deep Reinforcement Learning algorithms.

Motion planning for Autonomous Ground Vehicles (AGVs) in dynamic environments is an extensively studied and complex problem. State of the art methods provide approximate solutions that make conservative assumptions to provide safety and feasibility. We aim to outperform current methods by following a trajectory optimization-based approach, providing a Local Model Predictive Contouring Control framework. Our method allows AGVs to execute reactive motion while tracking a locally parametrized reference path, anticipating on the predicted evolution of the environment. Given the static environment configuration in an occupancy grid map and dynamic obstacles represented by ellipses, we formulate explicit collision avoidance constraints. Well-informed planning decisions are made through a cost function with trade-offs between competing performance variables such as tracking accuracy, maintaining the reference velocity, and clearance from obstacles.

An efficient implementation of the method is presented that satisfies the real-time constraint of online navigation tasks. Furthermore, we present an implementation of a complete navigation system to emphasize our ability to deal with real sensor data and onboard processing. We show that the general definition of the framework applies to both unicycle and bicycle kinematic models, commonly used to represent mobile robots and autonomous cars, respectively. Simulation results for a car and experimental results with a mobile robot show that our method is a feasible and scalable approach. Proposed improvements of the method include 1) considering obstacle velocities and positioning with respect to the AGV in the penalty term that creates clearance, 2) incorporating prediction uncertainty of obstacles, and 3) improving our method that deals with infeasible solutions of the optimal control problem.