Behaviour trees (BTs) serve as a powerful hierarchical structure for task execution, simplifying complex tasks but posing challenges in their manual design. The automatic generation of BTs addresses this concern, yet often lacks robust failure recovery options. This study presents Failure Recovery with Ontologically Generated Behaviour Trees (FROGBT), a novel approach bridging this gap by integrating ontological reasoning into the process of automatically generating BTs. This integration establishes a profound link between an agent’s knowledge and its capabilities, offering contextual insights into the agent’s skills. FROGBT enhances skill representation for planning and recovery. The approach’s effectiveness is indicated by its efficiency compared to the state-of-the-art framework for skill-based control, SkiROS, in a similar task. It showcases generality, uniting diverse skills, developed by various engineers, for recurring tasks, and introduces innovative failure recovery strategies. FROGBT highlights ontological reasoning’s potential to enhance BT generation with context-awareness and reasoning abilities, paving the way for future research on failure recovery concepts in generated BTs.

Unmanned Underwater Vehicles (UUVs) operate in complex environments and need to be able to adapt to sudden failures, or changes in the environment. To achieve autonomous operation, UUVs must have the ability to self-adapt in such cases. To effectively handle component failures and unexpected events, self-adaptation must be applied to both the architecture and task plan of the UUV. This allows the UUV to modify its architecture to accommodate component failures and adjust its task plan in response to unforeseen events that may render the current plan infeasible. The mutual dependencies between architectural adaptation and task planning pose a significant challenge when determining how to apply adaptation. As a result, the task planner must take into account the implications of architectural adaptation when generating a plan. This paper proposes Metaplan, a modular ROS2-based framework for applying architectural and task plan co-adaptation in a reusable way. Metaplan extends Metacontrol, an architectural self-adaptation framework, with a task planner based on Planning Domain Definition Language (PDDL). The effectiveness of Metaplan is demonstrated by evaluating it on SUAVE, an exemplar for evaluating self-adaptation frameworks for UUVs. Metaplan is shown to outperform a baseline which only makes use of a task planner. Architecture and task plan co-adaptation is demonstrated by presenting the UUV with a sudden drop in battery level, requiring the UUV to adapt both its architecture and its task plan. Furthermore, the reusability of Metaplan is showcased by applying it to a mobile manipulator scenario.

In household and retail store environments, humans efficiently identify grasp regions of an object to perform everyday tasks. To enable robots to understand the requirements of a task and the properties of an object, a novel grasp framework, called the Shape Primitive and Reasoning Grasping Engine (SPaRGE), is proposed in this thesis. SPaRGE combines a data-driven approach, a superquadric algorithm, and a logic-based approach to find task-specific object grasp poses from a partial point cloud. Multiple grasp poses are recovered using the data-driven approach, while objects are represented as multiple primitive shapes using the superquadric algorithm and logic-based approach. Logic rules are used to decide which primitive shape meets the criteria of a given task, resulting in a primitive shape indicating a grasp region. The data-driven approach then finds a grasp pose that meets the task requirements. Two versions of the logic-based approach are presented, differing in their design choices that affect the generalizability and scalability of the logic rules. The SPaRGE framework is evaluated on common household and retail store objects, demonstrating an average true positive rate of 81.2% and 86.0%, respectively, in enabling a robot to grasp objects at a desired region. Results show that the SPaRGE framework enables real-time object grasping without prior knowledge of the objects. Further experiments show that the SPaRGE framework can be adapted successfully for real-world object grasping scenarios. The proposed framework provides a significant contribution toward enabling robots to perform human-like grasping in real-world environments.

The problem of assisting users in comprehending the robotic scenario information in a retail setting has been studied. To design the system, an integrated ontology composed of several IEEE standard ontologies and a labelled property graph (LPG)-based ontology modified from the Web Ontology Language (OWL)-based ontology was proposed to symbolize information in the robotic environment. Then, a knowledge graph (KG)-based chatbot was developed to provide natural language interaction with users. A case study in a retail setting was designed, and the results were analyzed. The effectiveness of our designed system has been experimentally validated in both static and dynamic scenarios, with at least 1.5 times improvements.

Achieving human-like action planning requires profound reasoning and context-awareness capabilities. It is especially true for autonomous robotic mobile manipulation in dynamic environments. In the case of component failure, the autonomous robotic system requires reliable adaptation capabilities combined with a consistent understanding of the task, environment, and the robot’s capabilities for successful task completion. Recent research has shown that
Active Inference, a unifying neuroscientific theory of the brain, has the potential to intrinsically handle substantial uncertainties in the system, resembling the adaptability of humans. These works, however, have the following limitations: (1) no distinction is made between actions with some commonality, capable of satisfying similar tasks, and (2) actions are assumed to be always feasible when preconditions are satisfied, regardless of their context. Given the situation, certain actions satisfying a task might not lead to task succession. This work
proposes the AI for retail (Airet) framework, a novel extension of action planning through Active Inference for mobile manipulation. The Airet framework uses Bayesian networks and Ontological Reasoning to facilitate context-awareness in action planning through Active Inference. Reasoning on robot components, action-, manipulation- and environmental constraints is facilitated through a description-logic-based reasoner and an OWL-based ontology containing concepts relevant for action selection in a retail context. The capabilities of the Airet framework are demonstrated through the following cases (1) irrecoverable task & component. Failure prevention when dealing with ill-defined tasks, (2) Selection of the best action given the situation & the component capabilities through context-awareness (3) failure recovery & adaptation when dealing with component failure. Lastly, these situations are compared with
research on reactive task planning through Active Inference without context-awareness. This thesis represents a leap forward from the current state-of-the-art in Active Inference for task planning in robotics, laying the foundations for further research in the direction of this thesis

Suction based robotic actuators have potential for the bin-picking industry, but are currently not usable due the needed speed, accuracy and ability to handle novel and adversarial objects. An evaluation of the state of the art grasp pipeline developed by Mahler et al. [1] for detecting grasps on novel objects lead us to split the problem of robotic grasp generation into a global and a local component. The state of the art solution had the ability to fill the role of global evaluator leaving a local evaluator to be developed. This local grasp evaluator was obtained by training a neural network on a suction based grasp dataset, which was created using a newly developed annotation tool. The proposed grasp pipeline obtained by combining these two showed a 95.27% pick success rate for a random setup and a 90.28% success rate for solely novel objects.

Pick and place systems that operate in a warehouse setting have been studied a lot recently due to the high economic value for e-commerce companies. In this thesis, the focus is on the perception pipeline that performs object recognition given a certain input data stream (typically RGB-D images). Impressive results regarding object recognition have been reported in the last years, mainly driven by the development of convolutional neural networks. However, only very few proposed perception pipelines are suitable to adapt quickly to recognize new objects. This is considered a problem since large e-commerce companies add many new products to their inventory every day. In this thesis, efforts are made to solve this problem by proposing the use of a neuro-symbolic model in which concept learning is combined with symbolic reasoning. First, visual attributes are obtained from an input image by passing it through the neural part of the model, which consists of two β-Variational Autoencoders. The key element of the model is that the visual attributes can be recognized even if a particular combination of visual attributes have not been in the training dataset. Next, given a knowledge base and the visual attributes as inferred from the input image, a symbolic reasoner infers the most likely object ID. Hereby, the knowledge base is manually constructed and describes the relationships between object IDs and the corresponding visual attributes. The implementation of the neuro-symbolic model is first tested on a synthetic dataset which is similar to the dataset as used in the study that the neural part is based upon. Thereafter, the neuro-symbolic model is tested on real RGB-D images of the pick-and-place dataset and several iterations of the baseline model are evaluated. Hereby, the main research question is formulated as: Can a neuro-symbolic model be used to recognize unseen objects given RGB-D data as typically seen in pick and place scenarios? The best top-1 accuracy score on unseen images of the synthetic dataset was 79.5%. However, using the same neuro-symbolic model on the pick-and-place dataset, the top-1 accuracy score on unseen images was only 25.5%. In the following iterations of the model, the top-1 accuracy score was improved up to 32.4%. Analysis of the results of the pick-and-place experiments shows that the neural part is not very capable of recognizing the correct visual attributes. This likely be due (to some extend) to the simulation-to-real gap. However, further research is required to identify the exact cause(s) of the performance drop. Concludingly, the proposed neuro-symbolic model is capable of recognizing unseen images of the synthetic dataset, but is not very capable of recognizing unseen images of the pick-and-place dataset.

Robotically manipulating objects can be very challenging when not all of the environment can be fully observed, e.g. in environments which are physically and visually accessible from only a single side. By using multimodal sensory feedback and symbolic reasoning, conclusions can be drawn about the presence of objects that cannot be observed directly. This paper presents the Symbolic Reasoning for Partially Observable Environments (SyRePOE) system, which uses an ontology to maintain its world model and a reasoner to infer information about unobservable objects. SyRePOE is demonstrated in simulation and on a real robot, where it is tasked with stocking a retail shelf.

Substantial efforts are being made to make robots more reliable and safe to work around humans. Robots often perform flawless demos in a controlled environment under the supervision of an operator but tend to fail in the real world when deployed for a long period of time due to faults and environmental disturbances. A robotic system is composed of different physical and software components whose characteristics are likely to change over time. Assumptions made about the system during the design phase may change over time, especially when a system is deployed for long periods. Such changes that are often ignored, need to be considered. Environments in which a robot operates are dynamic with high uncertainty and unpredictability. In such scenarios, capabilities such as situational awareness and self-adaptation will be useful to create more robust, resilient and reliable solutions. The objective for this thesis work is to develop a framework which will embed capabilities such as situational-awareness, context-awareness and self-adaptation within a robot. This research provides a novel, reusable and generalised localisation framework called Situation-Aware Self-Adaptive (SASA) localisation framework for robotics application. This framework is developed using knowledge representation and reasoning which will provide a robot with the capability of adapting according to the situation. We have demonstrated the applicability of the SASA framework to a mobile robot localisation use case. In this research work, we have demonstrated the performance of the framework during environmental disturbances due to poor illumination and featureless environment and internal fault due to component failure. We have also demonstrated the reusability, changeability and the consistency of SASA framework. This work showed that the situational-awareness and self-adaptation capability enhances the robot’s localisation ability and provides reliable localisation even in the case of environmental uncertainties and internal faults where conventional localisation systems fail. This thesis represents a leap forward in the direction of creating more reliable and resilient solutions for robotic applications and it lays the foundations for further research in this direction.

Performing tasks in dynamic environments is still an open challenge in robotics. To be able to perform a task reliably in such scenarios, the state of the world has to be continuously monitored. In this context, most state-of-the-art perception methods focus on the recognition and classification of individual objects. However, these methods require extensive data collection and artificial neural network training, especially in complex scenes when the number of unique objects to recognise is large. This is for instance the case of retail stores, where there can be as many as 120,000 different products. Applying the state-of-the-art learning methods in this domain is not only expensive in terms of data gathering, but it will also require models so complex that product recognition would be significantly slow. This research tackles the problem of cabinet classification in a retail store, introducing a method to identify cabinets with missing or misplaced products without individual object recognition. Prior knowledge on the layout of the retail store is used to generate an image of what a cabinet is supposed to look like when it is correctly stocked. Taking an image of the current state of the cabinet and comparing this to the previously created image allows for a verification network to verify whether the cabinet is still fully and correctly stocked or not.
This research provides three main results. First, verification learning is demonstrated to transfer well to the retail store cabinet domain, maintaining high speed and accuracy. Second, this work shows that the verification network generalises well to both unseen cabinet configurations as well as unseen products, eliminating the need to include every product in the dataset used to train the network. Lastly, this research shows that verification learning transfers well from simulation to the real world to classify cabinets with missing products. However, this last result does not hold for cabinets with misplaced products, due to the smaller difference between a correctly stocked cabinet image and an incorrectly stocked cabinet image. Furthermore, while the verification network is very fast on the hardware used for this research, it will be significantly slower when applied on the less powerful hardware more commonly found in robots. This thesis represents a starting point for the detection of missing and misplaced products in retail store products, and it serves as a foundation for future research in this domain.

There is a growing demand for autonomous mobile robots in industry. The robots need to solve, among other things, the problem of navigation. Since the robots operate in semi-structured and (partially) unknown environments, the local path planning sub-problem has received a lot of attention from researchers. The result is a large variety of algorithms to solve this sub-problem, which are often developed and optimized for a specific scenario of environment. This gives the robots a large set of different possible system configurations. Traditionally, the engineer makes the choice at design-time about which system configuration to use. However, there is a demand for more autonomy in robotic systems, and it is therefore desired to have the robot make this choice at run-time. Self-adaptive systems are typically used to switch between system configuration at run-time.
The aim of this research is to improve the knowledge of the system about the navigation quality in different environments. This knowledge will help the self-adaptive system to make a better informed decision about adaptation. The contribution of this research is an approach to construct this knowledge for the system. First, metrics for navigation quality in terms of safety and performance and relevant environment characteristics in relation to the navigation quality are developed. Machine learning methods are used to train quality models that can be used to predict the safety level of a system configuration based on the measurements of the environment characteristics. These quality models are implemented in the MROS self-adaptive framework, and experiments showed that the use of quality models in this framework is useful to make better informed adaptation choices and and improves the navigation quality.

Dealing with inherently unmodeled dynamics and large parameter variations or faults, is a challenging task while controlling robot manipulators. Classical control techniques cannot usually provide satisfactory responses, and often external supervision systems have to be designed to handle the faults. Recent research has shown that active inference, a unifying neuroscientific theory of the brain, bares the potential of intrinsically coping with strong uncertainties in the system, mimicking the adaptability capabilities of humans. However, the current state-of-the-art regarding active inference in robotics is very narrow and limited. This thesis presents a novel active inference controller as a general adaptive fault tolerant solution for control of robot manipulators. The goal of this work is threefold. First, we demonstrate the applicability of active inference in robotics, deriving a control scheme which is computationally efficient and with high performance. Second, we verify the claimed adaptability properties of active inference against a model reference adaptive controller, in a simulated on-line pick and place task with a 7 degrees-of-freedom robot arm. Third, we propose a method to exploit the controller's structure to perform fault detection, isolation and recovery, without the use of external supervision systems. This work showed that not only active inference is applicable to robotics, but it also outperforms the model reference adaptive controller, and it allows to efficiently deal with sensory faults. This thesis represents a leap forward with respect to the current state-of-the-art of active inference for robotics, and it lays the foundations for further research in this direction.

The Active Inference framework is a neuroscience theory based on the Free Energy Principle by Karl Friston that has gained considerable prominence as a general theory to explain action and perception. Despite its promising future and use in different fields, its applicability for robot control has been barely investigated in the literature so far. This thesis aims at discerning if the current Active Inference implementation can be used for the control of robot manipulators and which are the requirements to do so. Firstly, an analysis of the current implementation of robot control under the Active Inference is made to understand its accomplishments and limitations. Secondly, both offline and online control schemes under the Active Inference framework are proposed. To continue with, a low-level controller is designed to be used in the Active Inference control scheme as a prior of the robot behaviour. The performance of the designed controller and the Active Inference scheme using this controller as a prior is compared to a benchmark controller. This comparison shows that the proposed implementation of the Active Inference scheme does not perform better than the current methods, but with further research on the how to implement this scheme using a real process it may become a valid alternative to these methods. Finally, recommendations on future work are given to bring this framework to online implementations and to verify the results obtained regarding the applicability of the Active Inference framework to robot manipulator control.

Kinodynamic planning is motion planning in state space and aims to satisfy kinematic and dynamic constraints. To reduce its computational cost, a popular approach is to use sampling based methods such as RRT with off-line machine learning for estimating the steering cost and inputs. However, scalability and robustness are still open challenges in these type of Learning-RRT algorithms. We propose the use of generative adversarial networks (GAN) for learning of the steering cost and inputs. Furthermore, a novel data generation method is introduced, which is easy to learn and, in terms of parameter count, scales linearly to higher degrees of freedom. In our experiments, we show that the GAN has excellent generalisation capabilities, resulting in a considerable improvement in performance compared to the state-of-the-art. Consequently, we show that our method can scale to a planar arm and is robust to data dimensionality.