This master thesis explores the possibilities of using a quadruped robot for manipulating a milking cup, using only body movments. This shows the capabilities of a quadruped for manipulating an object, without the need of higher degree of freedom manipulator arms.

With mobile robotics being applied for more and more complex applications, their autonomy should be preserved. While a lot of research is performed into the direction of failure prediction for autonomous processes or systems, the field of mobile robots has received less attention. Proactive failure prediction for mobile robots is a useful tool to prevent unwanted downtime and undesired damages. This work attempts to fill this research gap by showing the applicability of anomaly detection methods for failure prediction in the field of mobile robots. Specifically, we employ an unsupervised Variational Autoencoder to predict failures in the operational data from the Discovery Collector, a manure cleaning robot developed by Lely Industries. We elaborately showcase the feature engineering steps which yield the best performance, provide the performance of three general datasets, and state promising next steps for root cause classification which is enabled by accurate failure prediction. All in all, our work shows that the use of feature offsets, calculated from desired values compared to actual values, enhances the model performance tremendously. The provided datasets showcase F1-scores ranging from 0.64-0.76, showing the proposed solution is able to solve the failure prediction problem in the field of mobile robots, while highlighting the encountered limitations for future improvement.

Large Language Models (LLMs) possess significant semantic knowledge about the world, making them valuable for high level control for robots through Vision-Language-Action (VLA) models. These models integrate an LLM to deduce semantic knowledge from a robot’s vision and natural language inputs, facilitating real-world actions. Despite their potential, VLA models are a relatively new research area, with applications mostly limited to simulations or household tasks and insufficient validation in broader contexts. This study aims to develop a LLM-Enhanced Robotic Affordance and Control System (LERACS) for robotic manipulation in applied cases such as the management and maintenance of electrical grid infrastructure. LERACS is designed to visually ground manipulable objects and decompose tasks based on user instructions within a human-robot interaction chat interface, using ChatGPT. A system validation and an AI user experiment were conducted to evaluate its effectiveness in interpreting and performing actions based on pre-made and synthetically generated user instructions. These assessed LERACS’ performance across various settings and instructions. Results indicate high success rates in environmental interpretation and task execution, with robust labeling accuracy, especially in complex settings. Feedback from the AI user experiment highlighted LERACS’ adaptability, identified areas for improvement, and demonstrated its practical utility across diverse settings and task complexities.

Despite significant advancements in computer vision models, their ability to generalize to novel object-attribute combinations remains limited. In Compositional Zero-Shot Learning (CZSL), the goal is to recognize all possible attribute-object combinations while training on only a subset of these compositions. We demonstrate that Vision-Language Models (VLMs) are suitable for CZSL due to their ability to understand both text and image modalities, and we show the importance of cross-modality fusion. Existing methods for CZSL mainly focus on image classification. In this thesis, we enhance CZSL in object detection without forgetting prior learned knowledge. We use Grounding DINO, a VLM for object detection, and create auxiliary tokens for the words present in the training dataset, training only these to mitigate forgetting. This technique, referred to as Compositional Soft Prompting (CSP), was initially used for image classification, and we incorporate it into Grounding DINO to leverage it for object detection. We show that for CZSL, it is beneficial to anticipate the orientation of embeddings in the embedding space, promoting independence between attributes and objects. Additionally, by anticipating possible compositions that could exist based on the attributes and objects present in the training data, we assign soft labels for partial correctness using a method we call Compositional Smoothing. This approach guides the model to learn what the compositions are composed of rather than learning the compositions themselves. We refer to the combined approach of these methods as Compositional Anticipation. Our approach achieves a 70.5% improvement over CSP on the harmonic mean (HM) between seen and unseen compositions on the CLEVR dataset. Furthermore, we introduce Contrastive Prompt Tuning to incrementally address model confusion between similar compositions. We demonstrate the effectiveness of this method, achieving an increase of 14.5% in HM across the pretrain, increment, and unseen sets. Collectively, these methods provide a framework for learning various compositions with limited data, as well as improving the performance of underperforming compositions when additional data becomes available.

With the advancement of Artificial Intelligence leading to increasingly human-like outputs, assessing a machine’s ability to exhibit human-like intelligence has become more essential than ever. This study aims to investigate how human-like chess players perceive four conditions: one human opponent and three different types of algorithms. One of these algorithms, Maia, has been trained on human data and aims to play the most human-like move. In a custom-designed experiment similar to a Turing test, chess players faced off against Maia, Stockfish and a human without knowing their opponent’s nature. After each game, the chess player assessed how human-like the moves of the opponent were and estimated whether they played against an engine or a human opponent. During the game, participants were asked to think aloud about their next move and react towards the moves of the opponent. Additionally, the gaze of the player was captured with the SR EyeLink Portable Duo at 1000Hz, with the goal of finding differences within the player’s gaze while participants tried to discover the nature of their opponent. Results from the experiment revealed that, based on responses to a subjective questionnaire, the perceived humanness of Maia is statistically similar to a human and different from the other two chess engines. From the analysis of the voice recordings, categories of sentences were identified that could suggest recognition of the opponent, specifically: "expected", "unexpected", "human-like" and "engine-like". From the eye-tracking results, the average fixation duration and pupil diameter changes following the opponent’s move were compared for each condition, but showed no statistical differences between conditions. In summary, Maia was perceived more human-like compared with other chess engines. However, differences in underlying cognitive processes on how the human perceived this difference in a Turing Test experiment were not identified.

The increasing presence of robots calls for a more seamless and information-rich communication method between humans and robots. This paper explores how natural user interface (NUI) modalities, particularly speech and gesture controls, can be used through augmented reality (AR) to operate robots. The increasing presence of robots calls for proper evaluation methods of how to use AR for operating mobile robots. The study uses the Microsoft HoloLens and the robot, named Spot, from Boston Dynamics as primary technologies. The research consists of a user study consisting of 218 participants, one of the largest participant pools for this field to date. The experiment consists of walking the robot over a trajectory with discrete steps, with the perspective of following the robot or standing on a predetermined stationary point. To support the control of the robot, visual information and feedback are included in the HoloLens. Speech control showed the best time performance of the experiment, regardless of the perspective condition. Conversely, errors made during the trials were the majority for the speech condition, due to the waiting time of the speech recognition that caused participants to repeat the commands. The walking condition gave participants the impression that control commands were more intuitively mapped to the robot's motion. Overall, the participants preferred the speech control method while walking with the robot, and the least preferred method was using gestures in a stationary perspective. Even though the speech was the preferred control method and perspective-taking was preferred by participants, this was only for the experiment and task discussed in this paper. Both control methods have different characteristics that make them favorable to be used for specific tasks. Speech and gestures can be used for different tasks when operating a robot with Augmented Reality glasses; preference will depend on the task at hand and the control method design.

This study investigates the influence of synthetic data on the accuracy of 6D pose estimation in RGB images compared to RGB-Depth image-based methods. Additionally, it aims to examine how this performance varies across different types of small chess pieces during a picking task with a robotic arm. The methodology involves 3D scanning the chess pieces and generating a dataset of 61,910 synthetic images with diverse domain randomization. Using this synthetic dataset, six unique 6D pose estimation models were trained for each type of chess piece. The models were validated using a real-world 6D pose estimation dataset, and the obtained results were compared with those from the estimations on a synthetic dataset. It was observed that synthetic data can be used for bridging the visual simulation to reality gap. However there is superior performance on synthetic data compared to real-world data. This implies that results obtained in a synthetic environment cannot be directly projected to real-world scenarios. Also a noticeable decrease in accuracy in both object detection and 6D pose estimation was observed with respect to camera distance for real-world images, primarily linked to the reduced size of chess pieces in the images. Notably, each chess piece exhibited improvement after depth refinement, with accuracy closely linked to the performance of the depth camera. A picking experiment was also conducted, revealing that models relying solely on RGB data achieved a positive grasping rate of approximately 52%, while RGB-Depth-based methods reached around 70%. The findings underscore the potential for successfully picking chess pieces with and without depth refinement, emphasizing the feasibility of bridging the visual simulation to reality gap. Additionally, the study suggests several avenues for future research, including comparing synthetic data with real-world data, further exploring the training process, and introducing domain randomization in the picking experiment, such as background changes or different distractors. Furthermore, it suggests investigating diverse approaches to improve depth accuracy.

As the global population continues to increase, so does its dependance on food, which has led to the development of more efficient food production methods. Greenhouse horticulture and genetically modified seeds have increased crop yield and decreased water and energy consumption. However, delicate greenhouse-grown vegetables continue to necessitate manual labour. The purpose of this thesis is to design and evaluate a vision system capable of detecting the spatial structure of greenhouse vegetation in order to automate plant repositioning, pruning, and leaf removal. The system employs a robotic arm and a single camera to generate custom plant features, match them, and generate a 3D model of the plant’s structure. The thesis discusses the system’s requirements, design, and evaluation, including the segmentation of plant outlines using deep learning networks, the construction of a 2D plant structure using constrained Delaunay triangulation, and triangulation-based depth calculation. Analysis of three-dimensional precision reveals a correlation between camera positional errors and the algorithm’s depth estimation. The findings support the viability of using a vision system to automate greenhouse duties if certain modifications are made. Contributing to ongoing efforts to increase greenhouse agriculture’s efficiency and reduce manual labour.

In the engineering industry, all structural parts have to be designed as efficient and lightweight as possible. Traditionally, the design process has been carried out through manual design iterations, which can be time-consuming and require significant engineering expertise. Over the last decades however, several computational design techniques like Topology Optimization and Generative Design have been developed to support engineers in the structural part design process. Even though these techniques can have a positive influence on the design process, they both also have their downsides. Topology Optimization only gives a single result that is often a local optimum, influenced by boundary conditions and numerical settings. Commercial Generative Design tools explore multiple design options in a single run using AI algorithms, but need cloud-based systems to carry out their demanding simulations which still take several hours per run. It is however expected that a combination of the two, a Topology Optimization based Generative Design approach in the form of an auxiliary tool, has potential to improve the early stages of the design process even more. With such a design approach, multiple design solutions are explored quickly to study the effect of boundary conditions or numerical settings. This can help designers by giving direction and insight in trade-offs between multiple objectives, early on in the design process when design decisions still have the highest impact.

The goal for this research was therefore to research the effect of such a Topology Optimization based Generative Design approach on the design performance and experience. In order to do so, a robust and user-friendly TOP-GD tool was created. In this tool, multiple design solutions are explored quickly by implementing a batch-run setup that varies several chosen parameters, without needing to manually run several optimizations consecutively. Calculations are done with a simple TO script using coarse geometries, and without taking into account manufacturing methods yet. This asks for less demanding, detailed and complicated calculations than AI-based Generative Design tools currently offer, while at the same time moving from a single TO result to generating a range of candidate solutions. A lot of effort was put in the user-friendliness of the TOP-GD tool, enabling an easy workflow for the setup of design problems and a clear presentation of the results by means of a simple GUI.

The use of the TOP-GD tool in the design process was evaluated in an experiment, where it was compared with a more simple TO tool and a basic manual design approach using just pen and paper. This was done by giving the participants of the experiments three simple design assignments, that they had to carry out using each of the design approaches one by one. Evaluation of the approaches was done by comparing the design performance, and assessing the design experience with a survey and using Eye-tracking techniques.

The results of this experiment did not show enough evidence to conclude that the different design approaches had an effect on the design performance for the simple assignments executed during the experiment. However, the results of the survey show a clear positive impact of both the TO tools on the design experience, compared to manually designing. Furthermore, the TOP-GD tool has the largest positive impact on the design experience and its use in the design process is considered a big improvement, especially in quickly exploring new design directions and creating overview. This confirms the expectation that a Topology Optimization based Generative Design approach has a positive effect on the early stages of the design process. The differences found with Eye-tracking between the TO tools support this, although a more extensive experiment should be done to convincingly confirm this conclusion.

Different paradigms can be used to evoke brainwaves. These brainwaves can be interpreted as commands that can be used to control different applications. These frameworks that interpret the brainwaves are called brain-computer interfaces. Steady-state visually evoked potentials is one of these paradigms that uses external stimuli flickering at fixed frequencies to evoke brainwaves. This paradigm is fast to issue, is reliable, and needs no training time of the user. However, to be able to distinguish multiple commands it is important to distinguish between multiple commands reliably. The fundamental metric that determines the signal quality is the signal-to-noise ratio. The signal-to-noise ratio is measured in decibels and is the ratio in power between the signal that is evoked around the stimulus frequency and the power of some baseline, referred to as noise.
To extract the power the discrete Fourier transform is calculated from the measured brainwaves. The brainwaves are measured in millivolts over time. These brainwaves can be recorded using implants in the head named intracortical or external, such as electroencephalography. External methods such as electroencephalography are much safer for the user.

Important to recognize is that one of the contributing factors to signal-to-noise ratio are the characteristics of the external stimuli displayed. Research is still trying to figure out the exact relationship between stimuli characteristics and signal-to-noise ratio. However, the experiments of previous research lack the context of the gaze of the subjects to explain the electroencephalography recordings. In this research, this is attempted to be solved using eye tracking. Furthermore, there is a research gap in the
the scientific field surrounding signal-to-noise ratio and stimuli characteristics as the effect of surrounding
stimuli on the measured signal-to-noise ratio of the target stimulus has never been investigated.

This research attempted to solve these problems by performing 2 experiments with 6 participants. One experiment shows a single stimulus across various shapes (triangles, squares, and circles), colors (red, green, and white), frequencies (9, 13, 19, and 25Hz), and sizes (10.000, 20.000, 30.000 pixels). The other experiment simulates the natural environment of the external stimuli across the same frequencies, colors, and shapes. The natural environment of a single stimulus is actually surrounding stimuli at different frequencies, as applications often require the ability to distinguish between multiple different commands. Thus, to state the main research question: "What is the relationship between the stimuli characteristics and the measured SNR?".
This question is answered by dissecting the effect that color, shape, size, frequency, and surrounding stimuli have on the signal-to-noise ratio.

Inadvertent bird nest destruction by autonomous mowing machines poses significant threats to the breeding success of meadow birds. Drone-based detection methods represent the current state-of-the-art for bird nest localization to attain mower circumvention. However, they only identify 80% of bird nests with average localization error of 3.344 meters and are restricted to specific application times. This paper introduces alternative, fully automated nest localization systems integrated with autonomous mowers. Two strategies are proposed, 1) Directly detecting bird nests using thermal data, or 2) Indirectly, by tracking birds and extrapolating their trajectories back to their nests using RGBD data. These methods were validated with warmed chicken eggs hidden in grasslands and with drones simulating bird flight. YOLOv8 models were modified for both approaches. The thermal localization method is able to detect all bird nests with an average confidence of 73.4%. It allows for real-time localization and yields one unnecessary nest circumvention for every ten bird nests saved due to false positives. This method is shown to be effective in all breeding season temperatures, both day and night. Conversely, the trajectory extrapolation method detects birds with an average confidence of 82.2% and has localization error of 0.794 meters. Birds taking flight prematurely or from locations other than nests impact the number of bird nests saved and the number of unnecessary circumventions. It is demonstrated that this method fails to detect birds during nighttime. In conclusion, an automated thermal-based localization system integrated with autonomous mowers outperforms both RGBD- and current state-of-the-art drone-based methods. This study highlights therefore the potential of thermal-based solutions for bird nest protection in grasslands.

This study investigated the human-robot interaction of three different augmented reality interfaces that enabled an operator to control a robot arm. The first control methods gives the operator complete manual control by moving a holographic object which the robot arm copies (Direct control). The second control method allows the operator to manually set holographic waypoints which the robot arm autonomously tries to execute (Waypoint control). And the third control method, gives the operator the capability to specify a target location which the robot autonomously moves towards (Command control). An experiment was designed to give 30 participants the capability to control a Franka Emika Research 3 robot arm with a Hololens. During the experiment, the participant had to use each control method to move the end-effector of the robot arm from a starting position to a goal position in 9 different environments. The 9 environments were divided into three manually created complexity levels (Easy, Medium, and Hard), which were based upon the performance of an informed RRT* path planner that was simulated in 67.500 unique environments. The experiment results showed that Command control outperformed the other two methods in the success rate, path length, and operation time. In terms of number of routes Command control had in general a lower number compared to Direct control, showcasing that Direct control has more flexibility in route selection. Furthermore, Direct control required more skill to get a similar performances in path length, operation time, and number of collisions as Command control. The design of Waypoint control made this method too unpredictable for the participants to use proficiently, which was underscored by the very low success rate, the long operation time, and large path length of this method. Comparing the performance of the control methods against the complexity levels did not show many significant differences, except for the success rate that decreased with a more complex environment. In summary, each control method will require unique features to completely unlock the possibilities of an augmented reality interface. The three control methods show different performances, each with their own strengths and weaknesses. And the choice of a control method is therefore dependent on the task at hand.

With the introduction of automated driving systems come benefits such as the improvement of traffic safety. However, with an increasing level of automation in vehicles also comes an increase in interaction with in-vehicle technology by drivers while they are meant to supervise the automated driving systems. Due to more interaction with in-vehicle technology and a vigilance decrement of the driver in Level 2 driving, an increase in reaction time of the driver is seen when intervention is needed by the means of a take over request. This delay in reaction by the driver opposes the benefit of the introduction of automated driving systems and causes hazardous situations. To try and circumvent the effect of vigilance decrement, this paper attempts to demonstrate the reduction of noticing time of the Hands-On-Wheel warning message for drivers of Level 2 vehicles while interacting with in-vehicle technology through the implementation of a gaze-contingent interface. The results of this experiment indicate a 79.3% lower noticing time of the Hands-On-Wheel warning message when the stimulus is placed in a gazecontingent manner, while the participants engage in secondary tasks on the in-vehicle technology. The placement of the stimulus on the head unit when the participant is already looking at it reduces the primary task load of touching the steering wheel and causes for the stimulus to be seen quicker as compared to a static interface. However, the performance of the secondary task seems to decrease when using a gaze-contingent interface. This is due to the intrusive nature of the placement of the stimulus, which demands the driver to store information regarding the secondary task in their working memory while they attend to the primary task. Despite the decline in secondary task performance, the reduction of noticing times of time critical messages when placed in a gaze-contingent manner could be beneficial to the safety of autonomous driving functions where the driver has a vigilance task and is engaging in secondary tasks.

Human operators who are tasked with monitoring automation systems may experience a high visual demand to process the information streams from these systems. The visual sampling behavior of human operators can be described using mathematical models. These models can help designers improve environments where multiple signals are present for human operators to monitor, to a configuration that can be processed properly.

This study consisted of two parts. The first part investigated how peripheral vision plays a role in visual sampling behavior and task performance, specifically in the experimental eye-tracking setup presented in Eisma et al. (2018). In this setup, participants were instructed to monitor a bank of six dials, of which each dial pointer had a threshold indicator, and press a response key whenever a dial pointer crossed the threshold indicator. In the second part, the sampling models as presented by Senders (1983) are implemented to predict sampling trajectories. The sampling characteristics that resulted from the
predictions were then evaluated.

The results of the experiments show that peripheral vision plays a role in visual sampling and task performance. More specifically, sampling behavior is more evenly distributed among dials, and task performance is lower when peripheral vision is absent. The main attractor in the peripheral vision is shown to be the pointer speed. Moreover, the learning effect presented in Eisma et al. (2018) is not apparent when peripheral vision is absent.

The results of the predictions showcase the sampling behavior characteristics, some of which show similarities with the results from the experimental data.

Background: Merging on a highway is a complex driving task that requires a lot of interaction with other road users. During these tasks, a driver is required to evaluate gaps in space and time between the themselves and other road users and obstacles in order to arrive at the right moment to merge onto the highway. To improve safety and increase road efficiency, it is necessary to understand the decision process during merging decisions. A way of achieving this, is to understand what visual information humans use during this decision process. This study investigated the relation between gaze location and gap acceptance decisions during highway merges.
Methods: An experiment was performed in which 26 participants monitored an automated vehicle (AV) that was driving on a highway on-ramp. The participants were given the task to train the AV in whether or not to merge in front of an upcoming vehicle that was already driving on the highway. An eye tracker was used to measure gaze data, which was used to find the relation between gaze behaviour and decision outcomes and response times. A mixed-effects logistic model was used for a statistical analysis with decision outcomes as a dependent variable and different gap sizes as predictor variables. A mixed-effects linear model was used to find the relation between response times and dwell times and the different gap sizes and decision outcomes as predictor variables. For both the decision outcome and response time model, dwell time was later included to find the effect on the predictive validity.
Results: The results show that a larger time and distance gap to the upcoming vehicle relate to a higher merging probability. For larger time gaps to the on-ramp, the probability of merging was found to be smaller. It was also found that time gaps to the end of the on-ramp significantly relate to response times, with an increase of 55ms per 1s. Larger time gaps to the upcoming vehicle significantly relates to larger response times, with an increase of 64ms per 1s. No significant relation was found between response time and distance gaps to the upcoming vehicle. The response time was found to be 0.60s longer for rejected gap decisions. The time gap to the end of the on-ramp significantly relates to dwell time, with an increase of 0.56% per 1s. The distance gap to the upcoming vehicle significantly relates to dwell time, with an increase of 0.60% per 10m. The time gap to the upcoming vehicle significantly relates to dwell time, with an increase of 0.52% per 1s. The presented results show as well that a significant relation exists between gaze behaviour and decision outcomes and response times. When analysing decision outcomes and response times, the interaction between dwell time and gap sizes should be taken into account. This improved the predictive validity of the used regression models.
Conclusion: Several pieces of evidence suggest that gaze behaviour assist in understanding the human decision making process during merging. This study can serve as a basis for cognitive models that can investigate how the relation between gaze behaviour and gap sizes, decision outcomes and response times can help to understand and potentially predict gap acceptance decisions.

With the increase in development of self-driving cars, research has been conducted to retain humanized interaction between cars and other road users, such as pedestrians. One way to retain this type of interaction is through the use of external Human-Machine Interfaces (eHMIs). This project aims to contribute to this field of research by exploring the case of eHMI-equipped, self-driving cars yielding for a crossing pedestrian. From literature it is known that light- and text-based signals are both promising ways of utilizing an eHMI. Due to the often simplified nature of these models, the goal of this project was to investigate if their findings hold up when increasing the cognitive load of the pedestrian. An eye-tracking experiment was conducted where two promising eHMI signal types (i.e., flashing lights and message) were tested in a realistic scenario, where the participant took the role of pedestrian. In the experiment, the cognitive load of the participants was varied, by applying different sizes of gaze contingent windows to the stimuli.

A range of 63 different trials of 10s each were shown to 23 participants. This set of trials included 7 different presets, aimed to test the effect of the signal type and the gaze window size independently. The participants' task during the experiment was to indicate when they deemed it safe to cross the road, by pressing the space bar. Their gaze was measured with an eye-tracker, after which it was transformed into three different metrics: saccade count, saccade amplitude and fixation duration. Additionally, a novel metric, the dispersion of the grouped gaze data, was introduced and explored. Finally, a questionnaire was conducted, investigating the self-reported clarity of the different signal types. Together with the reaction time, these metrics aimed to answer the following research question: What influence does raising the cognitive load for crossing pedestrians have on the effectiveness of text-based eHMIs as opposed to light-based eHMIs and no eHMI?

The results show that, as has been previously established, light- and text-based signals showed very similar response times and feature similar gaze characteristics when compared to each other. Both outperformed the condition where no signal was shown. Raising the cognitive load shows a decrease in saccade count and saccade amplitude, paired with a higher fixation duration. However, no proof could be found that raising the cognitive load has an influence on the effectiveness on any of the different signal types, meaning that either, the cognitive load was not raised by enough, or there is no actual effect. The dispersion showed that for the light-based signal, the focus on the stopping car is lost the quickest after the signal was shown.

The results of this study may help in explaining how pedestrians base their traffic decision on the type of signals being shown. This may also serve as a basis to further explore the possible effects of cognitive load on the effectiveness of these types of signals.

Abstract— Visual occlusion can cause object detection and classification systems to fail due to swift movement or a soiled, moist or dusty environment. Hence, this asks for a ’blind’ collision detection and classification method. This paper presents a novel blind collision detection and classification scheme for a
heavyweight, low-speed, wheeled mobile robot. By recognizing the target signal pattern in phase current data, the robot can detect whether the wheels are blocked caused by overload (G1). After this, using acceleration and phase current data, the robot will be able to differentiate between a collision against a wall
or a collision with an object between the wall, or no collision of interest (G2). Additionally, an algorithm based on the pitch angle can detect if the robot slides on top of a soft object (G3). Three different algorithms are developed that address these goals. Finally, the developed algorithms are validated in
dynamic and soiled environment (G4). The first two goals are reached by training machine learning classifiers that identify the signal pattern of its target event and place uneventful data in a separate class (open-world classification). During training the classifiers Random Forest, Multi-layered Perceptron, Decision
Tree and Logistic Regression are compared, and the two best perfoming classifiers are subsequently tested for a time-series open-world classification task. The third goal is reached with the development of a heuristic algorithm based on the running mean of the pitch angle.
Experiments were performed in a controlled environment, creating collisions with varying floor conditions, robot weight and collision objects. The best performing algorithm for blind collision overload detection has achieved 98.6% detection accuracy. The object classification algorithm can differentiate
between two types of soft objects, a wall or no event with an accuracy of 93.3%. The algorithm based on the deviating pitch angle can detect events with 100% detection rate and 0% false positive rate. Detailed implementation schemes are provided for real-time implementation, illustrating a robust framework in soiled environments. The proposed solutions can be used to improve collision detection on blind mobile robots, as well as mapping the environment using the object classification model.

A robotic vehicle must continuously determine its position within the map to traverse a path safely; this is called self-localization. Current localization methods use mainly sensors like LIDARS. However, a LIDAR does not return data points if the environment is an empty field; the laser scan of the LIDAR does not reflect without obstacles, making self-localization in those environments impossible. Self-localization with visual information from cameras can be an alternative for vehicles operating in an open environment, such as the Lely Juno, an agricultural robot operating on farms. A widely used visual Simultaneous Localization and Mapping (SLAM) method in literature, ORB-SLAM3, lacks robustness on Lely Juno's visual data; the track is lost multiple times due to a sudden drop in the number of feature matches.


This work aims to experimentally determine (a) the cause of the drop in the feature matches that eventually causes the tracking error and (b) the robustness and accuracy of ORB-SLAM3 compared to a simple visual odometry (VO) system.
The cause of the drop in feature matches is investigated outside the ORB-SLAM3 pipeline by replicating the matching thread of ORB-SLAM3 with OpenCV ORB feature matching. The robustness and accuracy of ORB-SLAM3 are compared with a simple visual odometry system by comparing the trajectories and the minimum number of matches found in the sequences.

The sudden drops in feature matches could not be replicated outside the ORB-SLAM3 pipeline with the brute-force-based ORB matcher in OpenCV. While in some cases, the decrease in feature matches is related to the image itself (e.g., blur, movement), the leading cause is complicated. In this work, the leading cause is further reduced to the quality of the disparity map, the type of matcher used, and the use of grids when extracting features. The simple VO system is more robust than ORB-SLAM3. However, the absolute pose error is worse and unsuitable for reliable navigation for long trajectories.

For the Lely Juno, a simple visual odometry system can only be used for short distances outside. Therefore, investigating the issues in ORB-SLAM3 is an excellent direction from the project's point of view. More understanding is needed of the exact effects of the disparity map, the type of matcher used, and the use of grids on the number of feature matches.

The contributions of this work are insights into the reasons behind the tracking errors, exploring the effect of different ORB parameters and datasets on the matching performance, and creating two new datasets in low-textured and repetitive agricultural environments. Investigating the exact impact of the disparity map, the type of matcher used, and the use of grids on the number of feature matches is left for future work.