To ensure safe operation of autonomous vehicles (AVs), trajectory planners should account for occlusions. These are areas invisible to the AV that might contain vehicles. Set-based methods can guarantee safety by calculating the reachable set, which is the set of possible states, for each potentially hidden vehicle. A recently published method proved in simulation experiments to reduce the cautiousness by reasoning about these occluded areas over time, assuming perfect input data. We present a novel algorithm that uses this reasoning and is applicable on a real AV with its accompanying uncertainties and imperfect sensor data. The uncertainties include sensor errors and noise, computation and communication delays and control errors in the trajectory following. This is achieved by modelling the error distributions and accounting for them in the calculations, where the confidence interval for each error is exposed as a setting. Experiments indicate that our algorithm can reduce the traversal time through an intersection by 2.2 seconds with reasoning. An ablation study of the different error measures shows that the errors in the construction of the field of view (FOV) limit the performance the most. Reducing the errors in the FOV construction is therefore the most important recommendation, besides making the method interaction-aware.

3D semantic understanding is essential for a wide range of robotics applications. Availability of datasets is a strong driver for research, and whilst obtaining unlabeled data is straightforward, manually annotating this data with semantic labels is time-consuming and costly. Recently, foundation models have facilitated open-set semantic segmentation, potentially aiding automatic labeling. However, these models have largely been limited to 2D images. This work introduces Label Any Pointcloud (LeAP), which leverages 2D Vision Foundation Models (VFMs) to automatically label 3D data with any set of classes in any kind of application. VFMs are used to create image labels for the desired classes which are then projected to 3D points. Using the Bayesian update, point-wise labels are combined into voxels to improve label consistency, and label points outside the camera fustrum. A novel Cross-Modal Self-Training (CM-ST) approach further enhances label quality. Through extensive experiments, we demonstrate that our method can generate high-quality 3D semantic labels across diverse fields without any manual 3D labeling. Models adapted to new application domains using our labels show up to 3.7× (12.9 → 47.1) mIoU improvement compared to the unadapted baselines. This ability to provide labels for any domain can help accelerate 3D perception research.

The training process of machine learning models for self-driving applications suffers from bottlenecks during loading and processing of LiDAR point clouds with large storage complexity.
Many studies aim to remedy this problem from an implementation perspective by developing efficient data loading and processing pipelines.
This study, on the other hand, explores an alternative approach by augmenting data representations to achieve lower storage complexity known as point cloud compression.

A broad analysis is presented on novel point cloud compression codecs using tensor decomposition methods.
Several point cloud representations and tensor decomposition methods are considered over a range of hyperparameter choices and compression values.
In order to assess the performance of the presented codecs: the compression rate, quality of the reconstruction, and time complexity is compared to the octree-based baseline model: TMC13.
Compared to the baseline model, the performance of the presented tensor decomposition-based codecs falls short. One of the presented codecs does notably outperform the others. This codec uses synthetic tensorization followed by sorting using z-location and decomposition using the TT-SVD algorithm.
Sorting by z-value isolates the ground plane, which is a dominant low-rank feature, which can effectively be decomposed using the TT-SVD algorithm yielding adequate results.

Several limitations of the presented tensor decomposition-based codecs are: the omission of bitwise compression on the factor matrices, and the trade-off between bitwise precision and truncation due to tensor decomposition.
Future work could improve in these areas along with considering the use of different heuristics and optimizing the tensor network topology.

The ever-increasing complexity of Artificial Intelligence (AI) models has led to environmental challenges due to high computation and energy demands. This thesis explores the application of tensor decomposition methods—CP, Tucker, and TT—to improve the energy efficiency of large Convolutional Neural Networks (CNNs) during inference by reducing energy consumption. The energy consumption of several convolution layers was measured using a watt meter across various CNN configurations and different hardware architectures (Central Processing Unit(CPU) and Graphics Processing Unit (GPU)). In addition, several regression models were fitted to estimate energy savings, incorporating memory usage. It was found that TT decomposition consistently provided the most significant energy savings across various compression ratios, influenced by CNN hyperparameters such as input/output channels, feature sizes, and kernel sizes, whereas CP decomposition was the least effective in reducing energy. The GPU implementations generally resulted in additional energy consumption, and the GPU regression models suggested a need for more complex relationships. The thesis also revealed that the efficiency of tensor decompositions might be highly dependent on the implementation details of software libraries, such as TensorlyTorch, which can significantly impact the computation and memory complexities. These findings underscore the importance of both hardware specific considerations and careful software implementation in achieving energy efficient CNNs, providing a foundation for further research in energy-constrained environments.

Humans are our best example of the ability to learn a structure of the world through observation of environmental regularities. Specifically, humans can learn about different objects, different classes of objects, and different class-specific behaviors. Fundamental to these human abilities is evolved sensory hardware and automatic pattern recognition systems thought to be powered in part by the leading neuroscience theory of predictive coding. Artificial intelligence research is often inspired by neuroscience and algorithms already exist that implement predictive coding. In this paper, we seek to evaluate a leading predictive coding video-prediction algorithm, PredNet, for its ability to perform human-like learning of the types mentioned. By successfully training PredNet on a custom Simple Shape Motion (SSM) video dataset that explicitly requires structure learning to occur in order to accurately predict the next frame, we establish that PredNet is capable of rudimentary structure learning. We investigate PredNet filters and feature maps but find scant evidence of truly symbolic knowledge, and propose instead that PredNet performs semi-symbolic learning. We perform ablation studies that reveal the aspects of PredNet that critically contribute to its structure learning ability. Finally, we detail a set of modifications made to PredNet to allow object-centric processing as a promising step change towards human-like structure learning. Evaluation results and investigations are provided. Performance was slightly worse than Baseline, likely due to a noted implementation flaw. Code and instructions to reproduce dataset creation and model training / evaluation are available at https://github.com/ofSingularMind/parallel_prednet.

Human Mesh Recovery (HMR) frameworks predict a comprehensive 3D mesh of an observed human based on sensor measurements. The majority of these frameworks are purely image-based. Despite the richness of this data, image-based HMR frameworks are vulnerable to depth ambiguity, resulting in mesh inaccuracies in 3D space. Several HMR frameworks in the literature use LiDAR data to avoid this depth ambiguity. However, these frameworks are either only LiDAR-based, which limits performance due to LiDAR sparseness and limited training data, or they are model-free, making the resulting meshes vulnerable to artifacts and limited detail. Therefore, this work introduces SMPLify-3D, an optimization-based HMR framework that combines the richness of image data and the depth information within sparse LiDAR data to improve the 3D mesh inaccuracies of image-based HMR frameworks. The proposed framework consists of three main steps: 1) a body part visibility filter based on the 2D detected keypoints, 2) rough alignment between the mesh and the observed LiDAR point cloud using the ICP algorithm, and 3) an optimization scheme, inspired by SMPLify, that modifies the actual pose and shape of the mesh to improve both the 3D and image alignment. SMPLify-3D is versatile compared to other methods and outperforms image-based and SMPL-compatible LiDAR-based HMR frameworks by improving the Per-Vertex-Error (PVE) with 45% and 26% on the 3DPW and HumanM3 datasets respectively. Multiple quantitative experiments are conducted to show the effects of LiDAR noise and sparsity on the framework’s performance. Additionally, qualitative results illustrate how the proposed method achieves superior results on out-of-sample data recorded by a mobile robot. The source code for this work is available at: https://github.com/guidodumont/SMPLify-3D

Forensic microtrace investigation relies on a time- and labour-intensive process of manually analysing samples via microscopy. To aid forensic experts in their investigations, an image recognition model for microtrace localisation and classification is needed. This work investigates the trace recognition accuracy that can be achieved by analysing images captured with automated microscopy through deep learning. Fibres, hairs, skin, glass and sand are pixel-wise classified in microscopy scans of tape-lift samples. As deep learning requires extensive amounts of annotated training data, pretraining is investigated to minimise the required annotation workload. ImageNet pretraining, pretraining with self-supervised learning and a sequential application of these approaches are tested. It is found that pretrained models are able to reduce the required annotated data twofold compared to models trained from scratch, while retaining the prediction accuracy. While the ImageNet-pretrained models outperform the self-supervised-pretrained models, the highest accuracy is achieved by combining the two approaches. With this, traces are recognised with a mean intersection over union of 0.56 when training on only 2.2 dm2 of annotated tape lift scans.

Monitoring wildfires using multiple unmanned aerial vehicles (UAVs) is essential for timely intervention and management of the fire while minimising risks to human lives. A research gap was identified for practical UAV solutions integrating critical features, such as localisation, fire safety and battery management. This thesis presents a novel rule-based algorithm designed for effective wildfire monitoring using cooperative UAVs. A simulation environment with a wildfire spread model was developed to test and evaluate various strategies under dynamic conditions. The proposed algorithm operates in two phases: first, locating the fire and then monitoring its progression. Key features include a heat avoidance mechanism to maintain UAV safety and recharging schedules to maximise operational uptime. A fire detection map is centrally created using the infrared data from the UAVs. The convex hull around the fire serves as the basis for path planning to track the fire perimeter. The best strategy is determined and optimised through a series of experiments in the simulation environment. The overall best-performing algorithm makes navigational decisions based on the importance of different sections of the fire perimeter. The importance of the perimeter is constantly updated based on the fire spread, visit times, and distances of the UAVs. The assignment of various UAV tasks was reviewed to enhance UAV cooperation, which showed performance benefits across different scenarios. Other key findings
include the recommendation of a small fleet size of up to six UAVs, with a flight speed of 10 m/s at an altitude of 100 meters, balancing costs, performance and safety. The algorithm demonstrated performance equal to state-of-the-art reinforcement learning techniques
while offering advantages in explainability. Additionally, the algorithm has been successfully validated in a lab environment, demonstrating its potential as a practical and cost-effective solution for wildfire monitoring. This work brings a fire monitoring system closer to real-world implementation and will possibly help fight wildfires effectively.

Unsupervised 3D object detection methods can reduce the reliance on human-annotations by leveraging raw sensor data directly for supervision. Recent approaches combine density-based spatial clustering with motion and appearance cues to extract object proposals from the scene, which serve as pseudo-annotations. However, density-based methods struggle with the uneven data densities seen in LiDAR point clouds, and fail to distinguish between foreground and background objects effectively. To address this issue, this thesis introduces MobileClusterNet, a learnable framework designed for 3D spatial clustering. MobileClusterNet incorporates a novel loss module which utilizes appearance embeddings alongside scene flow information, thereby learning to generate high-quality clusters consisting of both static and dynamic mobile objects. Annotations generated by MobileClusterNet can be used for training any existing supervised detector, without the need for extensive self-training. Experimental results on the Waymo Open Dataset demonstrate that MobileClusterNet outperforms traditional density-based methods like HDBSCAN in clustering performance by a large margin, and provides high quality proposals for training supervised detectors.

Loop Robots develops and operates the next generation of fully autonomous disinfection robots in hospitals and healthcare settings. Accurate localization is essential in order to navigate reliably and effectively disinfect the tight hallways and corners of a patient room, operating room, or intensive care unit in a hospital. Odometry, or dead-reckoning, is the process of estimating the robots pose with respect to a known initial pose. It forms the backbone of the robots localization strategy, as well as being critical to many other processes that run on the robot, such as control and mapping.
The current strategy of generating odometry using the wheel encoders is prone to drift due to the integration of errors into the estimate. Moreover, the estimation takes place on the horizontal plane due to the nature of the wheel encoders. To improve the odometry and extend it to the full 3D space, a solution that makes use of all of the onboard sensors in a tightly-coupled manner is required.
In this Thesis, we make the first steps towards this larger goal. The contributions of this thesis include: (1) an Extended Kalman Filter (EKF) algorithm to fuse data from the Inertial Measurement Unit (IMU) and wheel encoders to estimate position and orientation of the robot in 6-DoF; (2) a line feature extraction and tracking methodology to extract primitives from LIDAR data and (3) A Moving Horizon Estimation (MHE) scheme based on a factor-graph formulation to perform pose estimation on the horizontal plane using LIDAR data and wheel encoders.
We test the three modules individually using a combination of simulations and real-world data wherever possible. We found that the MHE scheme was able to reduce drift over the long term, but is sensitive to the effects of outliers in feature matching, motion distortion of LIDAR scans, and wheel slip. The EKF scheme is able to reduce the overall drift and correct for wheel slips.
Based on these results, promising avenues for the improvement of all the proposed modules are given, along with recommendations on how to combine them all in a tightly-coupled fashion.

Visual inspection of liquid medicine containers for contamination and defects is mandatory and crucial to ensure their safety for injection. This document presents research and development of three modules of the Visual Particle Inspection Subsystem (VPIS), an automatic inspection subsystem with the task of detecting and classifying particle contamination in liquid medicine containers. The proposed VPIS comprises three modules for which research and development is performed separately: the backgrounds subtraction and segmentation module, the classification module, and the tracking module.

For the background subtraction and segmentation module, a solution is proposed with filtered temporal background modelling and locally adaptive threshold segmentation. This solution works on the assumption that moving objects such as particles and bubbles are sparse and do not occur more than twice in a pixel in twenty frames. Therefore, a representative incomplete background cluster can be obtained by removing the two lowest or highest pixel values depending on the mode of illumination. Next, the average value of the background cluster is used for background subtraction and the standard deviation of the background cluster is used to determine a locally adaptive segmentation threshold. The solution has been positively evaluated for detection of low contrast objects, insensitivity to disturbances, processing time, and parameter configuration. Additionally, an add-on solution is proposed that makes it possible to detect objects in the challenging region near the rubber stopper of a syringe.

For the classification module, out of four candidate classification methods a Convolutional Neural Network (CNN) is chosen for the classification of single object detections. The CNN classifier achieves an accuracy of 0.93 and is used as a baseline classifier for the rest of this research. Next, with three exploratory research questions, research is performed into opportunities and pitfalls for this classification problem summarised below.
-It is found that certain handcrafted features correlate to the classification accuracy of a detection. A method is proposed to predict the classification accuracy of detections and filter out detections that cannot be classified reliably. Compared simple filters that filter out detections based on a single feature such as area or total contrast, the proposed method should achieve a higher subsequent classification accuracy and reject less detections.
-With the simplest classification strategy that is often used for inspection, a container will be rejected if a single detection is classified as a particle. It is shown through simulation that this classification strategy is not effective for this classification problem. This is as the large number of bubble detections in a clean container would result in a high false container rejection rate.
-Two multi-detection classification strategies are proposed that use multiple detections for classification. Median voting classification uses results from the tracking module to classify multiple detections of the same object. Multi-positive classification does not use tracking results but requires multiple detections to be classified as particles. Simulation results indicate that with both strategies, the classification accuracy for containers can be drastically improved. It is theorized that the best results can be achieved with median voting classification...

Understanding traffic participants’ behaviour is crucial for predicting their future trajectories, enabling autonomous vehicles to better assess the environment and consequently anticipate possible dangerous situations at an early stage. While the integration of cognitive processes and machine learning models has demonstrated promise in various domains, its application in trajectory forecasting of multiple traffic agents in large-scale autonomous driving datasets remains lacking. This work investigates the state-of-the-art trajectory forecasting model Trajectron++ which we enhance by incorporating a smoothing term in its attention module. This attention mechanism mimics human attention inspired by cognitive science research indicating limits to attention switching. We evaluate the performance of the resulting Smooth- Trajectron++ model and compare it to the original model on various benchmarks. Our results show improved performance on the large-scale nuScenes dataset, revealing the potential of incorporating insights from human cognition into trajectory prediction models.

Tracker-level fusion (TLF) is recognized as an effective approach to comprehensively improve visual object tracking performance by combining the capabilities of multiple baseline trackers. Although there is considerable interest in TLF, there are still challenges related to insufficient understanding, high cost, and unstable performance that make studying TLF difficult. In this thesis, I begin with an explicit summary of the overall pipeline of TLF which has significant guidance for TLF study. Additionally, I conduct a deep analysis of the positive and negative effects of baseline trackers to fully understand their influence on TLF. For visual object tracking, I propose three TLF frameworks based on three Distributed Kalman filters which are optimized for different scenarios and enable the fusion of different baseline trackers to enhance tracking performance. My TLF frameworks fuse the tracking results of different baseline trackers based on the principle of minimal trace to produce fusion results.Additionally, they exhibit superior and stable performance with general baseline tracker requirements, while also being simple, online, and real-time.Furthermore, the proposed frameworks can benefit from state-of-the-art baseline trackers over time, which will further improve their tracking performance. The proposed analysis and frameworks are studied extensively on three challenging benchmarks: generic tracking OTB2015, short-term tracking GOT-10k, and long-term tracking LaSOT. At over 240 FPS, the state-of-the-art success AUC score of 72.7% is achieved on OTB2015.

In the literature, neural network compression can significantly reduce the number of floating-point operations (FLOPs) of a neural network with limited accuracy loss. At the same time, it is common to manually design smaller networks instead of using modern compression techniques. This thesis will compare the two approaches for the object detection network YOLOv7. YOLOv7 can run in real time on a desktop GPU. For edge GPUs a smaller version, called YOLOv7-tiny, was manually designed by the authors of YOLOv7. This thesis answers the question: Can a state-of-the-art compression of YOLOv7 achieve higher accuracy than YOLOv7-tiny at the same number of floating-point operations?
First, two state-of-the-art compression methods are selected and compared on YOLOv7-tiny. Then the best performing method, GBIP, is used to compress YOLOv7 till it has the same number of FLOPs as YOLOv7-tiny. From the experiments it is determined that GBIP is not able to achieve higher accuracy than YOLOv7-tiny at the same number of FLOPs.

The rapid advancement in autonomous driving technology underscores the importance of studying the fragility of perception systems in autonomous vehicles, particularly due to their profound impact on public transportation safety. These systems are of paramount importance due to their direct impact on the lives of passengers and pedestrians. Additionally, their reliability can be easily compromised given the complexity and unpredictability of driving environments. However, current research and existing regulations often fail to adequately address the adversarial robustness of autonomous vehicle perception systems. This thesis delves into the adversarial robustness of camera-based perception systems of autonomous vehicles. Our research concentrates on developing and implementing evasion attacks that use black-box gradient estimation, as well as physical attacks in traffic sign detection and classification systems. Our findings indicate that even minor perturbations can impact the accuracy of these systems, leading to detection and classification errors. This finding highlights a critical vulnerability in the perception system's robustness against adversarial attacks. Moreover, the study extends to assess the transferability of adversarial examples across diverse perception models. Our results also expose significant gaps in the current regulatory frameworks of autonomous vehicles, necessitating the establishment of more rigorous and comprehensive safety standards.

Ship-to-Ship (STS) cargo transfers can significantly improve the efficiency of offshore installations while reducing their associated costs. However, the added complexity of cargo transfers involving ship-mounted cranes at sea poses significant challenges. To address this, crane manufacturers have introduced Relative Heave Compensation (RHC) systems to assist crane operators. Nevertheless, concerns have emerged among installation vessel companies regarding the relative heave measurement systems, which rely on Motion Reference Unit (MRU) sensors placed on both ships. Given that many supplier vessels lack the required sensors and the communication protocols pose bureaucratic hurdles, this research focuses on proposing novel, feeder-independent relative heave measuring solutions. This work introduces two unexplored sensor units, namely the LiDAR-MRU and RADAR-MRU. In the absence of actual sensor data, a simulation workflow using Simulink and Unreal Engine (UE) is presented to generate synthetic data. After, four distinct measurement solutions are developed, implemented, and evaluated using the simulated data in MATLAB. The implemented solutions estimate relative heave distance and speed with a Mean Absolute Error (MAE) ranging from 0.3-5.2% and 0.8-4.3% of the reference's maximum amplitude, and processing times from 1.8-91.4 [ms]. The results underscore their potential for field implementation. However, future validation is needed with actual sensor data.

A key important part of a warehouse operation is to keep track of the products in the warehouse. Traditionally, handheld scanners are used to scan the products to perform a stock count. The advancements in robotics have paved the way for new technologies that can improve the scanning process. This work shows how prior knowledge and warehouse structure can be used to perform scanning operations. The method uses a localized camera in the warehouse whose estimates drift over time and the knowledge about the environment to estimate the correct location of the product using factor graphs. The proposed method shows by exploiting the structure of the warehouse the drift in the VIO(Visual Inertial Odometry) can be reduced and the position estimation of the location labels can be improved

Camera-based patient monitoring is undergoing rapid adoption in the healthcare sector with the recent COVID-19 pandemic acting as a catalyst. It offers round-the-clock monitoring of patients in clinical units (e.g. ICUs, ORs), or at their homes through installed cameras, enabling timely, pre-emptive care. These are powered by Computer Vision based algorithms that pick up critical physiological data, patient activity, sleep pattern, etc., enabling real-time, pre-emptive care. In this work, we develop a person detector to deploy in such scenarios. These algorithms require huge quantities of training data which is often in shortage in the healthcare field due to stringent privacy norms. Therefore looking for solutions to enrich clinical data becomes necessary. An alternative currently popular among the Computer Vision community is to use synthetic data for training, created using 3D modeling software pipelines. However, this type of technique often has limitations in data diversity and data balancing as desired variations need to be provided explicitly. In this thesis, we propose a data augmentation method for enriching diversity in synthetic data without using any additional external data or software. In particular, we introduce a pose augmentation technique, which synthesizes new human characters in poses unseen in the original dataset using Pose-Warp GAN. Additionally, a new metric is proposed to assess diversity in human pose datasets. The proposed method of augmentation is evaluated using YOLOv3. We show that our pose augmentation technique significantly improves person detection performance compared to traditional data augmentation, especially in low data regimes.

Video Object Detectors (VID) are used in various applications such as surveillance, inspection, etc. Often in these applications there exists a spatial area of interest and a static background. The static backgrounds remain constant throughout the video sequence in the training data establishing an undesirable correlation with the moving object during training. To hide static backgrounds in the video, masking is an option. We create multiple synthetic datasets and reveal that
(i) VIDs detect moving objects better if the static background in the train and test set are similar or from the same distribution.
(ii) VIDs drop in performance if the static background in the train and test are different.
(iii) Adding more static backgrounds during training does not make VID robust to static background changes at test time.
(iv) Masking or removing static backgrounds cannot prevent VIDs from learning correlations with static backgrounds.
The experiments shed light on the usage of static backgrounds for detecting dynamic objects.