The training process of machine learning models for self-driving applications suffers from bottlenecks during loading and processing of LiDAR point clouds with large storage complexity.
Many studies aim to remedy this problem from an implementation perspective by developing efficient data loading and processing pipelines.
This study, on the other hand, explores an alternative approach by augmenting data representations to achieve lower storage complexity known as point cloud compression.

A broad analysis is presented on novel point cloud compression codecs using tensor decomposition methods.
Several point cloud representations and tensor decomposition methods are considered over a range of hyperparameter choices and compression values.
In order to assess the performance of the presented codecs: the compression rate, quality of the reconstruction, and time complexity is compared to the octree-based baseline model: TMC13.
Compared to the baseline model, the performance of the presented tensor decomposition-based codecs falls short. One of the presented codecs does notably outperform the others. This codec uses synthetic tensorization followed by sorting using z-location and decomposition using the TT-SVD algorithm.
Sorting by z-value isolates the ground plane, which is a dominant low-rank feature, which can effectively be decomposed using the TT-SVD algorithm yielding adequate results.

Several limitations of the presented tensor decomposition-based codecs are: the omission of bitwise compression on the factor matrices, and the trade-off between bitwise precision and truncation due to tensor decomposition.
Future work could improve in these areas along with considering the use of different heuristics and optimizing the tensor network topology.

The ever-increasing complexity of Artificial Intelligence (AI) models has led to environmental challenges due to high computation and energy demands. This thesis explores the application of tensor decomposition methods—CP, Tucker, and TT—to improve the energy efficiency of large Convolutional Neural Networks (CNNs) during inference by reducing energy consumption. The energy consumption of several convolution layers was measured using a watt meter across various CNN configurations and different hardware architectures (Central Processing Unit(CPU) and Graphics Processing Unit (GPU)). In addition, several regression models were fitted to estimate energy savings, incorporating memory usage. It was found that TT decomposition consistently provided the most significant energy savings across various compression ratios, influenced by CNN hyperparameters such as input/output channels, feature sizes, and kernel sizes, whereas CP decomposition was the least effective in reducing energy. The GPU implementations generally resulted in additional energy consumption, and the GPU regression models suggested a need for more complex relationships. The thesis also revealed that the efficiency of tensor decompositions might be highly dependent on the implementation details of software libraries, such as TensorlyTorch, which can significantly impact the computation and memory complexities. These findings underscore the importance of both hardware specific considerations and careful software implementation in achieving energy efficient CNNs, providing a foundation for further research in energy-constrained environments.

The shift to sustainable energy sources has increased demand for Energy Transition Metals such as nickel, copper, cobalt, and manganese. To satisfy this need while reducing the negative social and environmental effects of conventional mining, Deep-sea Nodule Collection (DSNC) appears to be a feasible option. Despite its potential, DSNC encounters difficulties with path planning since it depends on imperfect global paths, which result in inefficiencies and safety issues. This thesis looks into the possibility of using a local path planning algorithm, specifically an Interpolating Curve Planner (ICP).

To assess the ICP's performance under various model hyperparameters and environmental scenarios, a simulation tool is created. The study examines how well the ICP performs in terms of area coverage, collection rate, and reliability along different hyperparameters and presurvey measurement resolutions. It integrates novel production rate objectives, previous track following, and real-time sensor measurements within an environment generated from bathymetric data.

The examined local path planner shows a step towards optimised DSNC. Further research is needed to refine objectives and the structure of the local path planning algorithm to maximize its potential in addressing DSNC challenges effectively.

Automated driving has immense potential for improving road safety. Over the past decades, extensive research has been conducted in this field. Although the technological capability for highly automated driving exists today, its widespread application is not yet present. One major limiting factor of current automated driving solutions is that vehicle localization heavily relies on high-definition maps (HD maps), which are highly expensive to construct and maintain. This dissertation focuses on developing a more scalable solution for vehicle localization. It explores a novel technique that estimates the ego vehicle’s pose (location and orientation) by matching ground-level images captured by the vehicle’s onboard camera to publicly available geo-referenced aerial imagery...@en

This work addresses visual localization of intelligent vehicles as an alternative to traditional GPS- of HD map-based localization options. Specifically, the problem of Cross-View Pose Estimation (CVPE) is explored, which involves estimating the vehicle pose within an encompassing aerial patch, given a ground image from the on-board camera feed. The aerial patch containing the ground truth pose can be obtained through a rough localization prior, such as GPS. We find that existing CVPE methods start with a location prior that is too coarse given both the GPS performance and the required localization error. Therefore, we define a fine-grained localization setting and propose three approaches, targeting performance, interpretability, and data efficiency. Furthermore, the approaches have a unique capacity to predict a 6-DoF camera pose. Two approaches involve matching point-level local features in 3D space using a novel point cross-attention, while the last one aims to tailor an existing dense feature matching method to the fine-grained setting. Despite quantitative performance of the local feature matching approaches being inferior to the state-of-the-art, we establish a new state-of-the-art on the fine-grained setting with the improved dense-feature baseline. Nevertheless, we show the key limitations of the local feature matching, namely the influence of the “unmatchable” queries. Furthermore, using a 6-DoF projective transformation we discover severe issues with the ground truth quality on the KITTI dataset, commonly used in CVPE literature, potentially accounting to the large degree to the substandard performance of most available CVPE methods. Finally, our local feature matching methods demonstrate the capability of predicting pitch and roll angles of the camera, estimating which has not yet been attempted in CVPE.

Tracker-level fusion (TLF) is recognized as an effective approach to comprehensively improve visual object tracking performance by combining the capabilities of multiple baseline trackers. Although there is considerable interest in TLF, there are still challenges related to insufficient understanding, high cost, and unstable performance that make studying TLF difficult. In this thesis, I begin with an explicit summary of the overall pipeline of TLF which has significant guidance for TLF study. Additionally, I conduct a deep analysis of the positive and negative effects of baseline trackers to fully understand their influence on TLF. For visual object tracking, I propose three TLF frameworks based on three Distributed Kalman filters which are optimized for different scenarios and enable the fusion of different baseline trackers to enhance tracking performance. My TLF frameworks fuse the tracking results of different baseline trackers based on the principle of minimal trace to produce fusion results.Additionally, they exhibit superior and stable performance with general baseline tracker requirements, while also being simple, online, and real-time.Furthermore, the proposed frameworks can benefit from state-of-the-art baseline trackers over time, which will further improve their tracking performance. The proposed analysis and frameworks are studied extensively on three challenging benchmarks: generic tracking OTB2015, short-term tracking GOT-10k, and long-term tracking LaSOT. At over 240 FPS, the state-of-the-art success AUC score of 72.7% is achieved on OTB2015.

Inadvertent bird nest destruction by autonomous mowing machines poses significant threats to the breeding success of meadow birds. Drone-based detection methods represent the current state-of-the-art for bird nest localization to attain mower circumvention. However, they only identify 80% of bird nests with average localization error of 3.344 meters and are restricted to specific application times. This paper introduces alternative, fully automated nest localization systems integrated with autonomous mowers. Two strategies are proposed, 1) Directly detecting bird nests using thermal data, or 2) Indirectly, by tracking birds and extrapolating their trajectories back to their nests using RGBD data. These methods were validated with warmed chicken eggs hidden in grasslands and with drones simulating bird flight. YOLOv8 models were modified for both approaches. The thermal localization method is able to detect all bird nests with an average confidence of 73.4%. It allows for real-time localization and yields one unnecessary nest circumvention for every ten bird nests saved due to false positives. This method is shown to be effective in all breeding season temperatures, both day and night. Conversely, the trajectory extrapolation method detects birds with an average confidence of 82.2% and has localization error of 0.794 meters. Birds taking flight prematurely or from locations other than nests impact the number of bird nests saved and the number of unnecessary circumventions. It is demonstrated that this method fails to detect birds during nighttime. In conclusion, an automated thermal-based localization system integrated with autonomous mowers outperforms both RGBD- and current state-of-the-art drone-based methods. This study highlights therefore the potential of thermal-based solutions for bird nest protection in grasslands.

Robots are increasingly deployed in various locations to automate tasks, including in barns. However, in barns cows can obstruct the sensors such as LiDAR or camera, leading to a lack of environmental information. As a result, the robot’s localization system only relies on odometry at those moments, introducing additional uncertainty to the robot’s pose. When the visibility of the environment is restored, the robot may mistakenly believe it is in a location that does not correspond to its actual position.
The first contribution of this master thesis is a novel method for improving the self-localization in barns by implementing a line detection algorithm which is called Line Adaptive Monte Carlo Localization (LAMCL). The novelty is that only line segments are used to detect a localization error instead of corners between different line segments. It also retains the robot’s current pose to filter out fault-detected localization errors. In addition, the new approach is applied in a dynamic environment. The proposed method combines the Split-and-Merge line detection algorithm with AMCL. The detected line segments are compared with the walls in the environment to identify localization errors. When an error is detected, the robot’s pose is adjusted by placing a section of the particles at the location of the error. In this way, the robot can find its true location again.
The second contribution is a new dataset. This new dataset, called DataCow, consists of four recorded routes in a barn with GT on a handful of spots to evaluate the self-localization. DataCow includes the pseudo-2D LiDAR scans and the odometry of a robot driving through a barn. This dataset is used to evaluate the new self-localization method LAMCL. Through the experiments, it has been discovered that this new method improves the system’s recovery ability, but the accuracy and precision are compromised.

We explored the possibility of improving cross-view matching performance with self-supervised learning techniques and perform interpretations in terms of the embedding space of image features. The effect of pre-training by contrastive learning is verified quantitatively by experiments, and also exhibited by visualization of the feature space.

The automotive industry currently has been working on developing various levels of autonomy to assist in different Advanced Driver Assistance Systems (ADAS) with the ultimate aim of moving closer to the realization of an autonomous vehicle. For such ADAS, the industry has been using multiple sensors like Cameras, Radar, LiDAR, etc. LiDAR has been at the forefront of the research as it provides extremely rich and precise information about the surrounding. In research Motorized Optomechanical LiDAR has been used for almost a decade now. However, it can not be deployed in mass-produced vehicles because of how expensive the sensor is, thus, limiting it to academia without practical use. Therefore, the industry has been working on a different category of LiDAR namely Microelectromechanical Systems (MEMS) LiDAR. Which uses solid-state technology instead of servo motors. Thus, they are compact and much cheaper in cost, making them feasible to deploy in mass-produced vehicles.

There has been a considerable amount of research already done on Motorized Optomechanical LiDAR, therefore, one must take advantage of it to learn about MEMS LiDAR. The domain of Transfer Learning provides an opportunity to learn from the accumulated knowledge of Motorized Optomechanical LiDAR (Source Domain) and use it to reduce the learning time on MEMS LiDAR (Target Domain). Thus, reducing the effort, cost, and time to research MEMS LiDAR technology.

This thesis explores the Domain Mapping strategy of transfer learning for point clouds. A scan pattern-based domain mapping approach has been described in this thesis to reduce the domain gap between the source and target domain. Moreover, a point cloud densification pipeline that utilizes a depth completion network has been described to further reduce the gap between the two domains. Since semantic segmentation is one of the most actively researched tasks in the industry as it provides the researcher with in-depth information about the environment which could be used in activities like lane detection, parking assist, etc, therefore, this thesis utilizes it as the proxy for evaluating the performance of the domain mapping strategy. It compares a few domain mapping strategies like cropping the Field of View, an adaptation based on scanning pattern, and densification & adaptation to simulate the density of points in the source domain similar to the target domain. This thesis verifies that domain mapping strategy is a suitable solution to learn from Motorized Optomechanical LiDAR and use it to enhance performance on MEMS LiDAR and therefore reduce the data required for research of MEMS.

Multi-pedestrian tracking in camera networks has gained enormous interest in the industry because of its applicability in travel-flow analysis, autonomous driving, and surveillance. Essential to tracking in camera networks is camera calibration and, in particular extrinsic camera calibration. Extrinsic camera calibration incorporates the 3D position and orientation of the cameras in the camera network. Current methods for extrinsic calibration require special operators to place calibration objects in sight of all cameras in the network, which is impractical and limits the ease of tracking in camera networks.
In this thesis, an automatic extrinsic calibration model is proposed to calibrate the extrinsic camera parameters from the image data of all cameras in a network. The proposed method utilizes deep learning algorithms for feature extraction and matching. The feature extraction step is a human-pose estimator, extracting the key points of humans such as joints, eyes, and feet. The matching algorithm is a re-ID algorithm using an affinity-based feature extractor.
Two camera network datasets have been used to evaluate the proposed model. A four-camera fully overlapping dataset SALSA, and a more challenging seven-camera partially overlapping dataset WildTrack. The calibration accuracy of the model on both datasets is calculated by comparing the ground truth value with the calibrated extrinsic camera parameters. On dataset WildTrack, the model could calibrate three of the seven cameras, whereas the model had a near-perfect calibration on dataset SALSA. Dataset SALSA achieved a root mean squared error in translation of 0.02 meters and 0.0 radians in orientation, compared to the ground truth values.
The final product of this research is an automated extrinsic camera calibration model that eases the camera calibration in camera networks. The proposed model could not calibrate all datasets, but the model provides a baseline upon which future research can be done.

The world is heading more and more towards automation, that goes for transportation as well. Various car manufactures already have released level 2 autonomous vehicles meaning that the future is not that far away. An essential part of driving is of course detecting and obeying the traffic signs. Advanced Driver Assistance Systems (ADAS) like speed limit detection systems or a more advanced variant the Traffic Sign Detection (TSD) systems are integrated in top-of-the-line car models in order to increase safety and relieve some ofthe drivers workload. These systems attempt to make drivers aware of incoming signs on the road and warn them against possible danger or in some cases even take control over the vehicle. As in many computer vision systems differences in lighting, weather conditions, motion blur, and poor image quality can reduce the effectiveness of these systems. In case of the Traffic Sign Detection systems other challenges arise like differences in sign position and in the condition (dirty, deformed, discolored etc.) of the sign itself. In order to fully rely on such system all these challenges have to be overcome. The goal of this thesis is to enhance a traffic sign detection model that requires relatively low processing power in such a manner that it can compete with traffic sign models that show great results and achieve high accuracy but require high processing power maintaining the ability to operate in real time. The suggested BoBT method increases the accuracy of a model at the cost of inference speed. There is a small difference in precisely localizing the traffic signs compared to state of the art models but in terms of detecting traffic signs the difference is less than a percent with the highest accuracy model. Not to mention that it still outperforms these models in terms of speed.

For an Autonomous Vehicle (AV) to traverse safely in traffic, It is vital it can anticipate the behavior of surrounding traffic participants using motion prediction. Current motion prediction approaches can be categorized into object-centered and object-agnostic methods and are primarily based on deep learning. The former relies on a human-engineered pipeline of object detection and tracking, of which the errors can accumulate in the motion predictions. The latter does not rely on this pipeline; however, it lacks the ability to learn object representations causing blurriness and object disappearances for longer-term predictions which forms a safety hazard. This thesis proposes two methods to improve the performance of the object-agnostic sequence-to-sequence Occupancy Grid Map (OGM) prediction networks, trained on the Waymo Open Perception dataset. The first method uses inter-pixel loss functions, i.e. the SSIM and Sinkhorn losses, instead of the ubiquitously used per-pixel losses, to train the PredRNN++ Occupancy Grid Map (OGM) prediction network. Inter-pixel losses take into account the spatial relations between grid cells during the evaluation of OGMs, whereas per-pixel losses evaluate each grid cell’s value independently. The quantitative results demonstrate that using inter-pixel losses can improve short term predictions with a prediction horizon of T = 5 for the Mean Squared Error (MSE) by 4.3%, Image Similarity (IS) by 7.8%, Average Precision (AP) by 0.3%, metrics. For the longer term, T = 15, the predictions improve for the MSE by 5.0%, IS by 20.5%, AP by 0.1%, and Accuracy by 0.6%. Furthermore, the use of inter-pixel losses reduces blurriness and object disappearances. The second method is based on multi-task learning. By training the PredRNN++ to perform the prediction task together with the semantic segmentation task on the predicted OGMs, it is expected to learn object representations which it uses to improve the prediction quality. The quantitative results show that multi-task learning does not improve the OGM predictions. However, some qualitative results show that multi-task learning reduces blurriness and object disappearances.

A robotic vehicle must continuously determine its position within the map to traverse a path safely; this is called self-localization. Current localization methods use mainly sensors like LIDARS. However, a LIDAR does not return data points if the environment is an empty field; the laser scan of the LIDAR does not reflect without obstacles, making self-localization in those environments impossible. Self-localization with visual information from cameras can be an alternative for vehicles operating in an open environment, such as the Lely Juno, an agricultural robot operating on farms. A widely used visual Simultaneous Localization and Mapping (SLAM) method in literature, ORB-SLAM3, lacks robustness on Lely Juno's visual data; the track is lost multiple times due to a sudden drop in the number of feature matches.


This work aims to experimentally determine (a) the cause of the drop in the feature matches that eventually causes the tracking error and (b) the robustness and accuracy of ORB-SLAM3 compared to a simple visual odometry (VO) system.
The cause of the drop in feature matches is investigated outside the ORB-SLAM3 pipeline by replicating the matching thread of ORB-SLAM3 with OpenCV ORB feature matching. The robustness and accuracy of ORB-SLAM3 are compared with a simple visual odometry system by comparing the trajectories and the minimum number of matches found in the sequences.

The sudden drops in feature matches could not be replicated outside the ORB-SLAM3 pipeline with the brute-force-based ORB matcher in OpenCV. While in some cases, the decrease in feature matches is related to the image itself (e.g., blur, movement), the leading cause is complicated. In this work, the leading cause is further reduced to the quality of the disparity map, the type of matcher used, and the use of grids when extracting features. The simple VO system is more robust than ORB-SLAM3. However, the absolute pose error is worse and unsuitable for reliable navigation for long trajectories.

For the Lely Juno, a simple visual odometry system can only be used for short distances outside. Therefore, investigating the issues in ORB-SLAM3 is an excellent direction from the project's point of view. More understanding is needed of the exact effects of the disparity map, the type of matcher used, and the use of grids on the number of feature matches.

The contributions of this work are insights into the reasons behind the tracking errors, exploring the effect of different ORB parameters and datasets on the matching performance, and creating two new datasets in low-textured and repetitive agricultural environments. Investigating the exact impact of the disparity map, the type of matcher used, and the use of grids on the number of feature matches is left for future work.

Camera-based patient monitoring is undergoing rapid adoption in the healthcare sector with the recent COVID-19 pandemic acting as a catalyst. It offers round-the-clock monitoring of patients in clinical units (e.g. ICUs, ORs), or at their homes through installed cameras, enabling timely, pre-emptive care. These are powered by Computer Vision based algorithms that pick up critical physiological data, patient activity, sleep pattern, etc., enabling real-time, pre-emptive care. In this work, we develop a person detector to deploy in such scenarios. These algorithms require huge quantities of training data which is often in shortage in the healthcare field due to stringent privacy norms. Therefore looking for solutions to enrich clinical data becomes necessary. An alternative currently popular among the Computer Vision community is to use synthetic data for training, created using 3D modeling software pipelines. However, this type of technique often has limitations in data diversity and data balancing as desired variations need to be provided explicitly. In this thesis, we propose a data augmentation method for enriching diversity in synthetic data without using any additional external data or software. In particular, we introduce a pose augmentation technique, which synthesizes new human characters in poses unseen in the original dataset using Pose-Warp GAN. Additionally, a new metric is proposed to assess diversity in human pose datasets. The proposed method of augmentation is evaluated using YOLOv3. We show that our pose augmentation technique significantly improves person detection performance compared to traditional data augmentation, especially in low data regimes.

This work addresses visual localization for intelligent vehicles. The task of cross-view matching-based localization is to estimate the geo-location of a vehicle-mounted camera by matching the captured street view image with an overhead-view satellite map containing the vehicle's local surroundings. This local satellite view image can be obtained using any rough localization prior, e.g., from a global navigation satellite system or temporal filtering. Existing cross-view matching methods are global image descriptor-based and achieve considerably lower localization performance than structure-based methods with 3D maps. Whereas structure-based methods utilized global image descriptors in the past, recent structure-based work has shown that significantly better localization performance can be achieved using local image descriptors to find pixel-level correspondences between the query street view image and the 3D map. Hence, using local image descriptors may be the key to improving the localization performance of cross-view matching methods. However, the street and the satellite view do exhibit not only very different visual appearances but also have distinctive geometric configurations. As a result, finding correspondences between the two views is not a trivial task. We observe that the geometric relationship between the street and satellite view implies that every vertical line in the street view image has a corresponding azimuth direction in the satellite view image. Based on this prior, we devise a novel neural network architecture called SliceNet that extracts local image descriptors from both images and matches these to compute a dense spatial distribution for the camera's location. Specifically, the geometric prior is used as a weakly supervised signal to enable SliceNet to learn the correspondences between the two views. As an additional task, we also show that the extracted local image descriptors can be used to determine the heading of the camera. SliceNet outperforms global image descriptor-based cross-view matching methods and achieves state-of-the-art localization results on the VIGOR dataset. Notably, the proposed method reduces the median metric localization error by 21% and 4% compared to the state-of-the-art methods when generalizing, respectively, in the same area and across areas.

Driving is a challenging task. When people operate vehicles they utilize all their senses to assess the current traffic scenario and determine appropriate actions to take. Sensors in autonomous driving applications aim to mimic those human senses to build a similar understanding of these complex circumstances. Most scientific attention in the autonomous driving community has been put on camera and LiDAR solutions, which are already capable of constructing a cohesive three-dimensional representation of the surroundings with high accuracy. Instead, minimal research went into the utilization of the auditory landscape in traffic scenarios. With the use of sirens in emergency vehicles and upcoming regulations of minimum sound emissions for electric vehicles, it appears evident that acoustic perception deserves more attention. Auditory localization in particular is a task widely studied in the field of speaker recognition, where robust methods exist that can localize and track multiple speech sources in difficult acoustic environments. This raises the question of whether such principles can be applied in the domain of autonomous driving and help to make self-driving cars safer and more robust in navigating dense and complex urban cities. Especially in non-line-of-sight situations where oncoming traffic can be hazardous and conventional systems fail, employing acoustic sensing may prove instrumental in early detection and localization attempts. This study investigates, if, and to what extend, acoustic methods can be used to complement ordinary sensing and localization methods. Using the example of a T-crossing, where traffic is occluded by buildings, it is shown that acoustic sensing alone is capable of detecting oncoming traffic before it enters direct line-of-sight. To achieve this, a combination of a generic acoustic line-of-sight feature and two concepts of data-driven classification methods are used to infer from the surrounding soundfield at the ego-vehicle to other motorized vehicles in proximity. This investigation is aided by a supplementary real-world dataset that provides around two hours of data, gathered from five different locations. The performance of the methods is directly compared to a visual baseline and other acoustic line-of-sight methods. The results demonstrate that the proposed acoustic localization concepts can detect traffic about one second earlier than conventional line-of-sight sensors. Despite the complexity of the problem, it is shown that the more lightweight method in terms of parameter count is favourable and more performant in most of the tasks. Among others, these tasks include generalization across different environments and a larger time horizon.

The topic of automated driving is receiving increasing attention from the scientific community and automotive industry. A key task for an autonomous vehicle is the recognition of drivable area and, in an extension of this, detecting the road boundaries. State-of-the-art techniques often use camera and/or LIDAR sensors to perform this task. However, these sensors can be expensive and not suitable in low visibility situations such as during nighttime or in bad weather. Radar sensors are a viable alternative. The aims of this study are twofold: to introduce a technique of radar-based road boundary estimation (RBE) using radar sensors and to develop a suitable benchmark to compare different radar-based RBE techniques. The RBE algorithm proposed in this study, Clustering-based Urban Road Boundary Estimation (CURBE), clusters the radar detections and incorporates domain knowledge to determine which of the clusters describe road boundaries (so-called boundary clusters). Additionally, we propose a novel benchmark to compare radar RBE algorithms. The benchmark makes use of the nuScenes data set. Each radar reflection in the data set is given a ground truth label. The radar reflection is given the label of boundary point or non-boundary point based on several sources such as the measured radial velocity, object annotations, and a detailed road map. To quantify the performance of an RBE algorithm, the estimated labels outputted by the RBE algorithm are compared to the ground truth labels using the metrics precision, recall, and F1-score. CURBE is evaluated on the benchmark where it achieves an F1-score of 30.4%. We conclude that with the current version of CURBE and with the current method of measuring performance there are not enough grounds to consider this method as a reliable way of RBE. However, the approach shows promise. The method has the potential to significantly improve performance by using a clustering method that is designed for radar data or implementing a different way of selecting boundary clusters. Since the proposed benchmark uses the radar reflections in the data set as ground truth the performance measurement is less accurate in recordings with limited radar coverage, but overall it is concluded that the benchmark provides a fast and consistent way to measure and compare the performance of different RBE algorithms.

Self-driving vehicles have shown rapid development in recent years and continue to move towards full autonomy. For high or full automation, self-driving vehicles will have to be able to address and solve a broad range of situations, one of which is interaction with traffic agents. For correct and save maneuvering through these situations, reliable detection of agents followed by an accurate classification of the traffic gestures used by agents is essential. This problem has received limited attention in literature to date. The objective of this work is to establish and investigate a working traffic gesture pipeline by leveraging the latest developments in the fields of computer vision and machine learning. This work investigates and compares how well state-of-the-art methods translate to traffic gesture recognition and what application specific problems are encountered. Multiple configurations based on skeletal features, estimated using OpenPose, and classified using recurrent neural networks (RNN) were investigated. Skeleton estimation using OpenPose and feature representations were evaluated using an action recognition dataset with motion capture ground-truth. Three RNN network architectures, varying in complexity and size, were evaluated on traffic gestures. The robustness of the developed system regarding viewpoint variation is explored, combined with the viability of transfer learning for traffic gestures. To train and validate these methods, a new traffic gesture dataset is introduced, on which an mAP of 0,70 is achieved. The results show that the proposed methods are able to classify traffic gestures within reasonable computation time and illustrate the value of transfer learning for gesture recognition. These promising results validate the methodology used and show that this direction warrants further research.

The bottleneck of the maximum road volume in urban areas is the maximum capacity of the traffic flow on the intersection, which is coordinated with Traffic Light Controllers (TLCs). A promising method to decrease the number of stops are Green Light Optimal Speed Advice (GLOSA) systems. These systems will give a speed advice to arriving vehicles based on the schedule of TLCs, which needs to be known and fixed. However, most on-street controllers change their schedule until the last moment to maximize the performance. In this thesis a predictive controller is developed that is suitable for real-world application based on DIRECTOR; a state-of-the-art predictive controller. A prediction model is used to predict future arrivals based on available measurements to optimize and fix the schedule in advance. The proposed controller can enable GLOSA systems to improve performance. Appropriate pre-processing steps are implemented and the optimal input features are selected to improve the performance of a Long Short-Term Memory (LSTM) network to predict future arrivals. All detection data is stationary over time by using the differenced series. The time data is divided into workdays and weekend days to create a binary input and undesirable jumps during midnight are removed. The combination of stop line detectors, queue detectors, arrival detectors and signal states of the controlled and preceding intersections as input maximized the performance. The prediction horizon of the proposed prediction model could be extended. The Normalized Root Mean Square Error (NRMSE) decreased with 17% compared to DIRECTOR. The proposed controller extends the control horizon and uses multiple prediction models to predict the arrivals for the entire control horizon. The proposed controller outperforms DIRECTOR with 14 - 38% reduction in terms of vehicle delay and 5 - 32% reduction in terms of numbers of stops based on the scheduling mode. The GLOSA system is an add-on of the controller and is able to operate without the GLOSA system. The control horizon of the proposed controller always has a fixed length which is needed to determine the time until green. The implemented GLOSA system will determine the optimal speed based on the time until green and the expected delay due to the surrounding vehicles. The proposed controller is a cloud controlled application. Therefore, it is possible to adjust the setup (i.e. scheduling modes) during the day. Enabling GLOSA all day except during rush hours will lead to 3 - 4% reduction in terms of vehicle delay and 29 - 32% reduction in terms of numbers of stops based on the scheduling mode. This setup of the proposed controller is also competitive with the hand-crafted non-predictive on-street controller. Compared with this controller, the proposed controller will reduce the number of stops with 26% at the cost of 16% increase in vehicle delay. The proposed controller is designed conform the safety standards used on-street. Permission is received from the provincial government to do on-street pilots with the proposed controller. These results could give new insights into the performance of the proposed controller.