Recent advances in sensor technology have lead to increased resolution of novel sensors, while tracking applications where distance between sensors and objects of interest is very small have gained research interest recently. In these cases, it is possible that multiple sensor detections are generated by each object of interest. Extended Object Tracking (EOT) approaches consist of algorithms which make use of multiple sensor detections per object to jointly estimate their kinematic and shape extent attributes within the Bayesian tracking framework. In the last decade, various EOT algorithms have been proposed for different types of tracking applications. This M.Sc. thesis project addresses the problem of extended tracking of a single pedestrian walking in the area of a stationary vehicle (referred as ego-vehicle in this report) during a real automotive scenario. The objective is to achieve accurate estimation of both the kinematic attributes (2D centroid position/velocity), as well as its shape extent in x-y plane. In more detail, PreScan software is enabled to design a simulation scenario that is very close to a real automotive application, in terms of motion characteristics of objects of interest and sensor data acquisition. In the considered scenario, different sensor modalities are mounted on the ego-vehicle, namely a Lidar sensor and a mono camera sensor. Moreover, OpenPose library is employed to to obtain pose detections of human body parts from obtained camera images. Concerning shape extent representation, the simplest and most popular approach in previous studies, in general and especially for VRUs tracking, is to assume an elliptical shape. In fact, the Random Matrix Model (RMM), proposed originally by Koch, 2008, is a state-of-the-art EOT state modeling approach that allows for joint estimation of centroid kinematics and physical extent for considered elliptical objects of interest. Based on that, a RMM-based filter using Lidar position measurements has been proposed by Feldmann, 2011. In this project, this algorithm is used as a baseline filter for comparison with our proposed algorithm.
In addition, an alternative tracking algorithm is proposed in this study, which has the following differences with respect to the baseline filter:
State Initialization of the filter: In our proposed version of the tracking algorithm, human pose detections of shoulders and ankles are are associated with obtained Lidar position measurements in order to provide initial values for the kinematic state (2D position/velocity) and shape parameters (ellipse orientation and semi-axes lengths) of the pedestrian.Measurement Update step of the filter: In our proposed version of the tracking algorithm, camera-obtained pose detections of pedestrian shoulders are associated with obtained Lidar position measurements in order to create an extra measurement, for pedestrian heading angle. Subsequently, a nonlinear filtering update step fusing Lidar-obtained point cloud data for pedestrian position and human-pose-obtained measurement for pedestrian heading angle is implemented. Both considered tracking algorithms are evaluated for the designed simulation scenario. In detail, the following performance metrics are used for evaluation of each filter: RMSE for estimated pedestrian 2D position and velocity, respectively.
Modified Hausdorff distance for estimated pedestrian shape extent.
In more detail, Monte Carlo simulations with multiple runs are designed to evaluate performance of each state initialization approach and each tracking algorithm, where the following parameters change in each run:
Additive zero-mean Gaussian measurement noise on obtained Lidar position detections. Initial simulation timestep.

Object detection is one of the most important research topics in autonomous vehicles. The detection systems of autonomous vehicles nowadays are mostly image-based ones which detect target objects in the images. Although image-based detectors can provide a rather accurate 2D position of the object in the image, it is necessary to get the accurate 3D position of the object for an autonomous vehicle since it operates in the real 3D world. The relative position of the objects will heavily influence the vehicle control strategy. This thesis work aims to find out a solution for the 3D object detection by combining the Lidar point cloud and camera images, considering that these are two of the most commonly used perception sensors of autonomous vehicles. Lidar performs much better than the camera in 3D object detection since it rebuilds the surface of the surroundings by the point cloud. What’s more, combing Lidar with the camera provides the system redundancy in case of a single sensor failure. Due to the development of Neural Network (NN), past researches achieved great success in detecting objects in the images. Similarly, by applying the deep learning algorithms to parsing the point cloud, the proposed 3D object detection system obtains a competitive result in the KITTI 3D object detection benchmark.

This work presents a multi-sensor approach for weather condition estimation in automated vehicles. Using combined data from weather sensors (barometer, hygrometer, etc) and an in-vehicle camera, a machine learning and computer vision framework is employed to estimate the current weather condition in realtime and in-vehicle. The use of different sensor types is shown to improve robustness and reduce noise. The resulting modular framework allows it to be used with different sensor configurations, and allows changes in sensor configuration with minimal effort. Finally, a proof-of-concept experiment is presented; a dataset is recorded using a test vehicle and used for model evaluation.
The resulting datasets contains 20.000 pairs of video frames and sensor measurements recorded in different weather situations.