The pandemic showed the power of video calls, as it was used to accomplish a multitude of tasks that normally required physical attendance. Expanding on the video call’s transmission of sight and sound, haptic bilateral teleoperation aims to add the sense of touch and the ability to manipulate a remote envir- onment as if the user is physically present. Haptic bilateral teleoperation is a technique where a robot mimics the actions of a human operator, who in turn receives feedback from the robot’s sensors. The technique requires low-latency haptic feedback, in the range of single-digit milliseconds, which is completely im- practical when transmitting force measurements over a large distance network. Instead, the low latency requirements can be fulfilled with Model Mediated Teleoperation (MMT) to predict the user’s perceived force feedback. With the addition of force feedback, the latency of the video feedback becomes an area of greater concern. Force and video feedback in conjunction are necessary for true immersion, but using MMT to generate real-time video feedback is im- practical. This work explores a configuration where MMT is used to generate haptic feedback and video feedback is transmitted from the remote environment to the user. Transmittance of video feedback is made possible due to its relaxed latency requirements compared to force feedback. The sources of video latency can be divided into two, latency caused by the network transmission and latency from the non-network components. While minimization of both is valuable, any time saved on the non-network components can be spent on expanding the net- work and maximizing the technology’s range. In this work, we aim to answer the question ”How can we minimize the latency caused by the non-network components of a live video feedback system?” Our approach is to design a field- programmable gate array (FPGA) configuration to minimize all latencies in a live video system excluding the camera, network, and monitor components. We demonstrate a no-network live video system with 34 ms latency, of which around 5 ms originate from the FPGA. This is a marked improvement of the 63 ms latency our department achieved using a non-FPGA solution. Utilization of cameras and monitors designed for low latencies is recommended for bet- ter results, with a hypothesized latency of 11.5 ms for a high-end non-network approach.

This thesis explores the challenge of achieving haptic bilateral teleoperation across long distances, focusing on enhancing Model Mediated Teleoperation (MMT) to support dynamic environments under high network latency. Instead of striving for perfect model alignment, we acknowledge the inevitable mismatch between the remote environment and its model at the operator. We suggest a set of design considerations and a complementary framework to make operator intent a key factor in MMT solution development. By developing a system where a user can control a remote robot arm to draw on a moving whiteboard using a novel position-altering scheme that attempts to prioritize operator intent, we apply the framework to minimize the impact of network latency on user experience. Our findings reveal that the implemented position-altering scheme significantly reduces the effects of network delays on drawing accuracy, and a user study confirms that force feedback delays are more critical to user satisfaction than visual feedback delays. The results suggest that satisfactory teleopera-tion performance can be achieved with network latency up to 50 ms, challenging the conventional 10 ms requirement. This research provides design considerations, a comprehensive framework with a position-altering policy, and an open-source software package for advancing haptic teleop-eration technologies.

In recent years the emergence of Spiking Neural Net- works (SNNs) has shown that these networks are a promis- ing alternative to traditional Artificial Neural Networks (ANNs) due to their low-power computing capabilities and noise robustness. Nevertheless, in recent approaches, they have either strayed away from spike time backpropaga- tion, used discrete time, single spike neurons, and/or limited themselves to shallow networks. These approaches limit the potential of SNNs by sacrificing significant aspects of what makes them special in order to have a functional network. It is for this reason that we believe that, the implemen- tation of residual connections, allowing a model that has multi-spiking neurons, precise time and spike time back- propagation is the path to allow these networks to truly shine. As it will solve one of this model’s most severe lim- itations which prevent it from being used to build deeper networks, the banishing of spikes at a deeper depth. Hence we aim to remedy this problem by feeding inputs from fur- ther up the network to revitalize the spike counts at deeper layers. In this paper, we explore the implementation of residual connections in precise time multi-spiking neural networks as a way of solving the disappearing spikes problem that inhibits spike time backpropagation. We explore two al- ternatives in the implementation of the residual connection and we analyze how they affect the accuracy of the network as the depth increases in both Multi-Layer Perceptrons and Convolutional Neural Networks. We also developed an ar- chitecture to allow for swapping of the fuse function in or- der to take full advantage of the flexibility that precise time provides us. Results show that the implemented residual connections allow for deeper training, which has the poten- tial to aid in the network’s performance, although in some cases some hurdles remain.

This paper presents a method that shows how a lidar-based 3D static environment can be constructed from a driving scenario and is used to aid in the creation of a digital twin from that scenario. Built with limited computational resources in mind, the resulting 3D static background reconstruction method is lightweight and nonetheless up to par with similar works, proving itself to be a viable alternative to similar methods. It uses ground truth labels and open-source, out-of-the-box working building blocks to create the pipeline that filters the lidar frames, performs registration between those frames, and meshes the resulting point cloud into a 3D mesh of the static background. We performed experiments in the Siemens Prescan ADAS Simulator of a reenactment of the traffic scenario in combination with our 3D static background and measured the inference domain gap using the Bidirectional Chamfer Distance and a real-data trained object detector, comparing both the synthetic and original data. Using Prescan as the simulator, AD models van be evaluated closed-loop. This work opens up the possibilities of applying sensor domain shifts to existing data or creating reality-based traffic scenarios from existing traffic scenarios and their corresponding 3D static background of edge cases that do not, or too little, exist in real-world data.

Structure-from-Motion (SfM) and Neural Radiance Fields (NeRFs) have significantly advanced 3D reconstruction in multi-view scenarios. Despite their success in handling non-repetitive, texture-rich scenes, applying such techniques to real-world scenarios with texture-less and repetitive structures remains challenging. One such case is in industrial inspection processes, specifically the reconstruction of aircraft engine blades. The goal is to build a 3D model of all the engine blades from a single video. We explore the application of SfM in modeling repetitive and texture-less blades, identify common causes of failure, and improve upon the default SfM by replacing the commonly used exhaustive match with sequential match to handle ambiguity stemming from repetitiveness. Sequential matching enables more precise pose estimation and better 3D reconstruction in our scenario. In addition, we explore the importance of choosing the correct camera model and provide a comparative look at the existing 3D mesh reconstruction solutions, presenting tweaked versions that result in better performance. This work lays the foundation for 3D reconstruction of repetitive and texture-less objects by proposing sequential matching, enabling better 3D modeling of engine blades compared to classic SfM pipelines.

The modern workplace often exposes individuals to privacy risks, such as the unauthorised visibility of their computer screens. MARS (mmWave-based Assistive Rehabilitation System for Smart Healthcare), coupled with VideowindoW screens, offers an innovative solution to these threats by using mmWave radar to reconstruct human poses and estimate the position of 19 key joints. This enables the screens to become opaque based on the viewer's position, ensuring privacy. Although originally designed as a rehabilitation system, MARS can be utilised for its pose estimation capabilities to enhance workplace privacy. This research explores two modifications to the MARS architecture to assess their impact on the system's accuracy and performance. Specifically, we modify the MARS architecture by increasing the depth of its convolutional neural network (CNN) and integrating the PointNet architecture. Results establish that an optimal CNN configuration with two convolutional and two dense layers, followed by the output layer, modestly improves joint location estimation. However, integrating PointNet does not improve performance, likely due to PointNet's limitations in capturing the necessary local structural details of point clouds. These findings inform future research of possible improvements when leveraging the MARS dataset in the fields of privacy enhancement and smart healthcare applications.

Human Pose Estimation using Millimeter Wave radars has emerged as a promising alternative to traditional camera-based systems, addressing privacy and deployment constraints. While state-of-the-art Deep Learning models predominantly focus on spatial feature extraction to determine the positions of key points in the human body, this research investigates the effects of incorporating temporal dynamics in such models. It focuses of modifying an existing state-of-the-art spatial model to account for temporal dynamics and compares the performance of the two models. Long Short-Term Memory networks are used to capture temporal dependencies between frames of point clouds which significantly boosts the precision of key point detection. The proposed temporal model demonstrates a 53% reduction in Mean Absolute Error and a 45% reduction in Root Mean Squared Error compared to state-of-the-art model. Moreover, these improvements were achieved with a less complex model architecture and similar training times. The robustness of the model was further validated on a different dataset, showcasing its potential for broad application in fields such as healthcare, sports analysis, traffic monitoring and robotics. This study underscores the efficacy of temporal dynamics in pose estimation, and showcases the advantages of accounting for temporal dependencies when evaluating more complex movements.

Crowd-control is an emerging problem in urban areas and cameras are commonly seen as the solution, however, there are major concerns regarding privacy. To overcome these issues, while still maintaining the ability to keep track of people, mmWave sensors can be utilized instead, but this introduces new challenges when it comes to people counting. They work by recording the data as point clouds, making it difficult to determine the real number of people when they are occluded. To address this challenge, we combine the spatial and temporal information of the point clouds into graphs, and explore the possibilities of Graph Neural Networks. Our system classifies up to 5 people and bikes with an accuracy of 80.47%. This accuracy is 2.53% worse than the state of the art people counting system for mmWave radar point clouds.

Interactive video games often use vision-based systems or wearables to track player movements. Vision-based systems are privacy-invasive, and wearables require frequent recalibration and recharging. Frequently-Modulated Continuous-Wave (FMCW) radars have been proposed as an alternative tracking solution addressing these problems. Working in the millimeter-wave (mmWave) range, they capture scenes as point clusters, ensuring privacy without attaching sensors to the user. Previous research has shown their applicability in rehabilitation, gait recognition, and smart home appliances. This study focuses on integrating an mmWave sensing device with an interactive version of the Breakout video game. We propose a general framework with three main modules - generating points, clustering them, and reconstructing the player's movements as in-game commands. Our system enhances an already existing Kalman filter-based tracking algorithm. Online experiments were conducted to compare the proposed system to the baseline algorithm. The proposed system decreases the standard deviation on the estimated target location by 33% against motionless targets, while maintaining the baseline accuracy when tested on moving targets. Furthermore, it allows a higher game refresh rate, thus smoothing in-game movements. These results demonstrate the potential of FMCW radars in enhancing interactive video game experiences.

The People Counting Problem requires calculating the number of people in a region of interest. This is needed in crowd-monitoring scenarios but has become increasingly problematic when relying on video cameras, as they raise privacy concerns. Instead, we propose using a mmWave radar to detect people by creating point clouds from their radar signal reflections. This approach, however, can pose challenges when people walk closely together because their individual point clouds overlap and are seen as a single, larger cloud. It is difficult to count how many individuals this large point cloud holds, which can lead to miscounting the people in the scene. One approach to address this issue is leveraging the time dimension in people walking sequences, which can be done with Long Short-Term Memory (LSTM) models. Given this, we investigate how two state-of-the-art models, PointNet and MARS, perform for people counting from point clouds when extended through LSTMs. The results show how both PointNet and MARS improve performance when extended by LSTMs. Particularly, despite having over double the parameters, MARS+LSTM outperforms PointNet+LSTM in terms of accuracy and computational efficiency. MARS+LSTM can effectively capture small changes in the local structure of point clouds between frames, which PointNet loses due to max pooling. This highlights the importance of selecting a model architecture, like the CNN in MARS, that aligns with the data characteristics to maximise performance.

Gaussian Splatting is a successful recent method for generating novel views of a scene based on photographs taken from that scene [1]. It uses rasterization in order to render the scenes it generates, which consist of 3D Gaussians. However, modern hardware and tools are designed and optimized around rendering polygonal and texture based models [2]. This paper proposes a method of extracting both a 3D model and texture file from a Gaussian Splatting scene by using renders of that scene in Photogrammetry. It shows that this can be a viable method for generating a traditional 3D model from Gaussian Splatting scene, and can for certain cases generate a model of comparable quality while lowering the number of initial images required by up to three times.

3D Gaussian Splatting (3DGS) is a method for representing 3D scenes, but is prone to overfitting when trained with limited viewpoint diversity, of- ten resulting in artifacts like floating Gaussians at incorrect depths. This paper addresses this issue by introducing 3D Gaussian Splatting with Depth, which incorporates depth supervision from RGB Depth (RGB-D) cameras into the training process. By using depth data to guide the placement of Gaussians, the proposed method aims to reduce artifacts. Through quantitative and qualitative analysis, this paper demonstrates that depth-supervised Gaussian splatting mitigates overfitting artifacts, particularly in outdoor scenes with a mediocre cam- era point diversity. The depth-supervised model is able to reduce the depth loss by a factor of three times without substantially increasing the loss on regular views.

Current robotic perception systems utilize a variety of sensors to estimate and understand a robot's surroundings. This paper focuses on a novel data representation technique that makes use of a recent scene reconstruction algorithm, known as 3D Gaussian Splatting, to explicitly represent and reason about an environment using only a sparse set of camera views as input. To achieve this, I generate and analyze the first cross-modal dataset consisting of 3D Gaussians and views taken around ten household objects. I introduce the resulting 3D Gaussians and images to a self-supervised feature learning network, that learns robust 2D and 3D embedding representations, by optimizing for the cross-view and cross-modality correspondence pretext tasks. I experiment with several 3D Gaussian features as input to the model and two point sub-network backbones, and report results on the two pretext tasks. The learned features are subsequently fine-tuned for the 2D and 3D shape recognition tasks. Moreover, by leveraging the fast scene reconstruction capabilities of the algorithm, I propose the use of rendered views as a visual memory aid to support downstream robotic tasks. The proposed networks achieve comparable results to state-of-the-art methods for point and image processing. The code associated to this paper is available at https://github.com/SimiOy/Self-Supervised-Learning-for-3DGS.

3D Gaussian Splatting (3DGS) is a promising 3D reconstruction and novel-view synthesis technique. However, the field of semantic 3D segmentation of 3D Gaussian Splats scenes remains largely unexplored. This paper discusses the challenges of performing 3D segmentation directly on 3D Gaussian Splats, introduces a new dataset facilitating evaluation of 3DGS semantic segmentation and proposes use of PointNet++, initially developed for point cloud segmentation, as a 3DGS segmentation model. As the results show, PointNet++ is also capable of performing 3DGS segmentation with performance close to the performance achieved in point cloud segmentation tasks. When taking into account only the positions, 3D Gaussian splats appear to be more difficult for PointNet++ to process than point clouds sampled from mesh faces, possibly due to their irregularity. However, as shown in the paper, inclusion of size, rotation and opacity of each splat allows PointNet++ to achieve nearly 87% of accuracy, outperforming PointNet++ on point clouds sampled from meshes.

Haptic bilateral teleoperation is a concept where operators can feel, interact with and manipulate objects in a remote environment. The latency, often introduced by a network over a long distance, on force feedback poses a problem when trying to interact with objects. Model mediated teleoperation tries to mitigate this latency, but introduces new problems. Extending on model mediated teleoperation, this thesis uses a physics simulation based model to create an end-to-end system for haptic bilateral teleoperation, implements robotic control for a Novint Falcon and a model to predict force feedback when cutting through objects. Experiments on the implemented cutting model affirm the models accuracy.

Neural Radiance Fields (NeRFs) have showcased remarkable effectiveness in capturing complex 3D scenes and synthesizing novel viewpoints. By inherently capturing the entire scene in a compact representation, they offer a promising avenue for applications such as simulators, where efficient storage of real-world data, fast rendering and dynamic generation of new content are crucial. However, the potential for compression in NeRFs has been largely neglected in the existing literature. Moreover, the practical deployment of NeRFs in real-world scenarios, including simulators, faces significant obstacles such as constraints in training time, rendering speed, and scalability to large scenes. While recent advancements have tackled some of these hurdles individually, none have offered a comprehensive solution. In this paper, we introduce a new NeRF architecture based on a textured polygon-based method and augment this architecture by integrating encodings to expedite training. Additionally, we introduce learned pose refinement and an appearance embedding to enhance scalability to larger scenes. Through experimentation on the nuScenes dataset, we demonstrate that our method achieves competitive reconstruction performance with existing techniques while surpassing them in rendering speed. Furthermore, in terms of compression, our findings indicate that our method achieves competitive compression rates comparable to image-based compression techniques, while also enabling novel-view synthesis. This underscores its potential utility in applications like simulators.

The current state of teleoperation in Tactile Internet faces the problem of limited operating distance. To circumvent this limitation, a setup has been proposed which provides Ultra-Low Latency feedback to the operator from a local simulation of the remote environment. To be able to accurately simulate the remote environment and the objects within it, an initial estimation of the mass and Center of Mass of these objects is required. We explore three approaches to the estimation of these properties, with the use of Axis-Aligned Bounding Bounding Boxes (AABB), Oriented Bounding Boxes (OBB), and convex hulls. We also explore the impact of sensor resolution and object occlusion. The performance of these methods varies, and highly depends on the type of object. Sensor resolution has a high impact on the estimated mass, but not on Center of Mass. Object occlusion has a clear impact on both, suggesting the use of multiple view points being will improve accuracy.

Creating a local model of a remote environment is a way to reduce latency in tactile internet. This local model contains properties that are estimated by a nonintrusive estimation method. To prevent the model from deviating increasingly from reality, the estimates should be updated once an interaction begins. This research paper investigates how the physical properties of a remote object can be estimated. This is done by exerting forces and observing the resulting motion. The physical properties that are updated include the mass and center of mass. These values are calculated to converge toward the actual value. The mass can be estimated by observing the linear motion, for which the game engine Unity is used. For the center of mass estimation, it is advised to take a different approach, since the values received by Unity were not enough to base an estimate on. This paper also investigates the influence of update frequency on estimation accuracy and the inaccuracies that the Unity physics engine presents.

The Tactile Internet (TI) is a new paradigm for remote interactions, enabling the transmission of touch and physical sensations. One of the major challenges in achieving seamless remote interactions is latency. To circumvent strict latency requirements, the paper briefly introduces the approach of a Model Mediated Teleoperation scheme utilizing a locally run physics engine to simulate the remote environment.

The focus of this paper is on solving the problem of tracking objects in TI workspaces, to be able to simulate them.
We developed a pattern recognition-based pose estimation technique using OpenCV's Perspective-n-Point solver, which accurately estimates the pose of objects in real time.
Further contributions include the implementation of a virtual test bed in the Unity game engine. The solver and test bed were integrated with a Python Flask server. This approach proved to be effective in providing accurate position and rotation estimation.

Tactile Internet (TI) is a pioneering network paradigm that aims to communicate haptic feedback with ultra-low latency. It will enable new ways for us to interact with remote environments, such as transferring skills over the network and controlling remote objects. One crucial component of this framework is the ability to track the position and movement of remote objects. While there are many tracking algorithms, evaluating them in TI applications can be time-consuming, expensive, and sometimes impractical. Hence, we present a virtual platform for developing, evaluating, and comparing tracking solutions. We demonstrate how our platform can be used to tune tracking algorithms and determine trade-offs between different types of hardware. In turn, this provides insights that were previously difficult to obtain using real physical platforms, as real world coordinates may not align with virtual coordinates and experiments are not easily repeatable.