The pandemic showed the power of video calls, as it was used to accomplish a multitude of tasks that normally required physical attendance. Expanding on the video call’s transmission of sight and sound, haptic bilateral teleoperation aims to add the sense of touch and the ability to manipulate a remote envir- onment as if the user is physically present. Haptic bilateral teleoperation is a technique where a robot mimics the actions of a human operator, who in turn receives feedback from the robot’s sensors. The technique requires low-latency haptic feedback, in the range of single-digit milliseconds, which is completely im- practical when transmitting force measurements over a large distance network. Instead, the low latency requirements can be fulfilled with Model Mediated Teleoperation (MMT) to predict the user’s perceived force feedback. With the addition of force feedback, the latency of the video feedback becomes an area of greater concern. Force and video feedback in conjunction are necessary for true immersion, but using MMT to generate real-time video feedback is im- practical. This work explores a configuration where MMT is used to generate haptic feedback and video feedback is transmitted from the remote environment to the user. Transmittance of video feedback is made possible due to its relaxed latency requirements compared to force feedback. The sources of video latency can be divided into two, latency caused by the network transmission and latency from the non-network components. While minimization of both is valuable, any time saved on the non-network components can be spent on expanding the net- work and maximizing the technology’s range. In this work, we aim to answer the question ”How can we minimize the latency caused by the non-network components of a live video feedback system?” Our approach is to design a field- programmable gate array (FPGA) configuration to minimize all latencies in a live video system excluding the camera, network, and monitor components. We demonstrate a no-network live video system with 34 ms latency, of which around 5 ms originate from the FPGA. This is a marked improvement of the 63 ms latency our department achieved using a non-FPGA solution. Utilization of cameras and monitors designed for low latencies is recommended for bet- ter results, with a hypothesized latency of 11.5 ms for a high-end non-network approach.

Haptic bilateral teleoperation aims to create systems, where human precision and dexterity can be applied over long distances with the aid of robotic devices. These systems could make remote surgery or spacecraft repair a reality. The main challenge for such systems is latency. TU Delft's Networked Systems group is developing a teleoperation system, which combines predicted force feedback with delayed vision feedback. This project investigates the force feedback prediction of cutting interactions by implementing a simple physics based model to estimate the forces of cutting objects with a knife in a haptic simulation. The quality of the interaction and impact of visual delay is evaluated in a user study. It is found that a simple physics based cutting model can create convincing cutting interactions, which users find to be sufficiently controllable up to a delay of 75ms.

Natural disasters often present significant challenges for rescue operations due to the complex and hazardous environments they create. Teleoperation technology, particularly haptic bilateral teleoperation, offers promising solutions to enhance the efficiency, safety, and effectiveness of disaster response. This study explores the feasibility of approximating dynamic object movement to achieve satisfactory force feedback for haptic bilateral teleoperation systems, focusing on minimizing the impact of network delays on user experience. We conducted a series of experiments and user studies to assess the system’s performance under various delay conditions. Our findings indicate that dynamic object movement can be successfully approximated to provide realistic force feedback. The user study revealed that network delays greater than 75 ms significantly impact task difficulty, and delays beyond 125 ms affect system usability, although the system remains functional up to a total delay of 200 ms. The acceptable total delay, including system latency of approximately 135 ms, is found to be around 185 ms for optimal user experience. This research contributes to the development of teleoperation systems by providing insights into the
acceptable levels of network delay and the practical implementation of force feedback mechanisms, paving the way for future advancements in the field.

This paper presents a novel approach to simulating liquid interactions for haptic bilateral teleoperation without complex fluid dynamics simulations. We propose a model combining a simplified drag equation with a "position tail" mechanism to approximate force feedback for viscous and turbulent liquids. The model was evaluated through theoretical analysis, manual testing, and a user study. Results of the latter demonstrate the ability to convincingly simulate different liquid types, with participants correctly identifying viscous and turbulent liquid types. The study also revealed that visual feedback delays up to 125 ms did not significantly impact user experience. While promising, the system remains sensitive to noisy input data. This research contributes to the development of efficient and believable haptic simulations for teleoperation, paving the way for future developments.

Haptic bilateral teleoperation is a concept where operators can feel, interact with and manipulate objects in a remote environment. The latency, often introduced by a network over a long distance, on force feedback poses a problem when trying to interact with objects. Model mediated teleoperation tries to mitigate this latency, but introduces new problems. Extending on model mediated teleoperation, this thesis uses a physics simulation based model to create an end-to-end system for haptic bilateral teleoperation, implements robotic control for a Novint Falcon and a model to predict force feedback when cutting through objects. Experiments on the implemented cutting model affirm the models accuracy.

The current state of teleoperation in Tactile Internet faces the problem of limited operating distance. To circumvent this limitation, a setup has been proposed which provides Ultra-Low Latency feedback to the operator from a local simulation of the remote environment. To be able to accurately simulate the remote environment and the objects within it, an initial estimation of the mass and Center of Mass of these objects is required. We explore three approaches to the estimation of these properties, with the use of Axis-Aligned Bounding Bounding Boxes (AABB), Oriented Bounding Boxes (OBB), and convex hulls. We also explore the impact of sensor resolution and object occlusion. The performance of these methods varies, and highly depends on the type of object. Sensor resolution has a high impact on the estimated mass, but not on Center of Mass. Object occlusion has a clear impact on both, suggesting the use of multiple view points being will improve accuracy.

Tactile Internet (TI) is a pioneering network paradigm that aims to communicate haptic feedback with ultra-low latency. It will enable new ways for us to interact with remote environments, such as transferring skills over the network and controlling remote objects. One crucial component of this framework is the ability to track the position and movement of remote objects. While there are many tracking algorithms, evaluating them in TI applications can be time-consuming, expensive, and sometimes impractical. Hence, we present a virtual platform for developing, evaluating, and comparing tracking solutions. We demonstrate how our platform can be used to tune tracking algorithms and determine trade-offs between different types of hardware. In turn, this provides insights that were previously difficult to obtain using real physical platforms, as real world coordinates may not align with virtual coordinates and experiments are not easily repeatable.

Creating a local model of a remote environment is a way to reduce latency in tactile internet. This local model contains properties that are estimated by a nonintrusive estimation method. To prevent the model from deviating increasingly from reality, the estimates should be updated once an interaction begins. This research paper investigates how the physical properties of a remote object can be estimated. This is done by exerting forces and observing the resulting motion. The physical properties that are updated include the mass and center of mass. These values are calculated to converge toward the actual value. The mass can be estimated by observing the linear motion, for which the game engine Unity is used. For the center of mass estimation, it is advised to take a different approach, since the values received by Unity were not enough to base an estimate on. This paper also investigates the influence of update frequency on estimation accuracy and the inaccuracies that the Unity physics engine presents.

The Tactile Internet (TI) is a new paradigm for remote interactions, enabling the transmission of touch and physical sensations. One of the major challenges in achieving seamless remote interactions is latency. To circumvent strict latency requirements, the paper briefly introduces the approach of a Model Mediated Teleoperation scheme utilizing a locally run physics engine to simulate the remote environment.

The focus of this paper is on solving the problem of tracking objects in TI workspaces, to be able to simulate them.
We developed a pattern recognition-based pose estimation technique using OpenCV's Perspective-n-Point solver, which accurately estimates the pose of objects in real time.
Further contributions include the implementation of a virtual test bed in the Unity game engine. The solver and test bed were integrated with a Python Flask server. This approach proved to be effective in providing accurate position and rotation estimation.

The Tactile Internet (TI) aims to expand seamless interaction over the Internet by providing a new form of interaction through touch by providing haptic feedback. To realize this, the TI is limited by a round-trip latency of 1-10 ms, meaning that the TI is limited by a physical distance of 1500 km. A workaround to this requirement is the introduction of local simulations. To keep track of moving objects in these simulations, a stable tracking algorithm is needed. This algorithm is provided in the form of a particle filter. The TI requires high tracking accuracy from this algorithm, but to achieve that the algorithm becomes computationally expensive. If the movement of the to-be-tracked object is known a priori, however, the particle filter can be adapted to focus only on that movement, neglecting the other directions. This increases tracking accuracy with an equal amount of samples, thus requiring a lower amount of samples to achieve the same accuracy, reducing computational power. This paper explores how to achieve this adaptation and analyses the increase in accuracy. By adapting the filter, the tracking accuracy is significantly increased, even with a lower number of samples. This results in gaining a speedup with a factor of about $36$, while having similar tracking accuracy.

Tactile Internet (TI) is a networking paradigm with the goal of allowing for transfer of skill by transmitting haptic feedback. Accurate haptic feedback requires Ultra Low Latency (ULL), which severely limits the distance over which TI can be used. To address the ULL requirement, a local model can be used to generate haptics. For accurate haptic feedback generation, proper representation of smoothly curved objects is needed inside the model. Non
Uniform Rational B-Spline (NURBS) surfaces offer an expressive way of representing such objects, but must first be tessellated to a mesh. However, little is known about NURBS tessellation in the context of tactile internet. Therefore, this paper will discuss how to best tessellate NURBS surfaces for use in haptic feedback generation. A global spacing tessellation algorithm was implemented, and the resulting meshes were analyzed. Furthermore, a comprehensive user study was conducted to investigate the mesh quality necessary for smooth objects. Findings imply that careful selection of the tolerance is needed, to avoid over-burdening the model. A mesh tolerance of ϵ = 0.05mm is proposed as agood balance between spatial fidelity and simplicity of tessellated meshes

Tactile Internet (TI) is the next step in the evolution of our digital communication. It will extend the conventional audiovisual interaction by enabling the users to convey the sense of touch over vast distances in real time. The potential to transfer haptic (touch) information will have a significant impact on many industries as well as on everyday life. However, there is still a long way to go in achieving a fully immersive user experience. One of the most significant shortcomings of today’s haptic devices is their lack of ability to reproduce texture. The conventional way of high-quality texture reproduction is to control the vertical displacement of pins in a two-dimensional finger-mounted pin array. In such a Haptic Texture Device (HTD) the final quality of the texture will solely depend on the actuation method of a single pin. A Haptic Texture Device Actuator (HTDA) needs to match the fingertip’s force sensitivity and spatial resolution and the reaction time of a human for fingertip stimuli. It must also be small enough to be integrated into handheld or wearable haptic devices. We have based our work on an existing HTD design that is fast enough to match the human reaction time and has a small enough pitch between adjacent pins to match the fingertip’s spatial resolution. However, in other aspects of haptic perception, the design fell short of the expectations. We have improved the HTDA of this base design to match every requirement of haptic perception. To match the force sensitivity of the fingertip, we have added a Hall-effect sensor to the actuator design for position measurement and implemented a closed-loop actuator control based on the measured pin position. Using the same actuator concept in a new configuration allowed us to decrease the actuator length by 12.5% while increasing the force output by 50%. Besides not fully matching the requirements of human haptic perception, the base design had a serious flaw. Due to minimal design, the HTDA’s of the base design tend to interfere, causing unwanted pin actuation. We were able to mitigate this issue by adding a backiron to our design. A backiron is essentially a layer of ferromagnetic material around the actuator, which besides magnetic shielding, also contributed to the 50% increase in force output. Our improved HTDA design enables the development of a HTD, capable of high quality texture reproduction.

LoRaWAN is a public Wireless Sensor network with excellent properties like being long-range, low-energy radios and resulting in long battery life. Devices are connected to this network through gateways, and they will run in that deployment for years without replacement. Therefore, bugs and security issues in such devices' firmware are present for a long time, unless firmware updates are applied to fix them. Firmware-Updates Over-The-Air (FUOTA) is a firmware update application framework for LoRaWAN, but packet loss is an unsolved issue for such firmware updating frameworks. State-of-the-art packet dissemination uses the Low-Density Parity Checks (LDPC), but this has decoding overhead. Also, it is inflexible due to its fixed-rate nature. The code can not be dynamically adjusted to adapt to temporary changes in channel conditions.
In this work, we provide critical analysis to justify replacing Low-Density Parity Checks code (LDPC) proposed in FUOTA with the famous code Random Linear Network Coding (RLNC). The benefit of RLNC shows when scaling to multicast networks using LoRaWAN FUOTA as fewer messages need to be exchanged to serve the complete network of firmware updates. An analysis is presented on how to configure the finite field size, generation size, and redundancy for generation-based RLNC. The parameters are optimized to cope with the worst-case packet-error rate. As such, RLNC can optimally counter every lost packet with redundancy with near-zero decoding overhead. Since devices sleep after decoding a set of fragments, or the generation, sending additional fragments does not decrease efficiency. Finally, the decoding probability is provided as an analytical tool to show that a specific amount of redundancy can deal with a specific worst-case packet-error rate.
To evaluate and test the RLNC code for LoRa, embedded firmware and terminal software have been developed to do indoor and outdoor measurements. This testbed has been developed with a custom control plane replacing all FUOTA MAC layer modules (Fragmentation, Multicast, Synchronization, Device Management, etc.). Therefore, it can be shaped into custom network configurations meant for evaluating the firmware updating process without having to set up infrastructures like ChirpStack or TheThingsStack. Our work's testbed can isolate the decoding process and evaluate it for artificial packet drops.
Results are presented, which show that systematic coding phase of LDPC will perform poorly compared to RLNC. This systematic phase is at least as large as the firmware update. Between 38% to 88% improvement in decoding success is found by using RLNC in case of varying network conditions due to burst loss.

We can see and we can hear through the internet, but not yet touch. Tactile Internet (TI) is the paradigm that will enable the transmission of tactile feedback through the internet. TI requires Ultra Low Latency (ULL) networking to prevent desynchronization. Reaching the ULL requirements for medium to long distances is not possible without breaking the known physical limits of the speed of light. We propose instead to bypass these requirements by running a physical simulation of the controlled environment to perform predictions before the information can arrive. To this end, efficient low-polygon 3D meshes from the objects in the environment are required for physical simulation. The focus of this work is on reconstructing these meshes from point cloud scanners. Through a combination of Marching Cubes mesh reconstruction and Hoppe's Mesh Simplification, point clouds with up to 20 mm Gaussian noise can be accurately reconstructed into compact low polygon meshes.

Tactile internet enables communication in a new layer of immersion, touch. It has the potential to transform the landscape of digital communication. However, the achievable scale of tactile internet is severely limited by its 1 ms round-trip latency. We propose a workaround for the latency through tactile simulation, which can bypass the requirement by having the user interact with locally simulated force feedback instead of real ones. A real-time material estimation method is required to create such a simulation, as it needs material information such as friction and tactile texture. We, therefore, investigate whether material is estimable from point cloud scans since it provides readily available environmental data. We also explore how friction and tactile texture could be extracted from the material. Past material estimation methods rely heavily on point intensity; however, most existing point clouds do not have intensity information. Therefore, we study whether mocking intensity information could be sufficient, exchanging it with grayscale. The results demonstrate that without point intensity, the material estimation method can only discern object colour and not material properties. This finding implicates the importance of intensity and suggests future exploration of the viability of material estimation with intensity data for indoor objects.

As technology evolves, transmission speeds become faster. Tactile Internet requires ultra-low-latency (ULL) communications to further immerse humans in a remote environment by transmitting movement and force feedback, allowing them to interact with that environment in real-time. However, no transmission speed can be fast enough to support the "1 ms challenge" over long distances due to light speed limitations. A system with over 1ms of delay will feel unnatural to the user and cause "cyber-sickness". A novelty solution solves this by simulating the remote environment locally using point clouds. The system is then able to compute the force feedback immediately. This paper focuses on the desynchronization between simulation and reality that can build up due to disturbances. A framework for testing and observing desync caused by controlled disturbances is built for this purpose. The framework can also be used to test possible solutions to this issue. 1-dimensional simulations show that divergence happens slowly for friction and mass mismatches, providing a time frame during which it can be corrected. 2-dimensional simulations presented non-deterministic results, limiting the observations.

A problem faced by Tactile Internet (TI) is the distance limitation. A possible solution is to create a simulation of the remote environment, such that the delay experienced by a human operator is reduced. However, to obtain a model of the remote environment, before the user interacts with it, we need to be able to estimate the physical properties of objects in the environment from visual information.
This research paper investigates estimating an object's mass from visual information in the form of a point cloud. The object's volume is estimated by surfacing the mesh and dividing the resulting mesh into tetrahedrons, the volume of each tetrahedron can quickly be computed and summed to give an estimate of the total volume. The overall performance is better than the naive approach of using an OBB, however, it is less accurate than a volume slicing approach.