Soft robots are typically intended to operate in highly unpredictable and unstructured environments. Although their soft bodies help them to passively conform to their environment, the execution of specific tasks within such environments often requires the help of an operator that supervises the interaction between the robot and its environment and adjusts the actuation inputs in order to successfully execute the task. However, direct observation of the soft robot is often impeded by the environment in which it operates. Therefore, the operator has to depend on a real-time simulation of the soft robot based on the signals from proprioceptive sensors. However, the complicated three-dimensional (3D) configurations of the soft robot can be difficult to interpret using traditional visualization techniques. In this work, we present an open-source framework for real-time 3D reconstruction of soft robots in eXtended Reality (Augmented and Virtual Reality), based on signals from their proprioceptive sensors. This framework has a Robot Operating System (ROS) backbone, allowing for easy integration with existing soft robot control algorithms for intuitive and real-time teleoperation. This approach is demonstrated in Augmented Reality using a Microsoft Hololens device and runs at up to 60 FPS. We explore the influence that system parameters such as mesh density and armature complexity have on the reconstruction's key performance metrics (i.e., speed, scalability). The open-source framework is expected to function as a platform for future research and developments on real-time remote control of soft robots operating in environments that impede direct observation of the robot.@en

This paper analyzes the effective accuracy for close-range operations for the first and the second generation of Microsoft HoloLens in combination with Vuforia Image Targets in a black-box approach. The implementation of Augmented Reality (AR) on optical see-through (OST), head-mounted devices (HMDs) has been proven viable for a variety of tasks, such as assembly, maintenance, or educational purposes. For most of these applications, minor localization errors are tolerated since no accurate alignment between the artificial and the real parts is required. For other potential applications, these accuracy errors represent a major obstacle. The "realistically achievable"accuracy remains largely unknown for close-range usages (e.g. within "arms-reach"of a user) for both generations of Microsoft HoloLens.Thus, the authors developed a method to benchmark and compare the applicability of these devices for tasks that demand a higher accuracy like composite manufacturing or medical surgery assistance. Furthermore, the method can be used for a broad variety of devices, establishing a platform for bench-marking and comparing these and future devices. This paper analyzes the performance of test users, which were asked to pinpoint the perceived location of holographic cones. The image recognition software package "Vuforia"was used to determine the spatial transform of the predefined ImageTarget. By comparing the user-markings with the algorithmic locations, a mean deviation of 2.59 ±1.79 [mm] (HL 1) and 1.11 ±0.98 [mm] (HL 2) has been found, which means that the mean accuracy improved by 57.1% and precision by 45.4%. The highest mean accuracy of a single test user has been measured with 0.47 ±1.683 [mm] (HL 1) and 0.085 ± 0.567 [mm] (HL 2).

@en

How to visualize recorded production data in Virtual Reality? How to use state of the art Augmented Reality displays that can show robot data? This paper introduces an opensource ICT framework approach for combining Unity-based Mixed Reality applications with robotic production equipment using ROS Industrial. This publication gives details on the implementation and demonstrates the use as a data analysis tool in the context of scientific exchange within the area of Mixed Reality enabled human-robot co-production.

@en