For successful goal-directed human-robot interaction, the robot should adapt to the intentions and actions of the collaborating human. This can be supported by musculoskeletal or data-driven human models, where the former are limited to lower-level functioning such as ergonomics, and the latter have limited generalizability or data efficiency. What is missing, is the inclusion of human motor control models that can provide generalizable human behavior estimates and integrate into robot planning methods. We use well-studied models from human motor control based on the speed-accuracy and cost-benefit trade-offs to plan collaborative robot motions. In these models, the human trajectory minimizes an objective function, a formulation we adapt to numerical trajectory optimization. This can then be extended with constraints and new variables to realize collaborative motion planning and goal estimation. We deploy this model, as well as a multi-component movement strategy, in physical collaboration with uncertain goal-reaching and synchronized motion tasks, showing the ability of the approach to produce human-like trajectories over a range of conditions.@en

In the original version of the book, on page xi, one of the authors listed for Chapter 2 is “R. Schnmehl”, which should be “R. Schmehl”. On page 21, the same correction needs to be made twice, in the listed authors at the top of the page and also in the footnotes. “Schnmehl” should become “Schmehl” in both the cases. This has now been rectified and the author’s name has been corrected. The correction to this book have been updated with the changes.@en

In recent years, providing additional visual feedback about the interaction forces has been found to offer benefits to haptic-assisted teleoperation. However, there is limited insight into the effects of the design of force feedback-related visual cues and the type of visual display on the performance of teleoperation of robotic arms executing industrial tasks. In this study, we provide new insights into this interaction by extending these findings to the haptic assistance teleoperation of a simulated robotic arm in a virtual environment, in which the haptic assistance is comprised of a set of virtual fixtures. We design a novel method for providing visual cues about the interaction forces to complement the haptic assistance and augment visual feedback in virtual reality with a head-mounted display. We evaluate the visual cues method and head-mounted display method through human factors experiments in a teleoperated dross removal use case. The results show that both methods are beneficial for task performance, each of them having stronger points in different aspects of the operation. The visual cues method was found to significantly improve safety in terms of peak collision force, whereas the head-mounted display additionally improves the performance significantly. Furthermore, positive scores of the subjective analysis indicate an increased user acceptance of both methods. This work provides a new study on the importance of visual feedback related to (interaction) forces and spatial information for haptic assistance and provides two methods to take advantage of its potential benefits in the teleoperation of robotic arms.@en

During the learning of a new sensorimotor task, individuals are usually provided with instructional stimuli and relevant information about the target task. The inclusion of haptic devices in the study of this kind of learning has greatly helped in the understanding of how an individual can improve or acquire new skills. However, the way in which the information and stimuli are delivered has not been extensively explored. We have designed a challenging task with nonintuitive visuomotor perturbation that allows us to apply and compare different motor strategies to study the teaching process and to avoid the interference of previous knowledge present in the naïve subjects. Three subject groups participated in our experiment, where the learning by repetition without assistance, learning by repetition with assistance, and task Segmentation Learning techniques were performed with a haptic robot. Our results show that all the groups were able to successfully complete the task and that the subjects’ performance during training and evaluation was not affected by modifying the teaching strategy. Nevertheless, our results indicate that the presented task design is useful for the study of sensorimotor teaching and that the presented metrics are suitable for exploring the evolution of the accuracy and precision during learning.

@en

Despite a large body of research on robot learning, it has not yet been thoroughly studied how collaborating humans and robots learn reciprocally. In such situations, both humans and robots continuously learn about each other and the task through interaction. This letter addresses the research question: "How can human-robot co-learning be facilitated in physically embodied collaborative tasks?". First, we derived five requirements for successful human-robot co-learning from literature: shared goal, synchrony, interdependence, adaptability, and transparency. Based on these requirements, we designed a collaborative human-robot handover task and a robot Q-learning method. In an evaluation with six human participants co-learning was indeed found to emerge in the hand-over task. Particularly, for three of the human-robot dyads, our designed setup proved to facilitate co-learning in a way that met all five requirements. The task and robot learning method presented in this letter demonstrate how human-robot co-learning can be enabled in physically embodied tasks.

@en

In order for off-Earth top surface structures built from regolith to protect astronauts from radiation, they need to be several metres thick. In a feasibility study, funded by the European Space Agency, Technical University Delft (TUD aka TU Delft) explored the possibility of building in empty lava tubes to create rhizomatic subsurface habitats. With this approach natural protection from radiation is achieved as well as thermal insulation because the temperature is more stable underground. It involves a swarm of autonomous mobile robots that survey the areas and mine for materials such as regolith in order to create cement-based concrete reproducible on Mars through in-situ resource utilisation (ISRU). The concrete is 3D printed by means of additive Design-to-Robotic-Production (D2RP) methods developed at TUD for on-Earth applications with the 3D printing system of industrial partner, Vertico. The printed components are assembled using a Human--Robot Interaction (HRI) supported approach. The 3D printed and HRI-supported assembled structures are structurally optimised porous material systems with increased insulation properties. In order to regulate the indoor pressurised environment a Life Support System (LSS) is integrated, which in this study is only conceptually developed. The habitat and the D2RP production system are powered by an automated kite power system and solar panels developed at TUD. The long-term goal is to develop an autarkic, automated and HRI-supported D2RP system for building autarkic habitats from locally obtained materials.@en

Performing bimanual tasks with dual robotic setups can drastically increase the impact on industrial and daily life applications. However, performing a bimanual task brings many challenges, such as synchronization and coordination of the single-arm policies. This article proposes the safe, interactive movement primitives learning (SIMPLe) algorithm, to teach and correct single or dual arm impedance policies directly from human kinesthetic demonstrations. Moreover, it proposes a novel graph encoding of the policy based on Gaussian process regression where the single-arm motion is guaranteed to converge close to the trajectory and then toward the demonstrated goal. Regulation of the robot stiffness according to the epistemic uncertainty of the policy allows for easily reshaping the motion with human feedback and/or adapting to external perturbations. We tested the SIMPLe algorithm on a real dual-arm setup where the teacher gave separate single-arm demonstrations and then successfully synchronized them only using kinesthetic feedback or where the original bimanual demonstration was locally reshaped to pick a box at a different height.

@en

Many daily tasks exhibit a periodic nature, necessitating that robots possess the ability to execute them either alone or in collaboration with humans. A widely used approach to encode and learn such periodic patterns from human demonstrations is through periodic Dynamic Movement Primitives (DMPs). Periodic DMPs encode cyclic data independently across multiple dimensions of multi-degree of freedom systems. This method is effective for simple data, like Cartesian or joint position trajectories. However, it cannot account for various geometric constraints imposed by more complex data, such as orientation and stiffness. To bridge this gap, we propose a novel periodic DMP formulation that enables the encoding of periodic orientation trajectories and varying stiffness matrices while considering their geometric constraints. Our geometry-aware approach exploits the properties of the Riemannian manifold and Lie group to directly encode such periodic data while respecting its inherent geometric constraints. We initially employed simulation to validate the technical aspects of the proposed method thoroughly. Subsequently, we conducted experiments with two different real-world robots performing daily tasks involving periodic changes in orientation and/or stiffness, i.e., operating a drilling machine using a rotary handle and facilitating collaborative human–robot sawing.

@en

In this work, we propose a method of capturing the patient’s discomfort during robotic shoulder physiotherapy, creating "discomfort maps". These maps depict the personalized distribution of discomfort that each patient perceived across their shoulder range of motion, facilitating both robotic devices and human therapists to account for patient-specific characteristics during the therapeutic process. Our system enables a patient to communicate and map discomfort in space and time during movement via a handheld push-button device, while interacting with a robotic physical therapy device capable of moving the patient and estimating their pose. We validated our method through human factors experiments simulating shoulder physiotherapy sessions with 10 healthy participants. To avoid the risk of injury to the participants and to allow for ground truth map information, we emulate perceived discomfort via an auditory signal. Our experimental apparatus enabled participants to reconstruct synthetic discomfort maps, demonstrating the feasibility of automatically capturing and storing patient discomfort during robotic physiotherapy.@en

Real-world applications of Artificial Intelligence (AI) in architecture have been explored more recently at Technical University (TU) Delft by integrating AI in Design-to-Robotic-Production-Assembly and -Operation (D2RPA&O) methods. These embed robotics into building processes and buildings by linking computational design with robotic construction and/ or operation of building components and buildings. This paper presents two case studies in which AI-supported D2RA is implemented in a multidisciplinary approach that requires the integration of research domains such as architecture, robotics, computer and material science.@en

In this paper, we present a design and evaluation of a novel finger-operated teleimpedance interface used to command stiffness ellipsoids to the remote robot. The proposed interface provides a practical alternative to the state-of-the-art teleimpedance interfaces based on physiological signals that can be impractical in daily use. On the other hand, as opposed to existing practical interfaces that lack in terms of controlled degrees of freedom, the proposed interface enables control of 3D aspects of the ellipsoid. The remote robot stiffness ellipsoid is controlled with a single hand using the thumb, index, and middle fingers to operate two scroll wheels, a joystick, and a force sensor. These combinations of inputs can be mapped to control different aspects of the stiffness ellipsoid, i.e., orientation and shape/size. To investigate different modes of input mapping, we perform a human factors experiment to evaluate the performance and user acceptance of the proposed interface modes. The results of the experiments indicate that the participants can successfully operate the interface to complete 3D stiffness configuration alignment tasks in different modes. To further demonstrate the functionality of the proposed teleimpedance interface, we performed an additional experiment utilising a Force Dimension Sigma7 haptic device to control the motion of a KUKA LBR iiwa robotic arm while performing a complex physical interaction task.@en

This paper presents a method for semiautonomous teleimpedance where the control is shared between the human operator and the robot. The human commands the position of the teleoperated robotic arm end-effector while the robot autonomously adjusts the impedance depending on the object with which the end-effector interacts. We developed a vision system that calculates the appropriate robot stiffness based on the detected object geometry and material and object’s relation to the environment. This system uses an RGB-D camera near the robot’s end-effector to capture different perspectives of the scene. To validate the proposed method, we conducted experiments on a teleoperation system where a Force Dimension Sigma7 haptic device was used to operate a KUKA LBR iiwa robotics arm. At the same time, the Intel RealSense D455 depth camera provided the visual input. We examined two practical tasks: engaging with bolts on a plate and polishing a stripe.@en

Teleoperation is a crucial technology enabling human operators to control robots remotely to perform tasks in hazardous and difficult-to-access environments. Tasks in such environments often involve complex physical interactions with tools and objects of various softness. To this end, teleimpedance enables the operators to adjust the robot impedance in real-time to simplify such interactions. While the existing teleimpedance approaches provide several interfaces to command the robot impedance, there are no interfaces to visualize both the commanded impedance and that of the objects to be interacted with. This paper presents a novel interface to provide visual feedback on the impedance of remote robots and objects. To do so, we use virtual stiffness ellipsoids and different modes that display the individual impedance of the robot and objects as well as combined post-contact impedance. The key advantage of visual feedback on the impedance compared to force feedback is that the operator can see the interaction characteristics before the contact occurs. This enables the operator to act proactively before contact rather than just reactively after the contact. This paper also proposes a new intuitive way to command the robot impedance using mixed reality, interacting with these ellipsoids and modifying them as needed. To demonstrate the key functionalities of the developed interface, we performed proof-of-concept experiments on teleoperated tasks.@en

In this research, we propose a novel method to estimate and monitor rotator cuff tendon strains during active robotic-assisted rehabilitation. This is a significant step forward from our previous work which estimated these tendon strains during passive exercises (i.e., no muscle activity). Physiotherapists adopt a cautious approach to rehabilitation treatment to prevent (re-) injury given the limited available information about the shoulder's internal condition. By leveraging a robotic device and a musculoskeletal model, our approach provides quantitative information on the risk of re-injury by monitoring the strains of the rotator cuff tendons during shoulder movements with the application of external loads. Muscle strains depend on the shoulder kinematic state and muscle activations, which makes it crucial to obtain physiologically realistic joint kinematics to estimate real-time muscle function. To obtain the strains, we utilize our muscle redundancy solver that incorporates constraints on model accelerations, the glenohumeral joint reaction forces, and active muscle dynamics. Using this algorithm along with force and pose data from a collaborative robotic arm, we demonstrate the ability to update our tendon strain estimates based on muscle activation during robotic-assisted rehabilitation exercises. The findings of our research pave the way for establishing improved therapy that considers the risk of injury to individual muscles and explores a broader and more personalized range of motion. By providing physiotherapists with valuable quantitative information on rotator cuff tendon strains, our method empowers them to optimize rehabilitation protocols and deliver more personalized and effective care. In addition, the system and method presented here comprise a tool capable of offering new insights into the relationship between the rotator cuff muscles, external forces, and shoulder kinematics.@en

Despite the significant progress made in making robots more intelligent and autonomous, today, teleoperation remains a dominant robot control paradigm for the execution of complex and highly unpredictable tasks. Attempts have been made to make teleoperation systems stable, easy to use, and efficient in terms of physical interactions between the follower remote robot and the environment. In particular, the emergence of torque-controlled robots has permitted to regulate the interaction forces from a distance through direct force or impedance control, enabling them to engage in complex interaction tasks. Exploiting this feature, the concept of teleimpedance control was introduced as an alternative method to bilateral force-reflecting teleoperation. The aim was to create a feed-froward yet contact-efficient teleoperation by enriching the leader commands with desired impedance profiles while executing a task. Since then, the teleimpedance concept has found its way into a wide range of interface and controller designs, as well as application domains. Accordingly, after a decade of research progress, this survey aims to provide: first, a convenient introduction of the concept to new researchers in the field, second, consolidate the existing state-of-the-art for active researchers, third, and discuss the pros and cons of different methods in terms of interface and force feedback to provide guidelines for different applications and future developments.

@en

Robotic teleoperation is used in various applications, including the nuclear industry, where the experience and intelligence of a human operator are necessary for making complex decisions that are beyond the autonomy of robots. Human-robot interfaces that help strengthen an operators situational awareness without inducing excessive cognitive load are crucial to the success of teleoperation. This paper presents a novel visual interface that allows operators to simultaneously control a 6-DoF camera platform and a robotic manipulator whilst experiencing the remote environment through a virtual reality head-mounted display (HMD). The proposed system, Orbital Head-Mounted Display (OHMD), utilizes head rotation tracking to command camera movement in azimuth and elevation directions around a fixation point located at a robot's end-effector. A human factor study was conducted to compare the interface acceptance, perceived workload, and task performance of OHMD with a conventional interface utilizing multiple fixed cameras (Array) and a standard head-mounted display implementation (HMD). Results show that both the OHMD and HMD interfaces significantly improve task performance, reduce perceived workload and increase interface acceptance compared to the Array interface. Participants reported they preferred OHMD due to the increased assistance and freedom in viewpoint selection. Whilst OHMD excelled in usefulness, the standard HMD interface allowed operators to perform robotic welding tasks significantly faster.@en

This paper studies non-physical feedback mechanisms to guide human workers toward ergonomic body postures. Specifically, the focus is to solve the tasks that involve no direct physical interaction between the human and the robotic system, therefore tactile guidance by the robot body is not feasible. We propose a multi-modal ergonomic posture guidance system that comprises visual feedback and speech-based audio feedback. We hypothesise that the proposed multi-modal system leads to better performance compared to uni-modal feedback systems when trying to guide users from one pose to another. To test the hypothesis we conducted an experiment that compared conditions with only audio feedback, only visual feedback and multi-modal feedback. In addition, we examined speech-based audio guidance in joint space and in endpoint space. The results showed that the speech-based feedback in joint space came out as the preferred audio feedback due to its ability to allow users to carry out efficient and coordinated inter-joint movements, especially in cases of high redundancy. Furthermore, the proposed multi-modal feedback system was superior compared to the other feedback modalities both in terms of objective measures and subjective measures.

@en

Daily household tasks involve manipulation in cluttered and unpredictable environments and service robots require complex skills and adaptability to perform such tasks. To this end, we developed a teleoperated online learning approach with a novel skill refinement method, where the operator can make refinements to the initially trained skill by a haptic device. After a refined trajectory is formed, it is used to update a probabilistic trajectory model conditioned to the environment state. Therefore, the initial model can be adapted when unknown variations occur and the method is able to deal with different object positions and initial robot poses. This enables human operators to remotely correct or teach complex robotic manipulation skills. Such an approach can help to alleviate shortages of caretakers in elderly care and reduce travel time between homes of different elderly to reprogram the service robots whenever they get stuck. We performed a human factors experiment on 18 participants teaching a service robot how to empty a dishwasher, which is a common daily household task performed by caregivers. We compared the developed method against three other methods. The results show that the proposed method performs better in terms of how much time it takes to successfully adapt a model and in terms of the perceived workload.

@en

The complexity of the human shoulder girdle enables the large mobility of the upper extremity, but also introduces instability of the glenohumeral (GH) joint. Shoulder movements are generated by coordinating large superficial and deeper stabilizing muscles spanning numerous degrees-of-freedom. How shoulder muscles are coordinated to stabilize the movement of the GH joint remains widely unknown. Musculoskeletal simulations are powerful tools to gain insights into the actions of individual muscles and particularly of those that are difficult to measure. In this study, we analyze how enforcement of GH joint stability in a musculoskeletal model affects the estimates of individual muscle activity during shoulder movements. To estimate both muscle activity and GH stability from recorded shoulder movements, we developed a Rapid Muscle Redundancy (RMR) solver to include constraints on joint reaction forces (JRFs) from a musculoskeletal model. The RMR solver yields muscle activations and joint forces by minimizing the weighted sum of squared-activations, while matching experimental motion. We implemented three new features: first, computed muscle forces include active and passive fiber contributions; second, muscle activation rates are enforced to be physiological, and third, JRFs are efficiently formulated as linear functions of activations. Muscle activity from the RMR solver without GH stability was not different from the computed muscle control (CMC) algorithm and electromyography of superficial muscles. The efficiency of the solver enabled us to test over 3600 trials sampled within the uncertainty of the experimental movements to test the differences in muscle activity with and without GH joint stability enforced. We found that enforcing GH stability significantly increases the estimated activity of the rotator cuff muscles but not of most superficial muscles. Therefore, a comparison of shoulder model muscle activity to EMG measurements of superficial muscles alone is insufficient to validate the activity of rotator cuff muscles estimated from musculoskeletal models.@en

Skill propagation among robots without human involvement can be crucial in quickly spreading new physical skills to many robots. In this respect, it is a good alternative to pure reinforcement learning, which can be time-consuming, or learning from human demonstration, which requires human involvement. In the latter case, there may not be enough humans to quickly spread skills to many robots. However, propagation among robots without direct human supervision can result in robotic skills mutating from the original source. This can be beneficial when better skills might emerge or when a new skill is obtained to be used for other similar tasks. However, it can also be dangerous in terms of task execution safety. This letter studies the mutation of a robotic skill when it is propagated from one robot to another during a physically collaborative task. We chose the collaborative sawing task as a study case since it involves complex two-agent physical interaction/coordination and because its periodic nature can facilitate repetitive learning. The study employs periodic Dynamic Movement Primitives and Locally Weight Regression to encode and learn the motion and impedance required to execute the task. To explore what influences mutation, we varied several control and environment conditions such as the maximum stiffness, robot base position, friction coefficient of the sawed object, and movement period. The results showed that the skill varied over propagation steps and we identified several key aspects of mutation such as movement length, movement offset, and trajectory shape. Based on the results we identified possible benefits (skill mutations useful for different settings or different tasks, and energy efficiency) and dangers (high forces and skill mutations becoming useless for the original task) of the mutation.

@en