Manipulating a flexible ureteroscope is difficult, due to its bendable body and hand–eye coordination problems, especially when exploring the lower pole of the kidney. Though robotic interventions have been adopted in various clinical scenarios, they are rarely used in ureteroscopy. This study proposes a teleoperation system consists of a soft robotic endoscope together with a Guidance Virtual Fixture (GVF) to help users explore the kidney’s lower pole. The soft robotic arm was a cable-driven, 3D-printed design with a helicoid structure. GVF was dynamically constructed using video streams from an endoscopic camera. With a haptic controller, GVF can provide haptic feedback to guide the users in following a trajectory. In the user study, participants were asked to follow trajectories when the soft robotic arm was in a retroflex posture. The results suggest that the GVF can reduce errors in the trajectory tracking tasks when the users receive the proper training and gain more experience. Based on the NASA Task Load Index questionnaires, most participants preferred having the GVF when manipulating the robotic arm. In conclusion, the results demonstrate the benefits and potential of using a robotic arm with a GVF. More research is needed to investigate the effectiveness of the GVFs and the robotic endoscope in ureteroscopic procedures.@en

Minimally Invasive Procedures (MIPs) emerged as an alternative to more invasive surgical approaches, offering patient benefits such as smaller incisions, less pain, and shorter hospital stay. In one class of MIPs, where natural body lumens or small incisions are used to access deeper anatomical locations, Flexible Surgical and Interventional Robots (FSIRs) such as catheters and endoscopes are widely used. Due to their flexible and compliant nature, FSIRs can be inserted via natural orifices or small incisions, then moved towards hard-to-reach targets to perform interventional tasks. However, existing FSIRs are confronted with challenges in sensing, control, and navigation. These issues stem from the robot's non-linear behavior and the intricate nature of the lumens, where accurately modeling the complex interactions and disturbances proves to be exceptionally difficult. The rapid advances in Machine Learning (ML) have facilitated the widespread adoption of ML techniques in FSIRs. This article provides an overview of these efforts by first introducing a classification of existing ML algorithms, including traditional ML methods and modern Deep Learning (DL) approaches, commonly used in FSIRs. Next, the use of ML algorithms is surveyed per sub-domain, namely for perception, modeling, control, and navigation. Trends, popularity, strengths, and/or limitations of different ML algorithms are analyzed. The different roles that ML plays among tasks are investigated and described. Finally, discussions are conducted on the limitations and the prospects of ML in MIPs.

@en

Endovascular interventions are minimally invasive procedures that utilize the vascular system to access anatomical regions deep within the body. Image-guided assistance provides valuable real-time information about the dynamic state of the vascular environment. However, the reliance on intraoperative 2-D fluoroscopy images limits depth perception, prompting the demand for intraoperative 3-D imaging. Existing image registration methods face difficulties in accurately incorporating tissue deformations compared to the preoperative 3-D model, particularly in a weakly supervised manner. Additionally, reconstructing deformations from 2-D to 3-D space and presenting this intraoperative model visually to clinicians poses further complexities. To address these challenges, this study introduces a novel deformable model-to-image registration framework using deep learning. Furthermore, this research proposes a visualization method through augmented reality to guide endovascular interventions. This study utilized image data collected from nine patients who underwent transcatheter aortic valve implantation (TAVI) procedures. The registration results in 2-D indicate that the proposed deformable model-to-image registration framework achieves a modified dice similarity coefficient (MDSC) value of 0.89±0.02 and a penalization of deformations in spare space (PDSS) value of 0.04±0.01, offering an improvement of 3.5%-98.6% over the state-of-the-art image registration approach. Additionally, the accuracy of registration in 3-D was evaluated using a dataset obtained from an intervention simulator, resulting in a mean absolute error (MAE) of 1.51±1.02 mm within the region of interest. Overall, the study validates the feasibility and accuracy of the proposed weakly supervised deformable model-to-image registration framework, demonstrating its potential to provide intraoperative 3-D imaging as intervention assistance in dynamic vascular environments.

@en

Endovascular intervention is a minimally invasive method for treating cardiovascular diseases. Although fluoroscopy, known for real-time catheter visualization, is commonly used, it exposes patients and physicians to ionizing radiation and lacks depth perception due to its 2D nature. To address these limitations, a study was conducted using teleoperation and 3D visualization techniques. This <italic>in-vitro</italic> study involved the use of a robotic catheter system and aimed to evaluate user performance through both subjective and objective measures. The focus was on determining the most effective modes of interaction. Three interactive modes for guiding robotic catheters were compared in the study: 1) Mode GM, using a gamepad for control and a standard 2D monitor for visual feedback; 2) Mode GH, with a gamepad for control and HoloLens providing 3D visualization; and 3) Mode HH, where HoloLens serves as both control input and visualization device. Mode GH outperformed other modalities in subjective metrics, except for mental demand. It exhibited a median tracking error of 4.72 mm, a median targeting error of 1.01 mm, a median duration of 82.34 s, and a median natural logarithm of dimensionless squared jerk of 40.38 in the <italic>in-vitro</italic> study. Mode GH showed 8.5&#x0025;, 4.7&#x0025;, 6.5&#x0025;, and 3.9&#x0025; improvements over Mode GM and 1.5&#x0025;, 33.6&#x0025;, 34.9&#x0025;, and 8.1&#x0025; over Mode HH for tracking error, targeting error, duration, and dimensionless squared jerk, respectively. To sum up, the user study emphasizes the potential benefits of employing HoloLens for enhanced 3D visualization in catheterization. The user study also illustrates the advantages of using a gamepad for catheter teleoperation, including user-friendliness and passive haptic feedback, compared to HoloLens. To further gauge the potential of using a more traditional joystick as a control input device, an additional study utilizing the Haption Virtuose^TM robot was conducted. It reveals the potential for achieving smoother trajectories, with a 38.9&#x0025; reduction in total path length compared to a gamepad, potentially due to its larger range of motion and single-handed control.

@en

Objective: Navigation through tortuous and deformable vessels using catheters with limited steering capability underscores the need for reliable path planning. State-of-the-art path planners do not fully account for the deformable nature of the environment. Methods: This work proposes a robust path planner via a learning from demonstrations method, named Curriculum Generative Adversarial Imitation Learning (C-GAIL). This path planning framework takes into account the interaction between steerable catheters and vessel walls and the deformable property of vessels. Results: In-silico comparative experiments show that the proposed network achieves a 38% higher success rate in static environments and 17% higher in dynamic environments compared to a state-of-the-art approach based on GAIL. In-vitro validation experiments indicate that the path generated by the proposed C-GAIL path planner achieves a targeting error of 1.26±0.55mm and a tracking error of 5.18±3.48mm. These results represent improvements of 41% and 40% over the conventional centerline-following technique for targeting error and tracking error, respectively. Conclusion: The proposed C-GAIL path planner outperforms the state-of-the-art GAIL approach. The in-vitro validation experiments demonstrate that the path generated by the proposed C-GAIL path planner aligns better with the actual steering capability of the pneumatic artificial muscle-driven catheter utilized in this study. Therefore, the proposed approach can provide enhanced support to the user in navigating the catheter towards the target with greater accuracy, effectively meeting clinical accuracy requirements. Significance: The proposed path planning framework exhibits superior performance in managing uncertainty associated with vessel deformation, thereby resulting in lower tracking errors.

@en

Increased demand for less invasive procedures has accelerated the adoption of Intraluminal Procedures (IP) and Endovascular Interventions (EI) performed through body lumens and vessels. As navigation through lumens and vessels is quite complex, interest grows to establish autonomous navigation techniques for IP and EI for reaching the target area. Current research efforts are directed toward increasing the Level of Autonomy (LoA) during the navigation phase. One key ingredient for autonomous navigation is Motion Planning (MP) techniques. This paper provides an overview of MP techniques categorizing them based on LoA. Our analysis investigates advances for the different clinical scenarios. Through a systematic literature analysis using the PRISMA method, the study summarizes relevant works and investigates the clinical aim, LoA, adopted MP techniques, and validation types. We identify the limitations of the corresponding MP methods and provide directions to improve the robustness of the algorithms in dynamic intraluminal environments. MP for IP and EI can be classified into four subgroups: node, sampling, optimization, and learning-based techniques, with a notable rise in learning-based approaches in recent years. One of the review's contributions is the identification of the limiting factors in IP and EI robotic systems hindering higher levels of autonomous navigation. In the future, navigation is bound to become more autonomous, placing the clinician in a supervisory position to improve control precision and reduce workload.

@en

Navigation inside luminal organs is an arduous task that requires non-intuitive coordination between the movement of the operator's hand and the information obtained from the endoscopic video. The development of tools to automate certain tasks could alleviate the physical and mental load of doctors during interventions allowing them to focus on diagnosis and decision-making tasks. In this paper we present a synergic solution for intraluminal navigation consisting of a 3D printed endoscopic soft robot that can move safely inside luminal structures. Visual servoing based on Convolutional Neural Networks (CNNs) is used to achieve the autonomous navigation task. The CNN is trained with phantoms and in-vivo data to segment the lumen and a model-less approach is presented to control the movement in constrained environments. The proposed robot is validated in anatomical phantoms in different path configurations. We analyze the movement of the robot using different metrics such as task completion time smoothness error in the steady-state mean and maximum error. We show that our method is suitable to navigate safely in hollow environments and conditions which are different than the ones the network was originally trained on.@en

A major challenge during autonomous navigation in endovascular interventions is the complexity of operating in a deformable but constrained workspace with an instrument. Simulation of deformations for it can provide a cost-effective training platform for path planning. Aim of this study is to develop a realistic, auto-adaptive, and visually plausible simulator to predict vessels’ global deformation induced by the robotic catheter's contact and cyclic heartbeat motion. Based on a Position-based Dynamics (PBD) approach for vessel modeling, Particle Swarm Optimization (PSO) algorithm is employed for an auto-adaptive calibration of PBD deformation parameters and of the vessels movement due to a heartbeat. In-vitro experiments were conducted and compared with in-silico results. The end-user evaluation results were reported through quantitative performance metrics and a 5-Point Likert Scale questionnaire. Compared with literature, this simulator has an error of 0.23±0.13% for deformation and 0.30±0.85mm for the aortic root displacement. In-vitro experiments show an error of 1.35±1.38mm for deformation prediction. The end-user evaluation results show that novices are more accustomed to using joystick controllers, and cardiologists are more satisfied with the visual authenticity. The real-time and accurate performance of the simulator make this framework suitable for creating a dynamic environment for autonomous navigation of robotic catheters.

@en

Cardiologists highlight the need for an intra-operative 3D visualization to assist interventions. The intra-operative 2D X-ray/Digital Subtraction Angiography (DSA) images in the standard clinical workflow limit cardiologists’ views significantly. Compared with image-to-image registration, model-to-image registration is an essential approach taking advantage of the reuse of pre-operative 3D models reconstructed from Computed Tomography Angiography (CTA) images. Traditional optimized-based registration methods suffer severely from high computational complexity. Moreover, the consequence of lacking ground truth for learning-based registration approaches should not be neglected. To overcome these challenges, we introduce a model-to-image registration framework via deep learning for image-guided endovascular catheterization. This work performs autonomous vessel segmentation from intra-operative fluoroscopy images via a deep residual U-net and a model-to-image matching via a convolutional neural network. For this study, image data were collected from 10 patients who performed Transcatheter Aortic Valve Implantation (TAVI) procedures. It was found that vessel segmentation of test data results in median values of Dice Similarity Coefficient, Precision, and Recall of (0.75, 0.58, 0.67) for femoral artery, and (0.71, 0.56, 0.74) for aortic root. The segmentation network behaves better than manual annotation, and it recognizes part of vessels that were not labeled manually. Image matching between the transformed moving image and the fixed image results in a median value of Recall of 0.90. The proposed approach achieves a good accuracy of vessel segmentation and a good recall value of model-to-image matching.@en

Purpose: Planning a safe path for flexible catheters is one of the major challenges of endovascular catheterization. State-of-the-art methods rarely consider the catheter curvature constraint and reduced computational time of path planning which guarantees the possibility to re-plan the path during the actual operation. Methods: In this manuscript, we propose a fast two-phase path planning approach under the robot curvature constraint. Firstly, the vascular structure is extracted and represented by vascular centerlines and corresponding vascular radii. Then, the path is searched along the vascular centerline using breadth first search (BFS) strategy and locally optimized via the genetic algorithm (GA) to satisfy the robot curvature constraint. This approach (BFS-GA) is able to respect the robot curvature constraint while keeping it close to the centerlines as much as possible. We can also reduce the optimization search space and perform parallel optimization to shorten the computational time. Results: We demonstrate the method’s high efficiency in two-dimensional and three-dimensional space scenarios. The results showed the planner’s ability to satisfy the robot curvature constraint while keeping low computational time cost compared with sampling-based methods. Path replanning in femoral arteries can reach an updating frequency at 6.4 ± 2.3 Hz. Conclusion: The presented work is suited for surgical procedures demanding satisfying curvature constraints while optimizing specified criteria. It is also applicable for curvature constrained robots in narrow passages.

@en