Cerebral X-ray digital subtraction angiography (DSA) is a widely used imaging technique in patients with neurovascular disease, allowing for vessel and flow visualization with high spatio-temporal resolution. Automatic artery–vein segmentation in DSA plays a fundamental role in vascular analysis with quantitative biomarker extraction, facilitating a wide range of clinical applications. The widely adopted U-Net applied on static DSA frames often struggles with disentangling vessels from subtraction artifacts. Further, it falls short in effectively separating arteries and veins as it disregards the temporal perspectives inherent in DSA. To address these limitations, we propose to simultaneously leverage spatial vasculature and temporal cerebral flow characteristics to segment arteries and veins in DSA. The proposed network, coined CAVE, encodes a 2D+time DSA series using spatial modules, aggregates all the features using temporal modules, and decodes it into 2D segmentation maps. On a large multi-center clinical dataset, CAVE achieves a vessel segmentation Dice of 0.84 (±0.04) and an artery–vein segmentation Dice of 0.79 (±0.06). CAVE surpasses traditional Frangi-based k-means clustering (P < 0.001) and U-Net (P < 0.001) by a significant margin, demonstrating the advantages of harvesting spatio-temporal features. This study represents the first investigation into automatic artery–vein segmentation in DSA using deep learning. The code is publicly available at https://github.com/RuishengSu/CAVE_DSA.

Purpose: For tumor resection, surgeons need to localize the tumor. For this purpose, a magnetic seed can be inserted into the tumor by a radiologist and, during surgery, a magnetic detection probe informs the distance to the seed for localization. In this case, the surgeon still needs to mentally reconstruct the position of the tumor from the probe’s information. The purpose of this study is to develop and assess a method for 3D localization and visualization of the seed, facilitating the localization of the tumor. Methods: We propose a method for 3D localization of the magnetic seed by extending the magnetic detection probe with a tracking-based localization. We attach a position sensor (QR-code or optical marker) to the probe in order to track its 3D pose (respectively, using a head-mounted display with a camera or optical tracker). Following an acquisition protocol, the 3D probe tip and seed position are subsequently obtained by solving a system of equations based on the distances and the 3D probe poses. Results: The method was evaluated with an optical tracking system. An experimental setup using QR-code tracking (resp. using an optical marker) achieves an average of 1.6 mm (resp. 0.8 mm) 3D distance between the localized seed and the ground truth. Using a breast phantom setup, the average 3D distance is 4.7 mm with a QR-code and 2.1 mm with an optical marker. Conclusion: Tracking the magnetic detection probe allows 3D localization of a magnetic seed, which opens doors for augmented reality target visualization during surgery. Such an approach should enhance the perception of the localized region of interest during the intervention, especially for breast tumor resection where magnetic seeds can already be used in the protocol.

Identification and detection of thin-cap fibroatheroma (TCFA) from intravascular optical coherence tomography (IVOCT) images is critical for treatment of coronary heart diseases. Recently, deep learning methods have shown promising successes in TCFA identification. However, most methods usually do not effectively utilize multi-view information or incorporate prior domain knowledge. In this paper, we propose a multi-view contour-constrained transformer network (MVCTN) for TCFA identification in IVOCT images. Inspired by the diagnosis process of cardiologists, we use contour constrained self-attention modules (CCSM) to emphasize features corresponding to salient regions (i.e., vessel walls) in an unsupervised manner and enhance the visual interpretability based on class activation mapping (CAM). Moreover, we exploit transformer modules (TM) to build global-range relations between two views (i.e., polar and Cartesian views) to effectively fuse features at multiple feature scales. Experimental results on a semi-public dataset and an in-house dataset demonstrate that the proposed MVCTN outperforms other single-view and multi-view methods. Lastly, the proposed MVCTN can also provide meaningful visualization for cardiologists via CAM.

Background: X-ray digital subtraction angiography (DSA) is the imaging modality for peri-procedural guidance and treatment evaluation in (neuro-) vascular interventions. Perfusion image construction from DSA, as a means of quantitatively depicting cerebral hemodynamics, has been shown feasible. However, the quantitative property of perfusion DSA has not been well studied. Purpose: To comparatively study the independence of deconvolution-based perfusion DSA with respect to varying injection protocols, as well as its sensitivity to alterations in brain conditions. Methods: We developed a deconvolution-based algorithm to compute perfusion parametric images from DSA, including cerebral blood volume (CBV (Figure presented.)), cerebral blood flow (CBF (Figure presented.)), time to maximum (Tmax), and mean transit time (MTT (Figure presented.)) and applied it to DSA sequences obtained from two swine models. We also extracted the time intensity curve (TIC)-derived parameters, that is, area under the curve (AUC), peak concentration of the curve, and the time to peak (TTP) from these sequences. Deconvolution-based parameters were quantitatively compared to TIC-derived parameters in terms of consistency upon variations in injection profile and time resolution of DSA, as well as sensitivity to alterations of cerebral condition. Results: Comparing to TIC-derived parameters, the standard deviation (SD) of deconvolution-based parameters (normalized with respect to the mean) are two to five times smaller, indicating that they are more consistent across different injection protocols and time resolutions. Upon ischemic stroke induced in a swine model, the sensitivities of deconvolution-based parameters are equal to, if not higher than, those of TIC-derived parameters. Conclusions: In comparison to TIC-derived parameters, deconvolution-based perfusion imaging in DSA shows significantly higher quantitative reliability against variations in injection protocols across different time resolutions, and is sensitive to alterations in cerebral hemodynamics. The quantitative nature of perfusion angiography may allow for objective treatment assessment in neurovascular interventions.

Extracting the cerebral anterior vessel tree of patients with an intracranial large vessel occlusion (LVO) is relevant to investigate potential biomarkers that can contribute to treatment decision making. The purpose of our work is to develop a method that can achieve this from routinely acquired computed tomography angiography (CTA) and computed tomography perfusion (CTP) images. To this end, we regard the anterior vessel tree as a set of bifurcations and connected centerlines. The method consists of a proximal policy optimization (PPO) based deep reinforcement learning (DRL) approach for tracking centerlines, a convolutional neural network based bifurcation detector, and a breadth-first vessel tree construction approach taking the tracking and bifurcation detection results as input. We experimentally determine the added values of various components of the tracker. Both DRL vessel tracking and CNN bifurcation detection were assessed in a cross validation experiment using 115 subjects. The anterior vessel tree formation was evaluated on an independent test set of 25 subjects, and compared to interobserver variation on a small subset of images. The DRL tracking result achieves a median overlapping rate until the first error (1.8 mm off the reference standard) of 100, [46, 100] % on 8032 vessels over 115 subjects. The bifurcation detector reaches an average recall and precision of 76% and 87% respectively during the vessel tree formation process. The final vessel tree formation achieves a median recall of 68% and precision of 70%, which is in line with the interobserver agreement.

Optical-based navigation systems are widely used in surgical interventions. However, despite their great utility and accuracy, they are expensive and require time and effort to setup for surgeries. Moreover, traditional navigation systems use 2D screens to display instrument positions causing the surgeons to look away from the operative field. Head mounted displays such as the Microsoft HoloLens may provide an attractive alternative for surgical navigation that also permits augmented reality visualization. The HoloLens is equipped with multiple sensors for tracking and scene understanding. Mono and stereo-vision in the HoloLens have been both reported to be used for marker tracking, but no extensive evaluation on accuracy has been done to compare the two approaches. The objective of our work is to investigate the tracking performance of various camera setups in the HoloLens, and to study the effect of the marker size, marker distance from camera, and camera resolution on marker locating accuracy. We also investigate the speed and stability of marker pose for each camera setup. The tracking approaches are evaluated using ArUco markers. Our results show that mono-vision is more accurate in marker locating than stereo-vision when high resolution is used. However, this comes at the expense of higher frame processing time. Alternatively, we propose a combined low-resolution mono-stereo tracking setup that outperforms each tracking approach individually and is comparable to high resolution mono tracking, with a mean translational error of 1.8 ± 0.6mm for 10cm marker size at 50cm distance. We further discuss our findings and their implications for navigation in surgical interventions.

Purpose: The assessment of collateral status may depend on the timing of image acquisition. The purpose of this study is to investigate whether there are optimal time points in CT Perfusion (CTP) for collateral status assessment, and compare collaterals scores at these time points with collateral scores from multiphase CT angiography (mCTA). Methods: Patients with an acute intracranial occlusion who underwent baseline non-contrast CT, mCTA and CT perfusion were selected. Collateral status was assessed using an automatically computed Collateral Ratio (CR) score in mCTA, and predefined time points in CTP acquisition. CRs extracted from CTP were correlated with CRs from mCTA. In addition, all CRs were related to baseline National Institutes of Health Stroke Scale (NIHSS) and Alberta Stoke Program Early CT Score (ASPECTS) with linear regression analysis to find the optimal CR. Results: In total 58 subjects (median age 74 years; interquartile range 61–83 years; 33 male) were included. When comparing the CRs from the CTP vs. mCTA acquisition, the strongest correlations were found between CR from baseline mCTA and the CR at the maximal intensity projection of time-resolved CTP (r = 0.81) and the CR at the peak of arterial enhancement point (r = 0.78). Baseline mCTA-derived CR had the highest correlation with ASPECTS (β = 0.36 (95%CI 0.11, 0.61)) and NIHSS (β = − 0.48 (95%CI − 0.72, − 0.16)). Conclusion: Collateral status assessment strongly depends on the timing of acquisition. Collateral scores obtained from mCTA imaging is close to the optimal collateral score obtained from CTP imaging.

Intracranial vessel perforation is a peri-procedural complication during endovascular therapy (EVT). Prompt recognition is important as its occurrence is strongly associated with unfavorable treatment outcomes. However, perforations can be hard to detect because they are rare, can be subtle, and the interventionalist is working under time pressure and focused on treatment of vessel occlusions. Automatic detection holds potential to improve rapid identification of intracranial vessel perforation. In this work, we present the first study on automated perforation detection and localization on X-ray digital subtraction angiography (DSA) image series. We adapt several state-of-the-art single-frame detectors and further propose temporal modules to learn the progressive dynamics of contrast extravasation. Application-tailored loss function and post-processing techniques are designed. We train and validate various automated methods using two national multi-center datasets (i.e., MR CLEAN Registry and MR CLEAN-NoIV Trial), and one international multi-trial dataset (i.e., the HERMES collaboration). With ten-fold cross-validation, the proposed methods achieve an area under the curve (AUC) of the receiver operating characteristic of 0.93 in terms of series level perforation classification. Perforation localization precision and recall reach 0.83 and 0.70 respectively. Furthermore, we demonstrate that the proposed automatic solutions perform at similar level as an expert radiologist.

Purpose: In minimally invasive spring-assisted craniectomy, surgeons plan the surgery by manually locating the cranial sutures. However, this approach is prone to error. Augmented reality (AR) could be used to visualize the cranial sutures and assist in the surgery planning. The purpose of our work is to develop an AR-based system to visualize cranial sutures, and to assess the accuracy and usability of using AR-based navigation for surgical guidance in minimally invasive spring-assisted craniectomy. Methods: An AR system was developed that consists of an electromagnetic tracking system linked with a Microsoft HoloLens. The system was used to conduct a study with two skull phantoms. For each phantom, five sutures were annotated and visualized on the skull surface. Twelve participants assessed the system. For each participant, model alignment using six anatomical landmarks was performed, followed by the participant delineation of the visualized sutures. At the end, the participants filled a system usability scale (SUS) questionnaire. For evaluation, an independent optical tracking system was used and the delineated sutures were digitized and compared to the CT-annotated sutures. Results: For a total of 120 delineated sutures, the distance of the annotated sutures to the planning reference was 2.4 ± 1.2 mm. The average delineation time per suture was 13 ± 5 s. For the system usability questionnaire, an average SUS score of 73 was obtained. Conclusion: The developed AR-system has good accuracy (average 2.4 mm distance) and could be used in the OR. The system can assist in the pre-planning of minimally invasive craniosynostosis surgeries to locate cranial sutures accurately instead of the traditional approach of manual palpation. Although the conducted phantom study was designed to closely reflect the clinical setup in the OR, further clinical validation of the developed system is needed and will be addressed in a future work.

Augmented reality (AR) permits the visualization of pre-operative data in the surgical field of view of the surgeon. This requires the alignment of the AR device’s coordinate system with the used navigation/tracking system. We propose a multimodal marker approach to align an AR device with a tracking system: in our implementation, an electromagnetic tracking system (EMTS). The solution makes use of a calibration method which determines the relationship between a 2D pattern detected by an RGB camera and an electromagnetic sensor of the EMTS. This allowed the projection of a 3D skull model on its physical counterpart. This projection was evaluated using a monocular camera and an optical see-through device (HoloLens 2) (https://www.microsoft.com/en-us/hololens/) achieving an accuracy of less than 2.5 mm in the image plane of the HoloLens 2 (HL2). Additionally, 10 volunteers participated in a user study consisting of an alignment task of a pointer with 25 projections viewed through the HL2. The participants achieved a mean error of 2.7 1.3 mm and 2.9 2.9^∘ in positional and orientation error. This study showcases the feasibility of the approach, provides an evaluation of the alignment, and finally, discusses its advantages and limitations.

Craniotomy is a procedure where neurosurgeons open the patient’s skull to gain direct access to the brain. The craniotomy’s position defines the access path from the skull surface to the tumour and, consequently, the healthy brain tissue to be removed to reach the tumour. This is a complex procedure where a neurosurgeon is required to mentally reconstruct spatial relations of important brain structures to avoid removing them as much as possible. We propose a visualisation method using Augmented Reality to assist in the planning of a craniotomy. The goal of this study is to visualise important brain structures aligned with the physical position of the patient and to allow a better perception of the spatial relations of the structures. Additionally, a heat map was developed that is projected on top of the skull to provide a quick overview of the structures between a chosen location on the skull and the tumour. In the experiments, tracking accuracy was assessed, and colour maps were assessed for use in an AR device. Additionally, we conducted a user study amongst neurosurgeons and surgeons from other fields to evaluate the proposed visualisation using a phantom head. Most participants indeed agree that the visualisation can assist in planning a craniotomy and feedback on future improvements towards the clinical scenario was collected. (see https://www.acm.org/publications/class-2012)

The Thrombolysis in Cerebral Infarction (TICI) score is an important metric for reperfusion therapy assessment in acute ischemic stroke. It is commonly used as a technical outcome measure after endovascular treatment (EVT). Existing TICI scores are defined in coarse ordinal grades based on visual inspection, leading to inter-and intra-observer variation. In this work, we present autoTICI, an automatic and quantitative TICI scoring method. First, each digital subtraction angiography (DSA) acquisition is separated into four phases (non-contrast, arterial, parenchymal and venous phase) using a multi-path convolutional neural network (CNN), which exploits spatio-temporal features. The network also incorporates sequence level label dependencies in the form of a state-transition matrix. Next, a minimum intensity map (MINIP) is computed using the motion corrected arterial and parenchymal frames. On the MINIP image, vessel, perfusion and background pixels are segmented. Finally, we quantify the autoTICI score as the ratio of reperfused pixels after EVT. On a routinely acquired multi-center dataset, the proposed autoTICI shows good correlation with the extended TICI (eTICI) reference with an average area under the curve (AUC) score of 0.81. The AUC score is 0.90 with respect to the dichotomized eTICI. In terms of clinical outcome prediction, we demonstrate that autoTICI is overall comparable to eTICI.

Retinal vascular diseases are a leading cause for blindness and partial sight certifications. By applying adaptive optics (AO) to conventional imaging modalities, the microstructures of the retinal vasculature can be observed with high spatial resolution, hence offering a unique opportunity for the exploration of the human microcirculation. The objective of this systematic review is to describe the current state of retinal vascular biomarkers imaged by AO flood illumination ophthalmoscopy (FIO) and AO scanning laser ophthalmoscopy (SLO). A literature research was conducted in the PubMed and Scopus databases on July 9, 2020. From 217 screened studies, 42 were eligible for this review. All studies underwent a quality check regarding their content. A meta-analysis was performed for the biomarkers reported for the same pathology in at least three studies using the same modality. The most frequently studied vascular biomarkers were the inner diameter (ID), outer diameter (OD), parietal thickness (PT), wall cross-sectional area (WCSA), and wall-to-lumen ratio (WLR). The applicability of AO vascular biomarkers has been mostly explored in systemic hypertension using AO FIO and in diabetes using AO SLO. The result of the meta-analysis for hypertensive patients showed that WLR, PT, and ID were significantly different when compared to healthy controls, while WCSA was not (P < 0.001, P = 0.002, P < 0.001, and P = 0.070, respectively). The presented review shows that, although a substantial number of retinal vascular biomarkers have been explored in AO en face imaging, further clinical research and standardization of procedures is needed to validate such biomarkers for the longitudinal monitoring of arterial hypertension and other diseases.

Purpose: To present a novel methodical approach to compare visibility of percutaneous needles in ultrasound images. Methods: A motor-driven rotation platform was used to gradually change the needle angle while capturing image data. Data analysis was automated using block-matching-based registration, with a tracking and refinement step. Every 25 frames, a Hough transform was used to improve needle alignments after large rotations. The method was demonstrated by comparing three commercial needles (14G radiofrequency ablation, RFA; 18G Trocar; 22G Chiba) and six prototype needles with different sizes, materials, and surface conditions (polished, sand-blasted, and kerfed), within polyvinyl alcohol phantom tissue and ex vivo bovine liver models. For each needle and angle, a contrast-to-noise ratio (CNR) was determined to quantify visibility. CNR values are presented as a function of needle type and insertion angle. In addition, the normalized area under the (CNR-angle) curve was used as a summary metric to compare needles. Results: In phantom tissue, the first kerfed needle design had the largest normalized area of visibility and the polished 1 mm diameter stainless steel needle the smallest (0.704 ± 0.199 vs. 0.154 ± 0.027, p < 0.01). In the ex vivo model, the second kerfed needle design had the largest normalized area of visibility, and the sand-blasted stainless steel needle the smallest (0.470 ± 0.190 vs. 0.127 ± 0.047, p < 0.001). As expected, the analysis showed needle visibility peaks at orthogonal insertion angles. For acute or obtuse angles, needle visibility was similar or reduced. Overall, the variability in needle visibility was considerably higher in livers. Conclusion: The best overall visibility was found with kerfed needles and the commercial RFA needle. The presented methodical approach to quantify ultrasound visibility allows comparisons of (echogenic) needles, as well as other technological innovations aiming to improve ultrasound visibility of percutaneous needles, such as coatings, material treatments, and beam steering approaches.

We propose a consistent ultrasound volume stitching framework, with the intention to produce a volume with higher image quality and extended field-of-view in this work. Directly using pair-wise registrations for stitching may lead to geometric errors. Therefore, we propose an approach to improve the image alignment by optimizing a consistency metric over multiple pairwise registrations. In the optimization, we utilize transformed points to effectively compute a distance between rigid transformations. The method has been evaluated on synthetic, phantom and clinical data. The results indicate that our transformation optimization method is effective and our stitching framework has a good geometric precision. Also, the compound images have been demonstrated to have improved CNR values.

The collateral score is an important biomarker in decision making for endovascular treatment (EVT) of patients with ischemic stroke. The existing collateral grading systems are based on visual inspection and prone to subjective interpretation and interobserver variation. The purpose of our work is the development of an automatic collateral scoring method. In this work, we present a method that is inspired by human collateral scoring. Firstly, we define an anatomical region by atlas-based registration and extract vessel structures using a deep convolutional neural network. From this, high-level features based on the ratios of vessel length and volume of the occluded and the contralateral side are defined. Multi-class classification models are used to map the feature space to a four-grade collateral score and a quantitative score. The dataset used for training, validation and testing is from a registry of images acquired in clinical routine at multiple medical centers. The model performance is tested on 269 subjects, achieving an accuracy of 0.8. The dichotomized collateral score accuracy is 0.9. The error is comparable to the interobserver variation, the results are comparable to the performance of two radiologists with 10 to 30 years of experience.

In this work we present a fast approach to perform registration of computed tomography to ultrasound volumes for image guided intervention applications. The method is based on a combination of block-matching and outlier rejection. The block-matching uses a correlation based multimodal similarity metric, where the intensity and the gradient of the computed tomography images along with the ultrasound volumes are the input images to find correspondences between blocks in the computed tomography and the ultrasound volumes. A variance and octree based feature point-set selection method is used for selecting distinct and evenly spread point locations for block-matching. Geometric consistency and smoothness criteria are imposed in an outlier rejection step to refine the block-matching results. The block-matching results after outlier rejection are used to determine the affine transformation between the computed tomography and the ultrasound volumes. Various experiments are carried out to assess the optimal performance and the influence of parameters on accuracy and computational time of the registration. A leave-one-patient-out cross-validation registration error of 3.6 mm is achieved over 29 datasets, acquired from 17 patients.

Purpose: Compensation for respiratory motion is important during abdominal cancer treatments. In this work we report the results of the 2015 MICCAI Challenge on Liver Ultrasound Tracking and extend the 2D results to relate them to clinical relevance in form of reducing treatment margins and hence sparing healthy tissues, while maintaining full duty cycle. Methods: We describe methodologies for estimating and temporally predicting respiratory liver motion from continuous ultrasound imaging, used during ultrasound-guided radiation therapy. Furthermore, we investigated the trade-off between tracking accuracy and runtime in combination with temporal prediction strategies and their impact on treatment margins. Results: Based on 2D ultrasound sequences from 39 volunteers, a mean tracking accuracy of 0.9 mm was achieved when combining the results from the 4 challenge submissions (1.2 to 3.3 mm). The two submissions for the 3D sequences from 14 volunteers provided mean accuracies of 1.7 and 1.8 mm. In combination with temporal prediction, using the faster (41 vs 228 ms) but less accurate (1.4 vs 0.9 mm) tracking method resulted in substantially reduced treatment margins (70% vs 39%) in contrast to mid-ventilation margins, as it avoided non-linear temporal prediction by keeping the treatment system latency low (150 vs 400 ms). Acceleration of the best tracking method would improve the margin reduction to 75%. Conclusions: Liver motion estimation and prediction during free-breathing from 2D ultrasound images can substantially reduce the in-plane motion uncertainty and hence treatment margins. Employing an accurate tracking method while avoiding non-linear temporal prediction would be favorable. This approach has the potential to shorten treatment time compared to breath-hold and gated approaches, and increase treatment efficiency and safety.

Multimodal image fusion for image guidance in minimally invasive liver interventions generally requires the registration of pre-operatively acquired images with interventional images of the patient. Whereas rigid registration approaches are fast and can be used in an interventional setting, the actual liver deformation may be nonrigid. The purpose of this paper is to assess the magnitude of nonrigid deformation of the liver between pre-operative and interventional CT images in the case of tumor ablations, over the full liver and over parts of the liver that match the volumes typically imaged by a 3D ultrasound transducer. We acquired 3D abdominal CT scans of 38 patients that had undergone the radiofrequency ablation of liver tumors, pre-operative CT images as well as intraoperative CT images. To determine the magnitude of liver deformation due to pose changes and respiration, we nonrigidly registered the pre-operative CT scan with the intraoperative CT scan. By fitting the deformation to a rigid transformation in the region of interest and computing the residual displacements, the nonrigid deformation part can be quantified. We performed quantifications over the complete liver, as well as for two volumes of interest representative of sub-xiphoidal and intercostal 3D ultrasound acquisitions. The results showed that a substantial amount of nonrigid deformation was found, and rotation of the patient's pose and deep inhalation caused significant liver deformation. Hence we concluded that nonrigid motion correction in the interventions should be taken into account.

Multi-modal registration, especially CT/MR to ultrasound (US), is still a challenge, as conventional similarity metrics such as mutual information do not match the imaging characteristics of ultrasound. The main motivation for this work is to investigate whether a deep learning network can be used to directly estimate the displacement between a pair of multi-modal image patches, without explicitly performing similarity metric and optimizer, the two main components in a registration framework. The proposed DVNet is a fully convolutional neural network and is trained using a large set of artificially generated displacement vectors (DVs). The DVNet was evaluated on mono- and simulated multi-modal data, as well as real CT and US liver slices (selected from 3D volumes). The results show that the DVNet is quite robust on the single- and multi-modal (simulated) data, but does not work yet on the real CT and US images.