Source-linked AI summary
Concurrent Segmentation and Localization for Tracking of Surgical Instruments
Iro Laina, Nicola Rieke, Christian Rupprecht, Josué Page Vizcaíno, Abouzar Eslami, Federico Tombari, Nassir Navab
TL;DR
Surgical instrument tracking must remain accurate despite illumination changes, specular reflections, and motion blur. The paper jointly predicts segmentation and pose by reformulating landmark localization as heatmap regression in a CNN. This concurrent model improves localization over direct coordinate regression and outperforms state-of-the-art methods on retinal microsurgery and EndoVis benchmarks.
Problem
Marker-free surgical instrument tracking remains challenging in-vivo because vision-based methods must handle illumination changes, specular reflections, and motion blur.
Method
CSL jointly predicts semantic segmentation and instrument landmark localization by regressing Gaussian heatmaps with a CNN and shared end-to-end features.
Results
The model achieves over 90% accuracy for tool tips and 79% for the center joint at a 20-pixel threshold, and outperforms state-of-the-art methods on the evaluated benchmarks.
Takeaways & Limitations
The approach provides near-real-time, flexible instrument tracking without initialization, post-processing, or temporal regularization.
Takeaways & Limitations
Localization errors are higher for EndoVis sequences 5 and 6 because their tracking ground truth may be less accurate than for the other sequences.
Abstract
from arXiv · showhide
Real-time instrument tracking is a crucial requirement for various computer-assisted interventions. In order to overcome problems such as specular reflections and motion blur, we propose a novel method that takes advantage of the interdependency between localization and segmentation of the surgical tool. In particular, we reformulate the 2D instrument pose estimation as heatmap regression and thereby enable a concurrent, robust and near real-time regression of both tasks via deep learning. As demonstrated by our experimental results, this modeling leads to a significantly improved performance than directly regressing the tool position and allows our method to outperform the state of the art on a Retinal Microsurgery benchmark and the MICCAI EndoVis Challenge 2015.
1 Introduction and Related Work
Computer-assisted surgical interventions need marker-free instrument segmentation and localization, but in-vivo vision remains challenging under illumination changes, reflections, and blur. The paper therefore proposes concurrent segmentation and pose estimation using heatmap regression and shared deep-learning features.
- Marker-free tracking is desirable because markers can interfere with surgical workflow and require instrument modifications.
- Instrument segmentation supports graphical overlays and workflow analysis, while retinal landmark localization enables proximity estimation to the retina.
- Color features are computationally cheap but vulnerable to strong illumination changes, whereas gradient features are unreliable under motion blur.
- The proposed method performs tool segmentation and pose estimation simultaneously by representing localization as per-pixel landmark heatmap regression.
- Shared deep-learning features support joint training, and the method outperforms other CNN architectures and state-of-the-art methods on retinal microsurgery and EndoVis benchmarks.
2 Method
The method uses a CNN encoder-decoder to predict dense semantic labels and landmark locations from an input image. Its key design choice is Gaussian heatmap regression, which aligns localization with segmentation and incorporates spatial context.
- 2 Method: The CNN maps an input image to tool-landmark locations and dense semantic labeling, using global context to reduce false positives from reflections or shadows.
- 2.1 Encoder: The shared encoder progressively downsamples images, while model variants differ in their decoding stages and output formulations.
- Modeling Strategies: The L baseline directly regresses the x and y coordinates of n landmarks as a 2 × n vector without segmentation.
- Modeling Strategies: The SL model jointly predicts coordinate locations and semantic segmentation, sharing encoder weights before splitting into task-specific branches.
- 2.2 Decoder Tasks: CSL replaces exact landmark coordinates with Gaussian heatmaps whose confidence reflects proximity to each landmark, allowing shared spatial outputs with segmentation.
- 2.2 Decoder Tasks: At test time, the maximum-confidence point in each predicted heatmap determines the landmark location, while high predicted variance indicates a misdetection.
3 Experiments and Results
The experiments evaluate concurrent landmark localization and semantic segmentation on Retinal Microsurgery and EndoVis data. The proposed method shows strong localization, segmentation, state-of-the-art benchmark performance, and flexibility across instrument configurations, with a ground-truth caveat for two EndoVis sequences.
- Datasets and setup: The RM experiments evaluate landmark localization and segmentation using held-out sequences, with three annotated joints and two semantic classes.The RM dataset contains 18 in-vivo sequences divided into four instrument-dependent subsets.
- Modeling strategies: Over 90% accuracy was achieved for both tool tips and 79% for the center joint at a 20-pixel acceptance threshold.These results were obtained by CSL in the RM modeling-strategy evaluation.
- Modeling strategies: 75.4% DICE was achieved, compared with 74.4% for CSL without skip connections, 73.7% for SL, and 72.5% for U-net.The comparison evaluates segmentation performance alongside localization on the RM data.
- Retinal Microsurgery: CSL reached an average KBB score above 84% with α = 0.15 in the RM Half Split experiment and achieved state-of-the-art performance in Cross Validation.Half Split trains on the first halves of all 18 sequences and evaluates on the remaining frames; Cross Validation tests across instrument-dependent subsets.
- EndoVis Challenge: On EndoVis, the method significantly outperformed the previous state of the art and distinguished multiple instrument parts using five semantic classes and two joints.The experiments used leave-one-surgery-out evaluation and multi-class segmentation, with binary results obtained by merging instrument classes.
- EndoVis Challenge: The network successfully localized and segmented a previously unseen instrument and viewpoint in Sets 5 and 6, although tracking ground truth there was considered less accurate.The challenge may include two test instruments even when only one appears in training; horizontal flips were used as augmentation.
4 Conclusion
The method concurrently predicts surgical-instrument segmentation and articulated 2D pose by modeling landmark localization as heatmap regression. It operates near real time without initialization, post-processing, or temporal regularization, and achieves state-of-the-art results on two surgical benchmarks.
- Heatmap regression enables concurrent end-to-end prediction of instrument segmentation and articulated 2D pose.The approach uses deep-learned CNN features to model landmark localization as heatmaps.
- The method flexibly supports different numbers of tracked joints and semantic classes, including distinguishing left and right instruments.These objectives are specified during training by setting the number of semantic classes and heatmaps.
- Near-real-time inference requires no initialization, post-processing technique, or temporal regularization.
- The proposed approach delivers state-of-the-art results on two surgical intervention benchmarks.