Source-linked AI summary
Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning
Alexey Shvets, Alexander Rakhlin, Alexandr A. Kalinin, Vladimir Iglovikov
TL;DR
Robot-assisted surgery needs accurate pixel-wise instrument segmentation to support instrument tracking and pose estimation despite challenging surgical imagery. The paper develops U-Net-family deep learning models, including TernausNet and modified LinkNet, and reports state-of-the-art performance across binary and multi-class tasks. The approach also provides an end-to-end basis for efficient surgical-scene analysis.
Problem
Accurate semantic segmentation of robotic instruments is needed for tracking and pose estimation, but surgical images contain shadows, reflections, occlusions, fogging, and dynamic tissues.
Method
The paper modifies U-Net and evaluates TernausNet and modified LinkNet architectures for binary and multi-class robotic instrument segmentation.
Results
The approach achieves state-of-the-art performance in both binary and multi-class settings, including TernausNet-16 IoU=0.836 for binary segmentation and IoU=0.655 for instrument-part segmentation.
Takeaways & Limitations
The end-to-end models provide a foundation for efficient surgical-instrument position detection that can support tracking and pose estimation.
Abstract
from arXiv · showhide
Semantic segmentation of robotic instruments is an important problem for the robot-assisted surgery. One of the main challenges is to correctly detect an instrument's position for the tracking and pose estimation in the vicinity of surgical scenes. Accurate pixel-wise instrument segmentation is needed to address this challenge. In this paper we describe our winning solution for MICCAI 2017 Endoscopic Vision SubChallenge: Robotic Instrument Segmentation. Our approach demonstrates an improvement over the state-of-the-art results using several novel deep neural network architectures. It addressed the binary segmentation problem, where every pixel in an image is labeled as an instrument or background from the surgery video feed. In addition, we solve a multi-class segmentation problem, where we distinguish different instruments or different parts of an instrument from the background. In this setting, our approach outperforms other methods in every task subcategory for automatic instrument segmentation thereby providing state-of-the-art solution for this problem. The source code for our solution is made publicly available at https://github.com/ternaus/robot-surgery-segmentation
1 Introduction
Robotic instrument segmentation supports tracking and pose estimation from surgical video, but remains difficult under visual changes, occlusions, and dynamic tissue backgrounds. The paper presents deep learning methods achieving state-of-the-art binary and multi-class segmentation.
- Motivation: Semantic segmentation provides pixel-wise instrument masks needed for tracking and pose estimation in robot-assisted surgery.Surgical-console images and videos contain instruments and patient tissues used for intra-operative guidance.
- Challenges: Instrument segmentation is challenged by shadows, specular reflections, blood, lens fogging, occlusions, and dynamic background tissues.
- Prior work: Existing approaches frame the task as binary or instance segmentation, while semantic segmentation distinguishes different instruments or instrument parts.
- Prior work: Deep learning methods had shown competitive binary and promising multi-class robotic instrument segmentation before this work.
- Contribution: The paper presents a U-Net-based solution, further improved with TernausNet and modified LinkNet, achieving state-of-the-art results in binary and multi-class settings.
2 Methods
The study evaluates four deep segmentation architectures on stereo surgical-video data for binary and multi-class instrument labeling. Its U-Net-family models combine encoder-decoder processing, skip-connections, Jaccard-based loss, and pixelwise outputs.
- Dataset: The dataset contains 8 training sequences of 225 frames and test sequences sampled from porcine procedures using a da Vinci Xi system.Images are high-resolution stereo frames, with labels provided for left-camera images.
- Dataset: The tasks include binary instrument-background segmentation and multi-class segmentation by instrument parts or by seven distinct instruments.Instrument-part classes are rigid shaft, articulated wrist, and claspers.
- Architectures: The study evaluates U-Net, two TernausNet variants, and modified LinkNet architectures.
- Architectures: U-Net-like models use contracting and expanding paths with skip-connections to combine contextual and high-resolution features for pixelwise masks.
- Architectures: TernausNet replaces the standard encoder with pre-trained VGG11 or VGG16 networks, while LinkNet-34 uses a pre-trained ResNet34 encoder.
- Training: Training combines Jaccard index optimization with classification loss: binary cross entropy for binary segmentation and categorical cross entropy for multi-class segmentation.
- Training: The model outputs a same-sized image of per-pixel class probabilities, with binary predictions thresholded at 0.3.
3 Results
TernausNet-16 achieved the strongest binary and multi-class part-segmentation results, while LinkNet-34 provided the fastest inference. Seven-class instrument segmentation performed less well, with the authors attributing this to limited and imbalanced training data.
- Segmentation performance: TernausNet-16 achieved the best binary segmentation results, with IoU=0.836 and Dice=0.901.These values were reported as the best in the literature at the time.
- Segmentation performance: TernausNet-16 also achieved the best multi-class segmentation of instrument parts, with IoU=0.655 and Dice=0.760.
- Segmentation performance: IoU=0.346 and Dice=0.459 were achieved by TernausNet-11 for seven-class instrument segmentation.The authors describe this task's results as less optimistic.
- Segmentation performance: The authors suggest that seven-class performance could improve with a larger dataset because several classes appear only a few times during training.
- Inference efficiency: LinkNet-34 was the fastest model, taking around 90 ms for a 1280 × 1024 image in binary segmentation.Inference time was measured using one NVIDIA GTX 1080Ti GPU and was more than twice as fast as TernausNet.
- Overall comparison: The approach demonstrated state-of-the-art performance compared with other deep learning solutions in the MICCAI 2017 subchallenge.
4 Conclusions
The paper presents an end-to-end deep-learning pipeline for binary and multi-class robotic instrument segmentation, comparing improved U-Net-based architectures including LinkNet and TernausNet. The authors report competitive performance and propose the methods as a foundation for surgical instrument position detection supporting tracking and pose estimation.
- Conclusions: The solution compares U-Net-based architectures improved with LinkNet and TernausNet for robotic instrument segmentation.
- Conclusions: The networks form an end-to-end pipeline that analyzes the whole image resolution efficiently.
- Conclusions: The approach achieved competitive performance for both binary and multi-class robotic instrument segmentation.
- Conclusions: The authors propose the methods as a foundation for real-time surgical instrument position detection, tracking, and pose estimation.