Source-linked AI summary
2017 Robotic Instrument Segmentation Challenge
Max Allan, Alex Shvets, Thomas Kurmann, Zichen Zhang, Rahul Duggal, Yun-Hsuan Su, Nicola Rieke, Iro Laina, Niveditha Kalavakonda, Sebastian Bodenstedt, Luis Herrera, Wenqi Li, Vladimir Iglovikov, Huoling Luo, Jian Yang, Danail Stoyanov, Lena Maier-Hein, Stefanie Speidel, Mahdi Azizian
TL;DR
Robotic surgery lacked shared datasets and validation methods for comparing instrument-segmentation algorithms. The paper presents a 2017 challenge using manually labelled, more varied data across binary, parts, and type segmentation, with results showing different teams led on different evaluation aspects and dataset subsets.
Problem
Robotic assisted surgery lacked common datasets and benchmarking methods, while the prior dataset had limited background variation and inaccurate automatic annotations.
Method
The 2017 challenge used manually segmented porcine nephrectomy images from 10 procedures and evaluated binary, instrument-part, and instrument-type segmentation.
Results
TUM achieved the highest overall accuracy with mean IoU 0.751, while MIT achieved the highest mean IoU in 7 of 10 datasets.
Takeaways & Limitations
The challenge provides a benchmark spanning binary, parts, and type-based segmentation of articulated da Vinci instruments.
Takeaways & Limitations
The dataset’s relatively small size and sampling from few procedures could bias rankings toward models that overfit without properly testing generalization.
Abstract
from arXiv · showhide
In mainstream computer vision and machine learning, public datasets such as ImageNet, COCO and KITTI have helped drive enormous improvements by enabling researchers to understand the strengths and limitations of different algorithms via performance comparison. However, this type of approach has had limited translation to problems in robotic assisted surgery as this field has never established the same level of common datasets and benchmarking methods. In 2015 a sub-challenge was introduced at the EndoVis workshop where a set of robotic images were provided with automatically generated annotations from robot forward kinematics. However, there were issues with this dataset due to the limited background variation, lack of complex motion and inaccuracies in the annotation. In this work we present the results of the 2017 challenge on robotic instrument segmentation which involved 10 teams participating in binary, parts and type based segmentation of articulated da Vinci robotic instruments.
I. INTRODUCTION
The challenge addressed limited shared datasets and validation methods for robotic instrument segmentation by introducing manually labelled, more varied data and three segmentation tasks.
- I. INTRODUCTION: Robotic instrument segmentation supports scene understanding for instrument tracking and augmented-reality overlays that must avoid occluding instruments.Accurate pixel labelling is presented as an important component of these assistive applications.
- I. INTRODUCTION: The field lacked common data and validation sets for comparing methods and developing larger networks.The 2015 dataset also had limited background variability and annotation misalignments from forward-kinematics labelling.
- I. INTRODUCTION: The 2017 follow-up used hand-created labels from porcine nephrectomy procedures, 10 separate procedures, and more instrument type and part labels.These changes targeted label quality, background variation, and task coverage.
- I. INTRODUCTION: The challenge comprised binary instrument-versus-background segmentation, instrument-part segmentation, and instrument type segmentation.The background included an ultrasound probe, surgical clips, and porcine tissues in the binary task.
- I. INTRODUCTION: The dataset used 10 da Vinci Xi porcine-procedure sequences, sampled active motion and visibility, and supplied stereo images with calibration information.Eight sequences contributed training and test frames, while two full sequences were held out for testing.
C. Data labelling
The challenge labels were created manually at the instance level, while one submitted method used image preprocessing and augmentation before training.
- C. Data labelling: Intuitive Surgical’s dedicated segmentation team created polygon labels frame by frame with the open-source Viame software.Labelling was performed only on the left image of each stereo pair to reduce labelling time.
- C. Data labelling: Labels were provided separately for each object at the instance level.
- C. Data labelling: The NCT method cropped black borders, downsampled images, and applied pixel, flip, zoom, and rotation augmentations before training.
B. Universty of Bern
Teams used varied encoder-decoder and convolutional architectures for the challenge, including cascaded FCNs, U-Net ensembles, pretrained U-Net variants, and SegNet.
- B. Universty of Bern: The University of Bern method used a cascaded FCN whose first stage produced binary masks reused by part and type segmentation.Its encoder-decoder architecture used skip connections, residual computation, pooling, and transposed convolutions.
- B. Universty of Bern: The Beijing Institute of Technology trained an ensemble of three separate U-Nets and fused their outputs into a final prediction.
- B. Universty of Bern: TernausNet improved on vanilla U-Net by using pretrained VGG11 or VGG16 encoders and produced pixel-by-pixel class masks.The decoder connected to corresponding encoder blocks through concatenated skip features.
- B. Universty of Bern: SIAT used a symmetric SegNet encoder-decoder with an ImageNet-pretrained VGG16 and fine-tuned models for binary and multi-label segmentation.The decoder output was passed into a softmax classification layer.
- B. Universty of Bern: SIAT preserved aspect ratio through padding but resized images to 480 × 270 because of hardware limitations.
F. University College London
The UCL team used ToolNet, a custom multiscale instrument-segmentation network inspired by holistically nested edge detection, with real-time inference and post-processing to reduce noise.
- F. University College London: ToolNet adapts holistically-nested edge detection for instrument segmentation and uses an aggregated multiscale loss.The challenge replaced the original Dice loss with an intersection-over-union-based loss.
- F. University College London: ToolNet generates predictions at multiple scales and learns weights to combine them into an averaged probabilistic prediction.The network produces predictions at M = 6 scales, with weighting parameters initialized equally.
- F. University College London: The method uses an encoder-decoder design based on FCRN with ResNet-50 encoding, residual up-sampling, and long-range skip connections.Skip connections help gradients bypass part of the network and flow from encoding to decoding layers.
- F. University College London: Training adds standard photometric and geometric augmentations plus synthetic specularities along instrument shafts.The application-specific augmentation targets specular reflections that can cause misclassification.
- F. University College London: Post-processing removes predicted instrument components disconnected from image borders and applies morphological operations to fill holes and smooth masks.The border prior reflects that instruments enter the recorded scene from an image border.
- F. University College London: The resulting method runs with near real-time performance on an NVIDIA GeForce GTX TITAN X.
H. IIIT Delhi
The IIIT Delhi team tested a deliberately simple baseline: a patch-based CNN foreground detector followed by CRF post-processing.
- H. IIIT Delhi: The approach was designed to assess how well a simplistic baseline performs.
- H. IIIT Delhi: A VGG-19 CNN classified 51×51 image patches as instrument-containing foreground or background.Patches containing any portion of an instrument received label 1; other patches received label 0.
- H. IIIT Delhi: A Conditional Random Field was applied after CNN foreground detection as post-processing.
I. University of Alberta
The University of Alberta used a compact FCN-8s architecture for binary, part, and type segmentation, adapting training across the challenge videos and tasks.
- I. University of Alberta: The method repurposed FCN-8s for all three segmentation tasks and selected a smaller architecture than alternatives such as ResNet-101.The smaller network was motivated by the limited number of training frames per video.
- I. University of Alberta: For datasets 1–8, separate models were trained using seven videos and validated on the held-out target video.Datasets 9 and 10 were entirely test videos, so no training data were excluded for them.
- I. University of Alberta: The team reused the model trained for dataset 7 for datasets 9 and 10 because of time constraints.Dataset 7 was selected because it contained data rarely seen in the other datasets.
- I. University of Alberta: PASCAL-pretrained weights initialized binary segmentation, whose best weights then initialized part and type segmentation.Input images were resized to 320 × 256 for faster training.
- I. University of Alberta: Connected predicted regions smaller than 15000 pixels were removed during post-processing.The threshold was tuned on the training data.
J. University of Washington
The University of Washington explored a non-machine-learning instrument-segmentation pipeline that combines color filtering, GrabCut refinement, weighted image features, and a border prior.
- J. University of Washington: The UW team sought to determine how far traditional computer-vision methods could go without machine learning.The motivation cited the lack of massive pre-labeled surgical image datasets in general settings.
- J. University of Washington: Opponent-color-space and hue/saturation thresholds first generate an instrument mask, which GrabCut refines into tool-versus-tissue predictions.
- J. University of Washington: Shape, edge, border, and disparity-discontinuity features are combined through a weighted sum to estimate whether pixels belong to tools or tissue.
- J. University of Washington: The final probability mask applies a border constraint because instrument contours connect to at least one image edge.This prior removes erroneous isolated regions.
- J. University of Washington: Performance is evaluated using mean intersection-over-union, averaging class IoUs present in each frame and then averaging frame scores per dataset.Overall scores weight dataset scores by dataset size.
- J. University of Washington: The challenge kept test annotations private and allowed each team a single segmentation submission before public results were released.This prevented tuning methods on test data through multiple submissions.
B. Binary Segmentation
The binary segmentation challenge evaluated separating da Vinci instruments from background across datasets, revealing strong variation across methods and difficult cases involving probes, needles, and complex lighting.
- Binary segmentation separated any da Vinci instrument from anatomical or man-made background objects, including visually similar drop-in ultrasound probes and needles.
- Frames from methods averaging above 0.8 mean IoU still differed considerably in distinguishing a drop-in ultrasound probe.
- Dataset 7 produced visibly different method performance because it contained a Vessel Sealer and complex lighting.
- The parts-based challenge additionally required segmenting instrument shafts, wrists, and jaws while treating probes, devices, and anatomy as background.
- Qualitative parts-segmentation outputs showed inconsistent labelling throughout the Vessel Sealer shaft.
D. Type Segmentation
Type-based segmentation required identifying individual da Vinci instruments, making it substantially more difficult than the earlier segmentation tasks. Quantitative and qualitative results show frequent misclassification despite often accurate masks.
- D. Type Segmentation: Type segmentation identified six instrument classes: Large Needle Driver, Prograsp Forceps, Monopolar Curved Scissors, Vessel Sealer, Fenestrated Bipolar Forceps, and Grasping Retractor.
- D. Type Segmentation: Only six teams participated in type segmentation, reflecting the increased difficulty of recognizing many da Vinci instruments from one another.
- D. Type Segmentation: The challenge compared binary, parts-based, and type-based segmentation, with type segmentation adding instrument classification to pixel-wise labeling.
- D. Type Segmentation: The figures present qualitative outputs from selected datasets and randomly selected methods for binary and parts-based segmentation.
- D. Type Segmentation: Type-based segmentation used mean IoU over all classes, with MIT winning 7 datasets, SIAT 1, NCT 1, and UB 1.
- D. Type Segmentation: Qualitative type-segmentation results showed accurate masks for many frames but poorer identification, including partial or complete instrument misclassification.
A. Data Augmentation
The challenge analysis highlights augmentation choices suited to robotic surgery, while identifying dataset, annotation, and computational constraints that affect evaluation and generalization.
- A. Data Augmentation: Data augmentation increases training variability, but techniques can hurt performance when they create scenes absent from the test set.The passages describe augmentation as useful for correlated sequences while cautioning that inappropriate transformations may consume model capacity on irrelevant scenes.
- A. Data Augmentation: Most teams favored spatial over photometric augmentation, especially horizontal and vertical flips matching common robotic instrument orientations.The robot’s physical design commonly produces instruments entering from image sides toward a central point; zoom transformations were used less often despite common surgical image shifts.
- A. Data Augmentation: All highest-performing teams combined photometric and spatial augmentation, while TUM improved segmentation by adding random specular reflections.Specular reflections from directed surgical lighting affect both instruments and tissue, motivating the reflection-specific augmentation.
- B. Challenge Design: The challenge used 3000 frames, with 1800 for training, creating a relatively small dataset compared with standard computer-vision segmentation benchmarks.The authors state that data from few procedures can bias rankings toward models that overfit without properly testing generalization.
- B. Challenge Design: Instrument-type frequencies may not reflect surgical usage because the data came from few procedures; Monopolar Curved Scissors appeared in only 2 training sequences.The authors recommend fewer frames per dataset and more separate procedures in future video segmentation challenges.
- B. Challenge Design: Annotation quality remained constrained by motion blur, difficult lighting or smoke, single-annotator labeling with expert review, and inconsistent handling of instrument holes.The authors note that hole annotations could not be corrected consistently across training and test data before release.
- A. Data Augmentation: The augmentation table records team choices, including blank entries where teams did not augment or where augmentation information was unavailable.UW used a non-deep-learning approach, UA and UCL cited time limitations, and BIT’s augmentation choices were unavailable.
- B. Challenge Design: Submissions required 9 separate models for 10 datasets, and teams reported that the computational burden limited experimentation.The planned remedy was to release entirely separate training and test sets so that only one model would need evaluation.
C. Future Challenges
The authors argue that future surgical-vision challenges should extend beyond instrument segmentation to dense scene annotation, including anatomy and other devices. This could support more context-aware assistance within the paper’s stated scope.
- C. Future Challenges: Dense annotation of anatomy alongside instruments would provide more complete scene segmentation for surgical-vision challenges.The authors specifically identify tissue recognition as a valuable extension beyond robotic-instrument segmentation.
- C. Future Challenges: Recognizing different tissues could support more context-aware assistance to surgeons, motivating the 2018 dataset’s broader annotations.The later dataset included multiple tissue types and devices such as clips, thread, and suction/irrigation tools.