Source-linked AI summary
2018 Robotic Scene Segmentation Challenge
Max Allan, Satoshi Kondo, Sebastian Bodenstedt, Stefan Leger, Rahim Kadkhodamohammadi, Imanol Luengo, Felix Fuentes, Evangello Flouty, Ahmed Mohammed, Marius Pedersen, Avinash Kori, Varghese Alex, Ganapathy Krishnamurthi, David Rauber, Robert Mendel, Christoph Palm, Sophia Bano, Guinther Saibro, Chi-Sheng Shih, Hsun-An Chiang, Juntang Zhuang, Junlin Yang, Vladimir Iglovikov, Anton Dobrenkii, Madhu Reddiboina, Anubhav Reddy, Xingtong Liu, Cong Gao, Mathias Unberath, Myeonghyeon Kim, Chanho Kim, Chaewon Kim, Hyejin Kim, Gyeongmin Lee, Ihsan Ullah, Miguel Luna, Sang Hyun Park, Mahdi Azizian, Danail Stoyanov, Lena Maier-Hein, Stefanie Speidel
TL;DR
High-quality labeled data remains limited for surgical CNN segmentation, motivating an expanded robotic challenge with anatomical and medical-device classes. The challenge retains porcine data and evaluates methods across varied test sequences, including a 0.674 average score on test dataset 1.
Problem
High-quality labeled data remains limited for training and evaluating deep CNNs in medical surgery.
Method
The 2018 challenge expanded robotic instrument segmentation to anatomical objects and medical devices while retaining porcine surgical data.
Results
Test dataset 1 produced an average score of 0.674, with three teams exceeding 0.9 IoU for the kidney class.
Takeaways & Limitations
The challenge provides a broader segmentation setting in which methods must handle both device classes and anatomically ambiguous tissue.
Takeaways & Limitations
The dataset remains limited to porcine procedures, whose anatomy is simpler than human tissue because less fatty tissue occludes organs.
Abstract
from arXiv · showhide
In 2015 we began a sub-challenge at the EndoVis workshop at MICCAI in Munich using endoscope images of ex-vivo tissue with automatically generated annotations from robot forward kinematics and instrument CAD models. However, the limited background variation and simple motion rendered the dataset uninformative in learning about which techniques would be suitable for segmentation in real surgery. In 2017, at the same workshop in Quebec we introduced the robotic instrument segmentation dataset with 10 teams participating in the challenge to perform binary, articulating parts and type segmentation of da Vinci instruments. This challenge included realistic instrument motion and more complex porcine tissue as background and was widely addressed with modifications on U-Nets and other popular CNN architectures. In 2018 we added to the complexity by introducing a set of anatomical objects and medical devices to the segmented classes. To avoid over-complicating the challenge, we continued with porcine data which is dramatically simpler than human tissue due to the lack of fatty tissue occluding many organs.
I. INTRODUCTION
The challenge addresses the need for pixel-wise surgical image segmentation by expanding prior robotic instrument segmentation to include more classes without overcomplicating the problem.
- Robot-assisted MIS improves surgical precision and control through dexterous articulated instruments and high-fidelity 3D vision.
- Fusing medical imaging with the endoscopic view requires identifying which objects and image regions are currently visible.This understanding can support selective display of information and higher-level recognition of anatomy or surgical tasks.
- Pixel-wise endoscopic segmentation commonly relies on deep CNNs, but these models require large amounts of high-quality labeled data.Limited labeled medical data has constrained comparable performance advances in surgery.
- The 2017 challenge used 10 porcine endoscope datasets for instrument-part and instrument-type segmentation, with 10 teams participating.A modified U-Net won, and many binary and parts-based methods exceeded 0.7 mIoU.
- The 2018 EndoVis sub-challenge expanded the scope to additional classes without overcomplicating the segmentation problem.
II. DATA
The challenge dataset combines medical-device and anatomical segmentation across 19 porcine sequences, with technician annotations and veterinary quality control. Anatomical labeling required complex protocols because tissue coverage, label boundaries, and limited viewpoints create ambiguities.
- Participants segmented medical devices and anatomical classes, while other anatomical objects were grouped into background.Devices included instruments, ultrasound probes, needles, thread, suction-irrigation devices, and clips; anatomy included parenchyma, covered kidney, and small intestine.
- The dataset contained 19 sequences divided into 15 training sets and 4 test sets from porcine procedures recorded on da Vinci X or Xi systems.Sequences were extracted around camera motion or tissue interaction, subsampled to 1 Hz, and reduced to 300 frames.
- Annotations were polygon labels created by trained technicians, with quality control from in-house veterinarians; only the left stereo image was labeled.
- Anatomical annotation required more complex protocols than instrument annotation to resolve ambiguities and maintain consistency.
- The covered-kidney label combines connective tissue and fat over an anatomical structure, but stretched fascia can make its boundary difficult to define consistently.Some views also require examining an extended image sequence because a single frame may not permit identification.
III. PARTICIPATING METHODS
The participating methods include ImageNet-pretrained CNN architectures with augmentation and task-specific training choices. The supplied methods use ResNeXt-101 with squeeze-excitation blocks and a U-Net with a VGG-19 encoder.
- Method 1 used ImageNet-pretrained ResNeXt-101 with squeeze-excitation blocks and no post-processing.Training used half-resolution images, weighted cross entropy, stochastic gradient descent, cosine annealing, and geometric and contrast augmentations.
- Method 2 used a U-Net-style architecture with an ImageNet-pretrained VGG-19 encoder.Training augmentation included hue, saturation, brightness, and contrast jitter.
C. Digital Surgery
Digital Surgery’s method used DeepLab V3+ to predict both the original and merged class sets. Its design combines multi-scale features with boundary-aware loss and reports 1 FPS inference on a GeForce 1080 Ti.
- Digital Surgery based its model on DeepLab V3+ with multi-scale Xception features, atrous convolution, and deconvolution layers.
- The network predicted scoremaps for the 10 dataset classes and 5 additional sets formed by merging semantically related classes.
- The method optimized a loss combining pixelwise classification with a boundary-distance weighting term.The weighting uses boundary pixels, Euclidean distance, and the maximum batch distance as defined in the supplied method description.
- 1 FPS was reported for inference on an NVIDIA GeForce 1080 Ti.
D. Norwegian University of Science and Technology
The Norwegian team used StreoScenNet, a stereo encoder–decoder segmentation network with a novel sum-skip connection and ImageNet pretraining. Their approach takes paired left-right frames and outputs a mask for the left frame without pre- or post-processing.
- StreoScenNet uses two ResNet encoder blocks and stacked convolutional decoder layers connected by a novel sum-skip connection.
- ImageNet pretraining of one encoder was intended to prevent performance loss from domain shift.
- The network receives left and right frames through separate encoders and predicts a single segmentation mask for the left frame.
- The model is trained end-to-end and performs segmentation without pre-processing or post-processing.
- The challenge methods included weighted-loss, dense-network, and DeepLab V3+ approaches alongside the Norwegian submission.
G. University College London (UCL)
The UCL submission used a global convolutional network with a pretrained ResNet-152 backbone for simultaneous classification and localization. Other described submissions used class-specific U-Nets or DeepLab V3+ with augmentation and transfer learning.
- The UCL method used a global convolutional network with an 11 × 11 kernel and a pretrained ResNet-152 backbone.
- Its network performed classification and localization simultaneously, using semantic score maps generated by symmetric and separable large filters.
- A boundary-refinement component improved localization performance near object boundaries.
- The Dresden submission trained one U-Net with a VGG-16 encoder for each segmentation class and pretrained instrument classes on EndoVis 2017 data.
- The IRCAD submission also used DeepLab V3+, applying affine transformations, noise, and color transformations with 20 augmentations per image.
J. National Taiwan University
The National Taiwan University section describes grouped semantic segmentation strategies alongside several U-Net-based submissions. Methods included hierarchical class grouping, multi-model superclass prediction, ensembles, and varied loss functions.
- The National Taiwan University method used PSPNet and hierarchically grouped classes by semantic similarity before subdividing them into true labels.
- Its grouping first separated artificial from anatomical objects, then divided them into instrument, other-artificial, kidney, small-intestine, and other-anatomical groups.
- The Yale submission used a modified U-Net with a ResNet-101 backbone, focal loss, weighted cross entropy, and majority voting over four models.
- The Open Data Science submission combined a WideResNet-38 encoder with InPlace activated batch normalization and a DeepLab V3 decoder.
- Rediminds trained three U-Net models for progressively different class groupings, using cross entropy, Jaccard, and Dice losses.
N. Johns Hopkins University (JHU)
The Johns Hopkins submission used a Pix2Pix segmentation model with a U-Net generator, combining supervised, reconstruction, and adversarial losses. Other submissions in this group used a conventional U-Net or supplied no additional method details.
- Johns Hopkins trained a Pix2Pix model for segmentation with a U-Net generator.
- Its training combined class-weighted cross entropy, L1, and adversarial losses, with pretraining as a recolorization network.
- The DGIST submission used a five-level U-Net with two 3 × 3 convolutional layers, ReLU activation, and one max-pooling layer per level.
- DGIST trained with categorical cross entropy and Adam using a learning rate of 0.001/(1+0.002×epoch).
- The NUS submission provided no additional details about its method.
Q. Team Banana
Team Banana did not provide team information or submission details. The evaluation used mean intersection over union, averaged across present classes and frames, with overall scores weighted by dataset size.
- Team Banana provided no team information or submission details.
- Mean IoU evaluates segmentation by averaging class IoU values across classes present in each frame, then averaging across frames.Frames with none of the considered classes are excluded.
- Overall scores weight each per-dataset score by dataset size.
B. Test Dataset 2
Test dataset 2 focuses on kidney exposure as fascia and perirenal fat are removed. Parenchyma was segmented substantially more accurately than the covered-kidney class, although the two were often confused with background.
- Test dataset 2 follows a close-up kidney sequence while fascia and perirenal fat are gradually removed with two instruments.
- Kidney parenchyma achieved more than twice the average mean IoU of the covered-kidney label.
- Qualitative results show frequent confusion between the covered-kidney class and background.The passage reports that this class is often mistaken for background and vice versa.
- Test dataset 3 used a close-up exposed-parenchyma sequence with large needle drivers, but scores were mostly poor.The passage attributes this likely to the kidney surface being heavily covered.
D. Test Dataset 4
Test dataset 4 evaluates segmentation during kidney imaging with fascia, fat, intestine, instruments, and an ultrasound probe, followed by liver imaging. Kidney-class results were the worst across the test datasets because most of the kidney surface was covered.
- Test Dataset 4: Test dataset 4 begins with a fascia- and fat-occluded kidney alongside intestine, then uses Maryland Bipolar Forceps, Prograsp Forceps, and a drop-in ultrasound probe.
- Numerical Results: Table I reports test-dataset-1 numerical results, with the highest-scoring method identified in bold and class wins distributed across OTH Regensburg, NCT, and IRCAD.
- Qualitative Results: Figures 6–8 compare ground-truth images with submissions from randomly chosen teams across selected frames from test datasets 1–3.
- Test Dataset 4: The procedure then moves to the liver, where the ultrasound probe is used again; liver is categorized as background.
- Test Dataset 4: Kidney-class results were the worst across all test datasets, consistent with most of the kidney surface being covered.