PaperlayerView exact revision on arXiv

Source-linked AI summary

ToolNet: Holistically-Nested Real-Time Segmentation of Robotic Surgical Tools

Luis C. Garcia-Peraza-Herrera, Wenqi Li, Lucas Fidon, Caspar Gruijthuijsen, Alain Devreker, George Attilakos, Jan Deprest, Emmanuel Vander Poorten, Danail Stoyanov, Tom Vercauteren, Sebastien Ourselin

arXiv:1706.08126v2cs.CV

TL;DR

Robotic tool segmentation is a building block for visual-servo control, but difficult endoscopic scenes and constraints on fiducial coatings challenge its use. The paper proposes lightweight CNN architectures with Dice loss and PReLU, and reports improved test-set metrics over FCN-8s alongside real-time performance.

  • Problem

    Robotic tool segmentation is a building block for visual-servo control, but heterogeneous endoscopic scenes complicate segmentation and fiducial coatings constrain workflow and reuse of recorded data.

    Source: §I · p. 2

  • Method

    The paper presents two lightweight end-to-end CNN architectures for real-time tool segmentation, using Dice loss and PReLU.

    Source: B. Contributions · p. 2, B. Dice loss · p. 2, C. Parametric Rectified Linear Unit… · p. 3

  • Results

    ToolNetMS improved balanced accuracy (foreground) by 1.4 percentage points, mean IoU by 1.6 percentage points, and mean DSC by 1.6 percentage points on the testing set; the authors also report generalization to unseen instruments and recording conditions.

    Source: C. Quantitative & qualitative… · p. 5, §IV · p. 6

  • Takeaways & Limitations

    The authors report up to 15 times fewer parameters than widely used FCNs, enabling real-time performance and easier distribution to end-users and embedded-device deployment.

    Source: §IV · p. 6

    Further architecture optimization, data augmentation, transfer learning, and adversarial training are identified as future work.

    Source: §IV · p. 6

Abstract

from arXiv · show

Real-time tool segmentation from endoscopic videos is an essential part of many computer-assisted robotic surgical systems and of critical importance in robotic surgical data science. We propose two novel deep learning architectures for automatic segmentation of non-rigid surgical instruments. Both methods take advantage of automated deep-learning-based multi-scale feature extraction while trying to maintain an accurate segmentation quality at all resolutions. The two proposed methods encode the multi-scale constraint inside the network architecture. The first proposed architecture enforces it by cascaded aggregation of predictions and the second proposed network does it by means of a holistically-nested architecture where the loss at each scale is taken into account for the optimization process. As the proposed methods are for real-time semantic labeling, both present a reduced number of parameters. We propose the use of parametric rectified linear units for semantic labeling in these small architectures to increase the regularization ability of the design and maintain the segmentation accuracy without overfitting the training sets. We compare the proposed architectures against state-of-the-art fully convolutional networks. We validate our methods using existing benchmark datasets, including ex vivo cases with phantom tissue and different robotic surgical instruments present in the scene. Our results show a statistically significant improved Dice Similarity Coefficient over previous instrument segmentation methods. We analyze our design choices and discuss the key drivers for improving accuracy.

I. INTRODUCTION

Robotic tool segmentation can support haptics, virtual fixtures, and visual-servo control, but complex instruments and challenging endoscopic scenes make reliable segmentation difficult. Fiducial coatings are presented as a poor fit for existing workflows and recorded surgical data.

I. INTRODUCTION

  • Minimally invasive robotic surgery enables dexterous manipulation in reduced spaces, but its master-slave configuration removes the surgeon’s direct tissue contact.Haptic feedback is described as a desirable feature in this setting.Source: §I · p. 1
  • Complex actuation and unstable instrument kinematics make robotic tool control difficult, while adding embedded tracking technologies can be technically challenging or impractical in some settings.The text cites endoscopic fetal surgery, where increasing instrument size or the number of ports is linked to adverse outcomes.Source: §I · pp. 1–2
  • Tool segmentation is a building block for visual-servo control and may support haptics, virtual fixtures, and surgeon context-awareness.The text describes haptic feedback as helping surgeons perform safe manipulations and confirm treatment execution.Source: §I · pp. 1–2
  • Endoscopic scenes challenge segmentation with specular reflections, occlusions, blur, reflections, body fluids, and smoke; fiducial coatings would limit systems to customized tools and complicate reuse of recorded data.The passage also notes potential manufacturing costs and the need for markers to withstand sterilization.Source: §I · p. 2

A. Related work

Prior instrument-segmentation work includes handcrafted-feature classifiers and FCN-based approaches. The paper introduces lightweight end-to-end architectures, tailored losses and activations, and reports better performance than a baseline on a challenging dataset with instruments absent from training.

A. Related work

  • Earlier instrument-background segmentation used handcrafted features with classifiers, while later work applied CNNs and FCNs to detection, phase recognition, and segmentation.A prior real-time instrument-segmentation pipeline combined FCNs with optical-flow tracking.Source: A. Related work · p. 2

B. Contributions

  • The paper presents two lightweight end-to-end CNN architectures for real-time robotic-tool segmentation, with about an order of magnitude fewer parameters than the state of the art.Source: B. Contributions · p. 2
  • The authors propose Dice loss for their data and PReLU as an adaptively learned activation for semantic labeling, reporting improved accuracy at insignificant additional computational cost.Source: B. Contributions · p. 2
  • ToolNetMS aggregates predictions across scales before calculating final Dice loss, while ToolNetH imposes multi-scale consistency through a holistically-nested loss.Source: B. Contributions · p. 2
  • The proposed algorithms outperform the state-of-the-art baseline on a challenging instrument-background dataset containing robotic instruments absent from training.Source: B. Contributions · p. 2

II. METHODS

FCNs preserve spatial configuration for end-to-end pixel labeling, but the described FCN approach has limitations in edge fidelity, real-time performance, and handling class imbalance.

A. Fully Convolutional Network (FCN) architectures

B. Dice loss

The paper uses Dice loss for instrument-background segmentation, motivated by class imbalance and the use of Dice Similarity Coefficient to evaluate segmentations. Its weighting accounts for class pixel prevalence.

B. Dice loss

  • The authors adapt Dice loss to 2D instrument-background segmentation, motivated by class imbalance and the use of Dice Similarity Coefficient to evaluate segmentation quality.The passage contrasts this choice with widely used cross-entropy loss and notes that foreground and background pixel counts can differ by orders of magnitude.Source: B. Dice loss · pp. 2–3
  • The class weight α(ŷ, y, k) adjusts estimation-error contributions by class pixel prevalence so that errors in sparse instrument pixels do not become negligible.Source: B. Dice loss · p. 3

C. Parametric Rectified Linear Unit (PReLU)

PReLU learns negative activation slopes to strengthen regularization in the smaller networks. ToolNetMS and ToolNetH impose multi-scale consistency through different prediction and loss designs.

C. Parametric Rectified Linear Unit (PReLU)

  • PReLU learns negative activation slopes end-to-end, providing a more robust regularization strategy for smaller networks and training sets.The network uses PReLU in encoder layers with rectified units; its shared negative slope adds an insignificant number of parameters while improving accuracy.Source: C. Parametric Rectified Linear Unit… · p. 3

D. Proposed multi-scale and holistically-nested networks

  • ToolNetMS and ToolNetH are proposed as real-time segmentation architectures that improve accuracy and speed over FCN-8s.The paper describes the networks as having one order of magnitude fewer parameters than the state of the art, reducing memory requirements and allowing real-time inference.Source: D. Proposed multi-scale and… · p. 3
  • ToolNetMS sums predictions across scales in a cascade and uses Dice loss to sharpen tool–tissue boundaries while addressing class imbalance.The reduced parameter count targets inference at robotic-surgery video rates, typically 25 fps.Source: D. Proposed multi-scale and… · p. 3
  • ToolNetH replaces ToolNetMS’s cascaded sum with Multi-Scale Dice Loss, which accounts for prediction accuracy at multiple scales.Its scale-specific probabilistic predictions contribute to the combined loss, with weights learned for the prediction sum and loss weights set to 1 in the implementation.Source: D. Proposed multi-scale and… · pp. 3–4

E. Network training details

Training uses a triangular cyclical learning-rate policy with validation-tuned bounds and a fixed iteration budget. The implementation also specifies momentum, weight decay, batch size, and dropout.

E. Network training details

  • The cyclical learning-rate policy oscillates between two bounds; the authors use a triangular waveform for simplicity.The policy is reported to reach convergence with higher accuracy in fewer optimization steps than fixed or exponential policies [24].Source: E. Network training details · p. 4
  • The triangular policy uses a stepsize of 2 × epoch, bounds selected experimentally during validation, and a total of 6 × stepsize training iterations.The stepsize and iteration budget are described as values shown to work well in practice [24].Source: E. Network training details · p. 4
  • Training uses momentum 0.99, weight decay 0.0005, batch size 1, and dropout probability 0.5 only at the smallest-scale layer.The architecture is implemented in Caffe.Source: E. Network training details · p. 4

A. Datasets

The dataset comprises ex vivo robotic-surgery scenarios and articulated instruments. The Da Vinci Robotic dataset includes training and test videos, with some test instruments absent from training.

A. Datasets

  • The dataset was assembled to test robustness and generalization across ex vivo robotic-surgery scenarios and articulated instruments.It contains training and testing data.Source: A. Datasets · p. 4
  • Four 45-second videos form the DVR training set, featuring one or two da Vinci Large Needle Drivers.Ground-truth masks were generated automatically from joint encoder information and forward kinematics, with hand-eye calibration errors corrected manually.Source: 1) Da Vinci Robotic (DVR) dataset · p. 4
  • The DVR test set has four 15-second and two 60-second videos; one test video includes two LNDs, and two include an LND and curved scissors absent from training.All videos include articulated motions and were recorded at 720×576 resolution and 25 frames per second.Source: 1) Da Vinci Robotic (DVR) dataset · p. 4

B. Baseline method and evaluation protocol

FCN-8s serves as the state-of-the-art baseline, and evaluation uses mean IoU and mean DSC. Training omits augmentation and transfer learning to focus on comparing the proposed architectures with that baseline.

B. Baseline method and evaluation protocol

  • The fine-grained FCN-8s is the comparison baseline, selected as a state-of-the-art convolutional architecture for robotic surgical tool segmentation.Source: B. Baseline method and evaluation… · p. 4
  • Frames are randomly selected during training, with no data augmentation or transfer learning; CLR bounds are tuned using the fourth DVR training video for validation.FCN-8s uses bounds [1e −10, 1e −8], while ToolNetMS and ToolNetH use [1e −7, 1e −5].Source: B. Baseline method and evaluation… · p. 4
  • Mean IoU and mean DSC are the selected metrics for evaluating predicted segmentations.The metrics are averaged over two classes: background and foreground; TP, FP, and FN denote true positives, false positives, and false negatives for each class.Source: B. Baseline method and evaluation… · p. 4
  • Training, testing, and inference-time measurements use an Intel Xeon E5-1650 3.50GHz CPU and NVIDIA GeForce GTX Titan X GPU.Inference time includes CPU-to-GPU and return transfers and is averaged across 500 inferences.Source: B. Baseline method and evaluation… · p. 4

C. Quantitative & qualitative comparison to state-of-the-art

Both proposed architectures outperform FCN-8s on reported segmentation metrics; qualitative comparisons also show more accurate tool boundaries and greater spatial consistency.

C. Quantitative & qualitative comparison to state-of-the-art

  • ToolNetMS improves over FCN-8s by 1.4 percentage points in balanced accuracy (foreground), 1.6 in mean IoU, and 1.6 in mean DSC.Its model size is 7.3M parameters versus approximately 134M for FCN-8s, and inference time is 23.2ms versus 123.5ms.Source: C. Quantitative & qualitative… · pp. 4–5
  • ToolNetH improves over FCN-8s by 3.8 percentage points in balanced accuracy (foreground), 3.5 in mean IoU, and 3.4 in mean DSC.It runs at approximately 29fps with 7.4M parameters; Table I reports results averaged across testing frames and classes.Source: C. Quantitative & qualitative… · p. 5, Table I · p. 5
  • Qualitative results show more accurate tool-border delineation and enhanced spatial consistency; the left-frame instrument was absent from the DVR training set.The figure compares the proposed architectures with FCN-8s.Source: C. Quantitative & qualitative… · p. 5, Figure 3 · p. 5

IV. DISCUSSION AND CONCLUSIONS

The discussion presents compact, real-time multi-scale CNNs for surgical-tool segmentation and reports generalization to unseen instruments and recording conditions. It highlights intermediate-layer feature fusion and deep supervision, while identifying optimization and 3D localization as future work.

IV. DISCUSSION AND CONCLUSIONS

  • The paper proposes multi-scale and holistically-nested CNNs trained with Dice loss for real-time robotic-tool segmentation at approximately 29Hz.The discussion presents these as the paper's proposed architectures.Source: §IV · p. 5
  • The methods address FCN-8s upsampling and real-time limitations, edge accuracy needed for shape-based control, and foreground-background pixel imbalance.The authors state these considerations underpin their competitive advantage over the state of the art.Source: §IV · p. 5
  • Fusing convolutional feature maps from intermediate CNN layers is suggested to help segmentation, while deep supervision at those layers further improves performance.The discussion says test-data results suggest these effects.Source: §IV · p. 6
  • The trained model is reported to generalize to unseen robotic instruments and video recording conditions, with up to 15 times fewer parameters than widely used FCNs.The authors connect the parameter reduction to real-time performance and deployment on embedded devices.Source: §IV · p. 6
  • Future work includes improving performance within the computational budget and studying data augmentation, transfer learning, and adversarial training.Adversarial training is described as a possible way to further increase segmentation accuracy.Source: §IV · p. 6
  • Three-dimensional instrument localization remains future work, using stereo-image matches for dense triangulation when cameras are calibrated.The passage conditions this approach on stereo endoscopes becoming more widespread.Source: §IV · p. 6
Loading 1706.08126v2…