Source-linked AI summary
ISINet: An Instance-Based Approach for Surgical Instrument Segmentation
Cristina González, Laura Bravo-Sánchez, Pablo Arbelaez
TL;DR
Surgical-instrument segmentation supports computer-assisted surgery, while pixel-wise methods face label-consistency challenges within instruments and across frames. ISINet combines instance-based segmentation with temporal consistency and outperforms state-of-the-art methods on the evaluated datasets.
Problem
Pixel-wise methods predict each pixel independently, creating a label-consistency challenge within instruments and across frames.
Source: §1 · p. 3Method
ISINet identifies instrument candidates, assigns a category to each complete instance, and uses predictions across consecutive frames for temporal consistency.
Source: §1 · pp. 3–4Results
ISINet duplicates or triplicates the performance of state-of-the-art pixel-wise methods across the evaluated settings.
Source: §1 · p. 4Takeaways & Limitations
The authors report that more training data improves generalization on both datasets and that temporal consistency better preserves object identity across time.
Source: §4 · p. 9The 2018 dataset’s general instrument category cannot be used for the 2017 fine-grained instrument-segmentation task.
Source: §1 · p. 3
Abstract
from arXiv · showhide
We study the task of semantic segmentation of surgical instruments in robotic-assisted surgery scenes. We propose the Instance-based Surgical Instrument Segmentation Network (ISINet), a method that addresses this task from an instance-based segmentation perspective. Our method includes a temporal consistency module that takes into account the previously overlooked and inherent temporal information of the problem. We validate our approach on the existing benchmark for the task, the Endoscopic Vision 2017 Robotic Instrument Segmentation Dataset, and on the 2018 version of the dataset, whose annotations we extended for the fine-grained version of instrument segmentation. Our results show that ISINet significantly outperforms state-of-the-art methods, with our baseline version duplicating the Intersection over Union (IoU) of previous methods and our complete model triplicating the IoU.
1 Introduction
Surgical-instrument segmentation supports several computer-assisted surgery tasks, but pixel-wise methods can assign inconsistent instrument labels across an object and over time. ISINet addresses this with instance-based segmentation and temporal consistency, while the authors extend the 2018 annotations for fine-grained evaluation.
1 Introduction
- Instrument segmentation is used as an intermediate task for instrument tracking, pose estimation, and surgical-phase estimation.These applications range from operating-room optimization to personalization and preoperative planning.Source: §1 · p. 1
- The 2015 benchmark lacked instrument-type labeling and had annotation misalignment and limited background variation; the 2017 benchmark still had limited data and sparse temporal sampling.The 2017 data comprises 10 sequences, with eight for training and two for testing.Source: §1 · p. 2
- The 2018 dataset includes greater surgical-scene variability, but its general instrument category cannot support the 2017 fine-grained instrument-type task.The 2018 images come from surgical procedures and include variable backgrounds, movements, angles, and scales.Source: §1 · p. 3
- Prior pixel-wise methods can assign multiple instrument types within one object, while frame-by-frame predictions make label consistency through time difficult.Existing temporal approaches include a prior based on optical flow; ISINet instead uses redundancy across frames to correct mislabeled instruments.Source: §1 · p. 3
- ISINet identifies instrument candidates, assigns one category to each complete instrument, and uses consecutive-frame predictions to produce temporally consistent instance labels.The network builds on Mask R-CNN and adds a temporal consistency module.Source: §1 · pp. 3–4
- To address fine-grained annotation scarcity, the authors add 15 EndoVis 2018 sequences of 149 frames each, with instrument types and temporally consistent instance labels.The annotations follow the 2017 protocol and were created with specialist assistance.Source: §1 · p. 4
- ISINet duplicates or triplicates the performance of state-of-the-art pixel-wise methods across the evaluated settings.The evaluation includes both datasets, with and without additional training data.Source: §1 · p. 4
2 ISINet
ISINet adapts instance segmentation to surgical instruments and adds temporal matching and class assignment across frames. It uses optical flow to align prior detections before updating each instance’s class prediction.
2 ISINet
- ISINet adapts Mask R-CNN to fine-grained instrument segmentation and promotes temporal consistency in per-instance class predictions across consecutive frames.Its prediction layer is modified to match the number of classes in the EndoVis datasets.Source: §2 · p. 4
- Mask R-CNN supplies candidate objects, scores, and class predictions for each frame, retaining predictions with confidence above 0.75.Source: §2 · p. 5
- Backward optical flow from FlowNet2 warps candidates from the previous f frames into the current frame for matching.Candidates are iteratively warped from one frame to the next.Source: §2 · p. 5
- Reciprocal pairings above an IoU threshold link warped candidates into instances, which need not appear in every frame.Source: §2 · p. 5
- For each current-frame instance, assignment combines its classes and scores across the preceding f frames to update its class prediction.The final method uses f=6; its score-weighted class mode uses U=0 for 2017 and U=0.5 for 2018.Source: §2 · pp. 5–6
3 Experiments
ISINet’s baseline outperforms TernausNet and MF-TAPNet on overall IoU metrics in both datasets. The experiments also extend EndoVis 2018 with instrument-type annotations and assess dataset combination and temporal consistency.
3.1 Additional Annotations for EndoVis 2018
- EndoVis 2018 was manually extended with instrument types, yielding nine identified types; Stapler was excluded from evaluation because examples were limited.Annotations cover 15 sequences of 149 frames and maintain instance labels consistently throughout each sequence.Source: §3.1 · p. 6
3.2 Experimental Setup
- The experiments use EndoVis 2017 and 2018 robotic-surgery datasets; 2017 evaluation uses four-fold cross-validation and three increasingly stringent IoU metrics.The table compares ISINet with state-of-the-art methods and reports per-class IoU, mean IoU across classes, challenge IoU, and IoU; D and T denote additional data and temporal consistency.Source: §3.2 · pp. 6–7, Table 1 · p. 7
3.3 Experimental Validation
- The baseline outperforms TernausNet and MF-TAPNet on all overall IoU metrics in both datasets, duplicating or triplicating state-of-the-art performance depending on the metric.For some instrument classes, improvement exceeds 30.0 IoU on both datasets.Source: Table 1 · p. 7, §3.3 · p. 8
- Training with additional data improves performance on both datasets; 6 of 7 classes improve on 2017 and 4 of 7 on 2018, with some gains up to 20.0 IoU percentage points.The comparison trains with all available examples but predicts only classes present in each dataset.Source: §3.3 · p. 8
- The temporal module improves overall metrics with and without additional data on both datasets, correcting outlier predictions in all classes except Ultrasound Probe.Some 2017 classes increase nearly 8 percentage points; the authors hypothesize the gain is smaller on 2018 because of greater variability.Source: §3.3 · p. 8
4 Conclusions
ISINet approaches surgical instrument segmentation through instance-based segmentation and temporal consistency. The authors report improved results on EndoVis 2017 and 2018, extend annotations for instrument types, and find benefits from additional training data and temporal consistency.
4 Conclusions
- The method combines instance-based segmentation with a temporal consistency module that considers predictions across frames.The authors report that the module better preserves object identity over time.Source: §4 · pp. 8–9
- ISINet outperforms state-of-the-art semantic segmentation methods on EndoVis 2017 and 2018.The authors also report that more training data improves generalization on both datasets.Source: §4 · p. 9
- The authors manually extend EndoVis 2018 annotations with instrument types for the fine-grained segmentation task.Source: §4 · p. 9
- The authors plan to provide code, pretrained models for both datasets, and the EndoVis 2018 instrument-type annotations.Source: §4 · p. 9
Supplementary Material for ISINet
The supplementary material presents ablations of ISINet’s temporal consistency module and qualitative comparisons on EndoVis 2017 and 2018. The captions identify both the module parameters and representative strengths and error modes.
Supplementary Material for ISINet
- Most tested temporal-consistency parameters improve baseline ISINet, with selected settings chosen for gains on both datasets.The ablations cover group 1 and 2 splits and vary threshold, number of frames, and class-selection method.Source: Table 1 · p. 12
- ISINet’s qualitative comparisons show unique labels for complete instances, recovered overlooked instrument pixels, and no false-positive instruments.The comparisons are against TernausNet and MF-TAPNet on EndoVis 2017 and 2018.Source: Figure 1 · p. 13
- The reported common ISINet error modes are missing instruments, mislabeling instruments, and coarse segmentations.Source: Figure 2 · p. 13