PaperlayerView exact revision on arXiv

Source-linked AI summary

SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation

Wenxi Yue, Jing Zhang, Kun Hu, Yong Xia, Jiebo Luo, Zhiyong Wang

arXiv:2308.08746v2cs.CVcs.AIcs.RO

TL;DR

Surgical instrument segmentation is hindered by SAM’s natural-object domain gap and its dependence on precise explicit prompts. SurgicalSAM efficiently tunes SAM with class-prototype prompts and contrastive prototype learning, achieving state-of-the-art results on EndoVis2018 and EndoVis2017 with few tunable parameters.

  • Problem

    Limited surgical data, a natural-to-surgical domain gap, and precise-prompt requirements limit SAM’s direct application to surgical instrument segmentation.

  • Method

    SurgicalSAM efficiently tunes SAM using a prototype-based class prompt encoder and contrastive prototype learning for discriminative class prompting.

  • Results

    SurgicalSAM achieves state-of-the-art performance on EndoVis2018 and EndoVis2017 while requiring substantially fewer tunable parameters.

  • Takeaways & Limitations

    Class prototypes replace explicit points or boxes, enabling an end-to-end pipeline with improved prompt robustness and training efficiency.

Abstract

from arXiv · show

The Segment Anything Model (SAM) is a powerful foundation model that has revolutionised image segmentation. To apply SAM to surgical instrument segmentation, a common approach is to locate precise points or boxes of instruments and then use them as prompts for SAM in a zero-shot manner. However, we observe two problems with this naive pipeline: (1) the domain gap between natural objects and surgical instruments leads to inferior generalisation of SAM; and (2) SAM relies on precise point or box locations for accurate segmentation, requiring either extensive manual guidance or a well-performing specialist detector for prompt preparation, which leads to a complex multi-stage pipeline. To address these problems, we introduce SurgicalSAM, a novel end-to-end efficient-tuning approach for SAM to effectively integrate surgical-specific information with SAM's pre-trained knowledge for improved generalisation. Specifically, we propose a lightweight prototype-based class prompt encoder for tuning, which directly generates prompt embeddings from class prototypes and eliminates the use of explicit prompts for improved robustness and a simpler pipeline. In addition, to address the low inter-class variance among surgical instrument categories, we propose contrastive prototype learning, further enhancing the discrimination of the class prototypes for more accurate class prompting. The results of extensive experiments on both EndoVis2018 and EndoVis2017 datasets demonstrate that SurgicalSAM achieves state-of-the-art performance while only requiring a small number of tunable parameters. The source code is available at https://github.com/wenxi-yue/SurgicalSAM.

1 Introduction

Surgical instrument segmentation is important but challenging for SAM because surgical images differ from natural-object data and explicit prompts require high precision. SurgicalSAM addresses these issues with efficient tuning, class-based latent prompts, and contrastive prototype learning.

  • Surgical instrument segmentation precisely delineates instruments and supports surgeons and computer-assisted operation systems.
  • Limited surgical data and the natural-to-medical domain gap hinder SAM’s direct zero-shot generalisation to surgical instruments.
  • Surgical instruments’ specialised appearance, complex anatomical backgrounds, and high inter-category similarity challenge zero-shot SAM strategies.
  • Even minor bounding-box prompt jitters can significantly impair SAM segmentation accuracy, motivating robust prompt preparation.
  • SurgicalSAM integrates surgical knowledge into SAM through efficient tuning and directly generates latent prompt embeddings from class prototypes without explicit points or boxes.
  • Contrastive prototype learning discriminates fine-grained instrument categories, enabling more accurate class prompting and segmentation.
  • Experiments on EndoVis2018 and EndoVis2017 report state-of-the-art performance with substantially improved training efficiency.

2 Related Work

Prior surgical instrument segmentation work uses specialist pixel- or mask-classification models, while SAM adaptation methods remain constrained by prompt requirements or weak class discrimination. SurgicalSAM instead targets efficient, class-discriminative adaptation.

  • Early specialist methods use pixel classification to predict frame-level pixel-wise instrument class probabilities.
  • Pixel-classification approaches can produce spatial class inconsistency, assigning multiple instrument types to one instrument.
  • Mask classification predicts sets of masks with class labels, inherently reducing spatial class inconsistency.
  • SAM adaptation studies either require accurate point or box prompts or use universal embeddings lacking discrimination for fine-grained instrument categories.

3 Methodology

SurgicalSAM performs class-promptable surgical instrument segmentation by generating prompt embeddings from class prototypes rather than explicit point or box prompts. Its end-to-end tuning combines a prototype-based prompt encoder, mask decoder, and contrastive prototype learning while freezing the large image encoder.

  • 3.1 Overview: SurgicalSAM uses an image encoder, prototype-based class prompt encoder, and mask decoder to predict the mask for a prompted instrument class.The image encoder produces image embeddings, while the prompt encoder generates dense and sparse prompt embeddings for mask decoding.
  • 3.2 Prototype-based Class Prompt Encoder: Using all class prototypes supplies robust priors for distinguishing surgical instrument categories with high visual similarity.The strategy combines the prompted class prototype with other class prototypes to improve class discrimination.
  • 3.2 Prototype-based Class Prompt Encoder: The prompt encoder compares image embeddings with a prototype bank to activate class-specific regions and form class-conditioned features.Each class prototype is compared spatially with the image embedding, producing similarity maps used as spatial attention.
  • 3.2 Prototype-based Class Prompt Encoder: Dense prompt embeddings use the prompted class, while sparse embeddings incorporate positive information from that class and negative information from other classes.Positive and negative class embeddings receive separate learned embeddings before entering the mask decoder.
  • 3.3 Contrastive Prototype Learning: Contrastive prototype learning treats class prototypes as anchors and image-derived class embeddings as samples to make prototypes more discriminative.The contrastive loss increases similarity to positive same-class embeddings and suppresses similarity to embeddings from other classes.
  • 3.4 Efficient Tuning: During end-to-end tuning, the image encoder remains frozen while the lightweight prompt encoder and mask decoder are optimized with dice and prototype contrastive losses.The combined objective uses predicted and ground-truth pixel values for segmentation alongside prototype learning.

4 Experiments and Discussion

Experiments on EndoVis2018 and EndoVis2017 show that SurgicalSAM achieves strong segmentation performance against SAM-based and specialist baselines while using fewer tunable parameters. Ablations and cross-dataset tests support the roles of contrastive prototypes, generalisability, and efficiency.

  • Main Results: SurgicalSAM surpasses existing SAM-based models and matches or exceeds specialist models on EndoVis2018 and EndoVis2017 with only a few tunable parameters.The comparison includes detection-based, tracking-based, and reference-based SAM strategies.
  • Main Results: SurgicalSAM outperforms all three zero-shot SAM baselines while using 4.65M tunable parameters versus 57.67M for MaskTrack-RCNN + SAM.Learned prototypes replace single-instance references and explicit prompts, simplifying the pipeline.
  • Main Results: 20.07% and 25.52% improvements in Challenge IoU over ground-truth centroids are reported on EndoVis2018 and EndoVis2017, respectively.These oracle comparisons indicate performance above basic manual centroid guidance.
  • Main Results: SurgicalSAM achieves SOTA performance competitive with specialist models using 4.65M parameters versus 68.72M for MATIS Frame.The paper reports particularly significant improvements in mean class IoU and attributes them to foundation-model priors helping with class imbalance.
  • Ablation Study: Contrastive prototype learning addresses overly similar fixed prototypes caused by the similar appearance of different instrument categories.The ablation study removes contrastive learning and compares it with fixed prototypes computed from training embeddings.
  • Cross-Dataset Generalisation: SurgicalSAM improves cross-dataset generalisation over MATIS Frame in both transfer directions, including an 11.43% IoU gain from EndoVis2018 to EndoVis2017.The cross-dataset evaluation considers only instrument classes shared by both datasets.
  • Efficiency: SurgicalSAM uses less than 1/6 of MATIS Frame’s GPU memory while providing faster training and lower GPU memory consumption.Efficiency is evaluated across multiple training batch sizes and inference speed.

5 Conclusion

SurgicalSAM efficiently tunes SAM for surgical instrument segmentation by generating prompt embeddings from class prototypes and improving prototype discrimination. It achieves state-of-the-art performance on both EndoVis datasets with efficient training and inference.

  • Conclusion: SurgicalSAM generates prompt embeddings directly from class prototypes, eliminating explicit points or bounding boxes and enabling an end-to-end pipeline.The method uses a prototype-based class prompt encoder and contrastive prototype learning.
  • Conclusion: SurgicalSAM achieves state-of-the-art performance on EndoVis2018 and EndoVis2017 while demonstrating training and inference efficiency.The conclusion presents the method as an approach for adapting SAM to surgical instrument segmentation.

6 Background and Motivation

SurgicalSAM addresses two barriers to applying SAM in surgical instrument segmentation: weak transfer across the surgical–natural domain gap and dependence on precise frame-level point or box prompts. It replaces explicit prompting with category-based prompting to support a simpler, more efficient pipeline.

  • Motivation: Promptable segmentation aligns with surgeon-provided reference information and supports surgical training, planning, navigation, and post-surgical analysis.The paper positions category-based interaction as an intuitive way to specify areas of interest.
  • Motivation: SAM generalises poorly to surgical instruments because their specialised appearance and high inter-category similarity differ substantially from natural objects.These characteristics make surgical instrument categories difficult to distinguish using SAM’s natural-object knowledge.
  • SurgicalSAM: SurgicalSAM uses a prototype-based class prompt encoder to integrate surgical instrument knowledge with SAM’s pre-trained knowledge through model tuning.The encoder learns class prototypes by interacting with image embeddings and produces prompt embeddings for mask decoding.
  • Motivation: Existing SAM-based surgical segmentation requires precise explicit prompts for each frame, creating substantial human-input or detector requirements.The required points or bounding boxes make direct use in surgical practice impractical.
  • SurgicalSAM: Category-ID prompting eliminates point-or-box prompts and enables a simpler single-stage pipeline without frame-by-frame prompt inputs.A single class prompt can apply to multiple frames or an entire video.

7 Preliminary for Segment Anything Model

SAM encodes an image and a visual prompt separately, then combines their embeddings in a mask decoder. The decoder uses cross-attention and learnable output tokens to produce the segmentation mask.

  • SAM Architecture: SAM consists of an image encoder, prompt encoder, and mask decoder that process images, prompts, and masks respectively.The image encoder uses an MAE-pre-trained Vision Transformer to derive image embeddings, while the prompt encoder derives prompt embeddings.
  • SAM Architecture: The image encoder maps an input image I to an image embedding F_I, while the prompt encoder maps prompt P to prompt embedding T_P.These embeddings provide the two inputs needed by the mask decoder.
  • SAM Architecture: The mask decoder combines image and prompt embeddings with learnable tokens to produce the output mask M.Its cross-attention mechanism facilitates interaction between the image and prompt representations.

8 Experiment Details and Results

The experiments define class-promptable evaluation and implement detection-, tracking-, and reference-based zero-shot SAM baselines alongside SurgicalSAM. They also specify preprocessing, prompt construction, and multi-stage inference-speed measurement.

  • Evaluation: Challenge IoU equals IoU when class prompts are provided, and mean class IoU averages IoU across instrument classes.Evaluations use the official code implementation.
  • Data Preparation: EndoVis18 and EndoVis17 images are resized to 1024 × 1280 and augmented with flipping, scale-and-crop, rotation, and colour jitter.Preprocessing follows established protocols for these datasets.
  • Training: Class prototypes are initialised from N(0, 1) and updated gradually by the loss functions during training.This describes the prototype initialisation used in the implementation.
  • Baselines: Detection-based zero-shot SAM uses a trained detector to predict candidate bounding boxes before feeding them into SAM for mask prediction.MaskTrackRCNN and Mask2Former serve as detector backbones for the implemented baselines.
  • Efficiency Evaluation: Multi-stage inference speed is computed as Q/(T1 + T2) fps, combining detector time T1 and SAM mask-prediction time T2.The measurement captures both candidate-box acquisition and mask generation across all frames.
  • Baselines: Track Anything is evaluated with one-point and five-point prompts based on ground-truth mask centroids and extremity points.The five-point scheme uses leftmost, rightmost, topmost, and bottommost points with a 10-pixel boundary margin.
  • Baselines: PerSAM uses the first frame containing each class and its ground-truth mask as the reference input, with both training-free and fine-tuned versions evaluated.The implementation follows the official PerSAM settings.

8.4 Ablation Study of Prompt Embeddings

The prompt-embedding ablation removes dense, sparse, and positive or negative embeddings from SurgicalSAM’s prototype-based class prompt encoder. The study reports that each component contributes, with dense embeddings having the strongest observed effect.

  • Prompt Embeddings: Removing dense prompt embeddings causes a larger performance drop than removing sparse prompt embeddings.This result supports the reported importance of dense prompt embeddings within the encoder.
  • Prompt Embeddings: The ablation evaluates the dense, sparse, positive, and negative prompt embeddings of the prototype-based class prompt encoder.Results are reported on EndoVis2018 in Table 6.

8.5 Comparison with Text Promptable Baseline

The text-promptable baseline uses CLIP class-name embeddings without contrastive prototype training, while SurgicalSAM tunes only a small parameter subset and maintains low training and inference costs.

  • 8.5 Comparison with Text Promptable Baseline: 75.94% Challenge IoU and 51.76% mc IoU are achieved by the text-promptable baseline.The baseline uses CLIP text embeddings of class names as prototypes.
  • 8.5 Comparison with Text Promptable Baseline: The baseline omits training with the prototype contrastive loss L_P CL.Its prototypes come directly from CLIP text embeddings of the class names.
  • 8.5 Comparison with Text Promptable Baseline: The reported baseline results are presented as evidence for the superiority of the proposed method.This comparison is framed as an ablation of contrastive prototype learning.
  • 8.6 Complexity Analysis and Clinical Significance: 641.68M total parameters are used by SurgicalSAM, including 637.03M in the frozen ViT-H image encoder.The image encoder accounts for most of the model parameters but is not tuned.
  • 8.6 Complexity Analysis and Clinical Significance: Only 4.65M parameters are tuned in SurgicalSAM.The remaining parameters are attributed to the frozen ViT-H image encoder.
  • 8.6 Complexity Analysis and Clinical Significance: SurgicalSAM has low training and inference costs, supporting high training efficiency for real-world surgical applications.The passage links this efficiency to resource efficiency during model development and affordability for healthcare institutions.

8.7 Visualisation

Visualisations show SurgicalSAM’s predictions across EndoVis datasets, including category discrimination in multi-category images and similarity maps associated with prompted classes. Repeated-seed experiments report consistent performance on EndoVis2018.

  • Result Visualisation: Additional prediction visualisations cover EndoVis2018 and EndoVis2017, with samples from all videos in each dataset.The visualisations are intended to provide a comprehensive view of predictions.
  • Result Visualisation: SurgicalSAM distinguishes two or three instrument categories in an image and identifies the prompted category.The passage attributes this capability to discriminative prototypes learned through contrastive prototype learning.
  • Result Visualisation: Discriminative prototypes learned through contrastive prototype learning support prompted-category identification.This is the stated mechanism associated with the multi-category visualisation results.
  • Similarity Map Visualisation: Positive-class similarity maps S(c) are computed between image embeddings and the prototype of the prompted class c.The maps are normalised for visualisation, with the prompted class’s ground-truth mask provided for reference.
  • Similarity Map Visualisation: Similarity-map visualisations validate the capability of SurgicalSAM’s prototype-based class prompt encoder.The maps are presented alongside ground-truth masks to examine the class activation mechanism.
  • Random-Seed Robustness: Across repeated EndoVis2018 runs, Challenge IoU averages 80.16 with standard deviation 0.29.The experiments use different random seeds to assess robustness to seed randomness.

9 Enlarged Figures

The appendix provides enlarged versions of selected main-paper figures to improve clarity and viewing convenience. The figures are recommended for screen viewing with zoom-in.

  • 9 Enlarged Figures: Fig. 1, Fig. 3(b), and Fig. 5 are best viewed on screen with zoom-in for better clarity.The passage specifically identifies these three main-paper figures.
  • 9 Enlarged Figures: Enlarged versions of the selected figures are provided in Fig. 10, Fig. 11, and Fig. 12.These enlarged versions are offered to facilitate better visibility.
  • 9 Enlarged Figures: The enlarged figures are intended to improve visibility and convenience when inspecting the visual results.Their purpose is supplementary viewing rather than presenting additional experimental findings.
Loading 2308.08746v2…