PaperlayerView exact revision on arXiv

Source-linked AI summary

3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation

Shizhan Gong, Yuan Zhong, Wenao Ma, Jinpeng Li, Zhao Wang, Jingyang Zhang, Pheng-Ann Heng, Qi Dou

arXiv:2306.13465v2cs.CV

TL;DR

SAM shows suboptimal performance on medical image segmentation, with limited performance on irregular-shaped or low-contrast objects and unstable results under single-point prompts on 3D medical images. The method holistically modifies SAM for volumetric inputs while retaining pretrained weights and tuning lightweight spatial adapters. With one coarse click, 3DSAM-adapter outperformed nnU-Net on three of four datasets—8.25% for kidney tumor, 29.87% for pancreas tumor, and 10.11% for colon cancer—and was comparable on liver tumor.

  • Problem

    SAM shows suboptimal performance on medical image segmentation, with limited performance on irregular-shaped or low-contrast objects and unstable results under single-point prompts on 3D medical images.

    Source: §1 · p. 1

  • Method

    The method holistically modifies SAM for volumetric inputs while retaining pretrained weights and tuning lightweight spatial adapters.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    With one coarse click, 3DSAM-adapter outperformed nnU-Net on three of four datasets—8.25% for kidney tumor, 29.87% for pancreas tumor, and 10.11% for colon cancer—and was comparable on liver tumor.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The visual sampler and global queries improve tolerance to noisy prompts, and the model performs well with only one inference point.

    Source: §3.3 · pp. 5–6

    The reported metric is lower than official leaderboard results because the experiments do not use organ labels for multi-label prediction.

    Source: §5 · p. 10

Abstract

from arXiv · show

Despite that the segment anything model (SAM) achieved impressive results on general-purpose semantic segmentation with strong generalization ability on daily images, its demonstrated performance on medical image segmentation is less precise and not stable, especially when dealing with tumor segmentation tasks that involve objects of small sizes, irregular shapes, and low contrast. Notably, the original SAM architecture is designed for 2D natural images, therefore would not be able to extract the 3D spatial information from volumetric medical data effectively. In this paper, we propose a novel adaptation method for transferring SAM from 2D to 3D for promptable medical image segmentation. Through a holistically designed scheme for architecture modification, we transfer the SAM to support volumetric inputs while retaining the majority of its pre-trained parameters for reuse. The fine-tuning process is conducted in a parameter-efficient manner, wherein most of the pre-trained parameters remain frozen, and only a few lightweight spatial adapters are introduced and tuned. Regardless of the domain gap between natural and medical data and the disparity in the spatial arrangement between 2D and 3D, the transformer trained on natural images can effectively capture the spatial patterns present in volumetric medical images with only lightweight adaptations. We conduct experiments on four open-source tumor segmentation datasets, and with a single click prompt, our model can outperform domain state-of-the-art medical image segmentation models on 3 out of 4 tasks, specifically by 8.25%, 29.87%, and 10.11% for kidney tumor, pancreas tumor, colon cancer segmentation, and achieve similar performance for liver tumor segmentation. We also compare our adaptation method with existing popular adapters, and observed significant performance improvement on most datasets.

1 Introduction

SAM’s medical segmentation is limited and unstable on difficult tumor targets, while adapting a 2D model to isotropic 3D volumes remains unresolved. The paper proposes a parameter-efficient, holistic 3D adaptation and reports gains over nnU-Net on three of four tumor tasks.

1 Introduction

  • SAM’s medical performance is suboptimal and unstable, especially for tumors with irregular shapes or low contrast and under single-point prompts on 3D images.The introduction attributes this gap to limited coverage of medical distributions in SAM’s training data and positional prompts’ reliance on local features.Source: §1 · pp. 1–2
  • Prior 2D-to-3D approaches retain 2D transformer backbones and add fusion structures, a mismatch for isotropic medical volumes that also raises token-count and memory concerns.The paper identifies holistic adaptation of pretrained 2D parameters to capture 3D spatial information as an open design challenge.Source: §1 · p. 2
  • The holistic design modifies SAM to accept volumetric inputs, uses a visual sampler and global queries for prompts, and adds multi-layer aggregation in the mask decoder.The approach retains reusable pretrained weights and is designed for parameter-efficient fine-tuning.Source: §1 · p. 2
  • 8.25% for kidney tumor, 29.87% for pancreas tumor, and 10.11% for colon cancer: 3DSAM-adapter outperformed nnU-Net on those tasks and was comparable on liver tumor.The experiments cover four tumor segmentation datasets.Source: §1 · p. 2

2 Related Works

Related work spans vision foundation models, parameter-efficient fine-tuning, and tumor segmentation. The paper positions its approach against methods that handle the added dimension separately and focuses evaluation on challenging tumor targets.

Foundation models in computer vision

  • SAM [4] is presented as a generalist promptable segmentation model, alongside approaches such as SEEM [31] and SegGPT [32] that broaden prompting or in-context segmentation.The discussion situates these models within the pretraining and fine-tuning paradigm in modern vision.Source: §2 · p. 3

Parameter-efficient model fine-tuning

  • Parameter-efficient fine-tuning includes addition-based adapters or prompts [16] [19] [20], specification-based tuning [15] [35], and reparameterization with low-rank matrices [36].The paper contrasts prior volumetric methods that treat the added dimension as a separate group with its direct adaptation of transformer blocks for 3D patterns.Source: §2 · p. 3

Tumor segmentation in medical imaging

  • Tumors remain difficult segmentation targets because they are small, irregular, low-contrast, and bounded ambiguously; medical SAM applications also report worse and less stable tumor results than organ results.The paper therefore evaluates its adapter in tumor segmentation scenarios.Source: §2 · p. 3

3 Methods

The method adapts SAM’s image encoder, prompt encoder, and mask decoder for volumetric medical segmentation. It aims to capture 3D spatial information while reusing pre-trained parameters and keeping adaptation lightweight.

3.1 Overview of SAM

  • The 2D SAM processes volumes slice by slice, missing inter-slice spatial information, and its natural-image training creates a domain gap for medical images.Source: §3.1 · p. 3

3.2 Adapting Image Encoder for Volumetric Inputs

  • The image-encoder design targets direct 3D spatial learning while reusing pre-trained parameters, modifying patch embedding, positional encoding, attention, and the bottleneck for volumetric inputs.Source: §3.2 · p. 4
  • A depth-wise 3D convolution in the adapter adds spatial processing, while training tunes convolutions, spatial adapters, and normalization layers and freezes the remaining parameters.Source: §3.2 · p. 5

3.3 Prompt Encoding by Visual Sampler

  • The visual sampler retrieves image embeddings at prompt coordinates, then uses global queries with attention to reduce over-smoothing and increase tolerance to noisy points.Source: §3.3 · p. 5
  • Training includes background images with false-positive prompts, and the model performs well at inference with only one point prompt.Source: §3.3 · pp. 5–6

3.4 Lightweight Mask Decoder

  • The decoder replaces 2D convolutions with 3D convolutions and aggregates intermediate encoder outputs, then fuses the upsampled mask features with the original-resolution image to generate the final mask.Source: §3.4 · p. 6
  • The method removes SAM’s multi-mask generation and ambiguity awareness for the downstream task, and trains the lightweight 3D-convolutional decoder from scratch.Source: §3.4 · p. 6

4 Experiments

The experiments evaluate the method on four public volumetric tumor datasets against segmentation and adaptation baselines, then examine prompt and decoder design choices.

4.1 Setup

  • The evaluation uses four public CT datasets containing 300, 281, 118, and 126 scans, split randomly into 70% training, 10% validation, and 20% testing.Tumor labels are used for training and testing; results are evaluated with Dice and Normalized Surface Dice.Source: §4.1 · pp. 6–7

4.2 Comparison with State-of-the-Arts

  • With a single point prompt in the whole volume, the method achieves state-of-the-art accuracy, exceeding prior state of the art by 12.44% Dice for pancreas tumors and 10.11% for colon cancers.For liver tumors, performance is inferior to nnU-Net because a single click can leave other scattered tumors unsegmented.Source: §4.2 · p. 8

4.3 Comparison with Existing Adapters

  • The adaptation outperforms existing methods with comparable tuned parameters, including by 16.24% on kidney tumor Dice, 4.22% on pancreas tumor NSD, and 12.53% on colon cancer Dice.It also exceeds full SAM fine-tuning by 2.08%–29.80% with fewer than 16.96% of parameters tunable.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 Ablation Studies

  • Visual sampling beats positional encoding by 40.00% Dice, while prompt locations yield almost identical results across center, surrounding, and margin regions.The position analysis samples the largest tumor when cases contain multiple tumors.Source: §4.4 · p. 9, Figure 7 · p. 9, Figure 6 · p. 9
  • Multi-layer aggregation achieves 15.75% higher Dice than the decoder without it.The authors connect this gain to the importance of local texture features for tumors with tiny shapes and low contrast.Source: §4.4 · p. 9, Figure 7 · p. 9
  • Adding prompts at multiple encoder feature levels brings no gain or can reduce performance compared with prompts at the final bottleneck levels.Making prompts effective at multiple feature levels is left for future research.Source: §4.4 · p. 9

5 Discussion and Conclusion

The method adapts SAM for volumetric tumor segmentation and is reported to outperform domain state-of-the-art models with a single coarse click. The authors note limitations tied to leaderboard comparisons and the challenges of segmenting small tumors.

5 Discussion and Conclusion

  • With a single coarse click, the method is reported to outperform domain state-of-the-art models and existing volumetric adaptation methods.The authors describe the approach as a comprehensive adaptation of SAM for volumetric medical imaging, especially tumor segmentation.Source: §5 · p. 9
  • Reported metrics are lower than official leaderboard results because the experiments do not use organ labels for multi-label prediction.The authors say pure tumor segmentation can bypass the intensive work of organ labeling, while noting that multi-task segmentation brings performance gains.Source: §5 · p. 10
  • The authors identify small tumor size and texture-detail loss during ViT downsampling as obstacles, and suggest recovering texture details as future work.They report that CNN-based methods generally perform better and are easier to train for pure tumor segmentation.Source: §5 · p. 10
  • The work is described as having potential for clinical use and as a possible guideline for adapting foundation models to domain-specific tasks.The authors state that improved segmentation accuracy could benefit related diagnostic tasks.Source: §5 · p. 10

A Appendix

The supplementary material provides dataset and implementation details omitted from the main paper because of space limits.

A Appendix

  • The supplementary material supplies dataset and implementation details not included in the main paper due to space limits.Source: A Appendix · p. 14

A.1 Datasets and Implementation Details

The appendix describes public-dataset preprocessing, training, and inference procedures, with dataset details and training hyperparameters summarized in tables.

A.1 Datasets and Implementation Details

  • Public datasets are preprocessed by resampling anisotropic images, clipping foreground intensities at the 0.5th and 99.5th percentiles, and normalizing with global foreground statistics.Training crops use a 1:1 foreground-to-background ratio, with random rotation, flipping, zoom, and intensity shifting for augmentation.Source: §A.1 · p. 14, Table 5 · p. 14
  • Training uses AdamW for 200 epochs with a base learning rate of 1e−4 on NVIDIA A40 GPUs; dataset hyperparameters are summarized in Table 4.One training epoch takes around 16 minutes for Kidney Tumor, 13 for Pancreas Tumor, 6 for Liver Tumor, and 7 for Colon.Source: §A.1 · p. 14, Table 4 · p. 14
  • Automatic segmentation uses sliding-window inference, while point-prompt inference crops a training-sized patch around the prompt center.Source: §A.1 · p. 14
Loading 2306.13465v2…