Source-linked AI summary
SAM2-UNet: Segment Anything 2 Makes Strong Encoder for Natural and Medical Image Segmentation
Xinyu Xiong, Zihuang Wu, Shuangyi Tan, Wenxue Li, Feilong Tang, Ying Chen, Siying Li, Jie Ma, Guanbin Li
TL;DR
Unified segmentation across diverse natural and medical tasks remains challenging, while SAM2 requires adaptation for task-specific outputs without manual prompts. SAM2-UNet uses a SAM2-pretrained Hiera encoder, classic U-Net decoder, and adapters for parameter-efficient fine-tuning, achieving strong performance across five benchmarks and eighteen datasets.
Problem
SAM2 produces class-agnostic results without manual prompts, while unified architectures for diverse natural and medical segmentation tasks remain challenging.
Method
SAM2-UNet combines the SAM2-pretrained Hiera encoder with a classic U-Net decoder and adapters for parameter-efficient fine-tuning.
Results
SAM2-UNet delivers powerful performance across five challenging benchmarks and eighteen datasets, including state-of-the-art results on three of five polyp datasets.
Takeaways & Limitations
SAM2-UNet provides a simple U-shaped framework that the authors propose as a new baseline for future SAM2 variants.
Abstract
from arXiv · showhide
Image segmentation plays an important role in vision understanding. Recently, the emerging vision foundation models continuously achieved superior performance on various tasks. Following such success, in this paper, we prove that the Segment Anything Model 2 (SAM2) can be a strong encoder for U-shaped segmentation models. We propose a simple but effective framework, termed SAM2-UNet, for versatile image segmentation. Specifically, SAM2-UNet adopts the Hiera backbone of SAM2 as the encoder, while the decoder uses the classic U-shaped design. Additionally, adapters are inserted into the encoder to allow parameter-efficient fine-tuning. Preliminary experiments on various downstream tasks, such as camouflaged object detection, salient object detection, marine animal segmentation, mirror detection, and polyp segmentation, demonstrate that our SAM2-UNet can simply beat existing specialized state-of-the-art methods without bells and whistles. Project page: \url{https://github.com/WZH0120/SAM2-UNet}.
1 Introduction
Image segmentation supports diverse natural and medical vision tasks, but designing one unified architecture remains challenging. SAM2-UNet addresses this gap with a simple U-shaped framework using SAM2 components, adapters, and a classic encoder-decoder design.
- Image segmentation underpins downstream tasks including camouflaged, salient, marine animal, mirror, and polyp segmentation, yet unified architectures remain an open challenge.
- SAM2 still produces class-agnostic segmentation without manual prompts, limiting its direct transfer to task-specific or class-specific downstream segmentation.
- SAM2-UNet adopts a classic U-shaped encoder-decoder architecture designed for simplicity and extensibility.
- Adapters enable parameter-efficient fine-tuning, allowing training on memory-limited devices.
- Experiments on eighteen public datasets across five challenging benchmarks demonstrate powerful performance for SAM2-UNet.
2 Method
SAM2-UNet combines SAM2's hierarchical Hiera encoder with a classic U-Net decoder, adapters, receptive field blocks, and supervised segmentation outputs. The design removes nonessential SAM2 components while retaining multiscale features and parameter-efficient fine-tuning.
- SAM2-UNet contains an encoder, decoder, receptive field blocks, and adapters, while discarding SAM2 memory attention, prompt encoder, memory encoder, and memory bank.
- Encoder: The SAM2-pretrained Hiera backbone provides hierarchical multiscale features that are more suitable for a U-shaped network than SAM1's plain ViT encoder.
- Encoder: The simplified Hiera illustration represents one variant of the Hiera block rather than the complete set of possible variants.
- RFBs: Receptive field blocks reduce encoder feature channels to 64 while enhancing the lightweight features.
- Adapters: Freezing Hiera and inserting adapters before each multiscale block enables parameter-efficient fine-tuning when full fine-tuning is memory-infeasible.
- Decoder: The decoder uses three Conv-BN-ReLU blocks and 1×1 convolution heads to produce three upsampled segmentation results supervised by the ground-truth mask.
- Loss Function: Training combines weighted IoU and BCE losses with deep supervision applied to all segmentation outputs.
3 Experiments
Experiments evaluate SAM2-UNet across five segmentation tasks spanning eighteen datasets, with quantitative and visual comparisons plus a backbone ablation. The method achieves leading results across most benchmarks, while performance varies across individual polyp datasets and backbone sizes.
- Datasets and Benchmarks: Experiments cover five tasks and eighteen datasets, including camouflaged, salient, marine animal, mirror, and polyp segmentation.The benchmarks use task-specific datasets and metrics, including S-measure, F-measure, E-measure, MAE, mIoU, IoU, mDice, and related measures.
- Comparison with State-of-the-Art Methods: SAM2-UNet outperforms all comparison methods across four camouflaged-object datasets and every reported metric.Its S-measure advantage over FEDER is 1.1% on CHAMELEON, 4.8% on CAMO, 3.6% on COD10K, and 3.9% on NC4K.
- Comparison with State-of-the-Art Methods: SAM2-UNet achieves the top results across all salient-object metrics, exceeding MENet in S-measure by 1.4% to 3.4% across five datasets.The reported datasets are DUTS-TE, DUT-OMRON, HKU-IS, PASCAL-S, and ECSSD.
- Comparison with State-of-the-Art Methods: SAM2-UNet achieves the best performance across both marine-animal datasets, exceeding MASNet in mIoU by 5.7% on MAS3K and 0.7% on RMAS.The marine-animal evaluation uses MAS3K and RMAS.
- Comparison with State-of-the-Art Methods: SAM2-UNet outperforms comparison methods on mirror detection, exceeding HetNet in IoU by 9% on MSD and 3.8% on PMD.The evaluation covers MSD and PMD.
- Comparison with State-of-the-Art Methods: On polyp segmentation, SAM2-UNet is state of the art on three of five datasets and exceeds CFA-Net by an average of 2.34% in mDice.It reaches 92.8% mDice on Kvasir, while its mDice advantages over CFA-Net are 6.5% on ColonDB and 6.4% on ETIS; performance is weaker on ClinicDB and CVC-300.
- Visual Comparison: Visual comparisons show accurate camouflaged-object detection across varied scenes and reduced false-positive and false-negative rates for polyp segmentation.The examples include hidden faces, chameleons, caterpillars, and seahorses for camouflage detection.
- Ablation Study: Larger Hiera backbones generally improve performance, while smaller backbones still surpass or approach specialized methods through the SAM2 pre-trained representations.The smaller Hiera-Base+ backbone surpasses FEDER, and further reductions produce results comparable to PFNet and ZoomNet with parameter-efficient fine-tuning.
4 Conclusion
SAM2-UNet is a simple U-shaped framework for versatile segmentation across natural and medical domains. Experiments across eighteen datasets and five benchmarks demonstrate its effectiveness and support its use as a baseline for future SAM2 variants.
- SAM2-UNet combines a SAM2 pre-trained Hiera encoder with a classic U-Net decoder for segmentation across natural and medical domains.
- Experiments spanning eighteen datasets and five benchmarks demonstrate the framework’s effectiveness.
- The framework can serve as a new baseline for developing future SAM2 variants.