Source-linked AI summary
Input Augmentation with SAM: Boosting Medical Image Segmentation with Segmentation Foundation Model
Yizhe Zhang, Tao Zhou, Shuo Wang, Peixian Liang, Danny Z. Chen
TL;DR
Medical segmentation is a domain-expert task for which SAM alone is not sufficiently powerful. The paper proposes SAMAug, which uses SAM-generated masks and stability scores to augment inputs to downstream models. Experiments across three segmentation tasks show the method's effectiveness.
Problem
SAM produces segmentation for many natural-scene objects, but studies find it insufficient for tasks requiring domain expertise such as medical image segmentation.
Source: §1 · p. 1Method
SAMAug uses SAM-generated masks and stability scores to augment raw medical-image inputs for a trainable downstream segmentation model.
Source: §1 · pp. 1–2Results
Experiments report SAMAug's effectiveness across three medical segmentation tasks, with significant HSNet improvements on CVC-ClinicDB and CVC-ColonDB and unchanged performance on the other three polyp datasets.
Source: §4.2 · p. 7, §5 · p. 9Takeaways & Limitations
Abstract
from arXiv · showhide
The Segment Anything Model (SAM) is a recently developed large model for general-purpose segmentation for computer vision tasks. SAM was trained using 11 million images with over 1 billion masks and can produce segmentation results for a wide range of objects in natural scene images. SAM can be viewed as a general perception model for segmentation (partitioning images into semantically meaningful regions). Thus, how to utilize such a large foundation model for medical image segmentation is an emerging research target. This paper shows that although SAM does not immediately give high-quality segmentation for medical image data, its generated masks, features, and stability scores are useful for building and training better medical image segmentation models. In particular, we demonstrate how to use SAM to augment image input for commonly-used medical image segmentation models (e.g., U-Net). Experiments on three segmentation tasks show the effectiveness of our proposed SAMAug method. The code is available at \url{https://github.com/yizhezhang2000/SAMAug}.
1 Introduction
SAM is a general-purpose segmentation foundation model, but studies find it insufficient for medical tasks requiring domain expertise. SAMAug uses its masks and stability scores to augment medical-image inputs for trainable downstream models.
1 Introduction
- SAM produces masks for many objects but is not powerful enough for segmentation tasks requiring domain expertise, including medical imaging.It was trained on 11 million images and over 1 billion masks.Source: §1 · p. 1
- SAMAug uses SAM-generated masks and stability scores to augment raw inputs for a task-specific, trainable medical segmentation model while keeping SAM fixed.The fusion function is parameter-free, and training updates the downstream model using the task dataset.Source: §1 · pp. 1–2
- The paper presents SAM prior maps and SAM-augmented inputs, then evaluates the method with CNN and Transformer models across three medical segmentation tasks.The authors describe SAM's segmentation outputs as attention or prior maps for downstream tasks.Source: §1 · p. 2, Figure 1 · p. 2
2 Related Work
The paper frames SAMAug as data augmentation and semantic image enhancement, contrasting its use of foundation-model structures with hand-designed transformations and low-level enhancement. It also contrasts the method with SAM fine-tuning and adaptation approaches.
2 Related Work
- Unlike data augmentation methods using hand-designed transformations such as rotation or cropping, SAMAug adds semantic structures from a segmentation foundation model to raw images.The paper also contrasts this high-level semantic addition with traditional image enhancement aimed at low-level reconstruction or recovery.Source: §2 · pp. 2–3
- Prior work reports that SAM alone often gives unsatisfactory medical segmentation, while fine-tuning or adaptation methods modify SAM for medical tasks.The paper says SAMAug has lower computation and memory costs during training than these methods; all require test images to pass through SAM at inference.Source: §2 · p. 3
3 Methodology
SAMAug generates segmentation and boundary priors from SAM masks, adds them to image channels, and trains a medical segmentation model on the augmented inputs. At deployment, the paper describes augmented-only inference and alternatives using both raw and augmented inputs.
3 Methodology
- The methodology generates segmentation and boundary prior maps from SAM and uses them to augment medical images before model training and deployment.The section covers prior generation, input augmentation, training, and deployment.Source: §3 · p. 3, §3.1 · p. 4, §3.2 · p. 4
3.1 Segmentation and Boundary Prior Maps
- SAM draws each mask onto a segmentation prior map using its stability score and combines mask exterior boundaries into a boundary prior map.The masks are generated at plausible image locations using grid prompts.Source: §3.1 · p. 4
3.2 Augmenting Input Images
- The method adds the segmentation prior to channel two and the boundary prior to channel three, creating three-channel inputs for grayscale images.This corresponds to background, regions of interest, and boundaries in tasks reducible to three-class segmentation.Source: §3.2 · p. 4
3.3 Model Training with SAM-Augmented Images
- The model can train on SAM-augmented images alone or, when SAM priors are implausible, on both raw and augmented images; β and λ default to 1.The second objective weights losses for raw-image and augmented-image samples.Source: §3.3 · pp. 4–5
3.4 Model Deployment with SAM-Augmented Images
- Augmented-only training requires SAM-augmented test inputs, while dual-input inference can average predictions or select the input with lower prediction entropy.For the selection option, lower entropy is treated as greater prediction certainty.Source: §3.4 · p. 5
4 Experiments and Results
Experiments evaluate SAMAug on polyp, cell, and gland segmentation, using SAM-generated priors to augment medical-image inputs. Results show improvements on selected polyp datasets and advantages for cell and gland segmentation models.
4.1 Datasets and Setups
- Experiments use Polyp, MoNuSeg, and GlaS benchmarks, following HSNet training setup for polyp segmentation and a shared training configuration for MoNuSeg and GlaS.The latter configuration uses Adam, batch size 8, 256 × 256 crops, learning rate 5e −4, 50K iterations, and spatial cross entropy loss.Source: §4.1 · p. 6
4.2 Polyp Segmentation on Five Datasets
- SAMAug significantly improves HSNet Dice scores on CVC-ClinicDB and CVC-ColonDB, with performance unchanged on CVC-300, Kvasir, and ETIS.The comparison is reported across five test datasets and validated by t-tests.Source: Figure 3 · p. 6, §4.2 · p. 7
4.3 Cell Segmentation on the MoNuSeg Dataset
- SAMAug improves segmentation results for U-Net, P-Net, and Attention U-Net on MoNuSeg.SAM-generated masks are not immediately accurate cell segmentations, but provide a perceptual prior for task-specific models.Source: §4.3 · pp. 7–8, Table 1 · p. 8, Figure 5 · p. 8
4.4 Gland Segmentation on the GlaS Dataset
- U-Net with SAMAug performs considerably better than U-Net without SAMAug on the GlaS test set.The evaluation merges test parts A and B.Source: §4.4 · p. 9, Table 2 · p. 9
5 Conclusions
The paper proposes SAMAug, which uses SAM to augment inputs for commonly used medical image segmentation models. Experiments across three segmentation tasks show the method's effectiveness, while future work targets augmentation robustness, efficiency, and further applications.
- SAMAug uses the Segment Anything Model to augment inputs for commonly used medical image segmentation models.Source: §5 · p. 9
- Experiments on three segmentation tasks show the effectiveness of SAMAug.Source: §5 · p. 9
- Future work may improve the augmentation function and SAMAug efficiency, and investigate uncertainty estimation and clinically oriented applications.Source: §5 · p. 9