Source-linked AI summary
Polyp-SAM: Transfer SAM for Polyp Segmentation
Yuheng Li, Mingzhe Hu, Xiaofeng Yang
TL;DR
Limited colonoscopy data and costly manual annotation make large-scale polyp-segmentation training difficult, despite the clinical importance of accurate detection. The paper finetunes SAM as Poly-SAM and compares decoder-only with full-component transfer learning. Across five public datasets, Poly-SAM achieves state-of-the-art results on multiple datasets and strong performance across the evaluation suite.
Problem
Limited colonoscopy data and costly physician annotation make training large-scale polyp-segmentation models difficult, although accurate polyp detection is clinically important.
Method
Poly-SAM finetunes SAM on multi-center colonoscopy images and compares freezing both encoders with finetuning the image encoder, prompt encoder, and mask decoder.
Results
Poly-SAM achieves state-of-the-art results on CVC-ColonDB, CVC-300, and ETIS with Polyp-SAM-B, while cross-dataset evaluation reaches 90.6% DSC and 85.5% mIoU.
Takeaways & Limitations
The findings demonstrate the potential of adapting SAM to medical image segmentation and support the lightweight ViT-B model for transfer to polyp segmentation.
Takeaways & Limitations
Polyp-SAM struggles with multiple sporadic lesions and requires ground-truth bounding boxes as prompts for optimal performance.
Abstract
from arXiv · showhide
Colon polyps are considered important precursors for colorectal cancer. Automatic segmentation of colon polyps can significantly reduce the misdiagnosis of colon cancer and improve physician annotation efficiency. While many methods have been proposed for polyp segmentation, training large-scale segmentation networks with limited colonoscopy data remains a challenge. Recently, the Segment Anything Model (SAM) has recently gained much attention in both natural and medical image segmentation. SAM demonstrates superior performance in several image benchmarks and therefore shows great potential for medical image segmentation. In this study, we propose Poly-SAM, a finetuned SAM model for polyp segmentation, and compare its performance to several state-of-the-art polyp segmentation models. We also compare two transfer learning strategies of SAM with and without finetuning its encoders. Evaluated on five public datasets, our Polyp-SAM achieves state-of-the-art performance on two datasets and impressive performance on three datasets, with dice scores all above 88%. This study demonstrates the great potential of adapting SAM to medical image segmentation tasks. We plan to release the code and model weights for this paper at: https://github.com/ricklisz/Polyp-SAM.
1. Introduction
Colorectal cancer screening depends on detecting and removing polyps, but manual colonoscopy annotation is time-consuming and variable. Poly-SAM adapts SAM through transfer learning to improve polyp segmentation and support multi-institutional generalization.
- Motivation: Colorectal cancer has high morbidity, and early detection and removal of polyps can improve patient outcomes.The paper identifies polyps as an important factor in colorectal cancer and describes colonoscopy as the diagnostic standard.
- Motivation: Manual inspection of colonoscopy images is tedious, time-consuming, and subject to inter-reader variability.These limitations motivate computer-aided diagnosis and automated segmentation.
- Background: Computer-aided diagnosis can improve annotation efficiency and reduce time-to-diagnosis.
- Background: Limited physician-annotated medical datasets make training large-scale segmentation models difficult.Annotation time and cost remain obstacles to collecting and curating training data.
- Contribution: Poly-SAM finetunes SAM on multi-center colonoscopy images to target polyp segmentation and generalization across institutions.The paper reports comparisons against existing polyp-segmentation methods.
2. Method
The method adapts SAM to polyp segmentation using colonoscopy datasets, prompt generation, and alternative transfer-learning strategies. Training and evaluation use segmentation losses and overlap metrics, with experiments comparing decoder-only and full-model finetuning.
- Datasets: Five public colonoscopy datasets provide images and manually annotated ground-truth masks for evaluating polyp segmentation.The described datasets include Kvasir, CVC-ClinicDB, CVC-ColonDB, and ETIS.
- SAM: SAM combines an image encoder, prompt encoder, and mask decoder to generate masks for prompted objects.It accepts prompts including points, masks, bounding boxes, and text.
- Training: SAM is trained with high-resolution resized images, and decoder masks are resized before comparison with ground truth to calculate loss.
- Prompt generation: Bounding boxes are used as prompts because prior investigations found them most effective for generating high-quality SAM masks.Ground-truth boxes were available for Kvasir and automatically extracted from masks for the remaining datasets.
- Transfer learning: The transfer-learning strategies either freeze both encoders and finetune only the mask decoder or finetune all SAM components.The study evaluates ViT-B and finetunes both ViT-B and ViT-L models for polyp segmentation.
- Optimization: Training uses Dice Loss with AdamW, a 4e-6 learning rate, linear warm-up, cosine annealing, and an 80% training split.
- Evaluation: Evaluation uses DSC and mIoU to measure overlap between predicted and ground-truth masks.DSC is calculated from the harmonic mean of precision and recall.
3. Result
Polyp-SAM achieved strong multi-center and cross-dataset segmentation results, while full SAM fine-tuning improved performance only modestly over decoder-only fine-tuning.
- Transfer learning strategies: Full-component fine-tuning improved Polyp-SAM-B by only 0-3% in DSC and 0.1-4% in mIoU over mask-decoder-only fine-tuning.The comparison used multi-center data and found better performance when all SAM components were fine-tuned.
- Multi-center generalization: 89.4% DSC on CVC-ColonDB, 92.4% DSC on CVC-300, and 90.3% DSC on ETIS made Polyp-SAM-B state of the art on those datasets.Polyp-SAM-B was comparable with other methods on CVC-ClinicDB and Kvasir.
- Multi-center generalization: 92.9% DSC and 88.9% mIoU on CVC-300, plus 90.5% DSC and 86.0% mIoU on ETIS, made Polyp-SAM-L state of the art on both datasets.The reported comparisons used mixed images from multiple public datasets.
- Failure cases: Polyp-SAM still failed in some Kvasir cases involving multiple sporadic lesions.The authors identify further fine-tuning as necessary for this failure setting.
- Cross-dataset generalization: 90.6% DSC and 85.5% mIoU on held-out CVC-ColonDB made Polyp-SAM-B state of the art in cross-dataset evaluation.Polyp-SAM-L achieved the second-best result, with 88.1% DSC and 82.5% mIoU.
4. Discussion and conclusion
Polyp-SAM adapts SAM to colonoscopy polyp segmentation and performs strongly across five public datasets. The study also identifies prompt dependence, sporadic-lesion failures, and unresolved video-segmentation challenges.
- Contribution: Polyp-SAM fine-tunes SAM for polyp segmentation using colonoscopy images, with Polyp-SAM-B and Polyp-SAM-L evaluated across five public datasets.The study compares the two model sizes and transfer-learning strategies for medical image segmentation.
- Model comparison: Polyp-SAM-B was more suitable than Polyp-SAM-L in the authors’ experiments because the lightweight model did not lose to the larger model on three of five datasets.Polyp-SAM-L is described as much more computationally complex.
- Transfer learning: Decoder-only fine-tuning achieved satisfactory performance, while fine-tuning all SAM components produced better but limited gains.The comparison concerns transfer learning with and without encoder fine-tuning.
- Limitations: Polyp-SAM requires ground-truth bounding boxes as prompts for optimal performance and struggles with multiple sporadic lesions.The authors also note that combining spatial and temporal information across video frames remains to be investigated.