Source-linked AI summary
Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging
Ruining Deng, Can Cui, Quan Liu, Tianyuan Yao, Lucas W. Remedios, Shunxing Bao, Bennett A. Landman, Lee E. Wheless, Lori A. Coburn, Keith T. Wilson, Yaohong Wang, Shilin Zhao, Agnes B. Fogo, Haichun Yang, Yucheng Tang, Yuankai Huo
TL;DR
Digital pathology requires segmentation despite scarce and expensive expert annotations, motivating evaluation of SAM’s zero-shot capability on WSI tasks. The study tests SAM across tumor, tissue, and nuclei segmentation, finding strong performance for large connected objects but inconsistent dense-instance performance even with 20 prompts per image.
Problem
WSI segmentation needs well-delineated expert training data, but obtaining such annotations is difficult because of privacy, labor, reproducibility, and process constraints.
Method
The study evaluates SAM zero-shot segmentation across tumor, tissue, and cell nuclei tasks using varied point and box prompts and comparisons with domain-specific methods.
Results
SAM achieves remarkable performance for large connected objects but does not consistently achieve satisfying dense instance segmentation, even with 20 prompts per image.
Takeaways & Limitations
Zero-shot SAM can segment heterogeneous digital-pathology objects without a heavy training process, while dense instance segmentation remains challenging.
Takeaways & Limitations
The assessment identifies limitations involving image resolution, multiple scales, prompt selection, and model fine-tuning.
Abstract
from arXiv · showhide
The segment anything model (SAM) was released as a foundation model for image segmentation. The promptable segmentation model was trained by over 1 billion masks on 11M licensed and privacy-respecting images. The model supports zero-shot image segmentation with various segmentation prompts (e.g., points, boxes, masks). It makes the SAM attractive for medical image analysis, especially for digital pathology where the training data are rare. In this study, we evaluate the zero-shot segmentation performance of SAM model on representative segmentation tasks on whole slide imaging (WSI), including (1) tumor segmentation, (2) non-tumor tissue segmentation, (3) cell nuclei segmentation. Core Results: The results suggest that the zero-shot SAM model achieves remarkable segmentation performance for large connected objects. However, it does not consistently achieve satisfying performance for dense instance object segmentation, even with 20 prompts (clicks/boxes) on each image. We also summarized the identified limitations for digital pathology: (1) image resolution, (2) multiple scales, (3) prompt selection, and (4) model fine-tuning. In the future, the few-shot fine-tuning with images from downstream pathological segmentation tasks might help the model to achieve better performance in dense object segmentation.
1. Introduction
Digital pathology needs segmentation methods that reduce reliance on scarce, expensive expert annotations. SAM is evaluated as a zero-shot foundation model for representative WSI segmentation tasks.
- Expert-annotated training data for WSI segmentation are difficult to obtain because of privacy, labor, reproducibility, and annotation-complexity constraints.
- SAM was trained on over 1 billion masks from 11 million licensed, privacy-respecting images and supports point, box, and mask prompts.
- The study assesses SAM zero-shot segmentation for tumor, tissue, and cell nuclei tasks on whole slide imaging data.
- The study reports limitations and performance gaps relative to state-of-the-art domain-specific models.
2. Experiments and Performance
The experiments compare SAM with domain-specific state-of-the-art methods across tumor, tissue, and nuclei segmentation, using varied point and box prompting strategies. Evaluations use WSI-derived datasets and scalable Python-based execution.
- The study ran SAM experiments directly in Python and reported results in Figure 1 and Table 1.
- Tumor Segmentation: Tumor segmentation used TCGA skin-cancer WSIs, SimTriplet as the SOTA comparator, 80× downscaling, and one-point versus 20-point prompts.
- Figure 1 presents qualitative comparisons between SOTA methods and SAM under different prompt strategies.
- Tissue Segmentation: Tissue segmentation used 8,359 256×256 patches from 1,751 ROIs across 459 WSIs, with Omni-Seg as the SOTA comparator.
- Tissue Segmentation: Tissue experiments tested single-point, 20-point, all-point, and all-box prompts, with the latter two serving as theoretical upper bounds.
- Cell nuclei Segmentation: Nuclei segmentation used MoNuSeg H&E images and compared SAM with BEDs, a competitive model trained on the MoNuSeg training data.
3. Limitations on Digital Pathology
The assessment identifies four digital-pathology limitations: resolution mismatch, multi-scale requirements, prompt dependence, and the need for fine-tuning. These constraints affect scalability and dense or heterogeneous segmentation workflows.
- Image resolution: SAM’s average training resolution of 3300×4950 pixels is much smaller than gigapixel WSI data, making patch-level interaction potentially impractical.
- Multiple scales: At the optimal 5× scale for CAP segmentation, PTC segmentation is difficult, whereas 40× imaging produces nearly 100 times more patches.
- Prompt selection: Zero-shot segmentation requires a considerable number of prompts, depends heavily on prompt quality, and raises inter-rater and intra-rater reproducibility concerns.
- Model fine-tuning: Domain heterogeneity still requires tedious manual prompt placement, motivating online or offline fine-tuning to propagate prompt-derived knowledge to large-scale WSI segmentation.
4. Conclusion
Zero-shot SAM performs remarkably for large connected objects in digital pathology, but its performance is inconsistent for dense instance segmentation even with 20 prompts per image.
- Zero-shot SAM enables segmentation of heterogeneous digital-pathology objects without a heavy training process.
- SAM achieves remarkable performance for large connected objects but does not consistently satisfy dense instance segmentation requirements with 20 prompts per image.