Source-linked AI summary
One Click per Cell Type Suffices: Training-free Group Interaction for Cell Instance Segmentation
Sanghyun Jo, Seo Jin Lee, Seohyung Hong, Yoorim Gang, Hyeongsub Kim, Hyungseok Seo, Kyungsu Kim
TL;DR
Cell-specific segmentation methods degrade on out-of-distribution cell types, while per-instance prompting is costly for densely populated histopathology images. CoP addresses this with training-free group prompting through frozen SAM features, retaining over 90% of per-instance performance across eleven benchmarks while requiring up to 97% fewer clicks.
Problem
Cell-specific segmentation methods are tied to training tissue types and morphologies, causing severe degradation on out-of-distribution cells, while per-instance interaction is costly for densely packed images.
Method
CoP recursively propagates a single click through frozen SAM multi-scale features using Hierarchical Similarity Gating to find reliable points and Farthest Prompt Recursion to expand spatial coverage.
Results
Across eleven H&E and non-H&E benchmarks, CoP retains over 90% of per-instance performance, requires up to 97% fewer clicks, generalizes without adaptation, and surpasses fully supervised methods.
Takeaways & Limitations
Group prompting reduces interaction from per-instance O(N) to per-type O(T), making a single click per cell type a scalable alternative for segmenting same-type cells.
Takeaways & Limitations
CoP is bounded by its base model and cannot recover instances that SAM3 fails to segment even from correct point prompts.
Abstract
from arXiv · showhide
Cell instance segmentation models trained on cell-specific datasets suffer severe performance drops on out-of-distribution cell types, while interactive foundation models overcome this through per-instance prompting at a cost that is prohibitively expensive for histopathology images containing hundreds to thousands of densely packed instances. We introduce \textbf{Group Prompting}, a new paradigm that shifts interactive segmentation from per-instance $O(N)$ to per-type $O(T)$, where a single click per cell type suffices to segment all instances of that type. Our key observation is that the frozen image encoder of the Segment Anything Model (SAM) already clusters same-type cells in its feature space before any prompt is given, and that this clustering holds across staining modalities without any training. Exploiting this property, we propose \textbf{Chain-of-Prompts (CoP)}, a training-free framework that recursively expands a single user click by (1) identifying reliable same-type locations through non-parametric gating of multi-scale encoder features, and (2) selecting the most spatially distant reliable point as the next prompt to maximize coverage. On eleven benchmarks, CoP generalizes to both unseen cell types and unseen imaging modalities without any adaptation: with one click per type it retains over 90\% of per-instance performance on three cell-type-annotated datasets while surpassing fully-supervised methods, and with one click per image it retains over 95\% on eight datasets spanning both H\&E and non-H\&E imaging. Project Page: https://shjo-april.github.io/Chain-of-Prompts/
1 Introduction
Cell-specific segmentation methods degrade on unseen cell types, while per-instance prompting is too costly for densely packed histopathology images. CoP introduces group prompting, recursively extending one click per type while retaining high performance across diverse benchmarks.
- Cell-specific methods suffer severe performance degradation on out-of-distribution cell types because their representations reflect training tissue types and morphologies.
- Histopathology images contain hundreds to thousands of densely packed instances, making O(N) per-instance prompting prohibitively expensive.
- 93.9% of upper-bound performance is achieved with three clicks instead of 245, using 81.7× fewer prompts.
- CoP combines hierarchical similarity gating with farthest prompt recursion to expand reliable prompt coverage without learnable parameters.HSG maintains precision above 96%, while FPR selects spatially distant prompts to uncover unexplored regions.
- Group Prompting shifts interaction from per-instance O(N) to per-type O(T), reducing annotation cost to the number of cell types.
- Over 90% of per-instance performance is retained on three cell-type-annotated datasets and over 95% on eight untyped datasets spanning H&E and non-H&E imaging.The method generalizes to unseen cell types and modalities while outperforming fully supervised methods.
2 Related Work
Prior work reduces annotation through prompt-driven, pseudo-mask, or category-level approaches, but these methods trade interaction cost against robustness in dense cellular imagery. CoP instead exploits structure already present in a frozen SAM encoder.
- SAM-family models use a prompt-independent image encoder and lightweight decoders that turn point or box prompts into masks.
- Annotation-free methods derive pseudo masks from self-supervised signals, including magnification, texture, or gradient-based cues.
- Open-vocabulary and category-conditioned models can produce masks from text or exemplar prompts, but rely on alignment learned largely from natural images.
- Dense, low-contrast cellular imagery rarely provides the spatial separation and morphological distinctness assumed by category-level conditioning.
3 Method
CoP encodes each image once with frozen SAM features, identifies reliable same-type points through hierarchical gating, and recursively chooses distant prompts to expand coverage. The resulting points are decoded into instance masks until no new points are found.
- CoP uses frozen SAM high- and low-resolution feature maps to discover same-type cells from one click per type.
- Hierarchical Similarity Gating: Hierarchical Similarity Gating combines multi-scale features to produce a high-precision reliable point set through element-wise gating.High-resolution features provide localization, while low-resolution features provide type selectivity.
- Hierarchical Similarity Gating: HSG thresholds the gated similarity map non-parametrically and applies connected-component labeling to extract similarity-weighted centroids.
- Farthest Prompt Recursion: Farthest Prompt Recursion selects the reliable point farthest from previous prompts to explore spatially uncovered regions.
- The encoder is computed once, subsequent prompts reuse its features, and recursion stops when the reliable set no longer grows.Each final point is decoded into an instance mask.
4 Experiments
CoP is evaluated on eleven benchmarks spanning cell-type annotation and imaging modality, using one click per type for annotated datasets and one click per image otherwise. It consistently preserves near-per-instance performance, including across unseen modalities, while ablations show that multi-scale gating and farthest-point recursion are important for precision and coverage.
- Experimental Setup: Eleven benchmarks span H&E type-annotated, H&E untyped, and non-H&E untyped groups, with CoP using one click per type for the first group and one click per image for the others.The evaluation uses official test splits and compares against baselines under their intended prompting protocols.
- Comparison with State-of-the-art Approaches: CoP surpasses every open-vocabulary and fully-supervised baseline on all three H&E type-annotated benchmarks while retaining over 90% of SAM3’s per-instance performance.On CoNIC, CoP reaches 0.731 AJI versus 0.801 for the per-instance upper bound, using roughly one click per type instead of hundreds.
- Comparison with State-of-the-art Approaches: Roughly 98–99% of the per-instance upper bound is retained on H&E untyped benchmarks with one click per image for both µSAM and SAM3.CoP also outperforms every fully-supervised baseline in this setting.
- Comparison with State-of-the-art Approaches: Over 95% of the per-instance upper bound is preserved across all four non-H&E benchmarks without adaptation.H&E-trained supervised models can fail almost completely under modality change, while CoP relies only on frozen encoder features; Cellpose3 surpasses CoP only on LIVECell Dice.
- Ablation Study: Combining high- and low-resolution features is necessary: using high-resolution features alone lowers AJI to 0.545, while low-resolution features alone lowers it to 0.419.The combined gating suppresses spatial noise from high-resolution features and semantic uncertainty from low-resolution features.
- Ablation Study: Farthest-point recursion and iterative propagation improve coverage: replacing farthest selection lowers AJI to 0.612 or 0.658, while removing FPR lowers it from 0.731 to 0.482.Alternative selection rules repeatedly sample covered regions, whereas recursive re-prompting reaches distant instances.
5 Discussion
CoP’s performance is driven by prompt-mask-pretrained encoder structure, remains robust to click placement, and exposes bounded cross-type leakage and dependence on SAM3’s upper bound.
- 0.731 AJI is achieved by CoP with SAM3’s 0.45B-parameter encoder, while alternative frozen encoders remain at 0.328 to 0.435 AJI.The comparison attributes the difference to prompt-mask pretraining rather than encoder capacity.
- 0.731 ± 0.005 AJI across 100 random initial clicks per CoNIC image shows minimal sensitivity to click location.
- About 5% cross-type leakage occurs when visually similar cell types are clicked, though it remains harmless for cell-type-agnostic segmentation.Cell-type-specific use would require type-discriminative cues from the same frozen features.
- CoP cannot recover instances that SAM3 fails to segment from a correct point prompt, because the framework reuses the frozen base model without cell-annotation training.
6 Conclusion
The paper presents CoP as a training-free way to segment same-type cells from one click by recursively propagating prompts through frozen SAM features. Across eleven benchmarks, it retains over 90% of per-instance performance while reducing clicks and generalizing across cell types and modalities.
- CoP discovers and segments all same-type cells from a single user click by recursively propagating prompts through frozen SAM features.
- SAM’s frozen encoder clusters same-type cells before prompting, enabling CoP’s non-parametric gating without additional training.
- Over 90% of per-instance performance is retained across eleven benchmarks spanning H&E and non-H&E modalities, with up to 97% fewer clicks.The method also generalizes to unseen cell types and imaging modalities without adaptation and surpasses fully supervised methods.