Source-linked AI summary
Comprehensive Evaluation and Fine-Tuning of Foundational Cell Nuclei Segmentation Models in Renal Pathology
Ruijie Wu, Junlin Guo, Ruining Deng, Yu Wang, Shilin Zhao, Haichun Yang, Yuankai Huo
TL;DR
General-purpose models remain inconsistent on kidney regions with low contrast, dense nuclei, complex morphology, or strong background staining, and the value of annotating unresolved Medium cases was unclear. The study combined 5,901 Easy pseudo-labels, 860 expert-annotated Medium cases, and 198 Hard cases to evaluate annotation enrichment across nine cell-segmentation model configurations. Across nine configurations, at least one strategy improved F1 over baseline per model; LSP-DETR reached 0.8725 F1 with Hard-only, while StarDist rose from 0.7380 to 0.8332 with Medium-only.
Problem
General-purpose models remain inconsistent on kidney regions with low contrast, dense nuclei, complex morphology, or strong background staining, and the value of annotating unresolved Medium cases was unclear.
Source: §1 · pp. 1–2Method
The study combined 5,901 Easy pseudo-labels, 860 expert-annotated Medium cases, and 198 Hard cases to evaluate annotation enrichment across nine cell-segmentation model configurations.
Source: §1 · p. 2Results
Across nine configurations, at least one strategy improved F1 over baseline per model; LSP-DETR reached 0.8725 F1 with Hard-only, while StarDist rose from 0.7380 to 0.8332 with Medium-only.
Source: §6 · p. 10Takeaways & Limitations
Medium data featured in seven of nine optimal strategies, but no single annotation composition was optimal across all models.
Source: §5 · p. 9, §6 · p. 10Each model–strategy configuration had a single fine-tuning run, so variability from random initialization or stochastic optimization was not quantified.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Accurate nuclei instance segmentation is essential for quantitative renal pathology, yet general-purpose models often struggle with low contrast, dense nuclei, complex morphology, and strong background staining. In this work, we extended a human-in-the-loop framework by combining 5,901 foundation-model-generated pseudo-labels from well-segmented cases (Easy), 860 newly expert-annotated unresolved challenging cases (Medium), and 198 expert-annotated consensus failure cases (Hard). These annotations, spanning different levels of segmentation difficulty, enabled the systematic evaluation of seven single-source and mixed-source fine-tuning strategies across nine cell segmentation model configurations. Fine-tuning improved all models, with Medium data included in seven of the nine best-performing strategies. LSP-DETR achieved the highest F1 score of 0.8725 with Hard-only fine-tuning, while StarDist showed the largest improvement, increasing from 0.7380 to 0.8332 with Medium-only fine-tuning. These findings show that annotations spanning multiple difficulty levels support effective model adaptation, although the optimal annotation composition remains model dependent.
1. INTRODUCTION
Kidney-specific segmentation remains inconsistent on difficult nuclei, motivating domain-specific supervision. This study tests whether newly expert-annotated unresolved Medium cases add value beyond Easy pseudo-labels and Hard expert annotations across models.
1. INTRODUCTION
- Low contrast, unusual morphology, dense nuclei, and strong background staining can remain unresolved by newer general-purpose models in kidney regions unlike their pretraining images.Architectural advances alone do not guarantee adequate segmentation of difficult kidney nuclei.Source: §1 · pp. 1–2
- Earlier evaluations found that CellViT adequately segmented 63.6% of roughly 8,800 kidney patches, while later models left 1,079 patches rated Medium.The remaining Medium cohort was substantially larger than the 198 previously annotated Hard cases.Source: §1 · p. 2
- The study asks whether annotating unresolved Medium cases adds value beyond Easy pseudo-labels and Hard expert annotations, and whether benefits vary across models.The benefit was uncertain despite the substantial expert effort required to annotate Medium cases.Source: §1 · p. 2
- Experts annotated 860 challenging patches from 1,079 unresolved Medium cases, alongside 5,901 Easy pseudo-labels and 198 Hard cases, for evaluation across nine model configurations.The study tests seven single-source and mixed-source strategies; the optimal Easy, Medium, and Hard composition varies by model.Source: §1 · p. 2
2. METHODS
The methods benchmark nine segmentation configurations, construct Easy, Medium, and Hard fine-tuning sources, and rebalance mixed-source training data through weighted oversampling.
2.1 Comprehensive Cell Nuclei Segmentation Model Benchmark
- Nine model configurations released from 2022 to 2026 were evaluated, spanning previously studied models and newly incorporated architectures.The benchmark includes Cellpose-based models, StarDist, CellViT variants, LSP-DETR, and PathoSAM, with differing instance-reconstruction and post-processing approaches.Source: §2.1 · pp. 2–3
2.2 Human-in-the-Loop Annotation and Multi-source Data Enrichment
- Experts selected 860 of 1,079 unresolved Medium-rated patches and manually annotated nuclei from the original images rather than correcting model predictions.Fine-tuning used 5,901 Good-rated Easy patches with foundation-model pseudo-labels, 860 newly annotated Medium patches, and 198 expert-annotated consensus-failure Hard patches.Source: §2.2 · pp. 4–5, Figure 3 · p. 5
2.3 Weighted Oversampling for Annotation Source Rebalancing
- For four mixed-source datasets, source-wise weighted oversampling increased sampling probability for underrepresented sources and reduced majority-source dominance.This increased model exposure to challenging Medium and Hard patches; the rebalancing strength is controlled by γ.Source: §2.3 · p. 5
3. DATA AND EXPERIMENTS
The study fine-tuned nine cell segmentation model configurations using Easy, Medium, and Hard annotations, testing seven source strategies on a common kidney-pathology test set. It evaluated performance with patch-averaged F1 and used source rebalancing for mixed-source datasets.
3. DATA AND EXPERIMENTS
- Fine-tuning and inference used Python 3.12.13, PyTorch 2.12.1, CUDA 12.8, an NVIDIA RTX 5090, and AdamW.Source: §3 · p. 5
3.1.1 Composition of Fine-tuning Data
- The study used 5,901 Easy pseudo-label patches, 860 newly annotated Medium patches, and 198 expert-annotated Hard failure cases.Easy labels came from well-segmented cases; Medium patches were unresolved challenges, and Hard cases were consensus failures.Source: §3.1.1 · p. 6
3.1.2 Test Dataset
- All experiments used the same independent 185-patch expert-annotated test set, excluded from enrichment, validation, and fine-tuning.Source: §3.1.2 · p. 6
3.2 Fine-Tuning Strategies
- Seven single-source and mixed-source strategies were applied across nine configurations, yielding 63 fine-tuning experiments.For mixed datasets, γ was 0.85 for Easy + Medium, Easy + Hard, and Easy + Medium + Hard, and 0.55 for Medium + Hard; these settings increased sampling of underrepresented sources.Source: §3.2 · p. 6
3.3 Evaluation Metrics
- F1 was the primary metric, computed per test patch using an IoU matching threshold of 0.5 and averaged across patches.It jointly accounts for false-positive and false-negative predictions.Source: §3.3 · p. 7
4. RESULTS
Fine-tuning improved all nine model configurations, but the best annotation mix varied by model. Medium data featured in most winning strategies, while qualitative examples showed improved recovery and separation alongside persistent errors in difficult regions.
4.1 Effect of Annotation Strategies Across Model Configurations
- Seven of nine best strategies included Medium data, but no strategy was optimal across all models.Medium-only led four models; Medium + Hard led two, the full mix led one, and Hard-only led two.Source: §4.1 · p. 7, Figure 4 · p. 7
4.2 Overall Fine-Tuning Performance
- StarDist gained 0.0952, rising from 0.7380 to 0.8332 with Medium-only fine-tuning, the largest absolute improvement.Cellpose-SAM had the second-largest gain, 0.0841.Source: §4.2 · p. 8
- 0.8725 was the highest overall F1, achieved by Hard-tuned LSP-DETR, and fine-tuning improved best-achievable F1 for all nine models on the 185-patch hold-out.Baseline F1 ranged from 0.6748 to 0.8339, while best fine-tuned F1 ranged from 0.7378 to 0.8725.Source: Table 4 · p. 8, §4.2 · p. 8
4.3 Qualitative Results
- Fine-tuned predictions better captured weak-contrast, dense, and complex regions, but errors remained for overlapping nuclei and poorly defined boundaries.Source: §4.3 · p. 8
- Figure 5 illustrates recovered missed nuclei and improved instance separation in StarDist, Cellpose-SAM, and LSP-DETR examples.StarDist's Medium-only example recovers nuclei in low-contrast, dense regions; Hard-tuned LSP-DETR improves separation in morphologically complex regions.Source: §4.3 · p. 8, Figure 5 · p. 9
5. DISCUSSION
Medium annotations often improved fine-tuning, but the best annotation mix varied across models. The findings are limited by the renal-only evaluation, fixed hold-out set, and single-run experiments.
5. DISCUSSION
- Medium annotations appeared in seven of nine best-performing strategies, but the optimal annotation mix varied by model.Source: §5 · p. 9
- Medium-only was optimal for StarDist, Cellpose-SAM, and both CellViT configurations; Medium + Hard was optimal for CellViT++ [Virchow] and PathoSAM.Source: §5 · p. 9
- Hard-only was optimal for Cellpose-DINO and LSP-DETR, while Cellpose 3.0 performed best with Easy + Medium + Hard.Source: §5 · p. 9
- Evaluation was limited to renal pathology and a fixed 185-patch kidney hold-out; mixed-source rebalancing was predefined rather than optimized for each architecture.Source: §5 · p. 9
- Each configuration had one fine-tuning run, leaving random-initialization and optimization variability unquantified; future work could test adaptive, model-specific data selection.Source: §5 · p. 10
6. CONCLUSION
Fine-tuning strategies improved F1 across all nine renal nuclei segmentation model configurations, though the best annotation composition varied. Medium data featured in most optimal strategies, while LSP-DETR and StarDist achieved the reported highest F1 and largest improvement, respectively.
6. CONCLUSION
- 0.8725 was the highest F1, achieved by LSP-DETR with Hard-only fine-tuning; StarDist had the largest improvement, from 0.7380 to 0.8332 with Medium-only fine-tuning.Source: §6 · p. 10
- At least one fine-tuning strategy improved F1 over baseline for every model configuration, and Medium data appeared in seven of nine optimal strategies.Source: §6 · p. 10
- The evaluation compared seven single-source and mixed-source strategies using Easy pseudo-labels, newly annotated Medium cases, and expert-annotated Hard failures across nine configurations.Source: §6 · p. 10