Source-linked AI summary
The Multi-modality Cell Segmentation Challenge: Towards Universal Solutions
Jun Ma, Ronald Xie, Shamini Ayyadhury, Cheng Ge, Anubha Gupta, Ritu Gupta, Song Gu, Yao Zhang, Gihun Lee, Joonkee Kim, Wei Lou, Haofeng Li, Eric Upschulte, Timo Dickscheid, José Guilherme de Almeida, Yixin Wang, Lin Han, Xin Yang, Marco Labagnara, Vojislav Gligorovski, Maxime Scheder, Sahand Jamal Rahi, Carly Kempster, Alice Pollitt, Leon Espinosa, Tâm Mignot, Jan Moritz Middeke, Jan-Niklas Eckardt, Wangkai Li, Zhaoyang Li, Xiaochen Cai, Bizhe Bai, Noah F. Greenwald, David Van Valen, Erin Weisbart, Beth A. Cimini, Trevor Cheung, Oscar Brück, Gary D. Bader, Bo Wang
TL;DR
Cell segmentation must work across varied microscopy modalities, yet existing methods are often modality-specific or require manual configuration. This work introduces a diverse benchmark and challenge for universal automatic segmentation, and reports that Transformer-based and other top algorithms outperform prior methods while generalizing across imaging settings without user intervention.
Problem
Existing cell segmentation methods are often tailored to specific microscopy modalities, trained on limited image diversity, or dependent on manual model and channel selection.
Method
The paper organizes a multi-modality challenge with diverse labeled and unlabeled data, standardized Docker evaluation, and algorithms designed to segment varied images without manual parameter tuning.
Results
Transformer-based algorithms surpassed previous state-of-the-art methods by a large margin and generated accurate cell contours across diverse microscopy images without user intervention.
Takeaways & Limitations
The challenge provides a proof of concept for generalist cell segmentation algorithms, with top methods released through user-friendly interfaces for microscopy analysis.
Takeaways & Limitations
The benchmark is limited to 2D microscopy segmentation and does not currently support interactive user feedback or classification tasks.
Abstract
from arXiv · showhide
Cell segmentation is a critical step for quantitative single-cell analysis in microscopy images. Existing cell segmentation methods are often tailored to specific modalities or require manual interventions to specify hyper-parameters in different experimental settings. Here, we present a multi-modality cell segmentation benchmark, comprising over 1500 labeled images derived from more than 50 diverse biological experiments. The top participants developed a Transformer-based deep-learning algorithm that not only exceeds existing methods but can also be applied to diverse microscopy images across imaging platforms and tissue types without manual parameter adjustments. This benchmark and the improved algorithm offer promising avenues for more accurate and versatile cell analysis in microscopy imaging.
Introduction
Cell segmentation is essential across diverse biological imaging settings, but existing methods often lack robust generalization and require manual model or channel selection. The challenge addresses these limitations by evaluating universal, automatic algorithms on diverse microscopy data and unseen experiments.
- Motivation: Cell segmentation supports biological image analysis across many experimental settings and imaging modalities.It is a prerequisite for identifying tumor sub-types, composition, and organization in multiplexed fluorescence cancer-microenvironment analysis.
- Limitations of existing methods: Microscopy diversity arises from differences in cell origins, microscopy types, staining techniques, and cell morphologies, making universal automatic segmentation difficult.Specialized methods have demonstrated effectiveness for particular image or cell types, but selecting an appropriate model remains challenging for non-experts.
- Limitations of existing methods: Existing generalized algorithms were trained on relatively narrow image types and struggle with RGB microscopy images such as Jenner-Giemsa-stained bone marrow slides.They also commonly require users to choose the model type and image channel manually.
- Challenge design: The challenge evaluates algorithms intended to handle a broad spectrum of microscopy images with one model and without manual intervention.Its testing set includes images from new biological experiments to assess generalization to previously unseen data, along with whole-slide images for large-scale evaluation.
- Challenge design: The benchmark uses a development phase with labeled and unlabeled images, online tuning evaluation, and a separate Docker-based holdout test for final ranking.The development resources include 1000 labeled and 1725 unlabeled images, while the holdout testing set contains 422 images.
Results
The challenge assembled a diverse multi-modality benchmark and evaluated 28 algorithms on a hidden holdout set, where Transformer-based methods achieved leading, stable performance across microscopy settings. The best methods also outperformed established generalist segmentation algorithms on whole-test, modality-specific, and unseen-image evaluations.
- Benchmark design: The holdout evaluation compared 28 algorithms on 422 microscopy images using accuracy, efficiency, significance, ranking stability, and ranking-agreement analyses.The winning algorithm was evaluated through pairwise Wilcoxon tests, 1000 bootstrap samples, multiple ranking schemes, and Kendall’s tau.
- Challenge results: 89.7% median F1 was achieved by T1-osilab, exceeding the second- and third-ranked algorithms at 84.5% and 84.4%, respectively.T1-osilab also had fewer outliers in its score distribution, indicating more robust performance across testing cases.
- Challenge results: The winning algorithm ranked first in all 1000 bootstrap samples and across five ranking schemes, while the second- and third-ranked algorithms were significantly better than other teams.The second and third algorithms did not differ significantly from each other.
- Cross-modality comparison: The best-performing algorithms maintained superiority across brightfield, fluorescent, phase-contrast, and DIC images, and exceeded established generalist methods on the full testing set.On the full testing set, T1 algorithms achieved a median F1 of 89.7% and significantly outperformed KIT-GE, Cellpose, and Omnipose variants.
- Unseen-image evaluation: 95.0% median F1 was achieved by the top algorithm on a post-challenge set of unseen images, while fine-tuned Cellpose and Omnipose exceeded scratch-trained variants but remained below their original pretrained models.The authors attribute the latter discrepancy largely to catastrophic forgetting during fine-tuning on new sources.
Discussion
The challenge identified Transformer-based segmentation as substantially stronger than existing methods and highlighted strategies for preserving generalization and handling diverse microscopy data. It also exposed limits in unlabeled-data utilization, practical deployment, and current task scope.
- Transformer-based algorithms significantly outperformed existing state-of-the-art cell segmentation methods.The authors attribute this advantage to self-attention, larger model capacity, and transfer-learning capabilities.
- The winning algorithm surpassed the leading CTC algorithm even after retraining it on the challenge dataset.The authors associate this gap with the challenge dataset’s greater diversity and its emphasis on universal rather than dataset-specific models.
- Top-performing methods used multi-head outputs that converted instance segmentation into distance-map regression and foreground semantic segmentation before post-processing.This design was particularly important for whole-slide images because of RAM and GPU-memory constraints.
- Top teams explored consistency regularization, pseudo-label learning, and uncertainty-aware training to use unlabeled images, but none produced notable segmentation gains.The authors leave effective use of unlabeled data as an open question.
- Cell memory replay addressed catastrophic forgetting during fine-tuning by mixing images from existing and new datasets in training batches.This strategy aimed to retain performance on known images while learning from novel images.
- The algorithms were released publicly, but applying them to new images can still require basic computational expertise.The work identifies integration into routine biological practice as an ongoing challenge.
- The benchmark’s current scope excludes 3D microscopy, interactive user feedback, and classification tasks.The authors propose expanding future benchmarks toward 3D images, classification, and biologist-in-the-loop systems.
- The authors conclude that generalist segmentation can generate accurate cell contours across varied microscopy images without user intervention and may accelerate quantitative biological research.They also report that top algorithms are open-source and integrated into user-friendly interfaces.
Author Contributions Statement
The author-contribution statement assigns analysis design, data work, tool development, challenge administration, analysis, and manuscript responsibilities across named contributors. The statement also records competing interests and notes that other authors reported none.
- J.M. led analysis design, data collection and cleaning, tool contributions, challenge registration and evaluation, analysis, and manuscript preparation.J.M. also revised the manuscript.
- R.X. contributed to analysis design, challenge registration and evaluation, and manuscript revision.
- S.A., C.G., A.G., R.G., S.G., and Y.Z. contributed to analysis design, data cleaning, labeled images, and challenge administration.The supplied statement ends during the description of their contributions.
- Several authors disclosed employment, ownership, consultancy, grant, or equity interests related to biomedical companies, while all other authors reported no competing interests.The disclosures include Song Gu’s employment, Cancilico ownership, and David Van Valen’s company leadership and equity.
Methods
The challenge evaluated fully automatic cell-segmentation algorithms across diverse microscopy settings using accuracy, efficiency, and ranking-stability criteria. Leading approaches combined heterogeneous-image modeling, modality-specific classification, contour representations, and transparent comparative evaluation.
- Best-performing algorithm: The leading algorithm used a SegFormer encoder-decoder with skip connections, combining model-centric and data-centric strategies for heterogeneous microscopy modalities.Its inference used sliding windows and importance maps to prevent duplicate detections at patch boundaries.
- Second-best-performing algorithm: The second-best algorithm classified images into four groups before applying group-specific ConvNeXt-based segmentation models.Classification used pseudo-labels, image intensities, HSV features, cell area, and a ResNet18 classifier.
- Third-best-performing algorithm: The third-best algorithm treated instance segmentation as sparse contour detection, using pixel-anchored contour regression to recover partially superimposed objects.Its contour representation also supported inductive shape priors for challenging conditions.
- Evaluation: The evaluation balanced segmentation accuracy with efficiency to discourage impractical ensemble-heavy solutions for users with limited computational resources.F1 score measured accuracy, while efficiency accounted for deployment considerations.
- Evaluation: The benchmark used F1 score with IoU threshold 0.5, excluding boundary cells from metric computation because they are often incomplete and less useful downstream.Predictions were categorized as true positives, false positives, or false negatives to compute precision, recall, and F1.
- Ranking and statistical analysis: The evaluation code and ranking scheme were public, and the winning algorithm remained first under multiple alternative ranking schemes.Ranking stability was further assessed with bootstrap samples and Kendall’s τ, while Wilcoxon signed-rank tests evaluated paired algorithm comparisons.