PaperlayerView exact revision on arXiv

Source-linked AI summary

The Multi-modality Cell Segmentation Challenge: Towards Universal Solutions

Jun Ma, Ronald Xie, Shamini Ayyadhury, Cheng Ge, Anubha Gupta, Ritu Gupta, Song Gu, Yao Zhang, Gihun Lee, Joonkee Kim, Wei Lou, Haofeng Li, Eric Upschulte, Timo Dickscheid, José Guilherme de Almeida, Yixin Wang, Lin Han, Xin Yang, Marco Labagnara, Vojislav Gligorovski, Maxime Scheder, Sahand Jamal Rahi, Carly Kempster, Alice Pollitt, Leon Espinosa, Tâm Mignot, Jan Moritz Middeke, Jan-Niklas Eckardt, Wangkai Li, Zhaoyang Li, Xiaochen Cai, Bizhe Bai, Noah F. Greenwald, David Van Valen, Erin Weisbart, Beth A. Cimini, Trevor Cheung, Oscar Brück, Gary D. Bader, Bo Wang

arXiv:2308.05864v2eess.IVcs.CVcs.LGq-bio.QM

TL;DR

Cell segmentation remains challenging across microscopy settings because existing methods may be modality-specific or require manual parameter choices. This paper presents a multimodality benchmark and challenge whose Transformer-based algorithm surpassed previous state-of-the-art methods by a large margin across diverse microscopy images.

  • Problem

    Universal automatic cell segmentation remains challenging because microscopy images vary widely, while generalized methods have limited modality diversity and can require manual model and channel selection.

    Source: Introduction · p. 3

  • Method

    The paper organized a challenge with diverse training and tuning sets and a holdout test set, producing a winning Transformer-based segmentation algorithm.

    Source: Introduction · p. 4, Discussion · p. 18

  • Results

    The Transformer-based algorithm surpassed previous state-of-the-art methods by a large margin and generated accurate cell contours across a wide range of microscopy images without user intervention.

    Source: Discussion · pp. 16, 18

  • Takeaways & Limitations

    The challenge results provide a successful proof of concept for generalist cell segmentation algorithms.

    Source: Discussion · p. 18

    The benchmark was confined to 2D microscopy images and omitted classification tasks.

    Source: Discussion · p. 18

Abstract

from arXiv · show

Cell segmentation is a critical step for quantitative single-cell analysis in microscopy images. Existing cell segmentation methods are often tailored to specific modalities or require manual interventions to specify hyper-parameters in different experimental settings. Here, we present a multi-modality cell segmentation benchmark, comprising over 1500 labeled images derived from more than 50 diverse biological experiments. The top participants developed a Transformer-based deep-learning algorithm that not only exceeds existing methods but can also be applied to diverse microscopy images across imaging platforms and tissue types without manual parameter adjustments. This benchmark and the improved algorithm offer promising avenues for more accurate and versatile cell analysis in microscopy imaging.

Introduction

Cell segmentation is essential across biological imaging, but variation in cells, microscopy, staining, and morphology makes universal automatic methods difficult. The paper introduces a challenge designed to develop efficient algorithms that generalize across microscopy modalities and unseen images without manual intervention.

Introduction

  • Cell segmentation is a prerequisite for biological image analysis, but variation in cell origins, microscopy, staining, and morphology complicates universal automation.In multiplexed fluorescence cancer-microenvironment analysis, segmentation enables identification of tumor subtypes, composition, and organization.Source: Introduction · p. 3
  • Generalized algorithms trained mainly on grayscale and two-channel fluorescence images may fail on RGB images and often require manual model and channel selection.Bone marrow aspirate slides stained with Jenner-Giemsa are an example of an RGB image type on which these models have struggled.Source: Introduction · p. 3
  • Transfer learning offers a complementary route to customized segmentation; Cellpose 2.0 adapts a pretrained model to new images.It requires 500–1,000 user-annotated image patches to reach performance on par with models trained on thousands of patches.Source: Introduction · pp. 3–4
  • Earlier competitions focused on limited image types or nucleus segmentation, while accuracy-focused evaluation encouraged resource-intensive solutions that hinder deployment.Examples include dataset-specific top-performing CTC models and the DSB winning algorithm's ensemble of 32 models.Source: Introduction · p. 4
  • The NeurIPS challenge tested whether one algorithm could efficiently segment diverse microscopy images and generalize to new experiments without manual intervention.Participants received diverse training and tuning sets, then submitted Docker containers for evaluation on a holdout set that included two whole-slide images (Figure 1).Source: Introduction · p. 4, Figure 1 · p. 5

Results

On the 422-image holdout set, T1-osilab achieved the highest median F1 score and outperformed established segmentation methods. The challenge evaluated algorithms across diverse microscopy modalities, experiments, and cell types, including previously unseen test data.

Challenge design: towards universal and efficient cell segmentation algorithms

  • The challenge provided 1000 annotated development images, optionally supplemented by 1725 unlabeled images, and used a hidden 422-image holdout for final evaluation.Participants could validate against a 101-image tuning set through an online leaderboard; the top 30 teams were invited to submit Dockerized algorithms, and 28 submitted successfully. Accuracy and runtime were recorded for final ranking.Source: Challenge design: towards universal… · p. 6

Challenge data: a large and diverse multi-modality microscopy image dataset

  • The dataset comprised 1000 training images from over 20 laboratories and more than 50 experiments, spanning brightfield, fluorescent, phase-contrast, and DIC microscopy.It included 300 brightfield, 300 fluorescent, 200 phase-contrast, and 200 DIC images; testing drew on new experiments, including previously unseen tissues or cell types, and contained 422 images. The dataset encompassed more labeled cells and greater diversity than earlier cell- and nucleus-segmentation challenge datasets [14].Source: Challenge data: a large and diverse… · p. 8

Algorithm overview: the Transformer-based algorithm achieved superior performance

  • The top three teams used distinct encoder-decoder designs and training strategies, while T1-osilab’s best-performing framework combined a hierarchical Transformer with multiscale attention.T1 used cell-recognition and distinction heads, public-image pretraining, challenge-data fine-tuning with cell-aware augmentation, and cell memory replay. T2 used four category-specific U-Net-like models with ConvNeXT and morphology-specific decoder heads; T3 used a ResNeXt-101 contour-proposal network with uncertainty-aware suppression. All three normalized intensities and trained with patch sampling; T2 and T3 also used unlabeled data for pretraining.Source: Algorithm overview: the… · pp. 9–10
  • On 422 holdout images, the top three algorithms had median F1 scores of 89.7% (IQR: 84.1-94.8%), 84.5% (IQR: 70.6-92.3%), and 84.4% (IQR: 77.4-91.1%).T1-osilab was significantly better than every other team and ranked first in every bootstrap sample and ranking scheme. It took approximately 2 seconds per 1000 × 1000 image; median maximum GPU memory consumption was 3099MB.Source: Figure 3 · p. 11, Algorithm overview: the… · p. 12

The best-performing algorithms outperform state-of-the-art cell segmentation algorithms

  • T1-osilab achieved median F1 89.7%, beating KIT-GE, Cellpose-pretrain, Cellpose-scratch, Omnipose-pretrain, and Omnipose-scratch by 49.9%, 24.4%, 35.4%, 58.9%, and 48.7%, respectively.The top three challenge algorithms achieved significantly higher accuracy than the existing state-of-the-art methods on the whole testing set.Source: The best-performing algorithms… · p. 13, Figure 4 · p. 14
  • The top three algorithms led on phase-contrast and DIC images, with median F1 scores of 93.6%, 88.8%, and 90.3% on phase-contrast and 86.8%, 75.0%, and 80.3% on DIC.KIT-GE excelled on phase-contrast images but struggled on low-contrast DIC images from new experiments; the top three methods remained ahead of the state-of-the-art methods on DIC.Source: Figure 4 · p. 14, The best-performing algorithms… · p. 15
  • On brightfield images, the top two algorithms achieved median F1 scores of 91.4% and 91.0%, while T3 led fluorescent images at 80.8%.Cellpose-scratch improved over Cellpose-pretrain on brightfield but declined on other modalities with previously unseen images; the Cellpose-scratch and Omnipose-scratch declines were 44.1% and 7.4%, respectively, attributed to new cell types.Source: The best-performing algorithms… · pp. 13, 15, Figure 4 · p. 14
  • On the post-challenge set of unseen images, the top three algorithms achieved median F1 scores of 95.0%, 88.7%, and 93.3%.Fine-tuned Cellpose and Omnipose exceeded their scratch-trained counterparts by 12.2% and 11.7%, respectively, but remained below their original pretrained models, consistent with catastrophic forgetting.Source: Figure 4 · p. 14, The best-performing algorithms… · p. 15

Discussion

The Transformer-based winner substantially outperformed existing cell-segmentation methods, supporting generalist segmentation across diverse microscopy images without user intervention. The discussion also identifies strategies, limitations, and accessibility measures that shape the algorithms’ practical use.

Discussion

  • The Transformer-based winner significantly outperformed existing state-of-the-art segmentation methods and generated accurate cell contours across microscopy images without user intervention.The authors highlight self-attention for global context and long-range dependencies, greater model capacity for nuanced patterns, and transfer learning; the winning algorithm used this strategy.Source: Discussion · pp. 16, 18
  • The winner outperformed the CTC challenge leader even after retraining on this dataset; the authors link the gap partly to greater data diversity and a universal rather than dataset-tailored strategy [37].Unlike the CTC dataset, the challenge included both labeled and label-free images.Source: Discussion · p. 16
  • Top-performing approaches used multi-head distance-map regression and foreground segmentation, robust augmentation, efficient backbones, and slide-window inference for whole-slide images.Multi-head outputs followed by post-processing outperformed detection-then-segmentation. Augmentations included cell-wise intensity perturbations by Lee et al. and Mosaic augmentation by Li et al.; patch-based inference helps address RAM and GPU memory limits.Source: Discussion · pp. 16–17
  • Methods using unlabeled images—including consistency regularization, reconstruction, pseudo-label learning, and Listen2Student—showed no notable segmentation improvement, leaving effective use of unlabeled data unresolved.T1-osilab used consistency regularization and reconstruction; T1-osilab and T2-sribdmed [30] explored pseudo-label learning, while T3-cells used uncertainty-aware Listen2Student [28].Source: Discussion · p. 17
  • Fine-tuning generalist Cellpose and Omnipose models caused catastrophic forgetting of unseen-image capabilities, while the winner’s cell memory replay [5] retained performance on existing and new images.Replay combined images from the existing and new datasets in fine-tuning mini-batches; catastrophic forgetting is cited as.Source: Discussion · p. 17
  • The top teams released code and integrated their algorithms into Napari and Docker containers, lowering barriers to use; the authors aim to sustain the benchmark and expand its data diversity.Napari enables application without additional coding, and a single-line command can initiate batch segmentation in the Docker containers.Source: Discussion · pp. 17–18
  • Potential applications include characterizing breast-tumor phenotypes, discovering dermal-fibroblast regulatory mechanisms, and studying cancer microenvironments in MIBO-TOF tissue microarrays.The cited examples use cell segmentation for phenotyping, single-cell molecule quantification, and disease-progression research, respectively.Source: Discussion · pp. 17–18
  • The challenge covered only 2D segmentation, lacked interactive user feedback, and omitted classification; future work should include complex 3D images, classification, and biologist-in-the-loop systems.The authors note that 3D images pose challenges including large volume and anisotropic resolution.Source: Discussion · p. 18

Author Contributions Statement

The statement assigns analysis, data, algorithm, and manuscript contributions across the author groups and reports competing interests for several authors, while noting that all others have none.

Author Contributions Statement

  • J.M., R.X., and other authors led analysis and challenge work, while additional groups developed algorithms, contributed data, and revised the manuscript.J.M. and R.X. conceived and designed the analysis and managed challenge registration and evaluation; J.M. also collected and cleaned data, performed analyses, and wrote the initial manuscript. Other contributors supplied labeled or public/unlabeled data, developed the top-three or other algorithms, and/or made code publicly available; G.D.B. and B.W. conceived and designed the analysis and wrote and revised the manuscript.Source: Author Contributions Statement · p. 19

Competing Interests Statement

  • Disclosed interests include employment, company ownership or equity, consultancy fees, research grants, and stock ownership; all other authors report no competing interests.Song Gu is employed by Nanjing Anke Medical Technology Co., Ltd.; Jan Moritz Middeke and Jan-Niklas Eckardt co-own Cancilico; David Van Valen co-founded Barrier Biosciences, serves as its Chief Scientist, and holds equity. Oscar Brück reports consultancy fees and research grants outside the submitted work; the statement also notes stock ownership in Hematoscope Ltd. outside the submitted work.Source: Competing Interests Statement · pp. 19–20

Methods · Methods-only references

The methods describe how the challenge was organized, how its datasets and competing algorithms were prepared, and how segmentation accuracy, efficiency, and ranking stability were evaluated.

Challenge Organization

  • The challenge launched June 15, 2022, ran for 139 days, and provided participants a U-Net baseline, tutorial, tuning feedback, and method guidance.Participants could submit tuning-set segmentations and receive F1 scores during development.Source: Challenge Organization · p. 26

Dataset curation and pre-processing

  • The dataset combined publicly sourced and newly annotated images with challenge-acquired, withheld test images, and included more than 900,000 new cell annotations.Two experienced biologists checked annotations; images with fewer than five cells and cells smaller than 15 pixels were excluded. Post-challenge yeast and bacteria images tested performance on unseen morphologies.Source: Dataset curation and pre-processing · pp. 26–27

Top three best-performing algorithms

  • Lee et al. combined SegFormer–MA-Net, cell-aware augmentation, external-data pretraining, and oversampling of underrepresented modalities.The network predicted cell-probability maps and gradient flows for post-processing; sliding-window inference with an importance map supported whole-slide images and reduced duplicate boundary detections.Source: Top three best-performing algorithms · pp. 27–28
  • Lou et al. [30] classified images into four groups for group-specific segmentation, while Upschulte et al. used contour proposals to detect and separate instances.Lou et al. used ConvNeXt-Stardist for two groups and ConvNeXt-Hover for the others; Upschulte et al. added boundary supervision and uncertainty-aware proposal selection.Source: Top three best-performing algorithms · pp. 28–29

Evaluation metrics

  • The F1 metric matched predicted and ground-truth masks at IoU > 0.5, while running time measured efficiency on a shared workstation with size-based time tolerance.Boundary cells were excluded from metric calculations; F1 was computed from true positives, false positives, and false negatives.Source: Evaluation metrics · pp. 30–31

Ranking Scheme: Rank-then-aggregate

  • The rank-then-aggregate scheme ranked teams separately on F1 and running time for each test case, then averaged and normalized their ranks.The authors also compared alternative ranking schemes, including score aggregation and test-based ranking.Source: Ranking Scheme: Rank-then-aggregate · p. 31
  • Across alternative ranking schemes, most algorithms changed rank, but the winner consistently remained first.The evaluation code and ranking scheme were public at the challenge’s start.Source: Ranking Scheme: Rank-then-aggregate · p. 31

Ranking Stability and statistical analysis

  • Ranking stability was assessed using 1,000 bootstrap samples and Kendall’s τ; paired algorithm comparisons used Wilcoxon signed-rank tests with significance at p < 0.05.Kendall’s τ quantified agreement between rankings; values closer to 1 indicate greater stability.Source: Ranking Stability and statistical… · p. 32

Related papers

Loading 2308.05864v2…