Source-linked AI summary
QCell: Recombining and Aligning Cell Queries for Overlapping Instance Segmentation
Yaroslav Prytula, Anton Popov, Dmytro Fishman
TL;DR
Overlapping microscopy cells produce weak boundaries and mixed visual evidence, while existing approaches often lack global reasoning across complete overlapping objects. QCell decomposes and recombines query representations and aligns overlapping-cell queries contrastively, introducing an Organoids benchmark. It outperforms compared methods across multiple benchmarks, including +2.2 AP and +2.7 AJI on ISBI2014.
Problem
Overlapping semi-transparent cells create weak boundaries and mixed visual evidence, while existing methods typically rely on local RoI features or upstream predictions rather than jointly reasoning over the full scene.
Method
QCell decomposes each query into amodal, visible, and invisible representations, recombines them with consistency regularization, and uses DN-guided contrastive and alignment losses to separate overlapping-cell queries.
Results
+2.2 AP and +2.7 AJI over MaskDINO on ISBI2014, with QCell outperforming all compared methods on that benchmark.
Takeaways & Limitations
QCell provides a query-based approach that models complete object structure and distinctive instance representations for overlapping cell segmentation, alongside a new Organoids benchmark.
Takeaways & Limitations
The paper identifies single-mask prediction and the lack of explicit supervision for discriminative instance features as limitations in overlapping cell segmentation.
Abstract
from arXiv · showhide
Instance segmentation of overlapping cells in microscopy remains challenging due to semi-transparent structures that produce weak boundaries and mixed visual evidence in overlap regions. Existing methods address this through local regions of interest or shape priors but lack global reasoning across overlapping objects. We present QCell, a novel query-based model that de-overlaps cell instances in microscopy scenes. Our approach combines (i) an instance recombination module that decomposes and recombines query representations in latent space, enabling the model to reason about complete object structure under overlap, and (ii) a contrastive query alignment objective that combines distinctive instance feature learning and separation of overlapping cell queries. We additionally introduce a new Organoid dataset benchmark for overlapping cell segmentation. We show that QCell outperforms state-of-the-art methods across multiple benchmarks, achieving +2.2 AP and +2.7 AJI on ISBI2014. Code is available at https://github.com/SlavkoPrytula/QCell
5 Better Medicine OÜ, Tartu, Estonia
Figure 1 presents overlapping cells as producing weak boundaries and ambiguous visual evidence, while QCell better preserves complete structures and separates neighboring instances.
- QCell better preserves complete structures and separates neighboring instances than existing methods.
Introduction
Overlapping microscopy cells create ambiguous, entangled visual evidence that challenges complete instance segmentation. QCell addresses this with query-level structural decomposition and recombination, DN-guided contrastive query learning, and a new Organoids benchmark.
- Motivation: Overlapping microscopy cells create weak boundaries and mixed visual evidence because semi-transparent regions contain information from multiple instances.
- Motivation: Existing approaches use region decomposition, shape priors, or generative completion but typically rely on local RoI features or upstream predictions.
- QCell: QCell decomposes each instance query into amodal, visible, and invisible representations, then recombines them with consistency regularization.
- QCell: Contrastive query learning uses stable denoising representations with instance-discriminative and alignment losses to separate overlapping-cell queries in embedding space.
- Contributions: The paper introduces a new Organoids benchmark for evaluating overlapping cell instance segmentation.
2 Related Work
Related work spans structural decomposition, shape priors, query-based segmentation, biomedical query models, and contrastive learning for instance discrimination. QCell combines these directions in a global query-based framework for overlapping cells.
- Overlapping instance segmentation: Prior overlapping-instance methods include two-stage detection frameworks, region-level decomposition, shape priors, and multi-stage generative completion.
- Query-based segmentation: Transformer query models reformulate detection and segmentation around image-attending queries, with denoising training and mask prediction extending this framework.
- Biomedical segmentation: Biomedical methods such as IAUNet and PCTrans learn object-specific or discriminative query representations, but PCTrans does not explicitly model overlapping object structure.
- Contrastive learning: Contrastive learning has been used to shape query representations through attraction and repulsion, primarily for category-level discrimination.
- QCell: QCell’s overview combines query-level instance recombination with contrastive query learning using matched and denoising queries.
3 Method
QCell extends MaskDINO with objectives that model complete cell structure under overlap and keep overlapping-instance queries distinguishable. It decomposes queries into structural components, recombines them with consistency regularization, and adds DN-guided contrastive learning.
- Overview: QCell extends MaskDINO with complementary objectives for object-structure modeling and query discrimination in dense overlapping scenes.
- Instance Recombination: The model decomposes each query into amodal, visible, and invisible representations, whose masks are predicted from dot products with pixel features.
- Instance Recombination: The three sub-queries are fused into a refined full-instance embedding supervised against the full amodal mask, encouraging complementary information across visible and hidden regions.
- Instance Recombination: Consistency regularization constrains the refined mask to be recoverable from the union of visible and invisible predictions, using a stop-gradient pseudo-target so gradients flow only through the refined prediction.
- Contrastive Query Learning: DN-guided contrastive learning aligns matched queries with same-instance denoising representations while separating them from other instances, addressing representational collapse under heavy overlap.
- Contrastive Query Learning: The latent alignment loss reinforces identity consistency and pushes queries for different instances toward orthogonality across multiple decoder layers.
4 Experiments
QCell is evaluated on three overlapping-cell microscopy datasets, including the novel Organoids benchmark, with comparisons, qualitative analyses, and component ablations. It achieves the strongest reported performance across the benchmarks and shows complementary gains from instance recombination and contrastive query learning, particularly under severe overlap.
- Datasets and evaluation: QCell is evaluated on ISBI2014, Revvity-25, and the novel Organoids benchmark, spanning different imaging modalities, object counts, and overlap challenges.ISBI2014 evaluation uses cytoplasm annotations, where semi-transparent overlap is most prevalent; Organoids contains dense scenes with up to 105 training instances and an average of 96 test instances.
- Main results: 65.9 AP, 92.3 F1, and 78.6 AJI: QCell achieves the best reported ISBI2014 cytoplasm segmentation performance, improving over MaskDINO by +2.2 AP, +2.3 F1, and +2.7 AJI.The reported gains occur where neighboring cells share weak or semi-transparent boundaries.
- Main results: 52.9 AP, 86.4 F1, and 73.6 AJI: QCell achieves the best AP and AJI among compared methods on Revvity-25.The qualitative comparison reports cleaner segmentations with fewer false positives in dense regions, while preserving fine cell structures and boundaries.
- Main results: 51.0 AP, 71.6 F1, and 63.2 AJI: QCell achieves the best overall results on the densely populated Organoids benchmark.Figure 3 describes improved object consistency in dense overlapping scenes.
- Ablation studies: Instance recombination improves complete-object reasoning by decomposing queries into amodal, visible, and invisible parts, while consistency regularization produces more coherent overlapping-region predictions.Adding the instance recombination loss raises the MaskDINO baseline from 63.7 to 65.1 AP; consistency regularization further reaches 66.6 AP.
- Ablation studies: 13.68 AP and 10.97 AP75: combining instance recombination and contrastive query learning gives the strongest result under severe overlap, with gains of +2.01 AP and +3.33 AP75 over baseline.The severe-overlap subset contains instances with ground-truth pairwise IoU ≥0.5 with another instance.
5 Conclusions
QCell addresses overlapping cell instance segmentation through complementary structural and representational mechanisms. Across ISBI2014, Revvity-25, and Organoids, it achieves state-of-the-art performance while introducing a dense brightfield microscopy benchmark.
- 5 Conclusions: QCell de-overlaps semi-transparent cells using latent-space instance recombination with consistency regularization and DN-guided contrastive query alignment.The alignment objective combines instance-discriminative and cosine alignment losses to learn distinctive, separated instance features.
- 5 Conclusions: The Organoids dataset provides a new benchmark for overlapping cell instance segmentation in dense brightfield microscopy scenes.Experiments on ISBI2014, Revvity-25, and Organoids demonstrate state-of-the-art overlapping cell segmentation performance.