Source-linked AI summary

End-to-End Cell Detection via Instance-aware Graph Modeling

Ruochen Liu, Yalin Zheng, Jingxin Liu, Jianfeng Zhang, Shoujun Huang, Dexing Kong, Haofeng Li, Wei Lou

arXiv:2609.15354v1cs.CVcs.AI

TL;DR

Graph-based pathology methods often separate visual extraction from relational modeling, while query-built cell graphs can contain redundant or spurious nodes. This paper jointly models patch-level visual features and instance interactions with learnable-query graph construction and an instance-aware graph network; visualizations report more detected cells and fewer false positives, especially for dense, adherent cells.

  • Problem

    Graph-based pathology methods often separate visual extraction from relational modeling, limiting end-to-end joint optimization.

    Source: §5 · p. 13
  • Method

    The framework builds a dynamic graph over learnable-query cell instances using spatial and feature cues, then applies an instance-aware graph network with topology as a latent state.

    Source: §1 · p. 3, §5 · p. 13
  • Results

    Visual comparisons show more detected cells and fewer false positives, particularly for densely packed and adherent cells.

    Source: §4.4.1 · p. 10
  • Takeaways & Limitations

    The graph topology adapts with query embeddings rather than remaining fixed a priori by spatial heuristics.

    Source: §5 · p. 14

Abstract

from arXiv · show

Accurate cell detection and classification are crucial for pathological analysis, directly affecting diagnostic accuracy and treatment planning. To capture complex cellular interactions beyond visual appearance within the tumor microenvironment, several approaches have employed graph neural networks to model spatial and relational patterns among cell nuclei, yielding promising results. However, these methods typically adopt a two-stage paradigm of visual extraction followed by relational modeling, which necessitates separate tuning for each stage, thereby increasing pipeline complexity and hindering end-to-end joint optimization. In this paper, we propose an end-to-end framework for cell detection and classification that jointly models patch-level visual representations and instance-level interactions, which incorporates a dynamic graph construction module and an instance-aware graph network. Specifically, the graph construction module dynamically builds the graph structure using learnable queries derived from patch-level features as cell instance representations, with adjacency defined by integrating feature similarity and spatial distances. The instance-aware graph network performs adaptive instance filtering and feature reorganization, aggregating them over the cell graph into a topological latent state for a selective state-space transition driven by visual cues, fusing appearance and relational evidence. When evaluated on multiple datasets with different staining protocols for cell and nucleus detection, our method significantly outperforms existing approaches in both detection and classification performance. The code will be released at https://github.com/RuochenLiu23/IGM.

1. Introduction

Cell detection must handle dense, overlapping, and heterogeneous tissue, while existing instance and graph methods leave gaps in instance-level relational modeling and end-to-end optimization. The paper proposes a query-based end-to-end framework that dynamically builds cell graphs and models their topology with an instance-aware network.

  • Bounding-box instance detection balances richer localization than point methods with lower annotation and computational overhead than segmentation.Source: §1 · p. 2
  • Query-based detectors can struggle in dense or heterogeneous regions, and pathology-specific geometric methods require pixel-level annotations or decoupled training.Source: §1 · p. 2
  • Prior two-stage graph methods require separate stage optimization, while feature-map graphs can obscure instance information with dense, noisy structures.Source: §1 · p. 2
  • Learnable queries serve as candidate cell instances and graph nodes, with connectivity jointly determined by spatial proximity and feature similarity.Source: §1 · p. 2
  • The query feature network maps patch-level context to instance-level semantics, while DGC builds the graph and IGN models inter-cell interactions.IGN reorganizes and filters queries before graph aggregation, then uses visual features to drive state updates along the cell topology.Source: §1 · p. 3
  • The framework achieves state-of-the-art performance across multiple public benchmarks for cell and nucleus detection under different staining protocols.Source: §1 · p. 3

2. Related Work

Cell and nucleus detection methods trade off localization detail, annotation demands, and computational cost. Graph-based pathology methods also face challenges in constructing meaningful cell-level graphs and scaling relational modeling.

2.1. Instance Detection for Cells and Nuclei

  • Bounding-box instance detection offers richer localization than point detection while avoiding the annotation and computational overhead of full segmentation.Source: §2.1 · p. 3

2.2. Graph-Based Learning in Pathology

  • Two-stage graph construction derives nodes from detected or segmented cells, whereas end-to-end pixel- or patch-level graphs can contain redundant or biologically irrelevant edges.Source: §2.2 · pp. 3–4
  • GCNs have limited receptive fields and can over-smooth, while standard GAT attention scales quadratically with node count; Mamba offers global modeling at linear complexity.Source: §2.2 · p. 4

3. Method

The framework maps patch-level visual features to instance-level query representations, builds a graph using spatial and feature relationships, then models graph context together with visual evidence. Its components are query feature learning, Dynamic Graph Construction (DGC), and the Instance-aware Graph Network (IGN).

3. Method

  • The full framework maps patch features to instance-level features, builds an adaptive cell graph, and models local and global relational context alongside visual evidence.These components are designed to improve query-feature discrimination for instance separation and class prediction.Source: §3 · p. 4

3.1. Query Feature Learning Network

  • Queries gather multi-scale visual and tissue context, are iteratively refined by a Transformer decoder, and provide spatial reference points for graph construction.Initial proposals are selected after class-agnostic NMS; between decoder layers, another NMS step reduces overlapping queries.Source: §3.1 · pp. 4–5

3.2. Dynamic Graph Construction

  • DGC constructs query-node connections using a distance that combines spatial proximity and feature similarity, allowing graph structure to update with the queries.Spatial distance uses reference points, while feature distance uses L2-normalized query embeddings and cosine similarity.Source: §3.2 · p. 5

3.3. Instance-aware Graph Network

  • IGN combines selective feature reorganization and graph aggregation with a topology-structured state-space layer driven by instance-level query features.CGSE uses all-query context to steer parameters in both SFR and TSS.Source: §3.3 · p. 6

3.3.1. Instance-aware Graph Learning

  • SFR filters redundant queries and maps retained features into directional representations; graph aggregation combines them into a topological representation where dissimilar neighbors can remain non-interfering.The method uses the geometry of representations for selectivity rather than explicit edge pruning.Source: §3.3.1 · pp. 6–7

3.3.2. Topology-structured State Space

  • TSS uses graph-aggregated topology as the incoming state and repeatedly injects visual features, combining relational and appearance evidence in a single state.The resulting features feed classification and bounding-box regression heads; predictions are matched to ground truth with the Hungarian algorithm.Source: §3.3.2 · pp. 7–8

3.3.3. Context-Guided State Encoding.

  • CGSE generates state-space parameters from instance features, with a globally shared A initialized using HiPPO-LegS and B and C learned from cellular knowledge.A low-rank structure is used for Δ to control parameter count.Source: §3.3.3 · p. 8

4. Experiment

The evaluation uses three public histopathology datasets with different tissue types, stains, and annotation settings. Experiments use a shared ResNet-50 backbone and standardize query counts for general DETR-based comparisons.

4.1. Datasets

  • Evaluation spans CoNSeP, CytoDArk0, and OCELOT, covering colorectal adenocarcinoma, mammalian brain tissue, and cell nuclei from TCGA whole-slide images.The datasets include H&E- and Nissl-stained images with nucleus-level or cell-level annotations.Source: §4.1 · p. 8
  • Patch sizes are 128×128 for denser CoNSeP images and 256×256 for CytoDArk0 and OCELOT.Annotations were converted to bounding-box labels for evaluation.Source: §4.1 · p. 8

4.2. Implementation Details

  • Experiments use ResNet-50, and general DETR-based methods share 900 queries for fair comparison.The reported setup also uses an NVIDIA A40 GPU and AdamW with an initial learning rate of 2e-4 and weight decay of 0.05.Source: §4.2 · p. 8

4.3. Evaluation Setup and Metrics

The evaluation compares detection and classification methods using metrics selected for their output types. Detection methods are assessed with detection metrics, while cell-specific classification methods are compared using F1-score.

4.3. Evaluation Setup and Metrics

  • Detection-method comparisons use mAP, mAP50, mAP75, and AR across confidence thresholds.These metrics assess detection and classification performance for detection methods.Source: §4.3 · p. 9
  • Cell-specific classification methods are compared using F1-score because their supervision and output granularity differ from detection methods.The paper follows in using F1-score as a consistent classification measure.Source: §4.3 · p. 9
  • The comparison includes general-purpose and pathology-specific detectors, plus classification methods on the multi-class dataset.The classification comparison is intended to assess performance against pathology-specific models.Source: §4.3 · p. 9

4.4. Comparison with the Existing Methods

The framework leads detection benchmarks across three datasets, with the largest reported margins on CoNSeP; it also achieves state-of-the-art cell-classification performance across the compared methods.

4.4.1. Comparison with Detection Methods

  • On CoNSeP, the method surpasses DDQ-DETR by 2.0% in mAP and 2.2% in AR; gains over DDQ-DETR are 2.1% in mAP and 1.4% in AR on CytoDArk0.The authors report superiority across all metrics on OCELOT and statistically significant improvements (p < 0.05).Source: §4.4.1 · p. 10
  • Across Nissl- and H&E-stained examples, the model detects more cells and reduces false positives, particularly for densely packed and adherent cells.Figure 4 compares DINO, CO-DINO, DDQ-DETR, and the proposed model; panels A–B are Nissl-stained and C–E are H&E-stained.Source: §4.4.1 · p. 10, Figure 4 · p. 10

4.4.2. Comparison with Classification Methods

  • The method achieves state-of-the-art performance across reported classification metrics, outperforming mask-based and point-based approaches, including on the challenging rare class.It attains the highest F_cm, which varies substantially across methods; the authors note the rare class’s scarcity and atypical morphology.Source: §4.4.2 · pp. 10–11, Table 2 · p. 11

4.5. Ablation Study

Ablations on CoNSeP support using both spatial and feature similarity in graph construction and show gains from the IGN components.

Ablation study

  • Adding IGL to the baseline raises mAP by 0.8%, and adding TSS yields a further 1.2% gain; IGN also outperforms the tested alternative graph learners.The baseline is 30.0% mAP and 56.5% AR.Source: §4.5 · p. 11
  • Removing positional similarity reduces mAP from 32.0% to 31.4% and AR from 58.7% to 57.3%.Source: §4.5 · p. 11
  • Removing feature similarity lowers mAP to 30.6% and AR to 56.6%, larger drops than removing positional similarity.The reported decreases are 1.4% in mAP and 2.1% in AR.Source: §4.5 · p. 11

4.6. Efficiency analysis

The model’s reported computational footprint is modest relative to instance-segmentation methods, with little added cost over DDQ-DETR.

Efficiency analysis

  • The model uses 49.57M parameters and 276 GFLOPs, and its graph refinement adds only 1.25M parameters and 6 GFLOPs over DDQ-DETR.The authors describe it as substantially lighter than instance-segmentation methods and comparable to point-based models, while providing a more complex task than point detection.Source: §4.6 · pp. 11–12, Table 4 · p. 12

4.7. Investigation of Hyper-parameters

The hyperparameter studies select balanced spatial and feature weighting, 900 queries, and dataset-specific graph connectivity settings.

4.7.1. Investigation of 𝛼-values

  • The query-distance weighting achieves optimal performance at α = 0.5, with spatial and feature distance contributing equally and complementarily.The model adopts α = 0.5.Source: §4.7.1 · p. 12

4.7.2. Investigation of Query Numbers

  • Increasing queries from 100 to 300 raises AR by 2.3% and mAP50 by 1.9%; increasing to 900 adds 3.8% and 2.3%, respectively.Larger query counts continue to improve AR but decrease precision metrics, so the authors adopt 900 queries; larger values were not tested due to GPU memory constraints.Source: §4.7.2 · p. 12

4.7.3. Investigation of K-values

  • The preferred K varies by dataset: CoNSeP performs best at K = 8, CytoDArk0 at K = 3, and OCELOT at K = 4.On densely packed, highly overlapping CoNSeP nuclei, K = 8 outperforms sparser K = 5 by up to 2.5% in mAP; the study tests sparse and dense connectivity regimes.Source: §4.7.3 · p. 13, Table 5 · p. 13

5. Discussion

The discussion frames cell identity as relational and argues for learning graph structure jointly with detection. It connects query-based instance graphs to adaptive topology and selective state-space modeling of noisy, oversized query sets.

5. Discussion

  • Cells are shaped by interactions in their microenvironment, while appearance-focused detectors can falter in dense, overlapping, or heterogeneous regions and post-hoc graphs inherit upstream errors.The discussion motivates making relational reasoning intrinsic to detection rather than appending it afterward.Source: §5 · p. 13
  • Learnable queries serve as candidate-cell graph nodes, carrying decoded multi-scale visual evidence from patch representations into instance-level relational reasoning.This single representation lets feature extraction, instantiation, and graph reasoning be optimized under a unified objective.Source: §5 · p. 13
  • Graph connectivity combines spatial proximity and feature similarity, with query updates and adjacency shaping each other during learning.The resulting topology adapts to data rather than remaining fixed by spatial heuristics.Source: §5 · p. 14
  • A selective state-space model suppresses irrelevant signals in redundant query graphs while preserving global structure at linear node complexity, unlike dense attention’s quadratic cost.Its topological latent state is driven by re-injected query visuals, and CGSE generates transition parameters from instance-level context.Source: §5 · p. 14
  • NMS and KNN act as parameter-free structural selectors; learned query embeddings, rather than changes within KNN, make graph connectivity dynamic.Retained query features still participate in backpropagation, so these selectors do not interrupt end-to-end training.Source: §5 · p. 14

6. Conclusion

The paper presents an end-to-end cell detection and classification framework that builds graphs over learnable queries and models their topology with a state-space network. It reports state-of-the-art performance across multiple benchmarks.

6. Conclusion

  • Learnable queries act as graph nodes, and spatial and feature cues dynamically construct a context-aware topology for varying cell distributions.The framework is designed to improve instance discrimination in query-based detection through structured reasoning.Source: §6 · p. 14
  • The instance-aware graph network uses a Topology-structured State Space model with graph topology as its latent state to address noise and long-range dependencies in large query sets.Source: §6 · p. 14
  • The method achieves state-of-the-art performance across multiple benchmarks.Source: §6 · p. 14
  • The paper presents combining query-based detection with structured graph learning as a promising direction for precise and robust cell instance analysis in digital pathology.Source: §6 · p. 14
Loading 2609.15354v1…