PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Scalable Context-Aware Single-Cell Spatial Transcriptomics Prediction from Histology Images

Zijun Gao, Chunbin Gu, Jinxi Xiang, Xiangde Luo, Pheng-Ann Heng

arXiv:2609.36429v1cs.CV

TL;DR

Spot-level H&E-to-ST prediction aggregates expression across cells, limiting access to cell-specific molecular profiles. CELLO uses one pathology foundation model pass with grid sampling and context-aware cross-attention; across 52 Xenium whole-slide images, it reports state-of-the-art accuracy and up to 7.1× faster inference. Its OOD results are case studies rather than estimates of organ-level generalization.

  • Problem

    Spot-level H&E-to-ST prediction aggregates signals from multiple cells, limiting the resolution available for studying cellular heterogeneity.

    Source: §1 · pp. 1–2

  • Method

    CELLO uses one pathology foundation model pass per image, grid sampling to extract features at cell locations, and distance-decay cross-attention for local morphological context.

    Source: §1 · p. 2

  • Results

    Across 52 Xenium whole-slide images spanning 12 organs, CELLO reports state-of-the-art accuracy in ID and OOD settings and is up to 7.1× faster at inference.

    Source: §5 · p. 10

  • Takeaways & Limitations

    CELLO establishes a foundation for scalable single-cell morphological-to-molecular prediction from H&E images.

    Source: §5 · p. 10

    CELLO requires cell locations at inference, and single-slide OOD organ results—including organs represented in training under other health conditions—are case studies, not estimates of organ-level generalization.

    Source: K Limitations · p. 23

Abstract

from arXiv · show

Predicting gene expression from H&E-stained histology images offers a scalable alternative to costly spatial transcriptomics, yet most existing methods operate at the spot level, where signals from multiple cells are aggregated and critical cellular heterogeneity is obscured. Extending this paradigm to single-cell resolution is non-trivial. Naively applying pathology foundation models faces a scale mismatch: their patch-level representations mix multiple cells, whereas per-cell cropping or resizing distorts morphology and removes local context. Conversely, segmentation-based models without strong pretrained visual encoders often lack the morphological representation capacity needed for accurate molecular prediction and inherit errors from imperfect cell boundary masks. Here, we present CELLO, an efficient end-to-end framework that performs a single pathology foundation model forward pass per image and uses grid sampling to extract location-specific features for all cells simultaneously. We further introduce a distance-decay cross-attention module that refines each cell representation using spatially biased local morphological context. Using 52 public Xenium-H&E pairs from HEST-1k that span 12 organs and approximately 10 million cells, CELLO improves the average predictive accuracy over the evaluated baselines while reducing the mean whole-slide inference time compared to DeepSpot2Cell, a 14.0x speed-up on average that excludes upstream cell segmentation. Our work establishes a scalable foundation for single-cell gene expression prediction from H&E images.

1 Introduction

CELLO addresses the move from spot-level to single-cell H&E-to-ST prediction with a scalable framework that uses paired single-cell data and context-aware cell representations. The authors report state-of-the-art prediction and improved computational efficiency across paired Xenium–WSI samples.

1 Introduction

  • A Visium spot typically contains dozens of cells, so its averaged expression profile masks distinct cell types, rare states, and cell–cell interactions.The need is to move H&E-to-ST prediction to single-cell resolution as imaging-based platforms such as 10x Xenium emerge.Source: §1 · pp. 1–2
  • DeepSpot2Cell uses independent cell crops that can distort morphology and remove context, while mask-based models may lack pretrained visual representations and depend on segmentation quality.Per-cell PFM inference also makes DeepSpot2Cell computationally prohibitive for slides with hundreds of thousands to millions of cells.Source: §1 · p. 2
  • CELLO uses one pathology foundation model pass per image and grid sampling for all cell locations, then refines cell features with distance-decay cross-attention to local morphology.Its approach avoids per-cell cropping and boundary-mask dependence while preserving pretrained visual representations.Source: §1 · p. 2
  • 52 paired Xenium–WSI samples across 12 organs show CELLO achieves state-of-the-art predictive performance and improved computational efficiency, excluding upstream cell segmentation.The framework trains directly on paired single-cell ST and H&E images rather than weakly supervised spot-level expression.Source: §1 · p. 2

2 Related Work

Prior H&E-to-ST work largely predicts spot-level expression, while cell-level alternatives use super-resolution, weak spot supervision, or segmentation masks. CELLO combines direct paired single-cell training with pathology foundation model features.

2.1 Spatial Transcriptomics Prediction From H&E Images

  • Figure 1 contrasts per-cell PFM cropping and mask-based UNet3+ with CELLO’s one-pass feature extraction and context-aware cross-attention.CELLO uses grid sampling to obtain features for all cells simultaneously, with 2D RoPE and Gaussian distance-decay bias for local context.Source: Figure 1 · p. 3
  • Most H&E-to-ST methods predict aggregate spot-level expression; super-resolution methods produce superpixel maps, while weakly supervised cell approaches distribute spot-level expression to cells.GHIST uses segmentation masks for cell-aware representations but does not fully exploit modern pathology foundation model features; CELLO instead trains on paired single-cell ST and H&E.Source: §2.1 · pp. 2–3

2.2 Computational Pathology Foundation Models

  • Pathology foundation models provide transferable patch-level embeddings learned from large histology image collections, and this work evaluates them as feature extractors for cell-level expression prediction.Examples include UNI, Virchow2, and H-Optimus-0.Source: §2.2 · p. 3

3 Method

CELLO predicts gene expression for cells at their locations in an H&E tile, using spatial tokens from a pretrained pathology foundation model and refining location-specific features with local context. Its predictions are trained with cell-level and patch-level Huber losses.

3.1 Problem Formulation

  • CELLO formulates H&E-to-ST prediction per cell, using an image tile and cell locations to predict each cell’s gene-expression vector rather than one vector per spot.The input includes a variable number of cells with spatial coordinates and ground-truth expression vectors.Source: §3.1 · pp. 3–4

3.2 Visual Feature Extraction

  • A pretrained pathology foundation model encodes each H&E patch into a CLS token and spatial tokens reshaped as a 2D feature map for continuous cell-location queries.The grid preserves the correspondence between spatial tokens and image regions.Source: §3.2 · p. 4

3.3 Position-Aware Cell Query Construction

  • Cell coordinates are converted to patch-local, normalized positions, then bilinear interpolation extracts a continuous feature from the four nearest spatial tokens.This produces a sub-patch feature anchored to the cell’s location.Source: §3.3 · p. 4
  • Cross-attention refines each location-specific query using 2D RoPE and a distance-decay bias that gives spatially nearer tokens greater prior weight.The design combines relative spatial encoding with a bias on pre-softmax attention scores.Source: Figure 2 · p. 4, §3.3 · p. 5

3.4 Gene Expression Prediction and Training Objectives

  • A two-layer ReLU MLP predicts gene expression, trained with a weighted cell-level and patch-level Huber-loss objective.The patch-level target is the sum of cell expression vectors in the patch; Huber loss is used for normalized expression regression and is less sensitive to outliers than mean squared error.Source: §3.4 · p. 5

4 Experiments

The experiments benchmark CELLO across Xenium samples from 12 organs, assess encoder and context choices, compare inference costs, and visualize predictions across test tissues.

4.1 10X-Xenium-52 Benchmark

  • The benchmark covers 52 paired Xenium whole-slide images, 12 organs, and approximately 10 million cells; test cases include ID and OOD settings.OOD cases include four unseen organs and two organs with test-time health conditions absent from training.Source: §4.1 · p. 6
  • 58.5% and 71.3% relative improvements on HVG PCC-50 over DeepSpot2Cell in ID and OOD, respectively; CELLO also outperforms UNet3+ in both settings.The reported improvements are larger in the OOD setting.Source: §4.1 · p. 6
  • CELLO improves over Grid by 18.6% and 16.2% on HVG PCC-50 in ID and OOD, respectively, supporting the contribution of contextual refinement.Grid queries cell features from shared patch features but omits distance-decay cross-attention.Source: §4.1 · p. 6

4.2 Ablation Study

  • Virchow2 performs best among three evaluated encoders across MPG, HVG, and SVG; its PCC-50 scores are 0.5494, 0.4390, and 0.5425, respectively.The encoder comparison is reported without data augmentation; the authors select Virchow2 as CELLO’s default visual encoder.Source: §4.2 · pp. 6–7, Figure 3 · p. 8
  • Adding the CLS token consistently degrades performance, while the ablation compares mean ∆PCC across vision encoders.The authors interpret this result as evidence that prediction is predominantly driven by fine-grained local morphology rather than abstract global semantics.Source: §4.2 · p. 8, Figure 3 · p. 8

4.3 Computational Efficiency

  • CELLO averages 14.0× faster inference than DeepSpot2Cell and remains 2.5× faster including cell segmentation; against UNet3+, it has similar cost but higher accuracy.The 14.0× comparison is an average across 12 test slides; the segmentation-inclusive comparison is the median over slides.Source: §4.3 · pp. 8–9, Figure 3 · p. 8

4.4 Case Study

  • Across six representative test tissues, CELLO produces sharper, more spatially faithful expression maps and higher ground-truth correlations than UNet3+.CELLO’s reported correlations for the six markers are 0.746, 0.735, 0.886, 0.901, 0.829, and 0.646.Source: §4.4 · p. 9, Figure 4 · p. 9

5 Conclusion

CELLO predicts single-cell spatial transcriptomics from H&E using shared image features and spatial context. Experiments across 52 Xenium slides spanning 12 organs support its accuracy and scalability.

5 Conclusion

  • A single pathology foundation model pass per image and grid sampling extract features for all cells, while distance-decay cross-attention incorporates spatially biased local morphology.The conclusion reports state-of-the-art accuracy in ID and OOD settings and inference up to 7.1× faster.Source: §5 · pp. 9–10

A.1 Context-Aware Modeling in Computational Biology

  • The paper frames biologically motivated context as a complementary scaling axis alongside model parameters and training data.CELLO applies this perspective through explicit cell-position context and surrounding tissue morphology.Source: §A.1 · p. 13

B Dataset Details

The 10X-Xenium-52 dataset contains 52 public H&E–Xenium pairs, with 9.53 million cells across 12 organs and a 36/4/12 train-validation-test split. Its organ, sample-size, disease-state, and gene-panel distributions are heterogeneous.

B Dataset Details

  • 52 public H&E–Xenium pairs provide 9.53M cells in 520,402 patches, split into 36 training, 4 validation, and 12 test samples.Cells are represented by segmentation centroids; expression is log(1 + x) of counts normalized to 100 per cell.Source: B Dataset Details · p. 13
  • The 12-organ dataset is long-tailed: Lung contributes 22 samples, while Brain, Ovary, Bone, and Heart each contribute one.Lung is most represented, followed by Breast and Bowel with six samples each.Source: B Dataset Details · p. 13
  • Cell and patch totals do not track sample counts directly: Breast has the largest aggregate cell count, followed by Bowel and Lung.Lymphoid and Kidney also have substantial cell populations despite only three and two samples, respectively.Source: B Dataset Details · p. 13
  • Per-sample cell counts vary by more than an order of magnitude, from roughly 20–40 K in Heart and Bone to 1M in Lymphoid outliers.Brain has one sample with about 1M cells despite minimal training support.Source: B Dataset Details · p. 13
  • Disease-state composition varies across organs: Lung and Lymphoid include cancer, diseased, and healthy samples, while Pancreas and Ovary are cancer-only.Bone and Heart are exclusively healthy; Bowel, Skin, Liver, and Kidney mix cancer and healthy tissue.Source: B Dataset Details · p. 14
  • Xenium panels contain 280–480 measured genes per sample, and the model’s loss and metrics use only genes observed in that sample.The model predicts a union vocabulary of 1,915 features, excluding negative-control and antisense probes from panel counts.Source: B Dataset Details · p. 14

C.1 Implementation Details of CELLO

CELLO is trained with distributed mixed-precision optimization, progressive backbone unfreezing, combined cell- and patch-level losses, and patch-based whole-slide evaluation.

C.1 Implementation Details of CELLO

  • Training uses four NVIDIA H100 GPUs with an effective batch size of 512, FP16 mixed precision, gradient scaling, and gradient clipping at 1.0.The per-GPU batch size is 128.Source: §C.1 · p. 16
  • The pretrained backbone is frozen for 10% of epochs, progressively unfrozen over the next 30%, then fully fine-tuned.AdamW and cosine annealing run for 200 epochs, with separate learning rates and weight decay for the backbone and task-specific heads.Source: §C.1 · p. 16
  • The objective combines cell-level gene regression and patch-level gene prediction losses weighted 1.0 and 0.5, respectively.The best checkpoint is selected by validation gene-level PCC.Source: §C.1 · p. 16
  • Whole-slide evaluation uses non-overlapping 224 × 224 patches, assigning each cell to its containing patch and predicting from that patch’s visual tokens.The inference stride equals the patch size.Source: §C.1 · p. 16

C.2 Implementation Details of DeepSpot2Cell

DeepSpot2Cell follows its original frozen-pathology-foundation-model design because independently encoding each segmented cell makes end-to-end fine-tuning costly.

C.2 Implementation Details of DeepSpot2Cell

  • DeepSpot2Cell keeps its pathology foundation model frozen, following the method’s original design.The approach extracts foundation-model features independently for each segmented cell crop.Source: §C.2 · p. 17
  • Fine-tuning would require backpropagating through a separate large-encoder pass for every cell, making training prohibitively costly on million-cell slides.CELLO amortizes foundation-model computation at the patch level and can progressively unfreeze its backbone.Source: §C.2 · p. 17
  • CELLO’s task-specific representation adaptation may partly contribute to its performance gap over DeepSpot2Cell.The frozen-backbone choice also reflects the scalability limit of per-cell-crop foundation-model baselines.Source: §C.2 · p. 17

D Additional Gene Panel Performance

CELLO leads the evaluated methods on the full Xenium gene panel in both ID and OOD settings, and also improves over baselines on MPG and SVG subsets.

D Additional Gene Panel Performance

  • On selected subsets, CELLO improves over DeepSpot2Cell by 70.6% on ID MPG PCC-50 and 68.8% on ID SVG PCC-50.In OOD, relative improvements are 117.9% on MPG PCC-50 and 107.1% on SVG PCC-50.Source: D Additional Gene Panel Performance · p. 17
  • The full-panel evaluation tests whether gains persist across all measured genes, including weaker morphology-associated signals and sparser genes.Source: D Additional Gene Panel Performance · p. 17
  • CELLO achieves the highest gene-level PCC on the full panel in ID and OOD settings, exceeding DeepSpot2Cell and UNet3+ across six ID tissues and all OOD organs.OOD correlations are lower overall, but CELLO retains an advantage, particularly in Liver, Ovary, and Kidney.Source: D Additional Gene Panel Performance · p. 17

E Additional Ablation Study Results

The encoder ablations favor Virchow2 in both ID and OOD settings, while adding the CLS token reduces average performance. Short single-seed runs do not establish a clear effect of loss weight or distance-decay bias.

ID encoder ablation

  • In ID tests, Virchow2 ranked best across all nine metrics, improving over H0 by +28.9% on MPG PCC-50.It also improved MPG PCC-10/200 by +19.7%/+35.5%, HVG PCC-10/50/200 by +13.7%/+14.7%/+11.9%, and SVG PCC-10/50/200 by +19.2%/+28.1%/+25.3%.Source: E Additional Ablation Study Results · p. 17

OOD encoder ablation

  • In OOD tests, Virchow2 ranked best on eight of nine metrics and improved over H0 by +41.9% on MPG PCC-50.It also improved MPG PCC-10/200 by +17.4%/+62.1%, SVG PCC-10/50/200 by +16.1%/+34.8%/+26.7%, and HVG PCC-10/50 by +11.8%/+11.0%.Source: E Additional Ablation Study Results · p. 17

CLS token context

  • Adding the CLS token reduced Virchow2’s nine-metric average by 13.6% in ID and 13.1% in OOD tests.A few entries improved, but gains were not systematic and were outweighed by degradation across MPG and SVG metrics.Source: E Additional Ablation Study Results · pp. 17–18

Loss weight and distance-decay bias

  • In single-seed 50-epoch runs, tested loss weights and removing distance-decay bias showed no clear effect on HVG PCC.The settings differed by less than 0.03; these runs used a shorter budget than the 200-epoch main-table model.Source: E Additional Ablation Study Results · p. 18

F Discussion About Context Information

The discussion favors spatially local context over global CLS information. Additional analyses compare per-cell crops, test external samples and localization sensitivity, and report whole-slide inference times.

F Discussion About Context Information

G Per-Cell Crop Baseline

  • CELLO scored higher than the frozen per-cell crop baseline at every k in both settings, though the comparison also includes backbone fine-tuning.The crop baseline uses CLS embeddings from 224×224 crops centered on cells; CELLO fine-tunes its backbone, so the gap reflects both shared-token querying and representation adaptation.Source: G Per-Cell Crop Baseline · p. 21, Table 9 · p. 21

H External Validation

  • CELLO was applied without adaptation to five additional HEST-1k Xenium samples under the same evaluation protocol.The reported external-validation table covers PCC at k = 10, 50, and 200.Source: H External Validation · p. 22, Table 10 · p. 22

I Sensitivity to Cell Localization

J Computational Efficiency Comparison Details

  • CELLO averaged 67.5 ± 48.8 s per slide versus 947.2 ± 1068.3 s for DeepSpot2Cell, a 14.0× speed-up; its time was comparable to UNet3+.These inference times exclude cell segmentation; including segmentation, CELLO remained 2.5× faster than DeepSpot2Cell by median and was on par with UNet3+ (1.02×).Source: J Computational Efficiency Comparison… · p. 22

K Limitations

The reported cross-organ generalization estimates are limited by sparse OOD sampling, and CELLO still depends on upstream cell detection for inference.

OOD evaluation scope

  • Several OOD organs have only one slide each, limiting the precision of cross-organ generalization estimates.The benchmark contains 52 paired Xenium–WSI samples across 12 organs.Source: K Limitations · p. 23

Inference dependency

  • CELLO requires cell locations at inference, so predictions depend on an upstream cell detector.The authors note that improving cell-segmentation accuracy can reduce the probability of bad predictions.Source: K Limitations · p. 23

OOD evaluation scope

  • Kidney and Liver appear in training under other health conditions, so those OOD results are case studies rather than organ-level generalization estimates.Source: K Limitations · p. 23

Related papers

Loading 2609.36429v1…