PaperlayerView exact revision on arXiv

Source-linked AI summary

CytoBERT: A Foundation Model for Cytometry Data

Syed Abdul Haseeb Qadri, Bjarne C. Hiller, Felix Blanke, Vanja Sophie Cangalovic, Kutalmış Coşkun, Amin Mirzaei, Tom Siegl, Sebastian Bader, Thomas Kirste, Martin Becker

arXiv:2608.14414v1cs.LG

TL;DR

Cytometry datasets use heterogeneous marker panels and annotations, making models difficult to transfer across studies. CytoBERT standardizes markers and self-supervised pretrains a Transformer for variable-panel cytometry, then fine-tunes it for sample classification. Across evaluated datasets, it achieves the best average ranking under grouped subject splitting, although differences from supervised baselines are not statistically significant.

  • Problem

    Cytometry marker panels and annotations vary across datasets, while many existing analysis methods are supervised, dataset-specific, and assume fixed panels, limiting cross-study transfer.

    Source: §1 · p. 2, §2 · p. 2

  • Method

    CytoBERT standardizes marker vocabulary and self-supervised pretrains a Transformer on marker-expression tokens for variable-panel cytometry, then fine-tunes it for sample-level classification.

    Source: §1 · p. 2

  • Results

    Across evaluated datasets, CytoBERT achieved the best average ranking under grouped subject splitting, although differences from supervised baselines were not statistically significant.

    Source: §5 · p. 6

  • Takeaways & Limitations

    The findings indicate the potential of foundation-model-style pretraining for cytometry.

    Source: §5 · p. 6

    The main evaluation uses grouped subject splitting, so its results address generalization to unseen subjects rather than ungrouped splits.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Cytometry measures the complex characteristics of single cells (e.g., counts and protein expression of immune cells) and is widely used across immunological research and clinical settings. However, cytometry data is highly heterogeneous and unstandardized due to experimental protocols and the choice of measured features. While machine learning methods hold the potential to gain deeper insights into cell biology, these challenges make them difficult to apply and transfer across studies. Recent advances in foundation models can alleviate these issues, but corresponding approaches are still scarce in this field. To address this, we provide CytoBERT, a publicly available, open-source, open-weight foundation model for single-cell cytometry data with variable marker panels. CytoBERT is pretrained in a self-supervised manner on a large-scale cytometry corpus (15 human datasets with heterogeneous marker panels and more than 50 million cells) curated through marker standardization, enabling it to learn transferable inter-marker relationships within cells. Fine-tuning CytoBERT for sample-level classification demonstrates that transfer learning across heterogeneous cytometry datasets is feasible, providing a starting point for scalable, generalizable cytometry analysis. Code is available at GitHub.

1 Introduction

Variable marker panels and inconsistent annotations make cytometry data difficult to integrate across studies and hinder transferable modeling. CytoBERT addresses this with a self-supervised Transformer designed to learn marker co-expression patterns across heterogeneous datasets.

1 Introduction

  • Cytometry measures 5–50 protein markers per cell, but marker panels and annotations vary across studies, making cross-study integration and model transfer difficult.Markers may be assigned different detection channels, metal-isotope tags, and naming conventions.Source: §1 · pp. 1–2
  • Manual gating is laborious, subjective, time-consuming, and difficult to scale, while many machine-learning methods assume fixed panels, need labels, and require retraining.Gated populations are summarized per sample using population abundances and marker-expression intensities for phenotype associations or clinical-group comparisons.Source: §1 · p. 2
  • CytoBERT processes each cell as marker-expression tokens and uses masked expression prediction to learn within-cell co-expression patterns.The Transformer is inspired by ImmuneFM; the authors also provide implementation details and make code and data available.Source: §1 · p. 2
  • A shared vocabulary of 220 markers supports evaluating pretrained CytoBERT for transfer learning across heterogeneous datasets and fine-tuning it for sample-level disease classification.The model is presented as a foundation model for cytometry data.Source: §1 · p. 2

2 Related Work

Prior cytometry classifiers are often supervised, fixed-panel, and dataset-specific, while existing foundation models do not fully address cytometry’s variable marker panels and standardization challenges. CytoBERT learns transferable marker-expression representations from standardized, variable-panel cytometry data for downstream sample classification.

Sample classification in cytometry

  • CITRUS, FloReMi, and CytoDx, along with CellCNN, CytoSet, and CytoCoSet, are typically supervised, dataset-specific, fixed-panel, and require retraining.Classical approaches extract cluster- or marker-level features; deep-learning methods learn sample representations directly from single-cell data.Source: §2 · p. 2

Foundation models for single-cell data

  • scBERT, scGPT, scFoundation, Geneformer, CellFM, and GeneMamba learn reusable representations from large-scale RNA-seq, where genes provide a stable feature vocabulary.Cytometry instead measures varying subsets of protein markers and often lacks standardized marker annotations.Source: §2 · p. 2
  • CyMAE assumes the fixed MDIPA marker panel and is evaluated only on COVID studies, while ImmuneFM handles variable panels but lacks a publicly available implementation.These limitations leave a gap in publicly available foundation models for heterogeneous cytometry data.Source: §2 · p. 2
  • CytoBERT learns transferable marker-expression representations from standardized cytometry datasets with variable marker panels and evaluates them for downstream sample-level classification.It addresses the gap in cytometry foundation models described above.Source: §2 · p. 3

3 Method

CytoBERT encodes each cell as marker–expression tokens and learns contextual relationships by predicting masked expression bins. For sample-level classification, it pools pretrained cell representations and jointly fine-tunes the encoder and classifier.

3 Method

  • Each token sums learned embeddings for marker identity and discretized expression, allowing CytoBERT to process cells with variable marker panels.Expression values are arsinh-transformed, MinMax-scaled per marker, and discretized into 10 bins; the model uses 220 marker embeddings of dimension 128.Source: §3 · p. 3

3.1 Self-Supervised Pretraining

  • During pretraining, CytoBERT masks non-zero expression bins while retaining marker identity, then predicts masked bins from within-cell marker context.A classification token is retained as the downstream cell representation, while a linear head predicts expression bins and cross-entropy is minimized over masked positions.Source: §3.1 · pp. 3–4

3.2 Fine-Tuning for Sample Classification

  • For sample classification, CytoBERT mean-pools pretrained classification-token embeddings from 512 cells per sample and jointly fine-tunes the encoder and two-layer MLP using cross-entropy.Performance is evaluated using ROC-AUC.Source: §3.2 · p. 4

4 Results & Analysis

CytoBERT was pretrained on heterogeneous cytometry data and fine-tuned for sample classification across seven datasets. Under grouped subject splits, it achieved the best average rank, although model differences were not statistically significant.

Pretraining setup.

  • CytoBERT was pretrained on approximately 50 million cells from 15 of 18 curated datasets, masking 25% of eligible non-zero expression bins each epoch.Its Transformer used hidden dimension 128, six layers, eight attention heads, and ten expression bins, with Adam for 50 epochs; hyperparameters were primarily motivated by ImmuneFM.Source: Pretraining setup · p. 4

Fine-tuning setup.

  • For downstream fine-tuning, each file was subsampled to 512 cells, with multiple independent subsamples classified separately and minority classes oversampled.Evaluation covered sample-classification tasks across seven ImmPort CyTOF datasets.Source: Fine-tuning setup · p. 5

Evaluation.

  • Evaluation used 3-fold cross-validation and ROC-AUC, with grouped subject splits when identifiers were available to prevent subject overlap between training and test folds.CytoBERT used variable-marker inputs, while logistic regression and CytoSet were restricted to markers shared across samples; comparisons included both methods and CytoBERT trained from scratch.Source: Evaluation · p. 5

4.2 Results on Downstream Datasets

  • Under grouped subject splits, CytoBERT achieved the best average rank, though model differences were not significant (p = 0.093).Compared with CytoBERT trained from scratch, this suggests self-supervised pretraining provides a useful initialization; the greatest improvements were on heldout datasets SDY1708 and SDY997.Source: §4.2 · p. 5, Figure 1 · p. 6
  • Pretraining included some evaluated datasets but used marker-expression values only, not sample labels; the clearest gains were on heldout SDY1708 and SDY997.SDY788 had the weakest grouped-split performance because it had few subjects.Source: §4.2 · p. 6

5 Conclusion

CytoBERT is an open-source, open-weight foundation model for variable-panel single-cell cytometry, using marker standardization and self-supervised pretraining to support downstream sample classification. Its evaluation suggests potential benefits for subject classification, though differences from supervised baselines were not statistically significant.

5 Conclusion

  • Across evaluated datasets, CytoBERT achieved the best average ranking under grouped subject splitting, although differences from supervised baselines were not statistically significant.The results suggest that pretraining may improve subject classification performance.Source: §5 · p. 6
  • CytoBERT combines marker standardization across cytometry datasets with self-supervised pretraining to enable fine-tuning for downstream sample classification.The model is designed for variable-panel single-cell cytometry data.Source: §5 · p. 6
  • The authors publicly released CytoBERT's code and trained weights to support reproducibility and future work.Source: §5 · p. 6

A Data and Code Availability

The manuscript provides code through GitHub and offers a code snapshot, data artifacts, and a Docker image to support long-term reproducibility and accessibility. Its pretraining dataset uses preprocessed data from ImmPort.

A Data and Code Availability

  • The authors provide code through GitHub, plus a code snapshot, data artifacts, and a Docker image for reproducibility and accessibility.A general overview of manuscript information is also available online.Source: A Data and Code Availability · p. 9
  • The pretraining dataset contains preprocessed data from ImmPort datasets.Source: A Data and Code Availability · p. 9

B Results under Grouped and Ungrouped Splitting

Grouped subject splitting substantially lowers ROC-AUC performance, especially on SDY788, suggesting that ungrouped splitting can yield overly optimistic estimates by allowing models to exploit subject-specific patterns. Grouped splitting is therefore a more appropriate evaluation setting for sample-level classification.

B Results under Grouped and Ungrouped Splitting

C Comparison of Model Rankings

CytoBERT achieves the best average rank under both ungrouped and grouped subject splitting. However, models connected by a horizontal line are not significantly different, and additional datasets are needed for a more robust comparison.

C Comparison of Model Rankings

D Overview of CytoBERT

CytoBERT combines dataset curation and preprocessing with masked expression-bin pretraining and supervised fine-tuning for sample-level classification. Its shared vocabulary spans 220 biomarkers across 18 CyTOF datasets.

D Overview of CytoBERT

Loading 2608.14414v1…