Source-linked AI summary
CytoBERT: A Foundation Model for Cytometry Data
Syed Abdul Haseeb Qadri, Bjarne C. Hiller, Felix Blanke, Vanja Sophie Cangalovic, Kutalmış Coşkun, Amin Mirzaei, Tom Siegl, Sebastian Bader, Thomas Kirste, Martin Becker
TL;DR
Cytometry’s heterogeneous, nonstandardized marker panels hinder knowledge transfer across studies. CytoBERT standardizes markers and uses self-supervised pretraining to support variable-panel transfer, achieving the best average ranking across evaluated datasets under grouped subject splitting.
Problem
Nonstandardized marker panels and annotations make cytometry data difficult to integrate and models difficult to transfer across studies.
Method
CytoBERT standardizes markers and pretrains a Transformer with masked expression prediction on cytometry cells to learn transferable marker-expression representations.
Results
CytoBERT achieves the best average ranking across evaluated datasets under grouped subject splitting, although differences from supervised baselines are not statistically significant.
Takeaways & Limitations
The results indicate potential for foundation-model-style pretraining in cytometry and provide publicly available code and trained weights for reproducibility and future work.
Takeaways & Limitations
Main results use grouped subject splitting, which differs from prior ungrouped evaluations and provides a more realistic estimate of generalization to unseen subjects.
Abstract
from arXiv · showhide
Cytometry measures the complex characteristics of single cells (e.g., counts and protein expression of immune cells) and is widely used across immunological research and clinical settings. However, cytometry data is highly heterogeneous and unstandardized due to experimental protocols and the choice of measured features. While machine learning methods hold the potential to gain deeper insights into cell biology, these challenges make them difficult to apply and transfer across studies. Recent advances in foundation models can alleviate these issues, but corresponding approaches are still scarce in this field. To address this, we provide CytoBERT, a publicly available, open-source, open-weight foundation model for single-cell cytometry data with variable marker panels. CytoBERT is pretrained in a self-supervised manner on a large-scale cytometry corpus (15 human datasets with heterogeneous marker panels and more than 50 million cells) curated through marker standardization, enabling it to learn transferable inter-marker relationships within cells. Fine-tuning CytoBERT for sample-level classification demonstrates that transfer learning across heterogeneous cytometry datasets is feasible, providing a starting point for scalable, generalizable cytometry analysis. Code is available at GitHub.
1 Introduction
CytoBERT addresses the difficulty of transferring cytometry models across heterogeneous, nonstandardized datasets by introducing a Transformer-based foundation model for cytometry data. It represents each cell as a sequence of marker-expression tokens and learns within-cell marker co-expression through self-supervised masked expression prediction.
- Motivation: Single-cell cytometry measures 5–50 protein markers per cell, forming a cell-by-marker matrix used in immunological and clinical research.Applications include immunophenotyping, disease detection, and treatment-response analysis.
- Motivation: Nonstandardized marker panels and annotations make it challenging to train models that transfer knowledge across cytometry studies.The same biological marker can have different detection channels, isotope tags, and naming conventions.
- Motivation: Manual gating summarizes cell populations through laborious, subjective, time-consuming procedures that are difficult to scale.Analysts draw bounding boxes around populations in hierarchical bivariate dot plots, then summarize abundances and marker-expression intensities per sample.
- CytoBERT: CytoBERT is a Transformer-based foundation model that processes each cell independently as marker-expression tokens and uses masked expression prediction to learn marker co-expression patterns.The model is inspired by the ImmuneFM framework, while its implementation details, code, and data are made available.
2 Related Work
Prior cytometry sample-classification methods use engineered cluster- or marker-level features or learn sample representations directly from single-cell data. Single-cell foundation models are established in transcriptomics, whereas cytometry foundation models remain limited by fixed panels, restricted evaluations, or unavailable implementations; CytoBERT addresses this gap with variable-panel marker-expression representations.
- Sample classification in cytometry: Classical cytometry classifiers, including CITRUS, FloReMi, and CytoDx, extract cluster- or marker-level features for sample-level prediction.
- Sample classification in cytometry: Deep-learning methods such as CellCNN, CytoSet, and CytoCoSet learn sample representations directly from single-cell data using convolutional or permutation-invariant architectures.
- Foundation models for single-cell data: Single-cell transcriptomics has several foundation models, including scBERT, scGPT, scFoundation, Geneformer, CellFM, and GeneMamba, trained on large-scale RNA-seq datasets.
- Foundation models for single-cell data: Cytometry foundation models remain limited: CyMAE assumes a fixed MDIPA marker panel and COVID-only evaluation, while ImmuneFM lacks a publicly available implementation.
- Foundation models for single-cell data: CytoBERT addresses this gap by learning transferable marker-expression representations from standardized cytometry datasets with variable marker panels and evaluating downstream sample-level classification.
3 Method
CytoBERT represents each cell as marker–expression tokens processed by a Transformer encoder, using masked expression modeling to learn contextual relationships among markers. It then aggregates cell representations for end-to-end sample-level classification.
- Cell representation: CytoBERT uses a Transformer encoder with marker embeddings and discretized expression-bin embeddings, supporting cells with variable marker panels.The model represents each cell as a sequence of marker–expression tokens; it uses M = 220 distinct markers and embedding dimension d = 128.
- Pretraining: Masked expression modeling replaces selected non-zero expression-bin embeddings with [MASK] while keeping marker identities visible.The model predicts missing expression states from the remaining marker–expression context within the same cell, avoiding dominant zero-valued expressions.
- Pretraining: CytoBERT minimizes cross-entropy over masked positions by predicting expression-bin distributions from contextualized marker representations.A classification token is retained as a compact cell-level representation for downstream fine-tuning, while marker representations support masked-bin prediction.
- Pretraining: Reconstructing masked bins enables CytoBERT to learn contextual inter-marker relationships within individual cells.The objective uses the remaining measured markers to infer the expression state at each masked marker position.
- Fine-tuning: For sample-level classification, cell-level classification-token embeddings are mean-pooled into a sample representation and jointly optimized with a two-layer MLP head.The main experiments sample K = 512 cells per sample and evaluate performance using ROC-AUC.
4 Results & Analysis
CytoBERT was pretrained on a large, heterogeneous CyTOF corpus and evaluated through sample-level classification across seven datasets using grouped subject splitting. Fine-tuning achieved the best average ranking, with the greatest improvements on held-out datasets SDY1708 and SDY997, although model differences were not statistically significant.
- Pretraining: CytoBERT was pretrained on approximately 50 million cells from 15 curated datasets using masked marker-expression bins and a six-layer Transformer encoder.Pretraining masked 25% of eligible non-zero marker-expression bins per epoch; the encoder used d = 128, 8 attention heads, and 10 expression bins.
- Downstream evaluation: Downstream evaluation fine-tuned CytoBERT for sample classification across seven CyTOF datasets, using repeated 512-cell subsamples and sample-level minority-class oversampling.Multiple independent subsamples were generated from each sample and classified independently to reduce sampling variability.
- Baselines: Comparisons included logistic regression, supervised permutation-invariant CytoSet, and randomly initialized CytoBERT trained from scratch.Logistic regression averaged marker expression across cells, while CytoSet and CytoBERT used mean pooling; fixed-dimensional baselines used shared markers.
- Results: CytoBERT fine-tuning achieved the best average ranking across downstream datasets under grouped subject splitting.Grouped splitting prevented samples from the same subject appearing in both training and test folds, estimating generalization to unseen subjects more realistically.
- Results: The greatest improvements over training from scratch occurred on SDY1708 and SDY997, indicating transfer to datasets not seen during pretraining.The clearest gains were observed on truly held-out datasets, while SDY788 performed weakest because it had few subjects.
- Statistical analysis: The Friedman test found no significant differences between models despite CytoBERT achieving the best average rank (p = 0.093).Subject-level prediction results were reported under grouped subject splitting, with ungrouped-split comparisons provided in supplementary figures.
5 Conclusion
CytoBERT is an open-source, open-weight foundation model for variable-panel single-cell cytometry data that uses marker standardization and self-supervised pretraining. Across evaluated datasets, it achieved the best average ranking under grouped subject splitting, although differences from supervised baselines were not sta...
- Contribution: CytoBERT is an open-source, open-weight foundation model for variable-panel single-cell cytometry data.The model targets single-cell cytometry datasets with varying marker panels.
- Contribution: Marker standardization across datasets and self-supervised pretraining enable CytoBERT to be fine-tuned for downstream sample classification.The approach standardizes markers before pretraining and supports transfer to sample-level classification tasks.
- Results: CytoBERT achieves the best average ranking across evaluated datasets under grouped subject splitting.This result suggests that pretraining improves subject classification performance.
- Results: Differences between CytoBERT and supervised baselines are not sta...
A Data and Code Availability
CytoBERT’s manuscript, code, and reproducibility artifacts are publicly available through project and archival resources. Its pretraining data comprise preprocessed datasets provided by ImmPort.
- Data and Code Availability: Code is available on GitHub, alongside a manuscript overview, code snapshot, data artifacts, and Docker image for long-term reproducibility and accessibility.The project overview is hosted at bckrlab.org/p/cytobert, while the code is hosted at github.com/bckrlab/cytobert.
- Data and Code Availability: The publication uses data provided by ImmPort, and the pretraining dataset contains preprocessed data from ImmPort datasets.The listed datasets span studies of infections, vaccination, autoimmune disease, transplantation, allergy, cancer, and aging.
B Results under Grouped and Ungrouped Splitting
ROC-AUC performance drops substantially under grouped subject splitting, particularly for SDY788, indicating that ungrouped sample-level splitting can produce overly optimistic estimates. Grouped splitting is therefore a more appropriate evaluation setting because it keeps samples from the same subject out of both training and test folds.
- Grouped versus ungrouped splitting: Grouped subject splitting causes a substantial ROC-AUC performance drop, particularly on SDY788.The comparison spans models and datasets.
- Grouped versus ungrouped splitting: Ungrouped sample-level splitting may exploit subject-specific rather than disease-specific patterns, yielding overly optimistic performance estimates.This concern motivates grouped subject splitting for evaluation.
- Grouped versus ungrouped splitting: Grouped subject splitting prevents samples from the same subject from appearing in both training and test folds.Figure 2 reports ROC-AUC for both ungrouped and grouped splitting across CyTOF datasets.
C Comparison of Model Rankings
CytoBERT achieves the best average model rank across CyTOF datasets under both ungrouped and grouped subject splitting. Critical difference diagrams indicate that horizon-connected models are not significantly different, while additional datasets are needed for more robust comparisons.
- Comparison of Model Rankings: CytoBERT achieves the best average rank under both ungrouped and grouped subject splitting.Lower rank indicates better average performance.
- Comparison of Model Rankings: Models connected by a horizontal line are not significantly different.
- Comparison of Model Rankings: Additional datasets would provide a more robust comparison of model performance.
D Overview of CytoBERT
CytoBERT’s pipeline combines dataset curation and preprocessing, masked expression-bin self-supervised pretraining, and supervised fine-tuning for sample-level classification. It addresses heterogeneous marker panels through dataset-specific standardization and a shared marker vocabulary across 18 datasets.
- The project uses a shared marker vocabulary spanning 18 datasets and comprising 220 unique biomarkers.
- Dataset-specific standardization is used to address heterogeneous marker panels across datasets.
- The full CytoBERT pipeline comprises dataset curation and preprocessing, self-supervised masked expression-bin pretraining, and supervised fine-tuning for sample-level classification.