Source-linked AI summary
Towards a knowledge-enhanced single-cell foundation model
Hanqing Zhang, Jie Bao, Mei Ma, Shuai Liu, Jiaying Ma, Jiaguan Liu, Jiaxiao Li, Zhenbo Li, Wenwen Gong, Zhijun Ca
TL;DR
scFMs face diminishing gains from expanding pretraining corpora despite substantial computational costs. scKITE adds cell-annotation and gene-regulatory supervision through pretraining-only auxiliary decoders; with 179,067 samples, it outperformed representative large scFMs across diverse downstream tasks.
Problem
scFM pretraining corpora can plateau in downstream performance as they grow, while requiring substantial computational resources.
Source: Introduction · pp. 1–2Method
scKITE trains a shared transcriptomic encoder with auxiliary decoders for cell annotation and regulon prediction, discarding the decoders after pretraining.
Source: Introduction · p. 2Results
With 179,067 pretraining samples (<0.5% of representative scFMs’ data), scKITE outperformed Geneformer, scGPT, and scFoundation across diverse downstream tasks.
Source: Introduction · p. 2Takeaways & Limitations
Knowledge-enhanced pretraining offers a route to biologically grounded single-cell representations while reducing reliance on continued data scaling.
Source: Introduction · p. 3The observed data-efficiency pattern is specific to this corpus, architecture, and benchmark set, not a universal scFM scaling threshold.
Source: Discussion · p. 14
Abstract
from arXiv · showhide
Single-cell foundation models (scFMs) increasingly rely on large-scale transcriptomic pretraining, yet expanding pretraining data can yield diminishing gains while substantially increasing computational cost. Our data scaling analyses showed that incorporating biological knowledge, including cell-level text annotation and gene-level regulatory information, provided additional scaling dimension than simply increasing data size. Motivated by this observation, we present scKITE, a simple yet effective scFM that integrates cell-annotation and gene-regulatory supervision into a shared transcriptomic Transformer encoder through lightweight auxiliary decoders. These decoders are used only during pretraining and subsequently discarded, yielding a general-purpose encoder enriched with biological knowledge for downstream applications. With only 179,067 pretraining samples, i.e., less than 0.5\% of those used by previous strong scFMs, scKITE outperformed these models across diverse downstream tasks, highlighting knowledge-enhanced pretraining as a promising paradigm for biologically grounded scFMs.
Introduction
scKITE adds cell-annotation and gene-regulatory supervision to transcriptomic pretraining, addressing diminishing returns from simply expanding scFM corpora. With 179,067 pretraining samples, it outperformed representative large scFMs across diverse downstream tasks.
Introduction
- Large-scale transcriptomic pretraining can capture transferable information, but requires substantial computational resources and may yield limited downstream gains as corpora expand.A systematic evaluation found performance plateaus using only a fraction of the full corpus, without clear data-scaling behavior comparable to large language models.Source: Introduction · pp. 1–2
- Cell-identity text and TF-centered regulatory programs provide complementary supervision about cellular context and gene-regulatory relationships.The paper presents these as accessible sources of biological knowledge for scFM representation learning.Source: Introduction · p. 2
- Across data scales, knowledge-enhanced pretraining improved downstream performance by an average 28.8% relative to scKITE without knowledge enhancement, while gains from adding data rapidly diminished.The scaling analysis held model architecture and evaluation settings fixed.Source: Introduction · p. 2
- scKITE uses lightweight annotation and regulon-prediction decoders to supervise a shared transcriptomic encoder, then discards both decoders after pretraining.The retained encoder is intended for downstream cell- and gene-level applications.Source: Introduction · p. 2
- With 179,067 pretraining samples (<0.5% of the data used by representative scFMs), scKITE outperformed Geneformer, scGPT, and scFoundation across diverse downstream tasks.The reported pretraining sample counts were approximately 30M, 33M, and 50M, respectively.Source: Introduction · p. 2
- Against scKITE without knowledge enhancement, scKITE increased mean zero-shot macro-F1 64.0% across five benchmarks, increased batch-integration overall score 22.8% across four datasets, and reduced Top-20 DE MSE 19.8%.The reported gains span cell-level annotation and batch integration, plus gene-level perturbation-response benchmarks.Source: Introduction · pp. 2–3
Results
scKITE combines transcriptomic pretraining with cell-annotation and gene-regulatory supervision, addressing diminishing gains from corpus expansion. Its knowledge-enhanced representations improve cell- and gene-level downstream results across the reported tasks.
Knowledge-enhanced pretraining improves transcriptomic representations beyond data scaling
- 28.8% average relative downstream-task improvement over knowledge-ablation scKITE persisted across four data scales, while the ablation plateaued after 50% of the corpus.The subsequent experiments used 179,067 training profiles.Source: Knowledge-enhanced pretraining… · p. 3, Figure 1 · p. 5
- Stage 2 adds annotation and regulon sequence decoders to Stage 1's masked-expression encoder, then discards them after pretraining.The retained encoder supplies cell-level representations for annotation and integration and gene embeddings for perturbation prediction and regulatory interpretation.Source: Knowledge-enhanced pretraining… · pp. 3–4
Knowledge-enhanced pretraining improves cell identity representation
- Across five benchmarks, mean macro-F1 was 0.500 zero-shot and 0.668 after fine-tuning, exceeding scGPT and Geneformer in both settings.Immune-compartment ARI and NMI rose from 0.20 to 0.46 and 0.34 to 0.68; cross-tissue Top-1 retrieval rose from 42.8% to 66.2%.Source: Knowledge-enhanced pretraining… · pp. 4, 6, Figure 2 · p. 7
Knowledge-enhanced pretraining improves batch integration
- Across four integration benchmarks, scKITE exceeded Geneformer by 6.5% in AvgBIO and 4.2% in AvgBATCH while improving over its knowledge ablation.Mean AvgBIO increased from approximately 0.338 to 0.474 and mean AvgBATCH from approximately 0.836 to 0.938; individual-cell-type analyses also found high batch ASW and graph connectivity across most evaluated types.Source: Knowledge-enhanced pretraining… · pp. 6, 8, Figure 3 · p. 9
Knowledge-enhanced pretraining improves gene context embeddings for perturbation-response prediction
- scKITE achieved the lowest overall Top-20 DE MSE on Norman and Replogle, outperforming evaluated baselines in most perturbation generalization settings.It improved over GEARS in all 25 evaluated Replogle perturbations and over its knowledge ablation in direction-correct recovery across 97 Norman conditions (P = 0.010); for unseen single-gene perturbations, scFoundation had lower MSE.Source: Knowledge-enhanced pretraining… · pp. 8, 10, Figure 4 · p. 11
Biological interpretation of knowledge-enhanced representations
- Annotation-decoder attention favored marker genes in 196 of 198 cell types, while regulon-target attention showed a weaker tendency and varied by cellular context.Supported targets had higher attention than matched non-targets for 32 of 45 TF programs (P = 5.9 × 10^-2); retrieval patterns differed across cell types.Source: Biological interpretation of… · pp. 10, 12
Discussion
scKITE’s scaling results suggest biological supervision can improve data efficiency alongside transcriptomic data expansion. The discussion emphasizes matching knowledge to learned representations while noting limits in supervision quality and evaluation scope.
Discussion
- With about 0.18 million pretraining samples, scKITE matched or exceeded representative large-scale scFMs trained on tens of millions of samples.The authors report diminishing returns from expanding pretraining data alone, while biological supervision improved performance across data scales.Source: Discussion · p. 12
- Cell annotations and context-specific regulatory programs provide complementary supervision aligned with cell-level and gene-level representations.The authors connect this alignment to improved representations across cell- and gene-level tasks.Source: Discussion · p. 12
- Cell-identity descriptions may vary in completeness, specificity, and consistency, while inferred regulatory programs are predicted rather than experimentally established causal networks.These limits qualify the biological supervision used in scKITE.Source: Discussion · pp. 12, 14
- Decoder-to-encoder cross-attention offers an interpretive view of genes accessed during cell-annotation and regulatory-target reconstruction.Figure 5 evaluates marker-gene and TF-target attention patterns.Source: Figure 5 · p. 13
- The observed data-efficiency pattern comes from one combination of corpus, architecture, and downstream benchmarks, not a universal scaling threshold.Source: Discussion · p. 14
- The authors identify perturbation-derived causal relationships, chromatin accessibility, TF-binding measurements, curated regulatory resources, and standardized experimental descriptions as potential additional evidence.Source: Discussion · p. 14
1 Pretraining data construction
scKITE’s pretraining data combine pseudo-bulk transcriptomic profiles and their existing natural-language annotations with profile-specific regulatory supervision. The regulatory targets are derived from inferred regulons and prepared for decoder training.
1.1 Transcriptomic pretraining corpus
- The corpus contains 376,983 human transcriptome–annotation pairs, split into 358,134 training profiles and 18,849 fixed validation profiles.Profiles were derived from CELLxGENE datasets in the CellWhisperer corpus.Source: §1.1 · p. 18
1.2 Natural-language annotations
- CellWhisperer’s existing metadata-derived descriptions provide annotation supervision; no additional large language model regenerated them.Descriptions include cell identity and may include tissue, disease or physiological condition, and donor characteristics.Source: §1.2 · p. 18
1.3 Regulatory knowledge construction
- pySCENIC construction uses GRNBoost2 TF–target associations, cisTarget motif refinement, and merged motif-supported targets to define regulons ranked by interaction importance.When multiple motif-derived sets support a TF–target relationship, the maximum GRNBoost2 importance score is retained.Source: §1.3 · p. 18
- AUCell activity thresholds identify active regulons; invariant regulons are excluded, leaving 530 informative regulons with profile-specific active identifiers and scores.Source: §1.3 · p. 18
- For each profile, active-regulon targets are restricted to expressed genes, and up to three eligible regulons are selected per profile using cyclic-without-replacement training across epochs.Validation uses a fixed deterministic selection; target ordering by interaction importance is preserved.Source: §1.3 · pp. 18–19
- Regulon supervision serializes each selected TF with its importance-ranked profile-specific targets, distributing limited target-token capacity round-robin across regulons.The sequence uses task and separator tokens and terminates with an end-of-sequence token.Source: §1.3 · p. 19
2 Downstream benchmark datasets
The downstream benchmarks cover cell-type classification, batch integration, and perturbation prediction across multiple datasets and evaluation settings. Dataset splits are structured to test transfer across studies, tissues, donors, assays, or perturbations.
2.1 Cell type classification datasets
- Classification benchmarks include Tabula Sapiens (143,133 cells, 75 types), pancreas (16,382 cells, 14 types), multiple sclerosis, tumor-infiltrating myeloid, and cross-tissue immune cells.The cross-tissue atlas includes 324,458 cells across nine tissue environments and 43 immune-cell types.Source: §2.1 · pp. 19–20
- Classification tests include held-out MS individuals and tumor-query groups, while the cross-tissue atlas reserves lung for validation and liver plus mesenteric lymph node for testing.Source: §2.1 · p. 20
2.2 Batch integration datasets
- Batch-integration datasets test cross-assay integration in perirhinal cortex and renal tissue, and cross-donor integration in liver.The liver analysis interprets results as reducing donor-associated variation while preserving cell-type structure, not eliminating all donor-related biological differences.Source: §2.2 · p. 20
- COVID-19 integration uses a 20,000-cell subsample organized into 18 study/sample-level batches.Source: §2.2 · p. 20
2.3 Perturbation prediction datasets
- Perturbation prediction uses Norman CRISPRa data partitioned by the GEARS simulation split and Replogle CRISPR-interference data to test generalization to unseen single-gene perturbations.Replogle contains 105 perturbations and 102,148 cells after benchmark preprocessing.Source: §2.3 · p. 21
3 Model architecture and representations
scKITE combines expression-ordered gene inputs with a shared Transformer encoder and auxiliary annotation and regulatory decoders. The encoder yields cell and contextual gene representations for downstream analyses.
3.1 Model input
- Inputs use a shared vocabulary for gene, text, and structural tokens; expressed genes are expression-ordered, quantile-binned, and combined with gene identity and mask embeddings.Up to 2,048 expressed genes are retained; expression masking is 30% in Stage 1 and 10% in Stage 2.Source: §3.1 · p. 21
3.2 Encoder
- The backbone is a 12-block pre-normalization Transformer encoder with hidden dimension 512 and eight attention heads, producing contextualized hidden states from input sequences.Source: §3.2 · pp. 21–22
3.3 Knowledge decoders
- Stage 2 uses independent autoregressive regulon and annotation decoders that cross-attend to the encoder states to generate profile-specific TF–target sequences and paired descriptions.Source: §3.3 · p. 22
3.4 Learned representations
- The <cls> state represents the cell, while each expressed gene retains a context-dependent representation used for gene-level analyses and perturbation prediction.The same gene can have different representations across cellular states.Source: §3.4 · p. 22
4 Two-stage pretraining strategy
scKITE first learns transcriptomic representations through masked-expression reconstruction, then adds annotation and regulon supervision through auxiliary decoders. After pretraining, the decoders are discarded and downstream applications use the shared encoder alone.
4.1 Stage 1: transcriptomic self-supervised pretraining
- Stage 1 masks 30% of valid gene positions and trains a regression head to reconstruct their original expression levels.Source: §4.1 · p. 22
4.2 Stage 2: knowledge-enhanced pretraining
- Stage 2 initializes from the Stage 1 checkpoint and jointly trains the encoder with reconstruction, annotation, and regulon objectives.The reconstruction masking probability is reduced to 10%; loss weights for all three objectives are set to 1.Source: §4.2 · p. 23
4.3 Encoder extraction after pretraining
- After pretraining, both auxiliary decoders are discarded, so downstream applications use only the shared encoder and require transcriptomic input.Source: §4.3 · p. 23
5 Pretraining data scaling analysis
The scaling analysis compares nested pretraining subsets while holding evaluation data and key training settings constant. It summarizes performance across three downstream tasks using scores normalized to the full-data scKITE reference.
5 Pretraining data scaling analysis
- The analysis evaluates 10%, 25%, 50%, and 100% of 358,134 training profiles, using the same 18,849-profile validation set throughout.Source: §5 · p. 23
- Nested geometric sketching selects representative profiles in a shared latent space, so larger subsets add profiles without replacing those in smaller subsets.Profiles are normalized, log-transformed, reduced to 4,000 variable genes, and projected to 50 dimensions before sketching.Source: §5 · pp. 23–24
- Each fraction is trained to near convergence, and cell annotation, batch integration, and perturbation-response scores are normalized to the full-data scKITE reference and averaged equally.Source: §5 · p. 24
6 Downstream evaluation
The downstream evaluation covers cell-type classification and representation structure, batch integration, and genetic perturbation-response prediction. Protocols use frozen representations for zero-shot or integration analyses and task-specific training for fine-tuned classification and perturbation prediction.
6.1 Cell type classification and representation analysis
- Cell-type evaluation includes frozen-encoder five-nearest-neighbor zero-shot prediction and full fine-tuning, alongside analyses of immune-compartment structure and cross-tissue retrieval.The representation analyses compare cell-type centroids, relatedness-stratified cell pairs, and directional retrieval among tissues.Source: §6.1 · pp. 25–26
6.2 Batch integration
- Batch integration uses frozen encoder embeddings and combines biological-conservation metrics with within-cell-type batch-mixing measures.Batch ASW and graph connectivity are macro-averaged across eligible cell types; the overall score combines AvgBIO and AvgBATCH.Source: §6.2 · pp. 26–27
6.3 Genetic perturbation-response prediction
- For perturbation prediction, scKITE gene embeddings from control-cell profiles replace GEARS gene embeddings while its other components remain unchanged.Evaluation uses shared data splits and includes top-20 differentially expressed gene metrics and recovery of genetic-interaction magnitudes.Source: §6.3 · pp. 27–28
7 Mechanistic interpretation
The mechanistic analyses examine whether annotation and regulon decoders prioritize biologically relevant genes through cross-attention. They also compare regulatory attention across cell contexts using externally supported targets and matched non-target genes.
7.1 Annotation decoder cross-attention analysis
- Annotation-decoder attention is compared between training-derived marker genes and expression- and detection-matched background genes in validation cells.Marker and background attention summaries are compared across cell types using paired tests.Source: §7.1 · pp. 28–29
7.2 Regulon decoder cross-attention analysis
- Regulon-decoder attention is tested for preferential access to external TF target genes versus matched non-targets, with attention converted to within-profile percentile ranks.CollecTRI interactions provide external targets, and TF-level summaries exclude TFs with fewer than 10 evaluable profiles.Source: §7.2 · p. 30
7.3 Cell-context-specific regulatory attention
- Regulatory enrichment is aggregated by TF–cell-type pair, with positive values indicating preferential attention to external targets over matched non-targets.Combinations with fewer than three evaluable cells are excluded; heatmap entries are selected by data coverage, and insufficient combinations remain uncolored.Source: §7.3 · p. 31