PaperlayerView exact revision on arXiv

Source-linked AI summary

GeneICL: A Tabular Foundation Model for Bulk Transcriptomics

Michael Bohl, Alexander Theus, David Wissel, Valentina Boeva

arXiv:2610.08694v1cs.LG

TL;DR

Clinical transcriptomic prediction must contend with high-dimensional, correlated features and few labeled samples, while generic tabular models are not tailored to transcriptomic structure. GeneICL uses transcriptomics-aware in-context pretraining and a compact recurrent architecture, and is evaluated across clinical outcome tasks. It compares favorably with transcriptomic and general-purpose tabular models, including for survival prediction.

  • Problem

    Transcriptomic prediction involves tens of thousands of strongly correlated features but only tens to hundreds of labeled samples, and generic tabular models do not explicitly account for gene-expression structure.

    Source: §1 · pp. 1–2

  • Method

    GeneICL is a transcriptomics-specific tabular foundation model that uses a semi-synthetic prior with real bulk RNA-seq covariate structure and recurrent in-context learning.

    Source: §1 · p. 2

  • Results

    Across 80 translational tasks, GeneICL outperforms larger transcriptomic foundation models and matches general-purpose tabular models with up to 387× more parameters; its survival adaptation ranks best overall against the compared approaches.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    A prior with realistic covariates improves regression performance over generic-prior or real-profile alternatives, while one looped block matches eight untied blocks with under a quarter of their parameters.

    Source: §6 · p. 10

    The prior is derived from bulk RNA-seq, single-cell and other omics modalities remain untested, and robustness to batch or platform shifts between support and query samples is open.

    Source: §6 · p. 10

Abstract

from arXiv · show

Gene expression is widely measured in biomedicine, yet clinical outcome prediction remains challenging due to high dimensionality, strong feature correlations, and limited labeled data. Large self-supervised transcriptomic foundation models often fail to outperform simple supervised baselines. Tabular foundation models offer an alternative through in-context learning, but are typically pretrained on generic synthetic data rather than transcriptomic structure. We ask whether transcriptomics-aware pretraining, rather than scale, is the missing ingredient. Towards this end, we introduce GeneICL, a 4.2M-parameter tabular foundation model combining a semi-synthetic pretraining prior built from measured bulk expression profiles with a parameter-efficient recurrent architecture. We further enable right-censored survival prediction via a training-free reduction to regression using Cox partial-likelihood residuals. We evaluate GeneICL on 80 clinical outcome-prediction tasks spanning classification, regression, and survival. Tabular foundation models consistently outperform self-supervised transcriptomic models, while GeneICL achieves the best overall rank among evaluated foundation models and tuned baselines. GeneICL does so with up to 387$\times$ fewer parameters, no gradient updates at inference, and predictions within seconds on a laptop CPU.

1 INTRODUCTION

Transcriptomic outcome prediction is difficult in high-dimensional, low-sample settings, and generic tabular models do not explicitly account for gene-expression structure. GeneICL combines a transcriptomics-grounded prior and recurrent in-context learning, with reported gains across prediction and survival tasks.

  • Transcriptomic prediction combines tens of thousands of strongly correlated gene features with only tens to hundreds of labeled samples, while generic tabular models do not explicitly model this structure.The introduction also notes that benefits of self-supervised pretraining for supervised downstream prediction remain debated, with evaluations finding simple baselines can match or outperform larger transcriptomic models.Source: §1 · pp. 1–2
  • GeneICL predicts from transcriptomic profiles in principal-component space using a recurrent, weight-tied in-context learner and a semi-synthetic prior with realistic gene–gene correlations.The prior samples expression profiles from a VAE trained on real bulk RNA-seq data and assigns labels using random synthetic functions.Source: §1 · p. 2
  • The benchmark covers prognostic and diagnostic outcomes from bulk transcriptomic cohorts, including patient cohorts and cancer cell-line drug screens.Comparisons include tuned classical methods, transcriptomic foundation models, and general-purpose tabular foundation models.Source: §1 · p. 2
  • With 4.2M parameters, GeneICL matches or outperforms general-purpose tabular foundation models with up to 387× more parameters.The model uses a semi-synthetic prior derived from over 500,000 experimental RNA-seq profiles and a single in-context block applied recurrently.Source: §1 · p. 2
  • Cox partial-likelihood residual regression lets the frozen model rank patients by survival risk without architectural changes or survival-specific pretraining.The adaptation achieves the best overall rank against existing censoring adaptations, a dedicated survival architecture, and tuned conventional baselines.Source: §1 · p. 2

2 RELATED WORK

Prior work spans transcriptomic representation models, tabular in-context learners, synthetic priors, and recurrent transformers. GeneICL brings these directions together by conditioning jointly on labeled transcriptomes and adapting the learner’s prior and representation to bulk expression data.

2.1 TRANSCRIPTOMIC FOUNDATION MODELS

  • Transcriptomic foundation models commonly encode each sample independently, so downstream prediction typically requires fine-tuning or an added predictor; GeneICL instead processes labeled transcriptomes jointly for prediction without task-specific parameter updates.Prior evaluations have not consistently found pretrained transcriptomic representations to outperform simpler supervised or statistical baselines.Source: §2.1 · p. 3

2.2 TABULAR FOUNDATION MODELS

  • Tabular foundation models use in-context learning for supervised classification and regression, while GeneICL adapts their prior and input representation to bulk transcriptomic prediction.Prior survival adaptations have used specialized architectures, survival-specific priors, or target imputation; GeneICL instead regresses Cox partial-likelihood residuals for zero-shot survival prediction.Source: §2.2 · p. 3

2.3 SYNTHETIC PRIORS

  • Tabular foundation models use varied synthetic priors or real datasets for pretraining, and downstream behavior has been linked to the pretraining distribution.Source: §2.3 · p. 3

2.4 LOOPED TRANSFORMERS

  • Looped transformers repeatedly apply shared layers, an inductive bias proposed to align with iterative learning algorithms; prior work found that one repeated layer can retain comparable tabular in-context performance with fewer parameters.Source: §2.4 · p. 4

3 GENEICL

GeneICL frames bulk-expression prediction as fixed-parameter in-context learning over labeled support examples and unlabeled query samples. Its transcriptomics-grounded pretraining, recurrent architecture, dual-task objective, and Cox-residual regression cover classification, regression, and right-censored survival.

3.1 PROBLEM FORMULATION

  • GeneICL predicts query labels from labeled support examples with a fixed pretrained model and separate classification and regression heads.Downstream adaptation uses the support examples in context rather than task-specific parameter updates.Source: §3.1 · p. 4

3.2 ARCHITECTURE

  • Support-fitted PCA retains leading components explaining 90% of variance; a [CLS]-token bottleneck feeds a single weight-tied ICL block applied recurrently.Support labels are embedded; over eight steps, queries attend to support and thinking states but not other queries.Source: §3.2 · pp. 5–6, Figure 2 · p. 5

3.3 TRANSCRIPTOMICS-GROUNDED PRETRAINING PRIOR

  • The semi-synthetic prior decodes VAE samples fitted to 547,829 ARCHS4 profiles across 20,021 genes, then labels sampled PCA directions with randomized nonlinear functions.The corpus excludes benchmark contributors; paired categorical and continuous labels support effectively unlimited tasks while retaining structure learned from real expression profiles.Source: §3.3 · p. 6

3.4 TRAINING OBJECTIVE

  • Each task embeds either categorical or continuous support labels while supervising both query heads; a closed-form binormal ranking loss approximates macro one-vs-rest AUROC.The ranking term is computed from positive and negative query-score moments.Source: §3.4 · p. 7

3.5 COX-BASED TABULAR SURVIVAL PREDICTION

  • For right-censored survival, GeneICL regresses null-model Cox martingale residuals to rank query patients by risk; Breslow estimates yield survival curves.The residuals are gradients of the Cox partial log-likelihood with respect to log-risk at the null model.Source: §3.5 · p. 7

4 BENCHMARK

The benchmark spans classification, regression, and survival tasks drawn from clinical endpoints and cancer cohorts. It uses common cross-validation splits for classification and regression, tuned conventional baselines, and predefined survival splits.

4 BENCHMARK

  • Median sample sizes are 144 for classification, 491 for regression, and 202.5 for survival tasks.Figure 3 shows individual task sizes as dots, interquartile ranges as shading, and medians as ticks on a shared logarithmic scale.Source: §4 · p. 7, Figure 3 · p. 8
  • Classification and regression use identical five-fold splits, while survival uses the first five predefined SurvBoard splits and study-equal-weighted metric averages.Conventional baselines are tuned within training folds; foundation models use fixed inference settings.Source: §4 · p. 7
  • 27 classification, 25 regression, and 28 survival tasks comprise the benchmark.Classification covers treatment response, prognosis, and disease state or subtype; regression includes drug-sensitivity endpoints and other quantitative traits, while survival uses cancer cohorts.Source: §4 · p. 7, Figure 3 · p. 8

5 RESULTS

Tabular foundation models outperform self-supervised transcriptomic models on classification and regression, and GeneICL performs strongly across both these tasks and survival prediction. Ablations indicate that its transcriptomics-grounded prior benefits regression, while architectural simplifications do not reduce evaluated performance.

5.1 CLASSIFICATION AND REGRESSION

  • Self-supervised transcriptomic foundation models consistently trail the strongest general-purpose tabular models and conventional baselines, while tabular models are competitive with tuned supervised methods.Table 1 covers 27 classification and 25 regression tasks.Source: §5.1 · p. 8, Table 1 · p. 9
  • GeneICL’s ensemble achieves the best overall average rank, highest AUROC and AUPRC, and best results on all three regression metrics; tuned linear remains strongest in balanced accuracy.The standard GeneICL model has the second-best overall average rank, behind the ensemble.Source: §5.1 · p. 8

5.2 GENEICL CAN PREDICT SURVIVAL OUTCOMES

  • Across survival comparisons, GeneICL-Cox achieves the highest aggregate C-index and best D-calibration, while remaining within 0.001 of the best integrated Brier score.Figure 4 compares 28 SurvBoard datasets and reports study-weighted aggregates and mean within-dataset ranks.Source: §5.2 · p. 8, Figure 4 · p. 9

5.3 ABLATION STUDY

  • Replacing the recurrent block with eight untied blocks raises parameters from 4.24M to 19.25M without improving evaluated metrics; full attention raises feature-axis complexity from O(n) to O(n^2) with matched or lower performance.Table 2 presents ablations of GeneICL’s design choices.Source: §5.3 · p. 8, Table 2 · p. 10
  • Replacing the VAE-generated covariates with the TabICLv2 prior or real ARCHS4 profiles lowers all regression metrics, while classification ranking metrics remain largely unchanged.The reported R2 values are 0.238 and 0.248 for those alternatives, versus 0.271 for the VAE-generated covariates; VAE generation provides novel expression profiles without performance cost relative to real data.Source: §5.3 · p. 9
  • Generic inference-time ensembles provide modest gains on several metrics, whereas transcriptomics-specific ensembles based on highly variable genes or Hallmark pathways do not consistently improve performance.The evaluated generic strategies include PCA thresholds, support subsets, gene subsampling, and input transformations.Source: §5.3 · p. 10

6 DISCUSSION AND CONCLUSION

The authors conclude that supervised pretraining on transcriptomics-aware data can perform strongly across clinical outcome prediction tasks without matching the scale of competing tabular models. They identify modality, cohort-shift, and small-sample questions that remain open.

6 DISCUSSION AND CONCLUSION

  • Across 80 translational prognostic and diagnostic tasks, GeneICL outperforms larger transcriptomic foundation models and matches general-purpose tabular models with up to 387× more parameters.The benchmark includes classification, regression, and survival; the paper concludes that pretraining objective, rather than model scale, appears to limit current models.Source: §6 · p. 10
  • Replacing the transcriptomics-aware prior with the generic TabICLv2 prior or real profiles lowers regression performance, while one looped block matches eight untied blocks with under a quarter of their parameters.The resulting model runs in seconds on a laptop CPU.Source: §6 · p. 10
  • The bulk RNA-seq-derived prior leaves single-cell RNA-seq and other omics modalities untested, and single-cohort evaluation leaves robustness to support–query batch and platform shifts open.The authors also leave unresolved whether the small margin over linear baselines reflects limits of available profiles or of current priors and architectures.Source: §6 · p. 10

A EVALUATION DATASETS

The benchmark focuses on prognostic and diagnostic prediction from bulk transcriptomic cohorts rather than representation-quality tasks. It includes classification, regression, and survival tasks drawn from clinical cohorts, preclinical screens, and other longitudinal studies.

Scope

  • The benchmark comprises 27 classification, 25 regression, and 28 survival tasks, for 80 tasks across bulk transcriptomic cohorts.Table 3 lists task source, target, sample size, and type.Source: A EVALUATION DATASETS · p. 17, Table 3 · p. 17

Task categories

  • Classification and regression tasks cover treatment response, prognosis, non-invasive diagnosis, and some tumor or tissue phenotypes; survival is evaluated separately.The benchmark spans prognostic and diagnostic outcomes with direct relevance to clinical decisions.Source: A EVALUATION DATASETS · p. 17

Data sources

  • Classification and regression data come from large clinical cohorts, preclinical pharmacogenomic screens, and smaller GEO cohorts retained for baseline-first longitudinal designs.Candidate studies were excluded for small sample size, non-baseline sampling, or unpredictable targets; retained tasks are prospective in design.Source: A EVALUATION DATASETS · p. 17

Survival tasks

  • Survival tasks use 28 cancer datasets from TCGA, ICGC, TARGET, and METABRIC, using only the gene-expression modality.The datasets come from the SurvBoard benchmark, which standardizes preprocessing, splits, and evaluation.Source: A EVALUATION DATASETS · p. 17

B BENCHMARK DETAILS

The benchmark uses cross-validation for classification and regression and fixed SurvBoard splits for survival. Models receive standardized, PCA-projected expression inputs, with transcriptomic foundation models evaluated as frozen feature extractors followed by linear probes.

Splitting and cross-validation

  • Classification and regression use five-fold cross-validation; classification splits are target-stratified, and patient-level grouping prevents a patient from appearing in both support and query sets.Regression splits are random; R2 is computed from pooled out-of-fold predictions because fold-wise R2 is unstable for small folds.Source: B BENCHMARK DETAILS · p. 19
  • Survival evaluation uses the first five of SurvBoard’s 25 fixed splits, fitting all data-dependent preprocessing on support data only.This includes standardization and principal-component projection.Source: B BENCHMARK DETAILS · p. 19

Model inputs

  • GeneICL takes log(1 + CPM) expression over 20,021 genes and performs its own standardization and 90%-variance PCA; other tabular models receive the same steps beforehand.TabPFN-Wide is the exception and receives the log(1 + CPM) values directly.Source: B BENCHMARK DETAILS · p. 19

Transcriptomic foundation models

  • Transcriptomic foundation models are frozen feature extractors, with their embeddings replacing expression profiles before an elastic-net linear probe is fitted on each fold’s support set.Genes absent from a model’s vocabulary are set to zero; COMPASS receives a neutral cancer-type token for samples without a cancer-type token.Source: B BENCHMARK DETAILS · p. 19

C HOLDOUT TASKS FOR MODEL DEVELOPMENT

On five separate holdout tasks, the authors checked whether development choices transferred to the full benchmark and whether retaining more principal components helped. The section also records GeneICL hyperparameter choices.

C HOLDOUT TASKS FOR MODEL DEVELOPMENT

D HYPERPARAMETERS

  • The hyperparameter table marks configurations identical to TabICLv2; its footnotes describe the reasoning workspace, recurrence unrolling, and detached hidden states across segments.Source: Table 5 · p. 20, D HYPERPARAMETERS · p. 20

E COMPUTATIONAL RESOURCE REQUIREMENTS

GeneICL runs quickly in benchmark evaluation and on a laptop CPU. Its pretraining pipeline uses a VAE trained on bulk RNA-seq profiles to generate transcriptomic covariates for synthetic prediction tasks.

E COMPUTATIONAL RESOURCE REQUIREMENTS

  • GeneICL takes 0.01–0.4 seconds per fold, one to two orders of magnitude less than other tabular foundation models.Figure 5 plots runtime against support-set size across 52 tasks; in-context model runtime is a forward pass.Source: Figure 5 · p. 21, E COMPUTATIONAL RESOURCE REQUIREMENTS · p. 21
  • On a laptop CPU, GeneICL averaged 7.5 seconds per fold with 4.9 GB peak memory, versus 10.9 seconds per fold for elastic net.The comparison used the same five-fold protocol and no test-time ensembling.Source: E COMPUTATIONAL RESOURCE REQUIREMENTS · p. 21

F.1 VARIATIONAL AUTOENCODER

  • The VAE was trained on 547,829 filtered human bulk RNA-seq profiles and uses a 256-dimensional latent space.Source: §F.1 · p. 22
  • VAE training groups locally similar samples and combines reconstruction and KL terms with prior-sample covariance and within-group variance constraints.Generated and real gene–gene covariances correlate at r = 0.87, and generated samples retain 53% of real within-group variance.Source: §F.1 · p. 22

F.2 PRETRAINING PRIOR

  • Synthetic tasks use transcriptomic covariates and variance-weighted subsets of principal components to construct MLP scores and noisy classification or regression labels.Some regression targets are floored or capped to mimic detection limits and saturated clinical measurements.Source: §F.2 · p. 23

G SURVIVAL PREDICTION

GeneICL-Cox converts censored survival labels into residual targets for regression, while the evaluation compares alternative adaptations and survival models. The section also explains why GeneICL omits positional encodings and feature grouping.

G SURVIVAL PREDICTION

  • The comparison covers four censoring adaptations on a frozen GeneICL backbone, tuned conventional survival models, and SurvPFN.Source: G SURVIVAL PREDICTION · p. 24
  • GeneICL-Cox regresses Cox martingale residuals in one pass, using out-of-fold support predictions to estimate the Breslow baseline hazard for survival curves.In-sample support scores would be overseparated and bias the baseline hazard downward, decalibrating curves.Source: G SURVIVAL PREDICTION · p. 24
  • Alternative adaptations turn censored outcomes into targets through iterative Buckley–James imputation, Kaplan–Meier jackknife pseudo-observations, discrete-time hazards, or two-step imputation.Source: G SURVIVAL PREDICTION · p. 24
  • D-calibration is not reported for two-step imputation and sometimes failed for XGBoost, with missing XGBoost values imputed from same-model, same-dataset split means.Source: G SURVIVAL PREDICTION · p. 25

H.1 POSITIONAL ENCODINGS AND FEATURE GROUPING

  • GeneICL omits positional encodings and feature grouping because PC scores lack shared order or sign, are rarely permutation-identical, and support labels distinguish features by target relation.Source: §H.1 · p. 26

H.2 LOOPING

GeneICL’s recurrent formulation remains stable when inference extends far beyond training depth, but additional steps do not systematically improve benchmark performance. Generic test-time ensembles are generally competitive, whereas transcriptomic-specific feature subsets do not yield consistent gains.

H.2 LOOPING

  • GeneICL generally reaches maximum or near-maximum performance within the first few recurrence steps, and further iterations do not systematically improve it.Weight sharing alone is not sufficient for stable depth extrapolation.Source: §H.2 · pp. 26–27

H.3 INFERENCE-TIME ENSEMBLE

  • Generic ensembling strategies preserve or improve performance on at least some metrics, with the strongest strategy depending on the metric.The figure compares classification and regression metrics against a single-forward-pass baseline.Source: Figure 7 · p. 27, §H.3 · p. 28

H.2 LOOPING

  • At T = 128, GeneICL performance remains essentially unchanged from around its training depth, while plain recurrence deteriorates when iterated substantially beyond training.Declines for the plain baseline are particularly large for Pearson correlation, Spearman correlation, and R2.Source: §H.2 · p. 27

H.3 INFERENCE-TIME ENSEMBLE

  • The ensemble tests vary PCA thresholds, support subsets, gene subsets, expression transformations, and class-label permutations, alongside HVG and Hallmark gene-set views.Source: §H.3 · p. 28
  • Transcriptomic-specific ensembling gives no consistent gains: HVG-based ensembling substantially reduces classification performance, and Hallmark-pathway ensembling performs worse across reported metrics.Source: §H.3 · p. 28

Related papers

Loading 2610.08694v1…