Source-linked AI summary
TabBench-Bio: A Living Benchmark for Machine Learning on High-Dimensional Biomedical Tables
Jules Kreuer, Sofiane Ouaari, Julia Hellmig, Julius Braitinger, Nico Pfeifer
TL;DR
Biomedical tables often have thousands of measured variables but only tens or hundreds of labelled samples, a setting that general-purpose benchmarks cover poorly. TabBench-Bio evaluates models across biomedical datasets and feature-by-sample operating points under a shared protocol. Tabular foundation models generally rank highly, but the strongest configuration varies across operating points and modalities.
Problem
Biomedical HDLSS comparisons are heterogeneous, and general-purpose benchmarks provide limited evidence on the high-dimensional, low-sample settings common in molecular biomedicine.
Source: §1 · p. 1Method
TabBench-Bio evaluates 43 biomedical datasets across a shared feature-by-sample grid using common folds and saved fold-level predictions for reproducible aggregation.
Source: §1 · p. 2, §3.5 · p. 5Results
Takeaways & Limitations
Researchers should consider performance under conditions relevant to their data when selecting a model, since the strongest configuration depends on biological representation.
Source: §7 · p. 8, §5 · p. 8Full-sample and fixed-budget conditions use different target pools, so ranking differences cannot be attributed to training-set size alone.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Biomedical tables often combine thousands of measured variables with only tens or hundreds of labelled samples, a regime that is poorly represented in general-purpose tabular benchmarks. We introduce TabBench-Bio, a living and interactive benchmark of 43 biomedical datasets spanning multiple domains. Under a shared cross-validation protocol, we compare classical estimators, neural networks, and tabular foundation models across 28 feature-by-sample operating points. At the reference cell of 10,000 features and 100 training samples, RealTabPFN 2.5 has the highest point estimate, closely followed by TabPFN 3 and Logistic Regression. A paired bootstrap over the target pool separates RealTabPFN 2.5 from TabPFN 3 by 87 Elo (95\% interval [43, 129]). Tabular foundation models generally occupy the leading ranks, while the strongest configuration depends on the operating point and biomedical modality. The AutoML framework AutoGluon, using its one-hour "extreme" preset, is configured as a separate resource-intensive reference and reported here at the reference and full cell. Fold-level predictions, run status, and deterministic aggregations make every reported result reproducible and reusable. The benchmark is open to contributions of new biomedical datasets. The interactive leaderboard is available at https://tabbench-bio.eu.
1 Introduction
TabBench-Bio addresses the difficulty of comparing biomedical machine-learning results across heterogeneous cohorts, assays, preprocessing, and validation protocols. It evaluates models across a shared feature-by-sample grid and provides a versioned, reproducible benchmark of biomedical datasets and model configurations.
1 Introduction
- Prior biomedical comparisons often use limited cohorts and estimators, while preprocessing and validation differences impede comparison and single operating points obscure robustness in HDLSS settings.Source: §1 · p. 1
- Biomedical HDLSS data combine thousands of variables with tens or hundreds of samples, and model feasibility and estimation depend on feature-to-sample ratios.Source: §1 · p. 1
- A benchmark spanning measurement processes is needed because Raman spectra have an ordered wavenumber axis unlike the biomedical tables considered here.Source: §1 · p. 1
- A shared 4-by-7 grid with common folds keeps the held-out population fixed, making robustness across data geometries a primary benchmark outcome.The grid contains 28 operating points.Source: §1 · p. 2
- The benchmark registers 43 biomedical datasets spanning heterogeneous molecular and sequence-derived representations under a shared protocol.Source: §1 · p. 2
- Its common implementation-level comparison covers 19 predictive configurations and a constant baseline, with AutoGluon as a separate resource-intensive reference at two operating points.Source: §1 · p. 2
- The benchmark uses leak-resistant fold-aligned evaluation, explicit failure semantics, and cell-conditional Bradley-Terry rankings.Source: §1 · p. 2
- Fold-level predictions, run status, resource measurements, deterministic aggregations, and an interactive leaderboard support reuse of benchmark results.Source: §1 · p. 2
2 Related work
Prior work offers tabular evaluation frameworks, model comparisons, domain-specific benchmarks, and biomedical data resources. TabBench-Bio builds on these approaches while focusing on biomedical HDLSS tasks and comparisons across heterogeneous representations.
2 Related work
- TabArena and TALENT provide curated datasets, implementations, and leaderboards, while TabArena emphasizes living governance and reusable predictions (Erickson et al., 2025; Liu et al., 2025).Source: §2 · p. 2
- Prior tree and neural comparisons find relative performance depends on dataset scale, feature type, and tuning protocol.Source: §2 · p. 2
- Tabular foundation models make differing assumptions about how evidence scales with rows and columns, motivating evaluation across a two-dimensional grid.Source: §2 · p. 2
- RamanBench provides a live leaderboard precedent, but spectral-locality architectures need adaptation for omics tables without an analogous column ordering (Koddenbrock et al., 2026).Source: §2 · p. 2
- Biomedical resources span molecular, microbiome, drug-discovery, bioactivity, genomic-prediction, and sequence-derived data, which TabBench-Bio unifies in a shared evaluation protocol.Source: §2 · p. 2
3 TabBench-Bio
TabBench-Bio evaluates fixed model configurations on static numerical biomedical representations using a shared, leak-conscious protocol. Its dataset registry, operating-point grid, fold design, and aggregation rules define the scope and interpretation of the comparisons.
3 TabBench-Bio
- The benchmark covers supervised classification and regression on static, pre-extracted numerical biomedical representations in the HDLSS regime.Source: §3 · p. 2
- Its current scope excludes survival analysis, longitudinal modelling, multimodal integration, external-site validation, and supervised feature selection.Each excluded task would change the prediction unit, endpoint, or source of supervision addressed by the protocol.Source: §3 · p. 2
3.1 Dataset scope and selection
- The registry contains 43 datasets across seven modality groups, with classification and regression tasks drawn from multiple biomedical data sources.Source: §3.1 · p. 3
- Dataset screening combines automated signal checks and manual review of quality, leakage risks, metadata, and reproducible construction; classes with fewer than ten samples are removed.Source: §3.1 · p. 3
3.2 Models and fitting regime
- The peer comparison includes 19 predictive configurations and a constant baseline, uses default adapter configurations without per-dataset tuning, and disables benchmark-level bagging, stacking, and ensembling.Source: §3.2 · pp. 3–4
- Models share frozen splits and configurations under one-hour fitting and prediction timeouts; AutoGluon Extreme is a separate resource-intensive reference evaluated at two prespecified cells.The results described in the passage use the reference cell.Source: §3.2 · p. 4
3.3 Operating-point grid
- Sample budgets change training partitions while held-out folds remain fixed, and budget comparisons reflect the full procedure, including class allocation and training-dependent feature filtering.Source: §3.3 · p. 4
- Feature caps use nested prefixes of a seeded random feature order, with filtering refitted by sample budget; the grid tests retaining larger random subsets, not feature selection.Source: §3.3 · p. 4
4 Results
Across 28 operating points, TabBench-Bio compares models on biomedical datasets under a shared protocol. Foundation models generally rank highly, but the leading configuration varies with operating point and modality.
4.1 Reference-cell ranking
- The reference cell ranks 42 of 43 datasets because one dataset has fewer training rows than the cell’s 100-sample budget.The cell was selected before inspecting model results; 42 targets are sample-budget-bound.Source: §4 · p. 5, §4.1 · pp. 5–6
- At 10,000 features and 100 training samples, RealTabPFN 2.5 leads with Elo 1156, and its paired-bootstrap advantage over TabPFN 3 is 87 Elo [43, 129].RealTabPFN 2.5 wins 32 of 42 targets, ties two, and loses eight; Logistic Regression has the highest macro-F1 at 0.721 versus 0.713 for RealTabPFN 2.5.Source: Table 1 · p. 6, §4.1 · p. 6, Figure 2 · p. 7
- Memory and time-limit failures receive constant-predictor performance, so the strict ranking also reflects whether models run within the resource limits.Failure counts differ substantially across models, and design skips are not included.Source: §4.1 · p. 6, Table 1 · p. 6
4.2 Model order depends on the operating point and modality
- At 10,000 features, the linear baseline moves from shared first place at 20 training rows to 16th at 1,000.Figure 3 compares six sample budgets using strict fold-level Elo; eligible target coverage varies across budgets.Source: §4.2 · p. 6, Figure 3 · p. 7
- At 100 training rows, increasing the feature budget from 2,000 to 10,000 shifts TabICLv2 from rank 17 to 19 and Logistic Regression from 7 to 4.RealTabPFN 2.5 remains first despite sharing a regular 2,000-feature limit with TabICLv2.Source: §4.2 · p. 6
- At the reference cell, RealTabPFN 2.5 leads gene expression, metagenomics, and molecular properties; other modality leaders include TabFM, TabPFN Wide (8k), Random Forest, and Extra Trees.TabFM leads four embedding targets, TabPFN Wide (8k) leads methylation, and Random Forest and Extra Trees lead four genomic-prediction regression targets; these pools contain at most 10 targets.Source: §4.2 · p. 6
5 Discussion
Tabular foundation models generally occupy leading ranks, but the benchmark does not support one default model. The observed leaders vary by biomedical modality, and comparisons with general-purpose benchmarks are not directly comparable.
5 Discussion
- Tabular foundation models generally occupy the leading ranks, but no single default model leads across operating points and biomedical modalities.The modality-specific comparisons favor matching a model to the representation and endpoint; regression and embedding pools are smaller and warrant more caution.Source: §5 · p. 8
- TabArena and TabBench-Bio rankings are not directly comparable because they differ in dataset collection and use of tuned and ensembled variants.The effects of domain, metric, tuning, and ensembling cannot be fully disentangled.Source: §5 · p. 8
- Under AUROC, Logistic Regression leaves the leading group and tabular foundation models hold the top six places.The authors note that part of the contrast reflects threshold placement rather than domain alone.Source: §5 · p. 8
6 Limitations
The reported rankings are bounded by dataset composition, feature capping, default configurations, changing target coverage, and overlap between some benchmark data and TabDPT pretraining.
6 Limitations
- The overall ranking reflects this release’s classification-regression balance and uneven representation of biomedical modalities.The leaderboard allows users to restrict comparisons to a modality and inspect corresponding performance and resource measurements.Source: §6 · p. 8
- Uniform label-agnostic feature capping may omit sparse biological signals, and the five seeded feature orderings do not cover their full distribution.Feature-width comparisons do not estimate the value of supervised feature selection.Source: §6 · p. 8
- The comparison measures default behavior rather than performance after equal hyperparameter tuning, and AutoGluon’s one-hour Extreme preset covers only two operating points.Sample-budget comparisons may also reflect changes in class allocation and training-dependent feature filtering, rather than sample count alone.Source: §6 · p. 8
- Larger fixed sample budgets have lower target coverage, so rankings across budgets cannot be attributed to training-set size alone.Bradley-Terry intervals assume a single transitive ability per model, and rare-class filtering precludes conclusions about underrepresented classes.Source: §6 · p. 8
- Three benchmark datasets come from the GEMLeR/expO collection, which also supplied two datasets used in TabDPT pretraining.TabDPT results on the related datasets warrant this overlap caveat; neither RealTabPFN model has overlap.Source: §6 · p. 8
7 Conclusion and Outlook
TabBench-Bio compares learners on biomedical tables with many features and few samples, finding that leading configurations vary with operating point and biological representation.
7 Conclusion and Outlook
- Across 43 datasets, tabular foundation models generally occupy the leading ranks, while the strongest configuration depends on operating point and modality.At the reference cell, RealTabPFN 2.5 leads the point ranking, followed by TabPFN 3 and Logistic Regression, whose point estimates are nearly identical.Source: §7 · p. 8
- At the reference cell, a paired bootstrap separates RealTabPFN 2.5 from TabPFN 3, despite their close point estimates.The supplied conclusion does not report the size or interval of the separation.Source: §7 · p. 8
- TabBench-Bio exposes behavior on heterogeneous biomedical measurements with many features and few samples that general-purpose suites rarely measure.The benchmark is expanding to include new models and tuned classical baselines and welcomes datasets, particularly from underrepresented assays and clinical endpoints.Source: §7 · p. 8
- Researchers should consider model performance under conditions relevant to their data when selecting a model.The benchmark supports comparison across biomedical representations and operating points.Source: §7 · p. 8
Reproducibility and Data availability
The benchmark releases its code, registry, reporting materials, result snapshots, and leaderboard, with manifests connecting recorded runs to published outputs.
Reproducibility and Data availability
- The benchmark code, versioned registry, reporting scripts, and released result snapshots are available in the project repository.The interactive leaderboard and machine-readable exports are served at tabbench-bio.eu.Source: Reproducibility and Data availability · p. 9
- Locally generated embedding datasets are public, including target labels, cross-validation group identifiers, and construction scripts.Source: Reproducibility and Data availability · p. 9
- Frozen splits and deterministic aggregation are linked to raw predictions and terminal status records through a per-release manifest with cryptographic digests.The manifest connects these records to the generated tables, figures, and manuscript values.Source: Reproducibility and Data availability · p. 9
Ethics statement
The study evaluates existing secondary datasets and does not establish clinical validity or support use of its rankings in patient care.
Ethics statement
- The study evaluates published models on existing secondary datasets, including locally generated sequence embeddings from existing biological sequences.Source: Ethics statement · p. 9
- No new human-subject data were collected, no institutional review board approval was required, and no re-identification was attempted.Some source datasets derive from human cohorts, so results describe default model behavior on retrospective research data.Source: Ethics statement · p. 9
- The benchmark does not establish clinical validity, and its rankings do not support use in patient care.Source: Ethics statement · p. 9
AI use statement
This block documents AI assistance, model defaults, fit-time comparisons, full-data rankings, and leading models across the benchmark grid.
AI use statement
- LLMs assisted with coding, writing, and code review, but not research questions, protocol design, synthetic data, mathematical claims, or result interpretation.Authors reviewed AI-assisted work; assisted code was read by at least two authors and covered by the repository test suite.Source: AI use statement · p. 9
A Model configuration defaults
- Peer models use no hyperparameter optimization or AutoGluon bagging; unlisted parameters retain library defaults, with native early stopping and a common one-hour timeout.AutoGluon Extreme is configured as a separate external reference.Source: A Model configuration defaults · p. 13, Table 2 · p. 13
B Performance and fit time
- At the reference cell, RealTabPFN 2.5 takes 78.9 seconds per fold, versus 33.1 seconds for second-place Logistic Regression.Fit times include shared harness overhead and exclude prediction, pretraining, memory, and energy costs; Figure A1 plots mean macro-F1 against median fit time on a logarithmic time axis.Source: Figure A1 · p. 14, B Performance and fit time · p. 14
C Full-feature, full-sample ranking
- Figure A2 reports strict fold-level Elo after shared preprocessing using all available training rows and features, with target-bootstrap intervals relative to Random Forest at 1,000.Most models cover 43 targets; classification-only TabPFN-Wide models cover 30, and the vertical line marks the one-hour AutoGluon Extreme reference.Source: Figure A2 · p. 15
D Leading models across the grid
- Figure A3 shows the top three models at each grid point; colors identify models, numbers give median Elo, and rank differences do not establish significance.Target coverage varies across cells; AutoGluon is excluded as an external reference.Source: Figure A3 · p. 16
E Metric sensitivity
At the reference cell, the ordering changes under AUROC, and omitting failures also changes model ranks. The registry defines evaluated sample counts and native feature widths, and documents how local sequence datasets were constructed.
E Metric sensitivity
- RealTabPFN 2.5 has the highest AUROC point estimate, ahead of TabPFN Wide 5k (ne3) and TabPFN 3, with the three separated by 20 Elo.This is a post-hoc metric sensitivity analysis using fits selected or early-stopped based on macro-F1; AUROC-based selection could change results.Source: Figure A4 · p. 17, E Metric sensitivity · p. 17
F Recorded failure rates
- Failure counts are concentrated in a few models, so the deployability component of the reference-cell ranking affects some models more than others.Memory and time-limit failures enter the rankings as the constant predictor; design skips do not.Source: Table 3 · p. 18, F Recorded failure rates · p. 18
F.1 Failure-policy sensitivity
- The two failure policies have a rank correlation of 0.72, and 16 models change rank; TabICLv2 drops 17 places after losing 31 targets.Omitting failures changes target coverage across models, so this sensitivity analysis does not establish bounds on ratings or ranks.Source: Figure A5 · p. 19, §F.1 · p. 19
G Dataset registry
- The registry records evaluated sample counts after removing classes with fewer than ten samples and native feature widths before cell-level feature caps.A dataset wider than a cell’s budget contributes only a random subset of its native columns in that cell.Source: Table 4 · p. 20, G Dataset registry · p. 20
G.1 Construction of local embedding datasets
- Three local sequence datasets use the shared registry; BRCA1 reference and variant vectors are concatenated, while protein embeddings are mean-pooled with residue-count weighting for long sequences.The BRCA1 variants are grouped by reference-sequence context; protein tasks use homology partitions or substitution position for cross-validation groups.Source: §G.1 · p. 21