PaperlayerView exact revision on arXiv

Source-linked AI summary

TabArena: A Living Benchmark for Machine Learning on Tabular Data

Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, David Salinas, Frank Hutter

arXiv:2506.16791v4cs.LGcs.AI

TL;DR

Existing tabular machine-learning benchmarks are rarely maintained despite concerns about dataset curation and evaluation. TabArena creates a living benchmark through manual curation and large-scale evaluation, finding that tuned and ensembled deep learning methods match or exceed gradient-boosted trees, foundation models dominate on small data, and cross-model ensembles advance the state of the art.

  • Problem

    Existing tabular machine-learning benchmarks are rarely maintained after publication, despite concerns about dataset curation and evaluation protocols.

    Source: §1 · p. 2

  • Method

    TabArena creates a living benchmark by manually curating datasets and models, evaluating them at scale, and sharing reproducible code and benchmark results.

    Source: §1 · p. 2, §2.2 · p. 4, §5 · p. 10

  • Results

    With tuning and ensembling, the best deep learning methods were equal to or better than gradient-boosted trees; foundation models dominated small data, and cross-model ensembles advanced the state of the art.

    Source: §1 · p. 2, §3.2 · p. 9

  • Takeaways & Limitations

    Leaderboard rank alone is an incomplete guide to ensemble value: a simulated pipeline using all TabArena models outperformed every individual model and AutoGluon.

    Source: §3.2 · p. 9

    TabArena-v0.1 focuses on IID classification and regression in the small-to-medium data regime.

    Source: §1 · p. 2

Abstract

from arXiv · show

With the growing popularity of deep learning and foundation models for tabular data, the need for standardized and reliable benchmarks is higher than ever. However, current benchmarks are static. Their design is not updated even if flaws are discovered, model versions are updated, or new models are released. To address this, we introduce TabArena, the first continuously maintained living tabular benchmarking system. To launch TabArena, we manually curate a representative collection of datasets and well-implemented models, conduct a large-scale benchmarking study to initialize a public leaderboard, and assemble a team of experienced maintainers. Our results highlight the influence of validation method and ensembling of hyperparameter configurations to benchmark models at their full potential. While gradient-boosted trees are still strong contenders on practical tabular datasets, we observe that deep learning methods have caught up under larger time budgets with ensembling. At the same time, foundation models excel on smaller datasets. Finally, we show that ensembles across models advance the state-of-the-art in tabular machine learning. We observe that some deep learning models are overrepresented in cross-model ensembles due to validation set overfitting, and we encourage model developers to address this issue. We launch TabArena with a public leaderboard, reproducible code, and maintenance protocols to create a living benchmark available at https://tabarena.ai.

1 Introduction

TabArena argues for replacing static tabular benchmarks with a sustainable, versioned, professionally maintained living system developed as an open-source community project.

1 Introduction

  • TabArena addresses outdated or flawed datasets and stale baselines in static benchmarks, which can reproduce prior shortcomings and miss the state of the art [10] [11].The authors propose a living system that is versioned, professionally maintained, and gradually improved by the community.Source: §1 · p. 2
  • After investigating 1,053 datasets, TabArena manually curates 51 datasets representing real-world tabular tasks.The benchmark examines common IID classification and regression problems in the small-to-medium data regime.Source: §1 · p. 2
  • TabArena curates 16 tabular models, including 3 foundation models, and trains approximately 25,000,000 model instances in well-tested pipelines.The pipelines are intended to reflect modeling practices used in practice.Source: §1 · p. 2
  • The project provides a public leaderboard, precomputed results, reproducible benchmarking code, and an experienced maintainer team to keep the benchmark current.The leaderboard initializes with TabArena version 0.1.Source: §1 · p. 2
  • TabArena-v0.1 leaves non-IID data, few-shot and very small or large datasets, and tasks such as clustering or survival analysis for future work.Its initial focus is IID classification and regression, excluding fewer than 500 or more than 250,000 training samples.Source: §1 · p. 2
  • TabArena-v0.1 finds tuned post-hoc ensembles improve individual models, deep methods match or surpass boosted trees, foundation models excel on small data, and cross-model ensembles lead.The best individual models do not contribute equally to the ensemble pipeline.Source: §1 · p. 2

2 TabArena

TabArena is designed as a living benchmark whose curation and evaluation protocols are continuously applied and refined to keep it current. It covers model selection and optimization, dataset curation, and evaluation design.

2.1 Models and Hyperparameter Optimization Protocol

  • TabArena benchmarks 14 state-of-the-art (foundation) models and two baselines using curated implementations, hyperparameter optimization, validation, and ensembling strategies.Models are run only within their stated restrictions; most tunable models receive one default and 200 sampled configurations, while TabICL and TabDPT are evaluated at default settings only.Source: §2.1 · pp. 2–4

2.2 Datasets Protocol

  • TabArena manually filters 1,053 datasets from prior benchmarks for a representative collection of IID, real-world predictive tabular tasks.Criteria include unique tasks, 500–250,000 training samples, usable licenses, public accessibility, and no obvious ethical concerns; several require human judgment.Source: §2.2 · pp. 4–5
  • Curation revealed inconsistent dataset versions across benchmarks and surprisingly few datasets suitable for IID tabular benchmarking; TabArena invites community review and additional data contributions.Contributions beyond its initial scope, including non-IID, tiny, or large datasets, are also sought.Source: §2.2 · p. 5

2.3 Evaluation Design Protocol

  • Evaluation repeats experiments more often on smaller datasets and ranks models primarily with Elo, giving each dataset equal weight in the aggregation.Datasets under 2,500 samples use 10 repeats of 3-fold outer cross-validation; others use 3 repeats. Elo uses ROC AUC, log-loss, or RMSE by task and 200 bootstrap rounds for 95% confidence intervals.Source: §2.3 · p. 6
  • TabArena includes AutoGluon as its first reference pipeline and defines a protocol for submitting models to its continuously updated leaderboard.The reference uses AutoGluon version 1.3, the best_quality preset, and four hours of training.Source: §2.3 · p. 6

3 Results

Across curated tabular datasets, TabArena shows that measured peak performance depends on validation and ensembling choices, while complementary models can improve results beyond individual methods.

3 Results

  • The initial leaderboard benchmarks 16 curated models on 51 curated datasets, with GPU runs for TabM, ModernNCA, and foundation models and CPU runs for the others and AutoGluon.The experiments used 32 GB RAM and 100 GB disk space; total compute time without parallelization was ∼15 wall-clock years.Source: §3 · pp. 6–7

3.1 Assessing Peak Performance: The TabArena-v0.1 Leaderboard

  • After post-hoc ensembling, neural networks are strongest single models on average; TabM, LightGBM, and RealMLP otherwise rank below CatBoost, which leads conventional tuning.The leaderboard compares defaults, a tuned single configuration, and weighted post-hoc ensembles of hyperparameter configurations.Source: §3.1 · p. 7
  • On restricted small datasets, TabPFNv2 outperforms related approaches by a large margin and, with tuning plus post-hoc ensembling, beats AutoGluon.Figure 4 covers TabPFNv2-compatible datasets with ≤10K training samples and ≤500 features, and TabICL-compatible classification datasets with ≤100K samples and ≤500 features.Source: §3.1 · p. 7, Figure 4 · p. 7
  • Tuned EBM and CatBoost offer strong inference-time efficiency, while stronger Pareto points cost ∼15× more inference time for TabM-TE and ∼100× for RealMLP-TE.RealMLP only starts to dominate gradient-boosted trees after considerable training time with an ensemble of 25+ configurations.Source: §3.1 · pp. 7–8, Figure 5 · p. 8

3.2 Holistic Benchmarking of Peak Performance with TabArena

  • Holdout validation instead of cross-validation greatly underestimates all models and biases comparisons toward models that already use ensembling.The authors cite this finding as support for nested cross-validation in the benchmark design.Source: §3.2 · p. 8, Figure 6 · p. 8
  • TabArena shares result artifacts for cheaper comparisons and maintains TabArena-Lite, a one-outer-fold subset intended to support academic studies and discovery of dataset-specific breakthroughs.The authors consider a model worthy of publication if it significantly outperforms others on at least one dataset, even without leading on average.Source: §3.2 · pp. 8–9
  • A simulated ensemble across all TabArena models outperforms every individual model and AutoGluon, advancing the state-of-the-art.Top leaderboard performance does not necessarily predict ensemble weight; validation-overfitting models such as ModernNCA and RealMLP may receive high weights.Source: §3.2 · p. 9, Figure 7 · p. 9
  • Because GBDTs and deep learning both contribute to stronger ensembles, the paper frames finding ensemble-compatible models as more important than simply beating GBDTs.The authors hypothesize that code quality and maintenance contribute to deep learning’s slower adoption, and TabArena wraps deep learning methods in an easy-to-use interface.Source: §3.2 · p. 9

4 Related Work

Prior tabular benchmarks lack a consistent shared framework, and studies have identified recurring data and evaluation flaws. TabArena addresses these gaps with a curated, actively maintained benchmark and public leaderboard.

4 Related Work

  • Tabular benchmarking spans many benchmarks and independent evaluation frameworks, leaving the community without a consistent shared understanding or platform for collective improvement.TabArena aims to provide both.Source: §4 · p. 9
  • TabArena addresses outdated or unsuitable datasets and frequent evaluation-protocol flaws through extensive data curation and evaluation in sophisticated ensembling pipelines.Reported concerns include non-IID data used inappropriately and datasets that are not originally tabular.Source: §4 · pp. 9–10
  • TabArena consolidates strong aspects of prior benchmarks and is distinguished by active maintenance protocols and a public leaderboard for tabular machine-learning models.AutoML and TALENT [36] are identified as the most similar benchmarks; both have received updates and are actively maintained, but lack TabArena’s stated combination of maintenance protocols and public leaderboard.Source: §4 · p. 10
  • TabArena draws inspiration from live leaderboards in other domains but differs through its focus on predictive machine learning for tabular data and its associated models, datasets, and evaluation design.Examples include ChatBot Arena [39], RewardBench [52], LiveBench [53], the Huggingface Open LLM Leaderboard, and GIFT-Eval [56].Source: §4 · p. 10

5 Conclusion

TabArena establishes a living, reproducible benchmark for tabular machine learning, intended to evolve with the research community. Its initial study finds strong performance from deep learning and foundation models, with ensembling further improving results.

5 Conclusion

  • TabM, RealMLP, and foundation models for small data perform similarly to or better than gradient-boosted trees, while post-hoc ensembling can dramatically improve performance.The reported assessment used 16 models and 51 representative real-world tasks.Source: §5 · p. 10
  • TabArena’s initial benchmark curated 16 models and 51 real-world tasks from 1,053 datasets and ran about 25 million experiments, sharing metadata and reproducible code.The runs totaled approximately 15 years of wall-clock time; the authors describe the study as their most representative assessment of tabular-data state of the art to date.Source: §5 · p. 10

Limitations and Societal Impact

  • Limitations include fixed random configurations, hardware-sensitive time limits, restricted dataset coverage, no feature engineering, and risks of leaderboard overfitting or data contamination.The authors propose keeping TabArena alive to address open-source benchmarking challenges; they also note that feature engineering could change model rankings.Source: §5 · p. 10
  • TabArena aims to improve benchmark trustworthiness and guide practitioners on small- to medium-sized IID tabular data, while shared artifacts are intended to offset its upfront environmental costs.The authors characterize the work as a step toward straightforward, reliable benchmarking that will grow through community cooperation.Source: §5 · p. 10

Future Work

  • TabArena-v1.0 aims to support classification and regression on any tabular dataset by adding tiny, large, and non-IID data, new models, and more datasets.The authors envision developing this broader benchmark with the entire community.Source: §5 · p. 10

Competing Interests

The authors disclosed competing interests involving authorship, development, maintenance, contribution, and company affiliation; they declared no other competing interests.

Competing Interests

  • Disclosed ties include roles involving RealMLP, TabICL, TabRepo, AutoGluon, FastAI MLP, Torch MLP, TabPFNv2, OpenML, and PriorLabs.D.H. authored RealMLP and TabICL; D.S. and N.E. authored TabRepo; N.E., L.P., and P.M.D. develop AutoGluon and maintain FastAI MLP and Torch MLP; L.P. and F.H. co-authored TabPFNv2. L.P. contributes to OpenML, and F.H. is affiliated with PriorLabs.Source: Competing Interests · p. 11

NeurIPS Paper Checklist · A Additional Experiments · A.1 Alternative Leaderboard Versions

This section maps TabArena’s contributions and findings to the paper’s sections and documents its reproducibility, limitations, and transparency practices. It also examines alternative leaderboard aggregations, metrics, and dataset subsets, including pairwise win rates among top models.

NeurIPS Paper Checklist

  • The checklist says the abstract and introduction’s claims about TabArena, its benchmark, results, and public resources are addressed across Sections 2–3 and Appendix E.These claims include curated datasets, a large-scale study and public leaderboard, validation and ensembling findings, foundation models on smaller datasets, cross-model ensembles, and individual model contributions.Source: NeurIPS Paper Checklist · p. 19
  • The authors report that limitations and positive and negative societal impacts are discussed in Section 5, and that the paper includes no theoretical results.The checklist also states that results and scope claims are intended to reflect the paper’s contributions and experimental results.Source: NeurIPS Paper Checklist · pp. 19, 20, 23
  • The authors report public release of all data, code, and result artifacts, with benchmark and experimental details in Sections 2–3 and user guides in Appendix E.They also say training and test details are provided in the appendix.Source: NeurIPS Paper Checklist · pp. 20–21
  • The checklist reports error bars in all result plots, statistical-significance analysis in Appendix A.5, and compute-resource details in Section 3.It also states that assets are selected under licenses permitting their use and that creators and licenses are credited in Appendix B.2.Source: NeurIPS Paper Checklist · pp. 22, 24
  • The authors state that they release no new models, their released code and result artifacts pose no perceivable misuse risk, and the work involves no human subjects or core-method LLM use.They report reviewing copyright and fair-use issues during data curation in Appendix B.1.Source: NeurIPS Paper Checklist · pp. 22–25

A.1 Alternative Leaderboard Versions

  • Appendix A.1 compares Elo, normalized scores, and average ranks using per-dataset errors averaged over outer folds.Errors are 1-AUROC for binary, logloss for multiclass, and RMSE for regression; normalized scores follow Salinas and Erickson [37], with scores below zero clipped to zero.Source: §A.1 · p. 27
  • The harmonic mean of ranks favors methods with very low ranks on some datasets, even when they perform very poorly on others, over consistently mediocre methods.Such methods may be useful in conjunction with other methods.Source: §A.1 · p. 27
  • Improvability measures how many percent lower the best method’s error is than the current method’s error on a dataset, then averages this quantity across datasets.It ranges from 0% to 100%.Source: §A.1 · p. 27
  • Among the best-performing models, win rates over strong competitors are rather moderate, undermining the idea of a one-size-fits-all solution.The appendix also reports task-specific leaderboards and comparisons on TabPFNv2- and TabICL-compatible datasets.Source: §A.1 · p. 27
  • The all-model leaderboard imputes inapplicable results with default RandomForest, while additional peak-average metrics can change rankings and reduce sensitivity to imputation-driven negative outliers.Default RandomForest is chosen as a fast baseline that is sufficiently, but not unreasonably, weak to penalize methods that do not apply to every dataset.Source: §A.1 · p. 27
  • The appendix presents leaderboards for binary classification, multiclass classification, and regression, plus comparisons on TabPFNv2- and TabICL-compatible datasets.It also presents a comparison restricted to datasets compatible with both foundation models.Source: §A.1 · p. 27

A.2 Analyzing Training Time Limit

The section evaluates a 1-hour training limit per configuration and finds that fewer than 1% of configurations reach it, while EBM runs were not stopped at the limit and may have benefited.

A.2 Analyzing Training Time Limit

  • Each configuration must finish training across all 8 inner folds within 1 hour on one train split, or training is gracefully stopped early.Figure A.9 visualizes training runtimes across hyperparameter configurations for all models.Source: §A.2 · p. 27
  • Fewer than 1% of configurations reach the 1-hour limit; EBM training was not stopped at the limit, positively influencing results, but was not rerun because this affected few trials.The appendix further investigates the time limit for GPU-optimized models in Appendix A.3.Source: §A.2 · p. 31

A.3 Tabular Deep Learning on GPU vs. CPU … B.2 Included Datasets Details

The appendices examine hardware and tuning effects, introduce a lower-cost benchmark, assess statistical significance, and document dataset curation. Together, they describe how TabArena’s results and dataset collection are evaluated and maintained.

A.3 Tabular Deep Learning on GPU vs. CPU

  • TabM and ModernNCA train much faster on GPU than CPU, and a one-hour CPU limit would non-marginally harm their predictive performance.RealMLP trains only slightly faster on GPU, with marginal predictive-performance impact; about 16% of TabM and ModernNCA CPU runs hit the time limit versus less than 0.1% on GPU.Source: §A.3 · p. 32, Figure A.10 · p. 32

A.4 TabArena-Lite

  • TabArena-Lite uses all datasets with one outer fold and 200 configurations per model, and its results are similar to TabArena’s, making it a performance indicator.The authors recommend starting new models with one default configuration, optionally adding fewer random configurations, and use Lite to prioritize integrations; a novel model outperforming others on at least one dataset is considered valuable.Source: §A.4 · p. 33, Figure A.11 · p. 33

A.5 Investigating Statistical Significance

  • Critical-difference analyses find a statistically indistinguishable top group containing at least one deep-learning model and one GBDT, plus TabPFNv2 and TabICL when compatible.The analyses use Friedman tests followed by Nemenyi post-hoc tests at α = 0.05 on tuned-and-ensembled peak performance across the full benchmark and model-compatible datasets.Source: §A.5 · p. 33, Figure A.12 · p. 34, Figure A.13 · p. 34

A.6 Investigating Tuning Trajectories and Validation Overfitting

  • Most models’ performance saturates by 201 tuning configurations, while ModernNCA peaks at 25 and then degrades; TabM is notably more resistant to validation overfitting.ModernNCA ranks sixth by test Elo but first by validation Elo; with 10 ensembled random configurations it overfits more than any non-deep-learning method with 201, whereas TabM’s resilience may relate to internal ensembling, joint early stopping, and a moderate search space.Source: §A.6 · p. 35, Figure A.15 · p. 35, Figure A.16 · p. 36
  • The TabArena ensemble favors validation-overfit ModernNCA and RealMLP over models such as TabM, yet still outperforms all methods, including AutoGluon.The authors suggest overfitting-aware configuration selection could improve the ensemble further.Source: §A.6 · p. 35

B Data Curation

  • TabArena-v0.1 curation surveyed 14 prior studies, combined their collections into 1,053 uniquely named datasets, and retained 51 after manual screening for realistic predictive tasks.Curators applied documented criteria, shared notes and metadata, and describe the resulting cleanup as imperfect and open to continued revision.Source: B Data Curation · p. 36

B.1 Dataset Selection Criteria

  • Dataset selection favors unique IID tabular predictive tasks, excluding transformed non-tabular or synthetic data, non-predictive or trivial tasks, samples outside 500–250,000, and datasets with quality or access problems.The criteria also address licensing and ethical concerns; datasets must be downloadable through an open-access structured repository to support automated benchmarking.Source: §B.1 · pp. 37–38

B.2 Included Datasets Details

  • The included-dataset appendix reports domain coverage and dataset-level details, and shares the tasks and datasets as OpenML suite 457, alias “tabarena-v0.1.”The curation repository also provides BibTeX and LaTeX citation code.Source: §B.2 · p. 38

B.3 Noteworthy Observations from Curation · C Model Curation

TabArena’s curation observations show why dataset selection requires careful judgment: repositories lack metadata, and existing datasets raise issues involving task fit, licensing, modality, duplication, and versioning.

B.3 Noteworthy Observations from Curation

  • Unavailable metadata made automated dataset selection impractical and, in the authors’ view, more biased than careful manual curation.Assessing appropriate splits and whether targets came from deterministic functions required substantial effort, sometimes including reading the original dataset papers; the authors introduced a checklist to make inclusion more objective.Source: §B.3 · p. 38
  • Many otherwise suitable datasets were excluded for restrictive licenses, especially Kaggle datasets.The authors hope more high-quality datasets with less restrictive licenses will become available, including on Kaggle.Source: §B.3 · p. 38
  • The curators excluded at least 16 outdated handwritten digit or letter image datasets, while retaining image-derived features when tasks were meaningful and tabular models were reasonable.Source: §B.3 · p. 39
  • Only four of 142 tiny datasets with publication dates appeared after 2010; many seemed to come from educational materials, books, or tutorial toy examples.Tiny datasets had fewer than 500 training samples, which the authors considered a likely artifact of earlier, smaller-scale data collection.Source: §B.3 · p. 39
  • Datasets from AutoML Challenge series often had obscure names and were ablated versions of other datasets, creating unintended benchmark duplicates.Many also came from non-tabular domains such as images or text.Source: §B.3 · p. 39
  • Of 254 datasets with listed alternatives, most were PMLB variants: 118 Feynman equations and 62 Friedman data-generation functions.Benchmarks also used inconsistent versions with altered binarization, features, or targets; TabArena searched by dataset name, then used raw versions with minimal preprocessing.Source: §B.3 · p. 39
  • 51 datasets met the IID criterion while 68 did not, supporting Rubachev et al.’s [10] finding that prior benchmarks inappropriately used random splits.TabArena aims to end this practice.Source: §B.3 · p. 39
  • Many tabular datasets were not designed for prediction and were filtered as scientific-discovery tasks or for quality issues, though some could merit future inclusion if predictive applications are justified.Some filtered tasks appeared relatively simple, including tasks found trivially solvable elsewhere or with few features.Source: §B.3 · pp. 39–40

C.1 Implementation Framework Details … C.5 TabArena Ensemble

TabArena standardizes model implementations and tuning, then builds ensembles from cross-validation folds, tuned configurations, and a learned model portfolio. Its implementation requirements and preprocessing choices aim to support reliable evaluation and deployment across diverse tabular models.

C.1 Implementation Framework Details

  • TabArena implements models in AutoGluon’s AbstractModel framework, requiring time-limited early stopping, external validation data, and default model-specific preprocessing.The framework also uses AutoGluon, TabRepo, and scikit-learn functionality, with a tested pipeline used in real-world applications.Source: §C.1 · p. 41
  • AutoGluon’s feature generator transforms text and datetime columns, encodes categoricals, drops constant or duplicate columns, and preserves missing values for model-specific handling.RandomForest and ExtraTrees use ordinal encoding and impute missing values to 0; TabDPT uses ordinal encoding.Source: §C.1 · pp. 41–42

C.2 Hyperparameter search spaces

  • Search spaces for CatBoost, LightGBM, XGBoost, RandomForest, and ExtraTrees were experimentally refined and verified to outperform the original TabRepo spaces.Gradient-boosted trees use AutoGluon early stopping with n_estimators=10_000; RandomForest and ExtraTrees use 8-fold CV with 50 estimators per fold, totaling 400.Source: §C.2 · p. 42
  • Other search spaces came from model authors or maintainers, AutoGluon, or TabRepo; TabICL and TabDPT use default configurations.RealMLP disables label smoothing by default, following Holzmüller et al. [20], because accuracy is not the evaluation metric.Source: §C.2 · pp. 42–43

C.3 Cross-validation Ensembles

  • TabArena uses 8-fold cross-validation ensembles, averaging fold-model predictions and selecting tuned configurations by average out-of-fold performance.This bagging-like approach avoids refitting and can improve predictions through fold diversity, but increases inference time.Source: §C.3 · p. 47

C.4 Post-hoc Ensembling

  • For post-hoc ensembling, TabArena applies GES to validation predictions from 200 tuned configurations and returns the weighted average of configurations with nonzero weights.GES runs for 40 iterations, adjusting weights to reduce validation error.Source: §C.4 · p. 47

C.5 TabArena Ensemble

  • The TabArena ensemble evaluates portfolio configurations sequentially until the time limit or portfolio end, then post-hoc ensembles the evaluated configurations.For Figure 7, the portfolio was simulated from result artifacts using leave-one-dataset-out cross-validation, with 200 configurations and 40 ensemble-selection steps.Source: §C.5 · p. 47

D Evaluation Design Details … E Using and Contributing to the Living Benchmark

TabArena describes how it estimates uncertainty in Elo comparisons, acknowledges residual randomness and environmental costs, and argues that shared predictions and efficient model portfolios may offset benchmark compute over time.

D.1 Elo Confidence Intervals

  • 200 bootstrap subsamples of the 51-dataset benchmark yield approximate 95% confidence intervals for mean-centered Elo scores, with the mean Elo of all methods set to 0.The benchmark datasets are treated as i.i.d. samples from an unknown dataset-generating distribution.Source: §D.1 · p. 48
  • EloRF intervals shift mean-centered intervals to place default random forest at 1000 on the benchmark, but omit shift randomness; this does not affect relative Elo differences.The shift is benchmark-specific, so the interval for default random forest need not have zero length; mean-centered intervals yield less variance and smaller intervals for strong methods.Source: §D.1 · p. 48

D.2 Sources of Randomness

  • Model, data, and evaluation randomness arise from training and configurations, dataset sampling and partitions, and metric calculation; repeated experiments, random configurations, and 100 bootstrap rounds address them only partially.A fixed model seed and static configuration set leave model randomness incompletely controlled, while a stable Bradley-Terry Elo implementation is used for evaluation.Source: §D.2 · p. 48

D.3 Environmental Impact of TabArena

  • TabArena acknowledges its negative environmental impact and says maintainers have not reached an official conclusion on the tradeoff between environmental impact and research.The paper argues that key contributions may offset the benchmark’s impact over time.Source: §D.3 · p. 49
  • Sharing model predictions and metadata, simulating post-hoc ensembles on saved predictions, and limiting per-split training time are presented as ways to avoid unnecessary computation.The paper notes that tighter training-time limits would save more energy.Source: §D.3 · p. 49
  • TabArena argues its one-time cost may be offset by more efficient models and portfolios implemented in widely used systems, potentially reducing compute use through future applications.The authors suggest these applications could yield a net reduction in compute usage and a positive environmental impact.Source: §D.3 · p. 49

E.1 Benchmarking with TabArena · E.2 Contributing Models

TabArena documents a four-step benchmarking workflow and invites model contributions through an ongoing process with defined implementation, verification, and maintenance requirements. The code snapshot is illustrative and may become outdated as the system changes.

E.1 Benchmarking with TabArena

  • Benchmarking proceeds through implementing a model in AbstractModel, creating a search space, running it on TabArena or TabArena-Lite, and analyzing the results.The paper provides code examples and documentation for these steps.Source: §E.1 · p. 49

E.2 Contributing Models

  • Model contributions begin with an issue in the benchmarking repository and are envisioned as ongoing interaction between contributors and maintainers.The issue is populated over time with information needed to integrate the model.Source: §E.2 · p. 51
  • A contributed model must be a publicly shared, standalone AbstractModel implementation that passes TabArena’s default model unit test.A scikit-learn API-like implementation can be wrapped with AbstractModel; pipelines combining existing models or other systems are reserved for future TabArena iterations.Source: §E.2 · p. 51
  • Contributors must specify model-specific preprocessing and recommend default hyperparameters and a search space for optimization.Source: §E.2 · p. 51
  • Model verification includes maintainer source-code review and at least promising TabArena-Lite results, with original-author input sought when contributors are not the model’s authors.Author input may involve issue participation, pull-request review, or result validation.Source: §E.2 · p. 51
  • Contributors must provide contact details for future maintenance, and two TabArena maintainers must approve a finalized issue before integration.Models may be deprecated if they become unmaintainable, are consistently outperformed, or have bugs that cannot reasonably be resolved.Source: §E.2 · p. 51

E.3 Contributing Data: New Datasets and Curation Feedback … E.5 Running TabArena Models in Practice

The appendices describe how the community can extend TabArena’s datasets and leaderboard, explain curation decisions through contrasting dataset examples, and show how integrated models can be used on new data. The processes aim to support transparent, reproducible contributions while allowing the benchmark to expand to underrepresented tasks.

E.3 Contributing Data: New Datasets and Curation Feedback

  • TabArena invites community feedback and dataset suggestions through maintainer-driven or user-driven routes, with maintainers reviewing submissions before inclusion in a later benchmark version.Maintainer-driven suggestions use a GitHub template with links to raw data and its license; user-driven submissions provide a dataset description and preprocessing pipeline.Source: §E.3 · pp. 51–52
  • The curation checklist is intended to inform dataset review, not generally exclude data, and TabArena seeks proposals for underrepresented domains, non-IID data, and supervised tabular tasks.The user-driven template requests a dataset-description YAML, preprocessing Python file, and checklist responses about the task and data.Source: §E.3 · p. 52

E.3.1 Checklist Examples

  • APSFailure was included as a borderline case because its legitimate, relatively recent source was judged to represent a meaningful tabular task despite anonymized features and uncertain modality.The dataset contains 76,000 samples; the example also notes no deterministic optimal mapping and describes potential practical value in detecting truck component failures.Source: §E.3.1 · p. 53
  • Socmob was excluded because it is a scientific discovery task whose goal is to quantify relationships while controlling for confounders, not maximize predictive performance.The example advises against using it to benchmark predictive modeling, while suggesting it might suit a future scientific-discovery benchmark.Source: §E.3.1 · pp. 53–54

E.4 Contributing Results: Leaderboard Submissions

  • Leaderboard submissions are made by pull request and include updated model results, reproducible documented code, a model description, and an attestation of appropriate test-data use.The process is guided by equality, transparency, reproducibility, fairness, and manageable verification.Source: §E.4 · p. 54
  • Verified submissions undergo manual code review and reproduction on a random sample of outer folds before appearing on the main leaderboard, while non-verified results are listed separately without recomputation.For TabArena-v0.1, the website did not yet have a non-validated leaderboard; the protocol was expected to change with the first community submission.Source: §E.4 · p. 54

E.5 Running TabArena Models in Practice

  • TabArena-integrated models can be used on datasets outside the benchmark, illustrated by running RealMLP on a scikit-learn toy dataset.The example code repository provides further usage examples.Source: §E.5 · p. 55

E.6 Handling Foul Play and Dataset Contamination

TabArena recognizes that foul play and dataset contamination can threaten its leaderboard, but excluding potentially contaminated foundation models would undermine its goal of benchmarking state-of-the-art methods. The section favors ongoing maintenance and transparent contamination reporting while acknowledging that stronger safeguards need research.

Avoiding Foul Play

  • Maintainers propose separate leaderboards with and without potentially contaminated models, investigating dataset-level outliers, and expecting LLM providers to perform memorization tests [122].They currently lack tools to detect contamination in tabular foundation models or determine whether such models significantly remember training data; further research and tools are needed.Source: §E.6 · pp. 55–56
  • Open code, data, and result artifacts make potential abuse easier for the community to spot, though this is no silver bullet if API-based closed-source models are benchmarked..Source: §E.6 · p. 56
  • Future datasets, potentially changed split seeds, and new detection tools may help the living benchmark detect foul play; maintainers consider keeping TabArena alive its strongest protection.Source: §E.6 · p. 56

Possible Data Contamination in TabArena

  • TabDPT was pretrained on seven TabArena tasks but outperformed by other models on most, with a clear advantage only on wine_quality.The authors therefore took no immediate countermeasures but will remain cautious when adding models that may have used TabArena datasets.Source: §E.6 · p. 56

Benchmarking Foundation Models with Dataset Contamination

  • Because foundation models may be pretrained on TabArena datasets, excluding them would conflict with the goal of benchmarking tabular state-of-the-art methods.The authors expect such evaluations and argue the benchmark must consider incorporating these models.Source: §E.6 · p. 56
  • TabArena could disclose suspected contamination, but leak-free model versions may not represent the checkpoints practitioners use, and more effective measures require research.A leaderboard could add a contamination indicator, as in GIFT-Eval [56], or show results with and without suspected models; proxy checkpoints are difficult to compare reliably because pretraining is complex and can require run-specific tuning.Source: §E.6 · p. 56

F Performance Results Per Dataset

This section reports TabArena-v0.1 performance for all methods on each dataset, comparing default, tuned, and tuned-and-ensembled configurations. The table summarizes average predictive performance and fold-to-fold standard deviation, with significance annotations for method comparisons.

F Performance Results Per Dataset

  • Table F.1 compares per-dataset average predictive performance and standard deviation over folds for all methods in Default, Tuned, and Tuned + Ens. regimes.The table reports results for default hyperparameters, tuned models, and ensembles after tuning.Source: F Performance Results Per Dataset · p. 57, Table F.1 · p. 58
  • Best-method annotations use paired Wilcoxon signed-rank tests with Holm-Bonferroni correction at α = 0.05, with AutoGluon excluded from Tuned + Ens. significance tests.Green marks the best method on average; bold marks methods not significantly worse than the best overall, and underlining marks methods not significantly worse than the best within the same pipeline regime.Source: Table F.1 · p. 58

Related papers

Loading 2506.16791v4…