PaperlayerView exact revision on arXiv

Source-linked AI summary

TabArena: A Living Benchmark for Machine Learning on Tabular Data

Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, David Salinas, Frank Hutter

arXiv:2506.16791v4cs.LGcs.AI

TL;DR

TabArena addresses the lack of standardized, reliable, and maintained benchmarks for tabular machine learning. It builds a living system from manually curated datasets and models, large-scale experiments, a public leaderboard, reproducible code, and maintenance protocols. The results show that tuning and ensembling alter model comparisons, foundation models excel on small datasets, and ensembles across models advance the state of the art.

  • Problem

    Existing tabular benchmarks are static and may contain outdated, problematic, unrepresentative, or leaked datasets while failing to incorporate later corrections, model updates, and new models.

  • Method

    TabArena manually curates representative datasets and models, evaluates them through large-scale experiments and defined protocols, and maintains the benchmark through a public leaderboard, reproducible code, and community-oriented processes.

  • Results

    Tuning and ensembling improve individual-model benchmarking; deep learning catches up with gradient-boosted trees, foundation models excel on smaller datasets, and cross-model ensembles achieve state-of-the-art performance.

  • Takeaways & Limitations

    Reliable tabular model comparisons require maintained curation, explicit evaluation protocols, and consideration of tuned and cross-model ensembles.

  • Takeaways & Limitations

    TabArena-v0.1 focuses on IID classification and regression in the small-to-medium data regime, leaving non-IID, very small, large, and other tasks for future work.

Abstract

from arXiv · show

With the growing popularity of deep learning and foundation models for tabular data, the need for standardized and reliable benchmarks is higher than ever. However, current benchmarks are static. Their design is not updated even if flaws are discovered, model versions are updated, or new models are released. To address this, we introduce TabArena, the first continuously maintained living tabular benchmarking system. To launch TabArena, we manually curate a representative collection of datasets and well-implemented models, conduct a large-scale benchmarking study to initialize a public leaderboard, and assemble a team of experienced maintainers. Our results highlight the influence of validation method and ensembling of hyperparameter configurations to benchmark models at their full potential. While gradient-boosted trees are still strong contenders on practical tabular datasets, we observe that deep learning methods have caught up under larger time budgets with ensembling. At the same time, foundation models excel on smaller datasets. Finally, we show that ensembles across models advance the state-of-the-art in tabular machine learning. We observe that some deep learning models are overrepresented in cross-model ensembles due to validation set overfitting, and we encourage model developers to address this issue. We launch TabArena with a public leaderboard, reproducible code, and maintenance protocols to create a living benchmark available at https://tabarena.ai.

1 Introduction

TabArena addresses shortcomings of static tabular benchmarks by establishing a continuously maintained benchmarking system with curated datasets, models, reproducible evaluation, and community maintenance. Its initial results show that tuning and ensembling materially shape model comparisons.

  • TabArena replaces static benchmarking with a versioned, professionally maintained system that can be gradually improved by the community.
  • The benchmark manually curates 51 representative real-world datasets from 1053 datasets used in tabular research.
  • TabArena evaluates 16 tabular machine learning models, including three foundation models, in well-tested modeling pipelines.
  • The initialization trained approximately 25,000,000 model instances and provides a public leaderboard, precomputed results, and reproducible code.
  • Experienced maintainers from different institutions keep the living benchmark up to date.
  • With tuning and ensembling, deep learning methods match or exceed gradient-boosted trees, foundation models perform strongly on small data, and cross-model ensembles achieve state-of-the-art performance.

2 TabArena

TabArena is a continuously maintained platform for reliable tabular benchmarking, combining curated datasets and models with standardized pipelines, validation, ensembling, and evaluation protocols. Its initial scope emphasizes realistic IID classification and regression tasks in the small-to-medium data regime, while its protocols support ongoing updates.

  • Models and Hyperparameter Optimization Protocol: TabArena-v0.1 benchmarks 14 state-of-the-art or foundation models alongside two simple baselines in an extendable platform.Models are implemented within well-tested frameworks and run only on datasets within their stated restrictions.
  • Models and Hyperparameter Optimization Protocol: The benchmark uses standardized modeling infrastructure with preprocessing, cross-validation, hyperparameter optimization, evaluation metrics, and parallelized fold-wise training.Implementations rely on AutoGluon’s AbstractModel framework aligned with the scikit-learn API.
  • Models and Hyperparameter Optimization Protocol: Eight-fold cross-validation, curated hyperparameter searches, and configuration ensembling are used to evaluate models near their full potential.The default search evaluates one default and 200 randomly sampled configurations, except for TabICL and TabDPT, which use default performance only.
  • Datasets Protocol: The initial benchmark focuses on IID classification and regression for small-to-medium datasets, leaving non-IID, tiny, and large-data use cases for future work.The dataset protocol excludes issues such as duplicates, non-tabular domains, unsuitable predictive tasks, quality or license problems, and non-IID data.
  • Evaluation Design Protocol: TabArena uses Elo ratings based on task-specific metrics, with 200 bootstrap rounds for 95% confidence intervals, while acknowledging that pairwise ratings omit performance magnitudes.ROC AUC is used for binary classification, log-loss for multiclass classification, and RMSE for regression.

3 Results

TabArena-v0.1 evaluates curated models and datasets using leaderboard regimes, efficiency analyses, validation comparisons, and ensemble simulations. Results show that post-hoc ensembling and cross-model combinations substantially affect apparent peak performance, while accuracy gains involve computational and validation-overfitting tradeoffs.

  • Leaderboard initialization: TabArena-v0.1 initializes its leaderboard by running 16 curated models on 51 curated datasets, alongside AutoGluon as a reference pipeline.The experiments used GPU hardware for TabM, ModernNCA, and foundation models, and CPU hardware for other models and AutoGluon.
  • Leaderboard results: Post-hoc ensembling makes neural networks the strongest single models on average, reversing the conventional tuned leaderboard led by CatBoost.Without post-hoc ensembling, TabM, LightGBM, and RealMLP would rank below CatBoost.
  • Foundation models: TabPFNv2 leads related approaches by a large margin on datasets within its constraints and outperforms AutoGluon after tuning and post-hoc ensembling.The compatible evaluation covers 33 TabPFNv2 datasets with no more than 10K training samples and 500 features.
  • Efficiency tradeoffs: Tuned EBM and CatBoost provide strong inference-time efficiency, whereas TabM-TE and RealMLP-TE require approximately 15× and 100× greater inference time for stronger improvements.RealMLP begins to dominate gradient-boosted trees only after substantial training time and an ensemble of 25 or more configurations.
  • Validation design: Holdout validation greatly underestimates model performance and favors models that already use ensembling, supporting nested cross-validation in the benchmark design.The comparison uses predictive performance after tuning and ensembling under holdout versus cross-validation.
  • Practical benchmarking: TabArena-Lite reduces evaluation cost by using datasets with one outer fold and is intended for studies identifying models that significantly outperform others on at least one dataset.The authors regard such dataset-specific improvements as worthy of publication even without state-of-the-art average performance.
  • Cross-model ensembles: A simulated ensemble using all TabArena models outperforms every individual model and AutoGluon, but its weights favor models that overfit validation data.ModernNCA and RealMLP receive high ensemble weights despite leaderboard performance not necessarily predicting contribution.
  • Model-family comparison: The benchmark frames GBDTs and deep learning as complementary ensemble contributors rather than mutually exclusive alternatives.TabArena also provides precomputed artifacts for simulating ensemble performance across models.

4 Related Work

Prior tabular benchmarks differ in curation, data scope, evaluation protocols, and maintenance. TabArena positions itself as a living benchmark that combines extensive curation, ensemble-oriented evaluation, active maintenance protocols, and a public leaderboard.

  • Benchmark limitations: Recent studies identify outdated datasets, inappropriate use of non-IID data, non-tabular datasets, and unsuitable evaluation protocols as recurring benchmark problems.These concerns motivate more careful dataset curation and evaluation design.
  • TabArena’s approach: TabArena addresses these concerns through extensive data curation and evaluation in sophisticated ensembling pipelines.The paper presents this combination as consolidating strengths found separately in previous benchmarks.
  • Living benchmark design: Unlike prior benchmarks described by the authors, TabArena includes active maintenance protocols and a public leaderboard for tabular machine learning models.The paper identifies AutoML Benchmark and TALENT as similar actively maintained efforts but characterizes TabArena as the first living tabular benchmark with these features.

5 Conclusion

TabArena is introduced as a living benchmark for small- to medium-sized tabular data, combining curated datasets, models, evaluation protocols, reproducible resources, and community maintenance. Its experiments find that deep learning and foundation models can match or exceed gradient-boosted trees in supported settings, while ensembling improves performance.

  • Conclusion: TabArena benchmarks small- to medium-sized tabular data using 16 models and 51 representative real-world tasks selected from 1053 datasets.The benchmark comprises approximately 25,000,000 individual runs and took approximately 15 years of wall-clock time.
  • Conclusion: Deep learning models and foundation models for small data perform similarly to or better than gradient-boosted decision trees.The conclusion specifically names TabM and RealMLP among the deep learning models.
  • Conclusion: TabArena shares metadata, reproducible code, and standardized evaluation resources so the community can benchmark new models.Future work includes supporting tiny, large, and non-IID data, integrating new models, and curating more datasets.
  • Conclusion: The benchmark is intended to grow through collaboration with the tabular machine learning community.The authors describe ongoing evolution and maintenance as part of TabArena's living-benchmark vision.

Competing Interests

The authors disclose affiliations with several tabular machine learning models, frameworks, and projects, including RealMLP, TabICL, TabRepo, AutoGluon, TabPFNv2, and PriorLabs. They report no other competing interests.

  • Competing Interests: Several authors develop or maintain tabular machine learning models, frameworks, or datasets represented in the paper.The disclosed connections include RealMLP, TabICL, TabRepo, AutoGluon, FastAI MLP, Torch MLP, and TabPFNv2.
  • Competing Interests: One author is affiliated with PriorLabs, a company focused on developing tabular foundation models.Another author is an OpenML core contributor.
  • Competing Interests: The authors declare no other competing interests.

NeurIPS Paper Checklist

The checklist addresses whether the paper's claims, reproducibility materials, experimental details, statistical reporting, and limitations are adequately documented. The supplied responses state that the claims reflect the paper's contributions and scope, public code and artifacts support reproduction, and limitations are discussed.

  • Claims and scope: The paper's abstract and introduction state its contributions and scope, including TabArena, dataset curation, benchmarking, a leaderboard, and maintainer organization.The checklist also identifies claims about validation, ensembling, deep learning, foundation models, and cross-model ensembles.
  • Limitations: The paper discusses limitations in Section 5, including fixed hyperparameter configurations, time-limit and hardware dependence, restricted dataset selection, and no feature engineering.The authors also identify future work involving broader data types, new models, and additional datasets.
  • Theory assumptions and proofs: The paper does not include theoretical results.Accordingly, the theory-assumptions-and-proofs checklist item is not applicable.
  • Experimental result reproducibility: The authors state that the benchmark and experimental setup are described in Sections 2 and 3, with additional appendix details and public result artifacts.They answer yes to the reproducibility question.
  • Open access to data and code: The authors state that all data and code are released through the public TabArena framework, with user guides and links in Appendix E.
  • Experimental setting/details: The benchmark design and experimental setting are described in the main sections and appendix, including the details needed to understand the results.

A.2 Analyzing Training Time Limit

The benchmark imposes a one-hour limit for evaluating each configuration on one train split, while runtime plots and leaderboard figures summarize results across tasks and model applicability. Less than 1% of configurations reach the limit, although an exception affects EBM results.

  • A.2 Analyzing Training Time Limit: Each configuration on one train split is limited to 1 hour, with training stopped early if all 8 inner folds do not finish.Figure A.9 plots the proportion of configurations against their training time in seconds.
  • Leaderboard context: The main leaderboard reports Elo and normalized scores, with TabPFNv2 and TabICL results imputed by RandomForest on datasets where those models are not applicable.Table A.1 also distinguishes default, tuned, and tuned-plus-ensembled performance.
  • Leaderboard context: Task-specific figures separately report binary classification, multiclass classification, and regression results using Elo and normalized scores.
  • Leaderboard context: Pairwise win-rate comparisons use model rows and columns across all datasets, with higher values indicating better win rates for the model on the y-axis.
  • Leaderboard context: Additional compatibility-restricted figures compare TabPFNv2 and TabICL on datasets where each or both models are applicable.Some figures impute the unavailable model's results with RandomForest.
  • A.2 Analyzing Training Time Limit: Less than 1% of configurations for all models reach the one-hour time limit.EBM training was not stopped early at the limit, positively influencing its results, and was not rerun because only a small fraction of trials was affected.

A.3 Tabular Deep Learning on GPU vs. CPU

The benchmark examines hardware effects, leaderboard efficiency, statistical comparisons, and tuning behavior across TabArena models and datasets. These analyses show that compute conditions and validation choices materially affect model evaluation.

  • GPU vs. CPU: Approximately 16% of ModernNCA and TabM CPU runs are early stopped by the 1-hour limit, versus less than 0.1% on GPU.
  • GPU vs. CPU: GPU training is faster than CPU training for TabM and ModernNCA, and slightly faster for RealMLP.A one-hour CPU limit would negatively affect TabM and ModernNCA more than RealMLP.
  • TabArena-Lite: TabArena-Lite uses all datasets with one outer fold and produces results similar to the full TabArena benchmark.The reduced protocol uses 200 hyperparameter configurations per model and is proposed for lower-cost academic benchmarking.
  • Statistical comparisons: Critical difference diagrams compare tuned and ensembled methods on the full benchmark and on TabPFNv2- and TabICL-compatible subsets.Lower ranks are better, and horizontal bars connect methods without statistically significant differences.
  • Tuning and validation overfitting: Most models saturate by 201 tuning configurations, while ModernNCA peaks at 25 and then degrades as validation overfitting increases.ModernNCA ranks sixth on test Elo but first on validation Elo, and neural networks overfit more than other methods.
  • Tuning and validation overfitting: The TabArena ensemble favors ModernNCA and RealMLP because validation optimization over-selects models that overfit, while under-selecting strong test-performing models such as TabM.Despite this selection imbalance, the ensemble still outperforms all methods, including AutoGluon.

B.3 Noteworthy Observations from Curation

TabArena’s curation process emphasizes realistic, predictive tabular tasks while documenting manual-selection challenges, licensing constraints, and excluded outdated modality-derived datasets.

  • Curation process: Manual curation was needed because repository metadata often lacked information required to judge appropriate splits and task origins.The authors therefore view automated curation as more biased than careful manual review in the current repository landscape.
  • Licensing: Restrictive licenses, especially among otherwise suitable Kaggle datasets, limited the pool of candidate benchmark datasets.
  • Selected datasets: TabArena-v0.1 reports domain coverage and dataset metadata for its selected datasets, including sample counts, feature counts, class counts, and categorical-feature percentages.
  • Excluded datasets: At least 16 outdated handwritten-digit or handwritten-letter image datasets were excluded, along with some simple image-derived feature tasks.The authors remain open to including modality-derived datasets when predictive applications can be justified and no other issues apply.

C.1 Implementation Framework Details

TabArena standardizes model implementations, preprocessing, hyperparameter spaces, and ensembling within a tested framework. Its pipeline combines model-specific handling with cross-validation and post-hoc ensembling.

  • Implementation framework: Models are implemented with preprocessing, training, inference, and unit tests inside AutoGluon’s AbstractModel framework.The framework supports standardized integration with AutoGluon, TabRepo, and scikit-learn functionality.
  • Implementation requirements: Implementations must support time-limit-based early stopping, externally provided validation data, and default preprocessing for common tabular data anomalies.
  • Preprocessing: TabArena combines model-agnostic preprocessing with minimal model-specific preprocessing, including encodings, imputations, scaling, and quantile transformations where required.Different models retain control over how categorical, numerical, missing, datetime, and text inputs are handled.
  • Hyperparameter optimization: Hyperparameter search spaces are drawn from experiments, author recommendations, AutoGluon, TabRepo, or original model papers, depending on the model.The benchmark includes dedicated search spaces for tree models, neural models, foundation models, linear models, and KNN.
  • Ensembling: TabArena uses 8-fold cross-validation ensembles and selects tuned configurations by average out-of-fold performance, while post-hoc ensembling combines previously evaluated models.The benchmark’s model pipeline therefore includes both within-model fold ensembling and ensembles built from evaluated predictions.

D Evaluation Design Details

TabArena evaluates methods with relative Elo scores, bootstrap uncertainty, and repeated experiments while acknowledging remaining randomness and broader environmental costs.

  • Evaluation target: The benchmark treats its 51 datasets as IID samples from an unknown distribution and seeks uncertainty estimates for performance on that distribution.
  • Elo scoring: Elo scores are interpreted through relative differences, with absolute values anchored either by the mean method Elo or default random forest at 1000.The two variants are Elomean and EloRF.
  • Uncertainty estimates: Two hundred bootstrap dataset subsamples provide approximate 95% confidence intervals for Elomean, with shifted intervals reported for EloRF.The shifted intervals do not account for randomness in the benchmark-specific random-forest shift, although that term does not affect relative differences.
  • Sources of randomness: Results remain affected by model, data, and evaluation randomness from training, dataset selection, partitions, metric calculation, and ranking precision.
  • Sources of randomness: The study repeats experiments and uses bootstrapping, but fixed model seeds and a static configuration set leave model randomness only partially controlled.The authors state that future TabArena versions will no longer use a fixed random seed.
  • Environmental impact: TabArena incurs substantial environmental cost, while shared predictions, time limits, and reusable efficiency improvements are presented as potential offsets.The benchmark also identifies portfolios that improve the quality-efficiency Pareto frontier.

E.1 Benchmarking with TabArena

TabArena provides a standardized workflow for adding, benchmarking, and comparing new tabular models through public implementations, search spaces, reproducible experiments, and maintainer review.

  • Benchmarking workflow: Users implement models within TabArena’s AbstractModel framework, create a search space, run experiments, and analyze results.The framework supports standardized preprocessing, cross-validation, hyperparameter optimization, metrics, parallelization, and customizable model pipelines.
  • Model integration: New model integrations require publicly shared code, default hyperparameters, a hyperparameter search space, and a standalone implementation.The implementation must pass TabArena’s default unit test and exclude ensembling pipelines or calls to other machine learning systems.
  • Model integration: Maintainers verify model source code, require promising TabArena-Lite results, and obtain two maintainer approvals before integration.Contributors may coordinate with original authors, while maintainers can later deprecate models that become unmaintainable, consistently underperform, or contain unresolved bugs.

E.3 Contributing Data: New Datasets and Curation Feedback

TabArena accepts community dataset suggestions through a structured, maintainer-reviewed process designed to assess data provenance, licensing, task relevance, and evaluation conditions.

  • Dataset contribution process: Users propose datasets through GitHub issues containing raw-data links, licenses, a dataset description, preprocessing code, and a curation checklist.The checklist covers availability, sample size, modality, task suitability, data generation, user value, temporal structure, group structure, preprocessing, and benchmarking recommendations.
  • Dataset contribution process: Maintainers verify the submitted information, discuss open questions, and include a dataset in the next benchmark version after determining that its task is reasonable.The process is user-driven and targets contributors with substantial knowledge of the suggested dataset.
  • Scope and coverage: The curation checklist does not generally exclude datasets; future versions aim to add underrepresented domains, non-IID data, and other supervised tabular tasks.The authors identify insufficient high-quality data or domain knowledge as reasons some task areas are not yet adequately covered.
  • Curation examples: The APSFailure example was included despite anonymized features and possible extraction from time-series because its legitimate source and meaningful predictive application supported its inclusion.The dataset contains 76,000 samples, and the application concerns detecting truck component failures to reduce manual effort and prevent accidents.
  • Curation examples: The socmob example was excluded because its scientific goal was to quantify feature–target relationships rather than maximize predictive performance.Although supervised models were applied, predictive accuracy was not the study’s objective.

E.6 Handling Foul Play and Dataset Contamination

TabArena treats foul play and dataset contamination as persistent threats to an open leaderboard and proposes ongoing disclosure, review, and benchmark updates as safeguards.

  • Threats: Open-source benchmarking can be compromised when developers overfit model hyperparameters to TabArena datasets or pretrain foundation models on them.The authors describe foul play and dataset contamination as fundamental limitations requiring future solutions.
  • Safeguards: Proposed safeguards include separate leaderboards for suspected contamination, maintainer investigation of outliers, open artifacts for community scrutiny, and new datasets or splits across benchmark iterations.The authors regard continuous maintenance as the strongest protection against foul play.
  • Foundation-model contamination: TabArena currently cannot detect contamination in tabular foundation models or determine whether they can meaningfully remember benchmark data.The authors call for future research and tools analogous to memorization tests developed for other model classes.
  • Observed contamination: The extent of contamination in TabArena-v0.1 cannot be estimated because most publications do not disclose training and evaluation data.The authors identify TabDPT as the only model directly pretrained on TabArena data in the initial version discussed.
  • Foundation-model contamination: Using leak-free foundation-model versions would avoid contamination concerns but might fail to represent the models practitioners actually use.The authors also note that independently trained checkpoints cannot be reliably guaranteed to perform similarly because deep-learning pretraining is difficult to reproduce robustly.

F Performance Results Per Dataset

TabArena reports per-dataset predictive performance across default, tuned, and tuned-plus-ensembled regimes, with uncertainty across folds and statistical comparisons among methods.

  • Performance reporting: Table F.1 reports average predictive performance per dataset with standard deviation over folds for Default, Tuned, and Tuned + Ens. regimes.The table compares unconfigured models, tuned configurations, and ensembles formed after tuning.
  • Statistical comparisons: The table marks the best-performing method on average and identifies methods not significantly worse using paired Wilcoxon tests with Holm-Bonferroni correction at α = 0.05.Separate underlining indicates methods not significantly worse than the best within the same pipeline regime.
Loading 2506.16791v4…