PaperlayerView exact revision on arXiv

Source-linked AI summary

TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applications

David Salinas, Nick Erickson

arXiv:2311.02971v3cs.LGcs.AI

TL;DR

Broad tabular benchmarks make repeated comparisons and ablations costly. TabRepo stores evaluations and predictions across many models and datasets, and the authors use them for low-cost ensemble analysis and transfer learning, reporting improvements over current tabular systems across accuracy, training time, and latency.

  • Problem

    Broad tabular benchmarks are expensive to run, limiting thorough ablations and exploration of research directions.

    Source: §1 · p. 1

  • Method

    TabRepo provides precomputed evaluations and predictions that support ensemble analysis and transfer learning with learned model portfolios.

    Source: §1 · pp. 1–2, §3 · p. 3, §6 · p. 9

  • Results

    Combining AutoGluon with a fixed learned portfolio achieved a 67% win-rate compared to the best system and established a new state-of-the-art.

    Source: §6 · p. 9

  • Takeaways & Limitations

    AutoGluon 1.0 adopted TabRepo’s learned portfolio as its default, replacing its prior hand-crafted default configuration.

    Source: §6 · p. 9

    The work does not use dataset features, which the authors identify as a possible avenue for further improving results.

    Source: §7 · p. 10

Abstract

from arXiv · show

We introduce TabRepo, a new dataset of tabular model evaluations and predictions. TabRepo contains the predictions and metrics of 1310 models evaluated on 200 classification and regression datasets. We illustrate the benefit of our dataset in multiple ways. First, we show that it allows to perform analysis such as comparing Hyperparameter Optimization against current AutoML systems while also considering ensembling at marginal cost by using precomputed model predictions. Second, we show that our dataset can be readily leveraged to perform transfer-learning. In particular, we show that applying standard transfer-learning techniques allows to outperform current state-of-the-art tabular systems in accuracy, runtime and latency.

1 Introduction

Broad tabular benchmarks make thorough comparisons and ablations costly. TabRepo provides precomputed model predictions and evaluations for studying tuning with ensembling and for transfer learning.

1 Introduction

  • Evaluating one method across AMLB’s 1,040 tasks requires 40,000 CPU hours, making thorough ablations and exploration expensive.The estimate covers 104 datasets, 10 folds, two time limits, and an 8-CPU machine.Source: §1 · p. 1
  • The repository contains 786,000 predictions from 1,310 models across 200 datasets, with 10 model families and 3 seeds.Source: §1 · p. 2
  • Precomputed predictions let researchers study tuned models alongside ensembles at marginal cost.Source: §1 · p. 2
  • TabRepo combined with transfer learning achieves state-of-the-art results compared with AutoML systems in accuracy and training time.The paper also reports that learned portfolio configurations improve latency.Source: §1 · p. 2

2 Related work

Prior benchmark repositories provide precomputed results, but TabRepo distinguishes itself by exposing predictions across a dense, broad set of tasks and using them for ensemble evaluation and transfer learning.

2 Related work

  • TabRepo exposes predictions, allowing errors for individual models and ensembles of any available subset to be evaluated nearly instantaneously.Earlier tabular benchmarks include Gorishniy et al. (2021), Grinsztajn et al. (2022), and McElfresh et al. (2023).Source: §2 · p. 2
  • Unlike Purucker and Beel (2022, 2023), TabRepo evaluates all models on all datasets and covers 200 classification and regression datasets.Purucker and Beel study sparse model evaluations and ensembling algorithms on 31 classification datasets.Source: §2 · p. 2
  • Offline evaluations support transfer learning, including portfolio learning, which greedily selects complementary models for a new dataset.Prior work has applied portfolio learning in HPO, time-series, computer vision, tabular deep learning, and AutoML.Source: §2 · p. 2
  • AutoGluon combines bagging, ensembling, and stacking, but its inference can be expensive and its default configurations require expert selection.TabRepo is used to learn default configurations that improve accuracy and latency when matching compute budget.Source: §2 · pp. 2–3

3 TabRepo

TabRepo records evaluations and predictions for model configurations across a broad set of tabular tasks, enabling ensemble evaluation from stored predictions rather than retraining.

3 TabRepo

  • The repository covers 200 OpenML datasets, with three tasks per dataset and eight cross-validation splits per task.The evaluation uses RMSE for regression, AUC for binary classification, and log loss for multiclass classification.Source: §3 · p. 3
  • The 1,310 base-model configurations span linear models, nearest neighbors, tree ensembles, MLP, TabPFN, and FT-Transformer; unfinished runs are early-stopped at one hour.The configurations include AutoGluon defaults and randomly sampled settings; TabPFN and FT-Transformer use GPUs.Source: §3 · p. 4
  • The evaluation also includes six AutoML frameworks, run with one-hour and four-hour fitting budgets.Source: §3 · p. 4
  • Stored predictions let the authors evaluate any ensemble of configurations by recomputing losses, without retraining or storing evaluations for every possible ensemble.This makes ensemble evaluation particularly fast.Source: §3 · p. 4
  • For each task and model, TabRepo stores predictions and losses, making the recorded base-model outputs reusable for on-the-fly ensemble evaluation.Source: §3 · p. 4
  • Caruana-style ensembling iteratively selects models whose averaged predictions minimize error, and may select a model repeatedly to give it weight.Final predictions average the selected models.Source: §3 · p. 4

4 Comparing HPO and AutoML systems

TabRepo supports low-cost comparison of tuned and ensembled model families with AutoML systems. Tuning and ensembling improve model-family results, but do not surpass state-of-the-art AutoML systems; model performance, runtime, and hyperparameter importance vary across families.

Effect of tuning and ensembling on model error

  • The comparisons use precomputed evaluations and predictions, so these tuning-and-ensembling experiments run at marginal cost.The analysis includes model families’ default hyperparameters, four-hour tuning, and post-tuning ensembles.Source: §4 · pp. 4–5

Model error and runtime distributions

  • CatBoost and LightGBM rank strongly overall, while MLP can suit some tasks; MLP and XGBoost rank correlations suggest potential ensemble complementarity.CatBoost also has the highest average rank but costs an order of magnitude more training time than MLP, XGBoost, and LightGBM.Source: §4 · p. 5, Figure 1 · p. 5

Effect of tuning and ensembling on model error

  • Tuning and ensembling improve every model family, but none beats state-of-the-art AutoML systems; ensembling lets LightGBM match CatBoost’s accuracy.The comparison uses the best configuration from a four-hour random search and an ensemble of its best 20 configurations.Source: §4 · p. 5, Figure 2 · p. 6

Hyperparameters importance

  • fANOVA identifies practitioner-familiar leading hyperparameters: neighbors for KNN, regularization for linear models, depth for CatBoost, and activation or batch normalization for MLP.The importance analysis is aggregated across tasks, and the Figure 3 y-axis is logarithmic.Source: §4 · p. 6, Figure 3 · p. 6

5 Portfolio learning with TabRepo

TabRepo’s offline evaluations support learning portfolios of configurations that can be evaluated under different fitting budgets. Portfolio ensembles outperform AutoML frameworks across metrics at 1-, 4-, and 24-hour budgets, while data requirements and portfolio size shape their performance.

Portfolio learning and anytime portfolio

  • Portfolio learning selects configurations sequentially, greedily adding the configuration with the lowest average error when combined with those already selected.Configurations are evaluated in an anytime fashion until the fitting budget expires; a fast baseline is used if the first selected configuration exceeds the budget.Source: §5 · p. 7

Evaluations

  • The evaluation uses leave-one-out transfer: each test dataset is excluded when computing portfolio configurations from the other datasets.Results are reported under a four-hour fitting-budget constraint.Source: §5 · p. 8

Portfolio results

  • Portfolio ensembles beat all AutoML frameworks on every metric at 1-, 4-, and 24-hour budgets without stacking, yielding lower latency than AutoGluon.Portfolio has better aggregate performance, but the statistical comparison ties it with AutoGluon; both are statistically better than all baselines.Source: Table 1 · p. 7, §5 · p. 8, Figure 4 · p. 8

How much data is needed?

  • Portfolio ensembles begin outperforming AutoGluon at around 10 offline configurations or training datasets, with gains saturating around 150.The study varies configurations and datasets independently under a four-hour budget.Source: §5 · p. 9

Portfolio size

  • A portfolio of 3 configurations beats all AutoML systems except AutoGluon, while a portfolio of 15 beats AutoGluon.The portfolio size shows consistent benefit as it increases.Source: Figure 5 · p. 8, §5 · p. 9

Ensemble size

  • Ensembles achieve their best performance with 15 selected models; adding more yields no measurable improvement.The comparison varies the number of ensemble members; AutoML systems typically use larger values, such as AutoGluon’s 100 iterations.Source: Figure 5 · p. 8, §5 · p. 9

6 Conclusion

TabRepo provides tabular-model evaluations and predictions for studying ensemble strategies efficiently. Learned portfolios improve performance on AutoML benchmarks and were adopted in AutoGluon 1.0.

Conclusion

  • TabRepo provides model evaluations and predictions, enabling efficient evaluation of ensemble strategies and analysis of tuning at marginal cost.The learned portfolios outperform state-of-the-art tabular methods on accuracy, training time, and latency.Source: §6 · p. 9
  • AutoGluon combined with a fixed learned portfolio achieves a 67% win-rate against the best system on AMLB and establishes a new state-of-the-art.AutoGluon 1.0 adopted the learned portfolio as its default, replacing the previous hand-crafted default configuration.Source: §6 · p. 9, Table 2 · p. 9
  • The authors identify combinations of CASH, multi-fidelity, and transfer-learning methods as directions for further work on tabular prediction.The paper provides code for accessing evaluations and assessing ensembles.Source: §6 · p. 9

7 Broader Impact Statement

TabRepo reduces the cost of analysis through precomputed evaluations and predictions, while the paper notes that larger model and dataset collections raise ethical-review challenges. It also identifies unexploited dataset features and random configuration selection as limitations, and documents its train-test split procedure.

7 Broader Impact Statement

  • 219136 CPU hours completed TabRepo’s model evaluations, versus 4066576 CPU hours estimated for the paper’s analysis without precomputed evaluations and predictions.Using TabRepo makes analyses requiring model interactions, such as ensembling, 10000x cheaper than retraining base models from scratch.Source: §7 · p. 10
  • As more models and datasets improve final performance, checking the ethical aspects of every dataset becomes more challenging; TabRepo checked each dataset’s license and ethical aspects.Verifying ethical aspects across a large dataset collection remains an open problem.Source: §7 · p. 10
  • The work does not use dataset features, unlike previous work, and suggests that future ensembling could exploit dataset statistics to improve results.The paper cites Jomaa et al. (2021) as previous work using dataset features.Source: §7 · p. 10
  • Random configurations restrict performance relative to potentially more efficient model-based approaches, which future work could investigate alongside multi-fidelity approaches.The paper points to Winkelmolen et al. (2020) for model-based or multi-fidelity configuration generation.Source: §7 · p. 10

C.1 Train-test splits

  • Each task uses 90% of the data for training and 10% for testing, using the first three folds of OpenML’s 10-fold cross-validation procedure.The splits are identical to those used by Gijsbers et al. (2024).Source: §C.1 · p. 18

D Raw Results and Reproducibility

The appendix provides resources for accessing and regenerating TabRepo results, and documents model configurations, implementation choices, and AutoML comparison settings. It also records exclusions and scope caveats relevant to interpreting those comparisons.

D Raw Results and Reproducibility

  • Raw results for model configurations, baselines, AutoML systems, and TabRepo experiments are published alongside reproduction instructions and a script for generating Fig. 2.Regenerating all precomputed model and baseline results requires a minimum of 500,000 CPU hours across 92,000 EC2 instances.Source: D Raw Results and Reproducibility · p. 18

E Model details

  • Each model family has 201 configurations: default hyperparameters plus 200 randomly selected configurations, using search spaces based on AutoGluon and expanded for variety.The paper notes that selecting an appropriate search space is complex and outside its focus; TabRepo supports arbitrary configurations.Source: E Model details · pp. 18–19
  • Models reuse AutoGluon preprocessing, initial hyperparameters, training, and prediction; early stopping sets maximum iterations to 10,000 for gradient boosting and epochs to 500 for MLP.The paper uses AutoGluon’s early-stopping logic rather than directly searching for optimal iteration or epoch limits.Source: E Model details · p. 19

E.2 MLP

  • The MLP uses AutoGluon’s default early stopping with a 500-epoch maximum, while CatBoost, LightGBM, and XGBoost each use a 10,000-iteration maximum.Extra Trees and Random Forest use 300 trees each.Source: §E.2 · p. 21, §E.3 · p. 21, §E.4 · p. 21

E.10 TabPFN

  • TabPFN uses its default setting without additional configurations; its results are not representative of its intended setting because many TabRepo datasets exceed 1000 training rows.FT-Transformer also uses default settings without additional configurations due to cost, uses the AutoGluon implementation, and runs on a GPU instance.Source: §E.10 · p. 22, §E.11 · p. 22

F.1 Excluded Frameworks

  • AutoML framework versions and configurations are documented; MLJAR and GAMA are omitted from the main results because of implementation errors, while NaiveAutoML is excluded over configuration issues.The paper notes FLAML version 2.0 was released after the experiments; NaiveAutoML had not been rerun after its configuration issue was resolved.Source: F AutoML Framework Details · p. 23, §F.1 · p. 23

F.2 AutoGluon

  • AutoGluon comparisons use the best_quality preset; Tab. 2 uses AutoGluon 1.0 with a size-100 portfolio, which trades slightly worse accuracy for faster latency, training, and lower disk use.The paper uses an m5.2xlarge instance for AutoGluon training on AMLB to match other systems’ compute.Source: §F.2 · p. 23

F.3 Auto-Sklearn 2

  • Auto-Sklearn 2’s meta-learning training datasets overlap with TabRepo, so its results may be overly optimistic; it is incompatible with regression, where Auto-Sklearn 1 results are used.The authors retain Auto-Sklearn 2 results as a useful comparison point.Source: §F.3 · p. 24

G Studying Portfolio Model Prioritization

Portfolio selection is consistent at the start: the first four model-family picks match across all leave-one-dataset-out evaluations. A three-family sequence of CatBoost, MLP, and LightGBM is sufficient to outperform all AutoML methods except AutoGluon.

G Studying Portfolio Model Prioritization

  • The first four model-family selections are identical across all 200 leave-one-dataset-out evaluations.CatBoost is first, MLP second, LightGBM third, and CatBoost fourth.Source: G Studying Portfolio Model… · p. 24
  • A size-3 portfolio of CatBoost, MLP, and LightGBM outperforms all AutoML methods except AutoGluon.The sequence is followed by a simple weighted ensemble.Source: G Studying Portfolio Model… · p. 24
  • At position five, about 90% of portfolios pick LightGBM and 10% pick KNN; their probabilities reverse at position six.After position 13, selections are no longer unanimous at individual positions, though overall family inclusion proportions stabilize.Source: G Studying Portfolio Model… · p. 24
  • All eight model families appear by position 13, including KNN at position four despite being the weakest standalone model.The authors suspect KNN’s portfolio value comes from diversity, while XGBoost competes with the architecturally similar CatBoost and LightGBM.Source: G Studying Portfolio Model… · pp. 24–25
  • The current selection logic ignores training time, leaving cheaper-first ordering as a proposed way to improve anytime performance.The authors leave investigating this change to future work.Source: G Studying Portfolio Model… · p. 25

H Impact of Time Limit

The one-hour training limit affects a small fraction of model runs, but the most expensive models account for most benchmark runtime. The authors expect removing the limit for those models would more than double total compute.

H Impact of Time Limit

  • 1.99% of models (15,531 of 782,060) neared or reached the 2,800-second threshold.Table 10 lists models exceeding 2,800 seconds per dataset.Source: H Impact of Time Limit · p. 25, Table 10 · p. 26
  • The time limit is reached mainly on large, high-dimensional datasets; early-stopped models are almost exclusively GBMs, particularly CatBoost.FT-Transformer is early-stopped in 7.5% of cases, reflecting its comparatively high compute requirement.Source: H Impact of Time Limit · p. 27
  • The 10% most expensive models account for 90% of total runtime, and the 40% most expensive account for 99%.CatBoost’s total runtime exceeds that of all other families combined.Source: Figure 7 · p. 26, H Impact of Time Limit · p. 27
  • Fully training the 2% most expensive models is expected to more than double benchmark compute, while the authors expect the limit does not alter their overall conclusions.For future work on larger datasets, they suggest a longer time limit, more CPU cores, and/or GPU acceleration.Source: H Impact of Time Limit · p. 27

I Portfolio results when including models requiring GPU

The GPU-model comparison adds TabPFN and FT-Transformer, evaluated separately because they require different hardware. Adding model diversity slightly improves portfolio win rate against AutoML baselines, while TabPFN remains constrained by its current limitations on larger datasets.

I Portfolio results when including models requiring GPU

  • The GPU comparison uses a single default configuration each for TabPFN and FT-Transformer, with FT-Transformer’s scope limited by the cost of running all tasks on a GPU.Results are reported separately because these models require additional GPU hardware.Source: I Portfolio results when including… · p. 27
  • FT-Transformer performs between MLPs and the best boosted-tree methods, while TabPFN does not reach the top methods’ performance.The authors attribute TabPFN’s result to high failure rates on large datasets and difficulty effectively exploiting large numbers of rows.Source: I Portfolio results when including… · p. 27
  • 56.8% to 57.5%: adding GPU-requiring models slightly improves the portfolio’s win rate against AutoML baselines.The comparison is for a four-hour time limit.Source: I Portfolio results when including… · p. 27
  • TabPFN’s failure rate is about 30% because it supports only 100 features and 10 classes.Failed runs use the timeout baseline’s predictions, which take less than five seconds to run.Source: I Portfolio results when including… · p. 27

J Additional results Portfolios

Across one- and four-hour limits, the Portfolio has the best win rate, re-scaled loss, and average rank among the compared methods. At a one-hour budget, the anytime portfolio matches AutoGluon on normalized error, outranks it, and has lower latency.

J.1 Portfolio win-rate comparison

  • At both one- and four-hour limits, Portfolio achieves the best win rate, re-scaled loss, and average rank across the compared methods.Tables 12 and 11 report the one- and four-hour comparisons, respectively.Source: §J.1 · p. 28

J.2 Performance on lower fitting budgets

  • At a one-hour budget, the anytime portfolio matches AutoGluon on normalized error, outperforms it on rank, and has around 30% lower latency.Table 15 reports normalized error, rank, training time, and inference time averaged over all tasks.Source: §J.2 · p. 28, Table 15 · p. 30
Loading 2311.02971v3…