PaperlayerView exact revision on arXiv

Source-linked AI summary

Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data

David Holzmüller, Léo Grinsztajn, Ingo Steinwart

arXiv:2407.04491v3cs.LG

TL;DR

Tabular MLPs remain an improvement target, while extensive hyperparameter optimization can be costly. The paper introduces RealMLP and tuned defaults for MLPs and GBDTs; across benchmarks, RealMLP mostly outperforms other neural models at moderate runtime and is competitive with GBDTs.

  • Problem

    Despite MLPs’ promise as tabular baselines, how to improve them and whether tuned dataset-independent defaults can reduce reliance on costly hyperparameter optimization remain open questions.

    Source: §1 · pp. 1–2

  • Method

    The paper develops RealMLP by improving MLP architecture, training, preprocessing, hyperparameters, and initialization, and compares tuned defaults for MLPs and GBDTs with library defaults and dataset-specific hyperparameter optimization.

    Source: §1 · pp. 1–2

  • Results

    Across the benchmarks, RealMLP mostly outperforms other neural models at moderate runtime and is competitive with GBDTs in benchmark scores.

    Source: §6 · p. 10

  • Takeaways & Limitations

    With good default parameters, trying both neural and tree algorithm families is worthwhile.

    Source: §6 · p. 10

    The evidence covers medium-to-large tabular datasets in standard settings; generalization to very small datasets, distribution shifts, missing numerical values, or log-loss remains unclear.

    Source: §5.2 · p. 10

Abstract

from arXiv · show

For classification and regression on tabular data, the dominance of gradient-boosted decision trees (GBDTs) has recently been challenged by often much slower deep learning methods with extensive hyperparameter tuning. We address this discrepancy by introducing (a) RealMLP, an improved multilayer perceptron (MLP), and (b) strong meta-tuned default parameters for GBDTs and RealMLP. We tune RealMLP and the default parameters on a meta-train benchmark with 118 datasets and compare them to hyperparameter-optimized versions on a disjoint meta-test benchmark with 90 datasets, as well as the GBDT-friendly benchmark by Grinsztajn et al. (2022). Our benchmark results on medium-to-large tabular datasets (1K--500K samples) show that RealMLP offers a favorable time-accuracy tradeoff compared to other neural baselines and is competitive with GBDTs in terms of benchmark scores. Moreover, a combination of RealMLP and GBDTs with improved default parameters can achieve excellent results without hyperparameter tuning. Finally, we demonstrate that some of RealMLP's improvements can also considerably improve the performance of TabR with default parameters.

1 Introduction

The paper develops stronger default MLP and GBDT models for tabular learning, evaluating them on disjoint meta-train and meta-test benchmarks. It also positions MLP improvements as a faster alternative to many specialized neural architectures and studies how better defaults affect the time-accuracy tradeoff.

1 Introduction

  • MLPs are strong tabular neural baselines that are often slower than GBDTs but faster than many other architectures, motivating improvements to MLPs and evaluation of their transfer to TabR.Source: §1 · p. 1

1.1 Contribution

  • Defaults are tuned as a meta-learning problem on 118 datasets and evaluated on 90 disjoint datasets, with separate classification and regression settings.The benchmarks cover 1K–500K samples and exclude missing numerical values.Source: §1.1 · p. 2
  • RealMLP combines architectural, training, preprocessing, hyperparameter, and initialization improvements, often outperforming similarly fast neural baselines and competing with GBDTs.The paper also introduces RealTabR-D, which applies some of these techniques to TabR and achieves strong benchmark results despite less extensive tuning.Source: §1.1 · p. 2
  • Tuned defaults for XGBoost, LightGBM, and CatBoost outperform library defaults on the meta-test benchmark but do not match HPO on average.Source: §1.1 · p. 2
  • Algorithm selection among default methods provides a better time-performance tradeoff than HPO, and the models are evaluated on both the meta-test and Grinsztajn et al. benchmarks.Source: §1.1 · p. 2

1.2 Related Work

  • The work expands MLP improvements rather than attention-based architectures, which are often significantly slower than MLPs or GBDTs.Prior MLP-related work includes SELU, bias initialization, regularization, and categorical and numerical embedding layers.Source: §1.2 · p. 2

Benchmarks

  • The evaluation addresses concerns about dataset-specific tuning by testing on datasets not used for tuning and extending comparison to larger benchmarks and AutoML and OpenML-CTR23 datasets.Unlike TabPFN [23], the approach meta-learns hyperparameters rather than a learning method, enabling use of fewer, larger, more realistic meta-train datasets.Source: Benchmarks · pp. 2–3

2 Methodology

The methodology evaluates hyperparameter configurations across separate meta-train and meta-test benchmarks, then optimizes defaults using an aggregate score based on repeated dataset splits. Benchmark selection and preprocessing define the evaluation scope.

2 Methodology

  • Evaluation scores fixed hyperparameter configurations across meta-train and meta-test benchmarks to limit tuning overfit, and uses the smaller Grinsztajn et al. benchmark for expensive baselines.Because meta-train includes related dataset variants, its weighting factors are inversely proportional to group size.Source: §2 · p. 3

2.1 Benchmark Data Selection

  • The meta-test benchmark includes datasets that are more extreme in several dimensions, allowing tests of whether default parameters generalize out of distribution.Across all datasets, rows with missing numerical values are removed, and missing categorical values are encoded as a separate category.Source: §2 · p. 3

2.2 Aggregate Benchmark Score

  • Default parameters are optimized using ten random 60%/20%/20% train-validation-test splits per dataset and classification error or 1-AUROC for classification.To avoid dependence on the compared methods and reduce sensitivity to zero errors, the aggregate is a shifted geometric mean with ε=0.01; meta-test and Grinsztajn datasets receive equal dataset weights, while meta-train weights account for related-dataset groups.Source: §2.2 · p. 4

3 Improving Neural Networks

Section 3 introduces RealMLP-TD, an MLP designed using meta-train experiments, alongside a simplified faster variant and a TabR adaptation incorporating selected RealMLP improvements.

Data preprocessing

  • Numerical columns are scaled using robust or min-max scaling and smoothly clipped, while categorical columns with at most eight values are one-hot encoded.Binary categories use values {−1, 1}, and missing categorical values are encoded as zero.Source: §3 · p. 4

NN architecture

  • RealMLP-TD uses three 256-neuron hidden layers, categorical embeddings, PBLD numerical embeddings, a learnable feature-scaling layer, and neural tangent parametrization.The parametrization scales weight contributions by input dimension to help prevent overly large steps when there are many columns.Source: §3 · pp. 4–5
  • RealMLP-TD uses learnable parametric activations, dropout after each activation, and data-dependent initialization of weights and biases.It uses SELU for classification and Mish for regression; the learnable activation parameters are initialized to recover the standard activations.Source: §3 · p. 6

Training

  • Training uses AdamW for 256 epochs with batch size 256, β1 = 0.9 and β2 = 0.95, plus task-specific classification or regression losses.Classification uses softmax cross-entropy with label smoothing ε = 0.1; regression uses MSE with targets affinely transformed to zero mean and unit variance.Source: §3 · p. 6

Hyperparameters

  • A four-cycle learning-rate schedule pairs scheduled dropout and weight decay with validation-based best-epoch selection instead of classical early stopping.Training always runs for the full 256 epochs, then reverts to the epoch with the lowest validation classification error or regression RMSE; ties favor the last epoch.Source: §3 · p. 6

RealMLP-TD-S

  • RealMLP-TD-S is a simpler, faster variant that omits embedding layers, uses non-parametric activations and simpler initialization, and drops dropout and weight decay.Source: §3 · p. 6

RealTabR-D

  • RealTabR-D adapts TabR-S-D with RealMLP numerical preprocessing, β2 = 0.95, a modified scaling layer, PBLD numerical embeddings, and classification label smoothing.The scaling layer amplifies the effective learning rate by a factor of 96.Source: §3 · p. 7

4 Gradient-Boosted Decision Trees

The paper derives tuned default hyperparameters for LightGBM, XGBoost, and CatBoost through semi-automatic meta-train optimization and manual adjustment. The tuned defaults share several trends across tasks and libraries.

4 Gradient-Boosted Decision Trees

  • The authors used hyperopt and SMAC3 [38] to explore configurations on meta-train benchmarks, then manually adjusted the best values; they fixed 1,000 estimators and used XGBoost’s hist method.They retained each library’s default tree-building strategy and report tuned defaults in Tables C.1–C.3.Source: §4 · p. 7
  • Across tuned defaults, row subsampling was universal, column subsampling rare, regression trees generally deeper than classification trees, and CatBoost’s Bernoulli bootstrap competitive with Bayesian while faster.The authors note that some hyperparameter values may be sensitive to the tuning and benchmark setup.Source: §4 · p. 7

5 Experiments

Experiments compare library defaults, meta-tuned defaults, and per-split hyperparameter optimization; RealMLP and RealTabR perform strongly, while benchmark metrics and design affect how results compare.

5.1 Methods

  • The experiments compare tree-based and neural methods in default, tuned-default, and per-split HPO variants, alongside validation-selected model portfolios and ensembles.The study omits FTT, RF-HPO, and TabR-HPO on some benchmarks because of computational cost or out-of-memory risks.Source: §5.1 · p. 7

5.2 Results

  • On meta-train and meta-test, RealMLP and RealTabR outperform GBDTs in shifted geometric mean error; on Grinsztajn, RealMLP trails CatBoost for classification but is comparable for regression.On Grinsztajn, RealTabR-D is comparable to CatBoost-TD for classification and better for regression.Source: Figure 2 · p. 8, §5.2 · p. 9
  • RealMLP’s tuned defaults transfer well to meta-test, but GBDT tuned defaults are less competitive with HPO there, despite improving substantially over untuned defaults.XGB-PBB-D from Probst et al. [50] outperforms XGB-TD on Btest classification; the authors suggest benchmark similarity or its 4,168 estimators and deeper trees may explain this.Source: §5.2 · p. 9
  • RealMLP-TD and RealTabR-D improve on MLP-PLR-D and TabR-S-D, respectively, while RealMLP-HPO improves on MLP-PLR-HPO.RealTabR-D performs better than RealMLP-TD on four of six benchmarks, especially regression; on the Grinsztajn et al. benchmark, TabR-HPO performs best by many aggregation metrics, particularly on electricity. TabR and FT-Transformer are notably slower than MLP-based methods on CPUs.Source: §5.2 · p. 9
  • Best-TD is faster on average than 50-step HPO and competitive with the best HPO models, while Best-D is often outperformed by RealMLP-HPO.Ensembling usually improves benchmark scores by 0–3% over selecting the best model; tuned CatBoost defaults are better and slower than other GBDTs on most benchmarks.Source: §5.2 · pp. 9–10
  • RealMLP’s component contributions depend on addition order, and non-architectural choices are at least as important as architectural improvements.Numerical preprocessing is often beneficial for other neural networks, while the scaling layer and PBLD embeddings also work effectively in RealTabR-D; label smoothing can hurt AUROC.Source: §5.2 · p. 10
  • Benchmark choices alter the apparent advantage of tuned defaults: alternative aggregation metrics reduce it, and AUROC rather than classification error favors GBDTs.Generalization of the defaults to very small datasets, distribution shifts, missing numerical values, or metrics such as log-loss is unclear; runtimes and tradeoffs may also vary with hardware, parallelization, or HPO algorithms.Source: §5.2 · p. 10
  • Additional experiments find that refitting multiple models is often better on average than bagging, and GBDTs benefit from high early-stopping patience for classification.For AUROC stopping, cross-entropy is preferable to accuracy.Source: §5.2 · p. 10

6 Conclusion

The paper evaluates improved GBDT defaults and an improved MLP on a separate meta-test benchmark and the benchmark by Grinsztajn et al.. RealMLP offers moderate runtime and competitive benchmark scores, while the results support trying both algorithm families with good defaults.

6 Conclusion

  • RealMLP mostly outperforms other literature NNs at moderate runtime and is competitive with GBDTs in benchmark scores.The evaluation includes a large separate meta-test benchmark and the benchmark by Grinsztajn et al..Source: §6 · p. 10
  • RealTabR demonstrates that RealMLP improvements can be combined with orthogonal neural-network improvements from other work.Source: §6 · p. 10
  • With good default parameters, it is worth trying both neural networks and GBDTs even with a moderate training-time budget.Source: §6 · p. 10

Appendices … A.2 RealMLP-TD-S Details

The appendices detail RealMLP-TD implementation choices, including data-dependent initialization, and describe the simplifications made for RealMLP-TD-S.

A Further Details on Neural Networks

A.1 RealMLP-TD Details

  • Linear weights use data-dependent initialization, rescaling each row for unit output variance; biases use he+5 [called hull+5 in 61], somewhat similar to LSUV [44].Initialization runs on the first training-set forward pass, with adaptive subsampling to stay within 1 GB of RAM; fit_transform() methods implement it.Source: §A.1 · p. 17
  • Weight decay follows PyTorch’s update θ ← θ − lr · wd · θ, so it includes the learning rate, unlike the original version [40].Source: §A.1 · p. 17

A.2 RealMLP-TD-S Details

  • RealMLP-TD-S one-hot encodes every categorical variable and omits both categorical and numerical embeddings.Source: §A.2 · p. 17
  • TD-S uses standard non-parametric SELU and Mish activations and omits dropout and weight decay.Source: §A.2 · p. 17
  • TD-S initializes weights and biases from N(0, 1), except for a zero-initialized last layer; it does not clip regression outputs and uses a different regression base learning rate.Source: §A.2 · p. 17

A.3 RealTabR-D Details … B More Experiments

These appendices specify RealTabR-D and cumulative-ablation procedures, discuss RealMLP’s speed, encoding, and scoring tradeoffs, and qualify additional experiments affected by an XGBoost bug.

A.3 RealTabR-D Details

  • RealTabR-D uses robust scaling and smooth clipping, Adam β2 = 0.95, γ = 96 scaling, PBLD numerical embeddings, 8-dimensional hidden embeddings, and classification label smoothing.The authors report γ = 96 performed significantly better than γ = 6 on meta-train benchmarks; the smaller hidden embedding dimension was chosen to fit experiments on a 24GB GPU, although 16 performed slightly better in meta-train experiments.Source: §A.3 · pp. 17–18
  • Because its learning-rate and weight-decay hyperparameters were adapted from TabR-S-D without meta-learning, the authors call the method RealTabR-D, not RealTabR-TD.Brief tests of NTP and Mish worsened performance; parametric activations and increased stopping patience offered small improvements but were excluded for runtime, while other tricks were not tried due to time, runtime, or implementation complexity.Source: §A.3 · p. 19

A.4 Details on Cumulative Ablation

  • The cumulative ablation starts from a three-layer, 256-unit ReLU MLP and selects a default learning rate from separate grids for standard and neural-tangent parametrizations.The listed ablation steps vary preprocessing, categorical encoding, stopping, schedules, Adam β2, label smoothing, output clipping, parametrization, activations, scaling, numerical embeddings, dropout, weight decay, and initialization.Source: §A.4 · p. 19

A.5 Discussion

  • Vectorized training across train-validation-test splits can speed GPU training, but can hinder patience-based early stopping and loss-based learning-rate schedules.The authors report that ablations favor their multi-cycle schedule over decreasing learning-rate schedules, while the latter may allow faster average training with low-patience stopping; larger-patience stopping remains an open question.Source: §A.5 · p. 20
  • The meta-train benchmark cannot resolve whether categorical embeddings help over one-hot encoding, while pure one-hot encoding is faster on most datasets.The authors cite potential embedding efficiency for high-cardinality categories, but note that such variables are uncommon in meta-train; they found size 4 adequate for numerical embeddings and chose size 8 for categorical ones.Source: §A.5 · p. 20
  • For faster networks, omitting embedding layers and parametric activations can help, while regression may benefit from more epochs or wider layers; label smoothing may misfit metrics such as AUROC.Larger batches can also help on larger datasets. Cross-entropy with label smoothing is not a proper scoring rule, so in the infinite-sample limit it does not recover true class probabilities.Source: §A.5 · p. 20

B More Experiments

  • An XGBoost categorical-encoding bug lowered meta-test SGM scores for XGB-TD and XGB-D by around 2% after fixing, but the authors say qualitative conclusions were unchanged.They retained the old XGB-TD and XGB-D results for comparison with XGB-HPO and XGB-PBB-D, which were not rerun because of computational cost.Source: B More Experiments · p. 20

B.1 MLP Ablations … B.4 Early stopping for GBDTs

The appendices examine RealMLP ablations and preprocessing, ensembling and refitting variants, and early-stopping choices for GBDTs. They report which choices matter across tasks and how model count, fitting strategy, and stopping settings affect benchmark performance.

B.1 MLP Ablations

  • Ablations often change regression scores more than classification scores; PBLD beats other numerical-embedding options, and PL outperforms PLR in this setting.PLR and no numerical embeddings appear especially poor for regression, but can be more benign with lower weight decay. The authors note that nRMSE may be more sensitive to outliers and that the classification benchmark has more datasets.Source: §B.1 · pp. 20–21
  • Adam β2 = 0.999 performs considerably worse than β2 = 0.95, especially for regression; cosine decay causes small deterioration, while constant learning rates perform worse.The β2 difference is less pronounced at lower weight decay. Omitting classification label smoothing is detrimental by around 1.8%, while the learnable scaling layer improves both benchmarks by around 1.2%.Source: §B.1 · p. 21
  • Parametric activations improve regression by 4.8% but are insignificant for classification, while activation-function differences are small and dropout and weight decay give comparable, not always significant, benefits.Mish performs best for regression; for classification it is competitive with SELU in this ablation but was worse in some other hyperparameter settings. Flat_cos scheduling of dropout and weight decay helps regression, not classification.Source: §B.1 · p. 21
  • Data-dependent initialization gains around 1% on both benchmarks; SP without weight decay performs insignificantly worse than NTP, and tie-breaking and one-hot encoding show no significant difference.Selecting the first rather than last best epoch yields a non-significant 0.4% deterioration. Using one-hot encoding for all categorical variables makes no significant difference, with few large-cardinality datasets in the benchmarks.Source: §B.1 · p. 21

B.2 MLP Preprocessing

  • Robust scaling with smooth clipping performs best for numerical preprocessing on both classification and regression benchmarks.The comparison uses RealMLP-TD-S without numerical embeddings and tests robust scaling, standardization, and quantile-based transformations before one-hot encoding.Source: §B.2 · pp. 21–22, Table B.2 · p. 23

B.3 Bagging, Refitting, and Ensembling

  • Five-model ensembles outperform single models, refitting is mostly better than bagging, and the benefit of five models appears larger for RealMLP-TD than LGBM-TD.Stopping preferences vary by scenario: individual stopping has a slight advantage for five-model bagging, while joint stopping favors single-model refitting; in the other two scenarios, joint stopping appears slightly better for RealMLP-TD and slightly worse for LGBM-TD.Source: Table B.3 · p. 23, §B.3 · p. 23, Table B.4 · p. 24

B.4 Early stopping for GBDTs

  • For GBDTs, low early-stopping patience harms classification more than regression; patience 300 is close to optimal when stopping on classification error.Classification-error stopping deteriorates strongly at patience ≲100. Cross-entropy stopping is less sensitive at low patience but gives roughly 2% worse benchmark score at high patience; Brier loss performs very well at high patience and is only slightly more patience-sensitive than cross-entropy.Source: §B.4 · pp. 23–24, Figure B.1 · p. 24

B.5 Results for AUROC · B.6 Results Without Missing-Value Datasets

On AUROC, RealMLP without label smoothing remains competitive with GBDTs but does not surpass them as it did on classification error. Excluding meta-test datasets that originally contained missing values slightly worsens RealMLP, while qualitative takeaways remain similar.

B.5 Results for AUROC

  • The AUROC analysis uses one-vs-rest AUC for multiclass tasks and evaluates 1 − AUC, although configurations and HPO were tuned for classification error.Source: §B.5 · p. 24
  • RealMLP without label smoothing is competitive with GBDTs on meta-test AUROC but does not outperform them, unlike on classification error.Source: §B.5 · p. 25
  • Stopping on cross-entropy generally performs better than stopping on classification error for AUROC.Source: §B.5 · p. 25
  • Label smoothing harms RealMLP on AUROC; possible explanations are a stopping metric without smoothing or near-constant logits where the model is relatively certain.The paper presents these explanations as possibilities.Source: §B.5 · p. 25
  • Tuned defaults mostly outperform library defaults on AUROC, with XGBoost on Btest class as the exception.Source: §B.5 · p. 25

B.6 Results Without Missing-Value Datasets

  • Removing the six meta-test datasets that originally contained missing values slightly worsens RealMLP, especially when fps_benchmark is excluded, but leaves qualitative takeaways similar.Source: §B.6 · p. 25

B.7 Comparing Preprocessing Methods for NNs

The appendix tests whether robust scaling and smooth clipping can improve several neural-network methods beyond RealMLP. Results are promising for some methods but not others without retuning, while trying both preprocessing approaches can improve performance on most benchmarks.

B.7 Comparing Preprocessing Methods for NNs

  • Elsewhere in the paper, RealMLP uses robust scaling and smooth clipping, while the other neural methods use the RTDL quantile transform.Appendix B.7 compares preprocessing choices across the methods.Source: §B.7 · p. 25
  • The evaluation tests whether robust scaling and smooth clipping benefit MLP, ResNet, MLP-PLR, FTT, and TabR-S for a more direct architecture comparison.The networks still differ in other aspects, including initialization and regularization.Source: §B.7 · p. 26
  • Trying robust scaling and smooth clipping (RS+SC) without retuning defaults looks promising for MLP and TabR, less so for MLP-PLR, and improves results on most benchmarks.The comparison also includes ResNet, FTT, and TabR-S.Source: §B.7 · p. 26

B.8 Results for Varying Architecture … Appendix C.5.

This appendix presents supplementary analyses and documentation on RealMLP’s architectural changes, GBDT tuning, benchmark dataset selection and design, evaluation choices, and related plots and implementation details.

B.8 Results for Varying Architecture

  • Around 5% or more improvement in benchmark scores across all benchmarks followed from RealMLP’s architectural changes, though they alone did not match TabR-S-D; non-architectural RealMLP-TD changes helped more.The comparison includes preprocessing and architecture variants.Source: Table B.5 · p. 28, §B.8 · p. 28
  • The MLP-D reproduction scored within 1% of MLP-D (RS+SC), while adding PL embeddings sometimes helped but was significantly worse on Btest class.The reproduction omitted weight decay and used marginally different early-stopping thresholding; its default learning rate was selected on the meta-train benchmark.Source: §B.8 · p. 28

B.9 Comparing HPO Methods

  • In the reported comparison, 50-step TPE often performed slightly better than 50-step random search, although the differences in benchmark scores were relatively small.TPE was implemented in hyperopt, and its first 20 of 50 steps used random search.Source: §B.9 · p. 28

B.10 More Time-Error Plots

  • Additional time-versus-error plots report arithmetic mean error, arithmetic mean rank, and arithmetic mean normalized error.Normalized errors are rescaled on each dataset split so the worst score is 1 and the best is 0.Source: §B.10 · p. 28

B.11 Critical Difference Diagrams

  • The meta-test benchmarks show that RealMLP-TD’s improvements over MLP-D are externally valid, although establishing external validity requires many datasets.The analysis concerns whether differences in average ranks generalize to new datasets from a distribution.Source: §B.11 · p. 28

B.12 Win-rate Plots

  • Win-rate plots compare the percentage of dataset-split combinations where each row method has lower error than each column method; ties count as half-wins.Wins are averaged across datasets and splits using dataset-dependent weighting, with methods sorted by average win-rate.Source: §B.12 · pp. 28, 34

C.1 Default Configurations; C.2 Hyperparameter Optimization

  • Default configurations are documented for the evaluated models, and HPO uses 50-step random searches with literature-adapted spaces for tree and neural methods.XGB-PBB-D uses Probst et al. [50] defaults with hist boosting and native categorical handling for high-cardinality features; RF-D uses scikit-learn defaults with ordinal-encoded categories and no validation set.Source: §C.1 · p. 40, §C.2 · p. 40

C.3.1 Meta-train Benchmarks

  • The meta-train benchmark contains 71 classification and 47 regression datasets, selected from UCI under sample, feature, task, and format criteria.Rows with missing values are removed, versions are weighted proportionally to 1/#versions, and regression targets are standardized.Source: §C.3.1 · p. 48

C.3.2 Meta-test Benchmarks

  • Meta-test dataset selection excludes datasets with fewer than 1,000 samples and classification datasets exceeding 10,000 features after one-hot encoding.The passages list excluded datasets across classification and regression, including Amazon_employee_access, Click_prediction_small, and sf-police-incidents for the feature-count criterion.Source: §C.3.2 · p. 50

C.3.3 Grinsztajn et al. [18] Benchmarks

  • The Grinsztajn et al. [18] benchmark uses its newer OpenML version, prefers versions with categorical features, and excludes eye_movements because of a reported leak.Tables C.24 and C.25 identify the classification and regression datasets in this benchmark.Source: §C.3.3 · p. 50, Table C.24 · p. 52, Table C.25 · p. 53

C.4 Comparison with Standard Grinsztajn et al. [18] Benchmark

  • The comparison distinguishes a revised setup using Grinsztajn et al. datasets from a slightly modified version of the original benchmark code.The revised version is used in plots except Figures C.2 and C.3; the passages also describe differences in preprocessing, baselines, stopping settings, and XGBoost configuration.Source: §C.4 · p. 54, Appendix C.5 · p. 54

Appendix C.5.

  • The revised benchmark is preferred for more realistic validation, identical baselines, and broader evaluation and plotting options.Figure C.1 reports the most comparable metrics for the revised version, while Figure C.2 shows the old version; the supplied passages do not give their results.Source: Appendix C.5 · p. 54
  • The revised setup uses ten splits per dataset, varies hyperparameter configurations across splits, and measures all runtimes on CPU.It subsamples downloaded datasets to 500K samples, standardizes regression targets, caps training plus validation at 13,333 samples with 25% for validation, and avoids separate validation sets for early stopping and HPO. It reports multiple aggregation strategies and uses nRMSE rather than R2 for regression, while attempting comparable aggregated metrics in Figure C.1.Source: Appendix C.5 · p. 54

C.5 Closer-to-original Version of the Grinsztajn et al. [18] Benchmark … C.9 Used Libraries

These appendices specify the adapted Grinsztajn et al. benchmark protocol and document how the paper estimates uncertainty and runtime, along with compute use and implementation libraries.

C.5 Closer-to-original Version of the Grinsztajn et al. [18] Benchmark

  • Figures C.2 and C.3 report closer-to-original benchmark results, using normalized accuracy/R2 or best validation-selected scores, while runtime averages use per-1,000-sample measurements.Figure C.2 normalizes scores by dataset before averaging; runtime comparisons use GPU neural-network runs and one CPU core for other models, with neural-network timing restricted to datasets having at least 15 steps on the same GPU model.Source: Figure C.2 · p. 55, Figure C.3 · p. 56, §C.5 · p. 56
  • Benchmark datasets follow Grinsztajn et al.; neural-network preprocessing varies by model, categorical features may be one-hot encoded, and training, validation, and test sizes are capped.Neural networks use Gaussian quantile transforms, except RealMLP, which uses robust scaling and smooth clipping; target standardization is also available, and training is limited to 10,000 samples and validation and test to 50,000 each.Source: §C.5 · pp. 55–56

C.6 Confidence Intervals

  • Confidence intervals treat datasets as fixed and splits as random, averaging within datasets and applying Student’s t intervals under a normality assumption for the resulting independent dataset-level variables.For shifted geometric means, the procedure uses f(x) = log(x + 0.01) and g = exp; the transformed interval endpoints follow because g is increasing.Source: §C.6 · p. 57

C.7 Time Measurements

  • Meta-train and meta-test runtimes were measured sequentially on a 32-core CPU node without GPUs, extrapolating short random searches to 50 steps and summing component runtimes for model combinations.Runtime per 1,000 samples was averaged across datasets without the usual dataset-dependent weighting.Source: §C.7 · p. 57

C.8 Compute Resources

  • The experiments required at least around 3,000 GPU hours and roughly 10,000 hours on 32–64-core HPC CPU nodes.The authors note that compute resources were not measured precisely.Source: §C.8 · p. 57

C.9 Used Libraries

  • The implementation used PyTorch [47], Scikit-learn [48], Ray [46], XGBoost [9], LightGBM, and CatBoost [51], adapting GBDT wrappers from CatBoost quality benchmarks [51].Source: §C.9 · p. 57

D Results for Individual Datasets

This section presents per-dataset error results in model-comparison scatterplots and tables spanning three benchmarks, classification and regression, and tuned and untuned methods.

Scatterplots

  • Figures D.1–D.5 compare Best-TD with CatBoost-HPO and Best-HPO, RealMLP-TD and RealMLP-HPO with CatBoost variants, and Ensemble-TD with Best-TD.Each plotted error is averaged across ten train-valid-test splits.Source: Figure D.1 · p. 59, Figure D.2 · p. 60, Figure D.3 · p. 61
  • Figures D.6–D.7 compare RealMLP-TD with RealMLP-HPO and CatBoost-TD with CatBoost-HPO, respectively.Each plotted error is averaged across ten train-valid-test splits.Source: Figure D.6 · p. 64, Figure D.7 · p. 65

Tables

Related papers

Loading 2407.04491v3…