PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond IID: How General Are Tabular Foundation Models, Really?

Lennart Purucker, Andrej Tschalzev, Nick Erickson, Gioia Blayer, David Holzmüller, Alan Arazi, Alexander Pfefferle, Mustafa Tajjar, Gaël Varoquaux, Frank Hutter

arXiv:2606.30410v1cs.LGcs.AI

TL;DR

Benchmarking for tabular foundation models is fragmented and concentrated on IID tasks. This paper introduces BeyondArena and DataFoundry to evaluate 11 models across 142 diverse datasets, finding that foundation models perform well on small- to medium-scale IID data but are outperformed in non-IID, large-scale, and high-dimensional settings.

  • Problem

    Tabular foundation models are evaluated across diverse communities, but tabular research benchmarks remain fragmented and largely limited to IID tasks.

  • Method

    BeyondArena unifies evaluation across task types, dataset scales, and feature types, while DataFoundry supports reproducible tabular dataset curation.

  • Results

    Tabular foundation models perform well on small- to medium-scale IID data but are outperformed by traditional methods on non-IID, large-scale, and high-dimensional data.

  • Takeaways & Limitations

    BeyondArena shows that current tabular foundation models are not yet fully general for tabular data and identifies broader challenges for future research.

  • Takeaways & Limitations

    The conclusions are limited by evaluating in-context learning without fine-tuning, selecting only three foundation models, and tuning models under compute constraints.

Abstract

from arXiv · show

Foundation models for predictive machine learning on tabular data have recently gained significant traction in academia and industry. Research communities across disciplines are increasingly evaluating tabular foundation models on diverse datasets and tasks. However, these task- and discipline-specific evaluations remain largely inaccessible to model researchers because benchmark software and evaluation protocols are fragmented. As a result, model researchers rely on standard benchmarks, which are mostly defined for tasks where tabular foundation models already excel. The most challenging scenarios are excluded, limiting meaningful progress in the field by focusing on marginal improvements on IID data rather than on broader, more demanding challenges. To overcome this, we introduce BeyondArena, the first unified holistic benchmark for tabular data that supports diverse task types (IID, temporal, grouped), across sample size and feature dimensionality scales, with diverse feature types (with text, with high cardinality) from a broad range of disciplines. To enable unified benchmarking beyond standard benchmarks, we introduce Data Foundry, a Python framework and metadata schema for curating tabular datasets for predictive machine learning. Our results across 11 models and 142 curated datasets show that existing tabular foundation models excel on tiny- to medium-sized IID data, while traditional tree-based and deep learning models still dominate on non-IID, large, and high-dimensional datasets. BeyondArena guides model research for the most demanding challenges in tabular data, enabling progress towards truly foundational tabular models.

1 Introduction

BeyondArena addresses fragmented and IID-limited evaluation of tabular foundation models by curating diverse datasets, standardizing benchmarking, and testing three open-source TFMs against eight traditional baselines. The results show TFMs dominate tiny, small, and IID data but fail to compete on non-IID, large-scale, high-dimensional, and high-cardinality categorical datasets.

  • Motivation: The study motivates broader evaluation by noting that tabular research has largely remained limited to IID tasks despite diverse TFM assessments in other communities.BeyondArena aims to close the gap between academic evaluations and practitioners’ real-world predictive applications.
  • Benchmark and evaluation: BeyondArena manually curates 142 datasets from 1128 and evaluates three state-of-the-art open-source TFMs against eight strong traditional baselines across diverse tasks.The benchmark targets predictive tabular classification and regression beyond standard IID evaluations.
  • Data curation: DataFoundry provides a Python framework and metadata schema for reproducible tabular dataset curation across scales and feature types.The broader ecosystem integrates these contributions into TabArena to reduce fragmentation in tabular-model evaluation.
  • Scope: BeyondArena focuses on classification and regression from 100 to 1M samples, including temporal and grouped data plus high-cardinality categorical and text features.Few-shot predictions below 100 samples, images as features, survival analysis, and relational learning are out of scope.
  • Results: TFMs dominate tiny, small, and IID data but fail to compete with traditional tree-based and deep learning models on non-IID, large-scale, high-dimensional, and high-cardinality categorical datasets.The evaluation covers temporal and grouped non-IID settings and diverse scales and feature types.

2 Background: IID and non-IID Tabular Data

BeyondArena classifies tabular data as IID or non-IID according to the application’s appropriate train-test split, rather than the dataset alone. Random splits suit unstructured test samples, whereas temporal or grouped splits are required when applications target future observations or unseen groups.

  • Application-dependent definition: The appropriate IID or non-IID classification depends on the practitioner’s application and its required train-test split.The same transaction data can require a random IID split for retrospective fraud investigation or a temporal non-IID split for real-time prevention; using the wrong split can misestimate performance, including through temporal leakage [15] [75].
  • Application-dependent definition: BeyondArena extends application-dependent split selection to data curation and academic benchmarking.This perspective connects practitioners’ evaluation choices with the benchmark’s organization of diverse application data.
  • IID Tabular Data: IID data permit a random split because test samples do not follow a particular structure.The definition holds out randomly sampled data.
  • Non-IID Tabular Data: Non-IID data require temporal or grouped splits to reflect their application structure.Temporal splits place test samples strictly after training data using a time index, while grouped splits keep every group entirely in either training or test using a group index.

3 Related Work

Prior tabular benchmarks predominantly target IID tasks, while non-IID benchmarks focus mainly on temporal data. BeyondArena unifies IID and non-IID evaluation through shared metadata, broader datasets and baselines, and rigorous assessment across tasks and multimodal data types.

  • IID Tabular Benchmarks: Prior tabular benchmarks predominantly focused on IID predictive tasks, including benchmarks for model development, tabular deep learning, AutoML, and data-centric evaluation [85] [87] [89] [90] [93] [94] [95] [96] [97] [99] [103].
  • Non-IID Tabular Benchmarks: Non-IID tabular benchmarks focused mainly on temporal data, despite decades of broader research on non-IID tabular data.
  • Non-IID Tabular Benchmarks: BeyondArena builds upon TabReD by supporting more temporal datasets, more baselines, and tabular foundation models [15] [165] [166].
  • BeyondArena unifies IID and non-IID benchmarking by curating representative datasets in one shared metadata format and evaluating state-of-the-art tabular models across tasks and multimodal data types.

4 Dataset Curation

BeyondArena curates datasets for challenging real-world tabular prediction by expanding coverage to non-IID tasks, unusual sample sizes, and text or date features. Data Foundry makes this curation reproducible through a Python framework, processing notebooks, and machine-readable metadata.

  • Dataset Selection Criteria: The curation criteria support IID, temporal, and grouped tasks; at least 100 training samples; and datasets with fewer than 500 or more than 100,000 training samples.Datasets and predictive tasks must be unique within the benchmark and use an appropriate random, temporal, or grouped validation protocol.
  • Dataset Selection Criteria: BeyondArena extends dataset coverage to tabular data containing text and date features, alongside non-IID applications and broader dataset-size scales.These criteria were designed to better represent challenging real-world predictive machine-learning applications than the prior TabArena selection criteria.
  • Data Foundry: Data Foundry provides a Python package for dataset checks and train-test splits, plus notebooks containing the exact processing code used for each dataset.The framework and metadata schema are intended to make extensive dataset curation reproducible.
  • Dataset Sources: Datasets were gathered from 21 tabular benchmark studies, including 304 datasets re-evaluated from 14 prior benchmarks and additional non-IID or multimodal sources.The source search also covered public data repositories and seven non-IID or multimodal tabular benchmarks.
  • Dataset Processing: Each selected dataset was standardized with unified formats, feature-type annotations, task metadata, and machine-readable curation notes, with feature engineering applied when needed.Stored metadata includes train-test splits, target columns, and group or time-index columns.
  • Curation Outcome: 142 of 1128 collected datasets met the selection criteria, with roughly 12.6% of investigated datasets selected for BeyondArena.The filtering process primarily excluded duplicates, non-real applications, and datasets not originally created for predictive tasks.

5 Experimental Design

BeyondArena evaluates 11 competitive tabular models using reproducible open-source implementations, task-appropriate validation and preprocessing, and robust metrics across IID and non-IID settings. Model coverage is constrained for TabPFN-2.6 and TabDPT by sample-size, pretraining, and computational limits.

  • BeyondArena Models: The benchmark compares 11 models spanning linear, tree-based, neural-network, and tabular foundation models, including Random Forest, CatBoost, LightGBM, XGBoost, TabDPT [193], TabPFN-2.6 [194], and TabICLv2 [196].Models were selected based on competitive performance on TabArena, TALENT [112], and TabReD [15].
  • BeyondArena Models: TabPFN-2.6 and TabDPT are run only on datasets with up to 100k training samples, with TabPFN-2.6 kept within pretraining limits and TabDPT restricted because retrieval-based inference is computationally infeasible.For a 1 million-sample dataset under cross-validation, TabDPT requires 8 million forward passes; missing results for both models are imputed using default Random Forest performance.
  • Open-source Implementation: The benchmark uses unit-tested, open-source AutoGluon-based implementations [181], modified to scale to larger datasets and correctly handle non-IID tasks such as unseen categories.BeyondArena builds on TabArena [11] and adds non-IID validation support.
  • Outer Splits and Inner Validation: Validation uses size-dependent repeated outer cross-validation and inner tuning splits, with stratified classification, random IID splits, grouped splits for grouped tasks, and time-binned intervals for temporal tasks.Datasets with fewer than 500 training samples use 5-repeated 5-fold inner cross-validation to avoid overtuning.
  • Multimodal and non-IID Preprocessing: A general model-agnostic preprocessing pipeline handles dates, text, and grouped data, while model-specific extensions retain operations such as scaling and categorical encoding.The pipeline is designed for multimodal non-IID data and extends TabArena’s preprocessing.
  • Metrics: The evaluation uses ROC AUC for binary classification, log-loss for multiclass classification, and RMSE for regression, aligning with prior benchmarks [11] [15] and machine-learning literature [210].ROC AUC avoids dependence on threshold tuning, log-loss is a proper scoring rule, and RMSE evaluates point prediction.

6 Results

Across 142 datasets, tabular foundation models excel on tiny-to-medium IID data but fail to match traditional models on non-IID, large-scale, high-dimensional, and high-cardinality settings. They nevertheless achieve peak performance or tie the best non-TFM on 70% of datasets, while ablations broadly validate the benchmark design.

  • Evaluation scope: BeyondArena evaluates generality through 12 sub-benchmarks spanning task type, dataset scale, and feature characteristics.The benchmark covers IID, grouped, and temporal tasks, multiple dataset scales, and feature settings including text and high cardinality.
  • Overall benchmark results: TFMs perform best on small, tiny, and IID datasets, but fail to compete with traditional models on temporal, grouped, large-scale, high-dimensional, and high-cardinality data.Traditional models, especially RealMLP, benefit substantially from tuning and ensembling, while TFM default training costs remain significantly lower.
  • Peak-performance coverage: 70% of datasets are viable for peak performance with TFMs: they significantly outperform other models on 49% and match the best non-TFM on another 21%.TabICLv2 ranks first on 19% of datasets, followed by TabPFN-2.6 at 10.5%.
  • Ablation studies: BeyondArena’s grouped-data preprocessing improves average performance, rankings remain mostly stable for L-P-G data, and long texts generally favor TF-IDF without materially changing model rankings.TabPFN-2.6 improves substantially when preprocessing is disabled, and calibration usually helps log-loss except for TabPFN-2.6 and RealMLP.

7 Conclusion

BeyondArena and DataFoundry establish standardized evaluation across diverse tabular tasks, datasets, feature types, and models, revealing limitations of current tabular foundation models. The conclusion also identifies methodological limitations and future extensions for broader evaluation.

  • BeyondArena and DataFoundry support reproducible evaluation across 142 datasets and 11 models spanning IID, temporal, and grouped tasks, dataset scales, dimensionality, and feature types.
  • Tabular foundation models perform well on small- to medium-scale IID data but are outperformed by traditional methods.
  • Limitations and Societal Impact: The study evaluates only 25 random model configurations, in-context learning rather than fine-tuning, and the top three TabArena TFMs.
  • Future Work: Future work could extend BeyondArena to few-shot prediction, multimodal tabular data, relational learning, and survival analysis.
  • BeyondArena establishes a broader evaluation standard that surfaces critical limitations of current tabular foundation models.

Competing Interests

The authors disclose extensive connections to the evaluated models, benchmark and software projects, and companies developing tabular foundation models.

  • Competing Interests: The authors include developers, maintainers, contributors, or founders associated with RealMLP, TabPFN-2.6, TabICLv2, TabArena, AutoGluon, skrub, and scikit-learn, alongside affiliations with Prior Labs and probabl.These relationships encompass model authorship, project maintenance, software contributions, founding roles, and company affiliations.

Appendices … B.4 Details on Curation Outcome

BeyondArena’s curation process distinguishes temporal tabular tasks from time-series forecasting, applies explicit dataset-selection and processing safeguards, and produces broad coverage across application domains and dataset scales. Because metadata are often undocumented and automated curation can hallucinate, the benchmark relies substantially on subject-matter expertise and documented decisions.

  • A Background: Using forecasting data for temporal tabular regression would ignore its real-world context and estimate performance under a validation setup with questionable practical relevance.The distinction therefore affects both data curation and the interpretation of benchmark performance.
  • B.1 Towards Automated Data Curation?: Automated dataset selection was unreliable because repositories lacked structured metadata and agentic LLMs produced hallucinated or faulty judgments, so subject-matter expertise was required.The authors also share curation insights and decisions publicly because criteria involve subjective human interpretation.
  • Appendices: Curation decisions and criteria are documented publicly because selection relies on subjective human interpretation and follows prior work by Erickson et al. [11].The public record is intended to expose dataset-specific curation insights and decisions.
  • A.1 Time-series Forecasting Tasks vs. Temporal Tabular Tasks: BeyondArena includes random, temporal, and grouped tabular tasks but excludes time-series forecasting because representative evaluation requires a different validation protocol.Forecasting predicts the future, whereas temporal tabular validation may or may not use future data depending on the task; conflating them can make validation unrepresentative [81] [82] [83] [84].
  • B.3 Details on Dataset Processing: Processing standardizes datasets by removing uninformative identifiers, converting reliably inferred proxy missing values to NA, standardizing names and feature types, and reconstructing dates when possible.Informative identifiers such as time indices are retained, skewed numerical targets may receive case-by-case logarithmic scaling, and object or string features are assigned categorical or string types.
  • B Dataset Curation: Temporal task construction manually defines prediction horizons and test times, verifies feature availability at prediction time, and filters data to prevent future-information and grouped temporal leakage.The procedure also analyzes repeated observations from the same entities over time, such as multiple transactions from one entity.
  • B.4 Details on Curation Outcome: Curated datasets span broad application domains and dataset scales, with healthcare, marketing, and finance prominent and medical datasets generally smaller.Figure B.1 reports domain counts, while Figure B.2 characterizes variation in rows and columns across domains.

B.5 BeyondArena Datasets Overview … D.2 BeyondArena Leaderboard as a Table

BeyondArena spans 142 datasets with varied domains, scales, feature types, and task structures, while its evaluation uses dataset-specific preprocessing and a four-hour configuration limit. Results additionally characterize inference-time trade-offs, with CatBoost leading the inference-time Pareto front and TabICLv2 and TabPFN-2.6 improving over tuned CatBoost at substantially higher inference cost.

  • B.5 BeyondArena Datasets Overview: BeyondArena comprises 142 datasets spanning diverse domains, sample and feature scales, text features, categorical cardinalities, and IID, temporal, and grouped tasks.Table B.1 provides per-dataset metadata including rows, columns, classes, problem type, and task type.
  • C.1 Time Limit Impact: The benchmark uses a four-hour limit for each configuration on a train split, allowing larger datasets than the one-hour TabArena limit.Training is stopped early when a model does not complete all inner folds within the limit.
  • C.1 Time Limit Impact: Only ∼0.003% of jobs, or 31 of 785,208, exceeded the four-hour limit, showing that the runtime cap was rarely reached.Approximately 1.72% exceeded one hour and ∼0.01% exceeded 3.5 hours.
  • C.2 Additional Details: The experimental setup reduces cross-validation folds when minority-class samples are insufficient and applies specialized datetime, text, and grouped-data preprocessing.Grouped preprocessing uses transductive 50-dimensional group encodings based on within-group aggregations.
  • D. Results: Inference-time comparisons include default, tuned, and post-hoc-ensembled models, with transformer foundation-model default performance equated to in-context learning performance.Figure D.1 presents the corresponding Improvability versus inference-time Pareto front.
  • D.1 Inference Time Overview: CatBoost dominates the inference-time Pareto front, while TabICLv2 and TabPFN-2.6 achieve 2.5% improvement over tuned CatBoost with significantly longer inference time.A tuned RealMLP incurs an even larger inference time.
  • D.2 BeyondArena Leaderboard as a Table: The results section reports per-model performance across all BeyondArena datasets in Table D.1.The supplied passage identifies the leaderboard table but does not provide its cell values or model ranking.

D.3 TabArena-v0.1 Datasets vs. BeyondArena Datasets in the Same Scope · E Statistical Analysis · E.1 Test 1: Are the methods really different from each other?

BeyondArena datasets within TabArena’s scope are more challenging for tabular foundation models, while statistical tests show that leaderboard differences are genuine and distinguish most method pairs.

  • D.3 TabArena-v0.1 Datasets vs. BeyondArena Datasets in the Same Scope: In the same-scope comparison, tabular foundation models dominate the 49 TabArena datasets but are outperformed by RealMLP on 20 new BeyondArena datasets.The comparison uses 49 accepted TabArena datasets and 20 datasets meeting TabArena’s IID and sample-size criteria.
  • D.3 TabArena-v0.1 Datasets vs. BeyondArena Datasets in the Same Scope: Table D.1 reports default, tuned, and tuned-plus-ensembled performances, with approximate 95% bootstrap confidence intervals for Elo values.TabPFN-2.6 and TabDPT results are imputed for datasets exceeding 100k samples.
  • E Statistical Analysis: The statistical analysis compares 27 methods using per-dataset normalized errors averaged across cross-validation folds, covering ROC AUC, log-loss, and RMSE.Methods receive scores from best (0) to worst (1) on each dataset, enabling comparison across heterogeneous tasks.
  • E.1 Test 1: Are the methods really different from each other?: A Friedman test evaluates whether the 27 methods’ rank distributions across 142 datasets are consistent with all methods being equivalent.Ranking methods within each dataset accounts for dataset-level variation in the global comparison.
  • E.1 Test 1: Are the methods really different from each other?: Pairwise Wilcoxon signed-rank tests compare all 351 method pairs using per-dataset paired normalized-error differences and Holm correction.The two-sided test evaluates whether paired differences are symmetrically distributed around zero and rejects H0 when p < 0.05.
  • E.1 Test 1: Are the methods really different from each other?: 260 of 351 method pairs (74.6%) show performance differences surviving Holm correction, indicating that most methods can be reliably distinguished.The result sharpens the global Friedman conclusion that the leaderboard ranking is not plausibly explained by random fluctuations alone.

E.2 Test 2: At what training-set size does using a TFM stop being a reasonable choice? … F.3 (Inner Splits, B.1) Using 8-fold Cross-Validation for Tiny Data

Across BeyondArena, TFM error changes sharply at the large-data boundary, while tuning and ensembling improve traditional models and regime-specific winners vary by dataset properties. Ablations show that reduced splits can preserve aggregate rankings, whereas validation and split choices materially affect benchmark fidelity.

  • E.2 Test 2: At what training-set size does using a TFM stop being a reasonable choice?: TFM degradation occurs at approximately 100,000 rows: tiny, small, and medium datasets are indistinguishable, but every large-scale comparison is significant with large effects.The Kruskal-Wallis test found H = 30.6 on 3 degrees of freedom (p = 1.0 × 10−6); pairwise large-scale comparisons had padj < 0.001 and r between 0.68 and 0.79.
  • E.3 Test 3: How much do tuning and ensembling help non-TFM models?: All 16 paired tests show that hyperparameter tuning and post-hoc ensembling significantly improve every traditional algorithm across the 142-dataset benchmark.Every test had padj < 0.001; RealMLP had the largest tuning-to-ensembling jump (r = 0.87), while LightGBM had the largest default-to-tuned jump (r = 0.89).
  • E.4 Test 4: What dataset properties predict that GBDTs will beat TFMs?: Dataset size and the number of high-cardinality categorical columns are the two meta-features that stand out as predictors of whether GBDTs outperform TFMs.The comparison uses the per-dataset advantage ∆ = eTFM − eGBDT and Spearman correlations with Holm correction across eight meta-feature tests.
  • E.5 Test 5: Which method should a practitioner pick, given their data regime?: TFMs lead on IID, tiny/small, and low-dimensional subsets; tuned ensembled RealMLP leads on temporal, grouped, medium, large, high-dimensional, and text subsets, while tuned CatBoost leads on high-cardinality categorical data.The grouped result is less decisive because only 5.1% of pairwise comparisons survive correction, despite RealMLP ranking first.
  • F Ablations: BeyondArena’s ablations test fewer outer splits, correct non-IID grouped splits, and alternative inner cross-validation for tiny datasets to assess benchmark validity.The ablation summary reports that fewer splits can preserve representative results, IID outer splits distort grouped-data results, and 5×5 versus 8-fold validation changes performance for all models.
  • F.1 (Outer Splits, A.1) Using Fewer Splits with BeyondArena-Core: Using fewer outer splits preserves aggregate rankings: the mean winrate delta stays below 1%, although individual dataset orderings become less stable as compute decreases.At Φ = 0, headline metrics recover the full speedup, which is 9x faster than Full, without meaningful fidelity loss.
  • F.2 (Outer Splits, A.2) IID Splits for non-IID Grouped Data: The grouped-data ablation compares grouped versus IID outer splits on musk and sat11_hand_algo_runtime using six representative models.The supplied passage states that Figure F.2 evaluates the split choice and that musk becomes trivial under IID splits, but does not provide the remaining numerical results.
  • F.3 (Inner Splits, B.1) Using 8-fold Cross-Validation for Tiny Data: For datasets with at most 500 training samples, the inner-split ablation compares 5×5 cross-validation with 8-fold cross-validation on selected IID tasks and models.It uses at most 100 features, only the first three outer folds, full tuning plus ensembling, and excludes TFMs because their validation protocol does not affect fitting.

F.4 (Inner Splits, B.2) Using IID Splits for non-IID Data … F.7 (Post-processing, D) Using Probability Calibration for Log-loss.

The ablations show that evaluation and preprocessing choices materially affect BeyondArena results: non-IID validation generally improves performance, grouped preprocessing helps on average, and SMS calibration benefits most methods while harming TabPFN-2.6 and RealMLP.

  • F.4 (Inner Splits, B.2) Using IID Splits for non-IID Data: For tiny datasets, 5 × 5 cross-validation consistently outperformed 8-fold cross-validation, while tuned RealMLP received a drastic boost.The comparison covered Linear, ExtraTrees, LightGBM, and RealMLP with tuning and post-hoc ensembling on datasets with fewer than 500 samples.
  • F.4 (Inner Splits, B.2) Using IID Splits for non-IID Data: Non-IID inner splits improved performance in almost all cases, often significantly, whereas IID validation distorted model rankings and grouped-dataset metrics.Grouped versus IID outer splits produced drastic changes in raw metrics and variances, with Kendall’s τ of 0.60 for musk and 0.49 for sat11_hand_algo_runtime.
  • F.5 (Grouped Data Preprocessing, C.1a/b) Disabling Preprocessing for Grouped Data.: The grouped-data preprocessing ablation evaluated default Linear, ExtraTrees, LightGBM, RealMLP, TabM, TabPFN-2.6, and TabICLv2 performance with preprocessing enabled versus disabled.The experiment was restricted to datasets with no more than 100k training samples and separated label-per-group from label-per-sample data.
  • F.5 (Grouped Data Preprocessing, C.1a/b) Disabling Preprocessing for Grouped Data.: Changing grouped-data preprocessing had model-dependent effects, but enabling it improved average performance across label-per-group and label-per-sample datasets.The ablation compared seven models on datasets with no more than 100k training samples.
  • F.6 (Text Data Preprocessing, C.2a/b) TF-IDF for Text Encoding.: The text-encoding ablation compared Qwen3-Embedding-8B with Matryoshka representation learning against TF-IDF with SVD using identical 32-dimensional output vectors.It evaluated Linear, ExtraTrees, and LightGBM across BeyondArena datasets containing text, using the first split and cheaper CPU models.
  • F.7 (Post-processing, D) Using Probability Calibration for Log-loss.: SMS probability calibration improved log-loss performance for most methods, with significant gains for tree-based models but worse results for TabPFN-2.6 and RealMLP.Calibration also produced significant average-rank gains for several models and changed the model ranking order.

G Per Dataset Results

This section reports per-dataset predictive performance across default, tuned, and tuned-plus-ensemble configurations, alongside hyperparameter-optimization trade-offs between metric error and cumulative training time.

  • Performance Per Dataset: Per-dataset tables compare average metric error and fold standard deviation for Default, Tuned, and Tuned + Ens. configurations, highlighting methods by statistical significance.Green marks the best average method, while bold marks methods not significantly worse than the best method.
  • HPO Pareto Trajectory: HPO Pareto trajectories show each method’s tuned and ensembled validation-error frontier as the trial budget increases, exposing compute-versus-performance trade-offs.Points farther down and left represent configurations that dominate others in validation error and training-time budget.

H NeurIPS Paper Checklist

The checklist states that the paper’s abstract and introduction reflect its contributions and scope, discusses limitations, and reports no theoretical results. It also documents reproducibility through experimental details, shared code and artifacts, compute information, and statistical error analyses.

  • Claims and scope: The abstract and introduction accurately state BeyondArena, DataFoundry, 142 curated datasets, diverse task and feature settings, and the reported model-performance scope.These claims include IID, temporal, and grouped prediction tasks across sample sizes, feature dimensionalities, and feature types.
  • Limitations: The paper discusses its limitations in Section 7.The checklist emphasizes that limitations and assumptions should be disclosed rather than omitted.
  • Theory assumptions and proofs: The paper presents no theoretical results.Accordingly, the checklist does not require theoretical assumptions or proofs for this work.
  • Experimental reproducibility: The experimental results are reported as reproducible through dataset-curation and experimental descriptions, appendix details, and shared code.The paper shares dataset-curation code, benchmarking code, and dataset artifacts.
  • Statistical significance and resources: All results include error bars, with additional statistical analysis provided in Appendix E.The paper also reports training and test details in Section 5 and compute-resource details in Section 5 and Appendix C.2.
Loading 2606.30410v1…