Unreflected Use of Tabular Data Repositories Can Undermine Research Quality

1 INTRODUCTION

• Repositories increasingly shape tabular research as studies scale evaluations across many datasets, sometimes taking benchmark quality and appropriateness for granted.OpenML is extensively used, while recent studies have expanded from 196 to 300 datasets and foundation-model evaluations include as many benchmark datasets as possible.Source: §1 · pp. 1–2
• The paper promotes reflective repository use by highlighting three benchmarking issues and proposing repository guidance to prevent related mistakes.Its proposed direction is to make repositories institutions that support appropriate dataset use and empirical research quality.Source: §1 · p. 2
• Different datasets require distinct validation strategies for model selection, yet these strategies are often absent from studies and repository-provided evaluation designs.The paper identifies this as the first critical issue in tabular benchmarking.Source: §1 · p. 2
• Repositories do not provide strong baseline performance for each dataset, allowing state-of-the-art comparisons to fluctuate across studies.The paper identifies missing dataset-level baselines as a second critical benchmarking issue.Source: §1 · p. 2
• Results depend heavily on dataset features and preprocessing, but repositories offer users only limited preprocessing guidance.This creates a third risk: inappropriate preprocessing can materially affect tabular evaluations.Source: §1 · p. 2
• Researchers often use repository datasets and evaluation setups without critically inspecting the data or task, making repository design influential for evaluation quality.The paper links this unreflective use to limited awareness of misuse consequences and repository design choices.Source: §1 · p. 2

2.1 Inappropriate validation and model selection strategy per-dataset

• 5-fold cross-validation is almost always superior to holdout selection on TabZilla, often producing strong gains, although its effect is stronger on small datasets and some large datasets.One dataset was excluded because holdout already produced near-perfect results, and four large datasets because 100 trials with 5-fold cross-validation were infeasible.Source: Figure 1 · p. 3, §2.1 · p. 4
• Holdout validation can select models that miss stronger hyperparameters and reverse model comparisons, making MLPs appear worse than XGBoost on colic and XGBoost worse than MLPs on profb.The paper cautions that 5-fold cross-validation does not eliminate validation overfitting and that other strategies may suit some small datasets better.Source: §2.1 · p. 4
• Repositories could reduce these problems by defining default validation procedures, sharing strong baselines in task or dataset schemas, and improving metadata for task-relevant preprocessing.These defaults should remain flexible because different research questions may require distinct validation strategies.Source: §2.1 · p. 5, §2.2 · p. 7, §2.3 · p. 8
• Better-tuned MLPs outperform all results from the TabZilla benchmark when evaluated with 100 HPO trials and 5-fold cross-validation, without changing the models themselves.The original evaluation underestimated both MLP and XGBoost performance, with MLPs affected more strongly.Source: §2.2 · p. 6

2.2 Obscured state-of-the-art due to missing objective baseline

• Restricted evaluation design made TabZilla’s conclusions about model differences unreliable, because correlations between dataset meta-features and neural-network versus tree-model performance did not persist under improved evaluation.Table 2 compares the original analyses with replications and results using better-tuned MLPs and XGBoost; no strong patterns persist after the advanced design.Source: §2.2 · p. 6
• Without strong, objective per-dataset baselines, studies can mistake improvements over underestimated benchmark results for modeling advances, even when no novel modeling contribution is introduced.The paper argues that systematic tracking of actual state-of-the-art performance is missing, despite reuse of longstanding OpenML tasks.Source: §2.2 · p. 6

2.3 Insufficient information for task-relevant preprocessing

• Repository datasets can contain predictive features unavailable at inference time, misleading version histories, and incomplete processing information, causing target leakage and inappropriate preprocessing.The authors propose separating non-predictive features from inputs and outputs and improving dataset metadata and processing-history information.Source: §2.3 · pp. 7–8

Vision

• Repositories can serve as intermediaries between dataset creators and users, even though they are not ultimately responsible for problems in research papers.The proposal frames repository improvements as support for more appropriate dataset use.Source: §3 · p. 8

Suggested improvement

• Simple preprocessing choices can entirely change model comparisons across classification and regression datasets.Examples include recoding ordinal features, adding date differences or feature fractions, and performing feature selection.Source: Figure 4 · p. 9
• Repositories should associate raw datasets with recommended default tasks covering target, preprocessing, estimation and validation protocols, metrics, post-processing, and strong baselines.These tasks would provide references rather than mandatory evaluation setups, allowing users to define alternatives for specific research questions.Source: §3 · p. 9

Benefits

• A default task could reduce user mistakes, support benchmark curation, and make improvements over relevant dataset-specific baselines explicit.It could also provide reference points for addressing issues such as data leakage.Source: §3 · p. 9

Who should define default tasks?

• The proposed starting point is to adapt evaluation setups from an established curated benchmark, then incorporate findings from subsequent research and community-maintained baselines.Datasets with approved default tasks could receive a separate “verified task” tag.Source: §3 · p. 9

Related work

• Recent work criticizes tabular benchmarks for outdated data, missing task-specific preprocessing, and inappropriate data splits.These concerns motivate broader skepticism about data use in tabular research.Source: §4 · p. 10
• Prior literature identifies model selection as important, and this paper highlights that holdout selection underestimates model performance on TabZilla.The paper connects this issue to concerns about external validity in existing studies.Source: §4 · p. 10
• The paper follows broader critiques of evaluation practices and supports recommendations that dataset creators restrict inappropriate applications.Its repository recommendations are presented as part of that critical research-quality agenda.Source: §4 · p. 10

Limitations

• The analysis is limited to supervised i.i.d. tabular learning, predictive state-of-the-art goals, OpenML, and three issues among many possible causes of reduced research quality.The authors also do not provide off-the-shelf solutions because they view the problems as structural.Source: §5 · p. 10

Conclusion

• The paper concludes that unreflected repository use can compromise prior studies’ conclusions and calls for dataset-specific model selection, strong baselines, preprocessing guidance, and expert-reviewed default tasks.The proposed tasks are intended to help researchers avoid pitfalls and repositories prevent them at larger scale.Source: §5 · p. 10

Software and hardware

• Deep learning models and XGBoost ran on NVIDIA H100, RTX A6000, A5000, or A40 GPUs, while LightGBM ran on an Intel Xeon E2640v2 CPU.Source: §A.1 · p. 14

• TabZilla-hard contains 36 classification datasets with benchmark-provided train/validation/test splits and generally no preprocessing beyond model requirements.Source: §A.2 · p. 14
• The Grinsztajn benchmark is reconstructed from 45 datasets, yielding 28 regression and 17 binary-classification tasks after selecting less-preprocessed variants.Source: §A.2 · pp. 14–15
• Binary features are treated as categorical in the original benchmark, but the study avoids using it to evaluate numeric and categorical models because few datasets retain substantial categorical structure.Source: §A.2 · p. 15
• Grinsztajn-style splits use 70% training data, with the remaining data divided into 30% validation and 70% testing, plus sample caps matching the original benchmark.Source: §A.2 · p. 15
• Deep-learning preprocessing normalizes regression targets, imputes numeric missing values with means, and applies feature normalization, while tree models mainly receive correct categorical datatypes.Source: §A.2 · p. 15

• Classification models are trained with cross entropy and regression models with mean squared error, while AdamW is used for deep-learning training.Source: §A.3 · p. 15
• Hyperparameter configurations are specified separately for XGBoost and LightGBM, with LightGBM’s possible leaf counts constrained when maximum depth is set.Source: Table 4 · p. 16, Table 3 · p. 16

• Model selection and hyperparameter optimization use logloss for classification and mean squared error for regression, making these the study’s main optimization metrics.Source: §A.4 · p. 16
• The paper reports logloss and RMSE as its main metrics throughout the evaluation.Source: §A.4 · p. 16
• Figures 3 and 4 additionally use AUC and R2 because these metrics make near-perfect performance easier to interpret.Source: §A.4 · p. 16
• The MLP and TabM configurations are documented as separate model-specific hyperparameter setups.Source: Table 5 · p. 17, Table 6 · p. 17

• Four TabZilla datasets were excluded because 5-fold cross-validation was infeasible under their training times, while one dataset was excluded after all models generalized perfectly with holdout selection.Source: §A.5 · p. 17
• The examples in Figure 2 were hand-picked to illustrate model-selection effects, so alternative examples could have produced a different local narrative.Source: §A.5 · p. 17
• For Table 1, the study selects each prior model’s best logloss from uploaded full hyperparameter-search results and aggregates those results with its own models.Source: §A.5 · p. 18
• For Table 2, the analysis reproduces prior results and then replaces LightGBM and MLP results with 100-trial holdout and 5-fold-cross-validation setups.Source: §A.5 · p. 18
• The Grinsztajn benchmark analysis corrects leakage by removing target-derived inputs, alternative targets, and other erroneous features from datasets such as NYC taxi, sulfur, and visualizing soil.Source: §A.5 · p. 18
• Other figure analyses change feature representations or selection, including categorical treatment of dates, datetime differences, feature ratios, and selecting 200 important features for guillermo.Source: §A.5 · pp. 18–19

• The road-safety prediction task is considered ethically dubious and should be discontinued as a default community benchmark.Source: B FULL RESULTS · p. 19
• Table 7 reports logloss performance on the TabZilla-hard benchmark using significance-adjusted comparisons against the best model.Source: Table 7 · p. 21
• Table 8 reports R2 performance on the Grinsztajn benchmark with Wilcoxon signed-rank tests and Holm-Bonferroni correction.Source: Table 8 · p. 22
• Table 9 reports R2 for seattlecrime6 and nyc-taxi-green-dec-2016 and AUC for the remaining datasets where preprocessing was helpful.Source: Table 9 · p. 22