PaperlayerView exact revision on arXiv

Source-linked AI summary

TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks

Ivan Rubachev, Nikolay Kartashev, Yury Gorishniy, Artem Babenko

arXiv:2406.19380v4cs.LG

TL;DR

Existing tabular benchmarks underrepresent temporal drift and feature-rich industrial data, making transferability of recent tabular methods unclear. The paper introduces TabReD, eight industry-grade datasets with time-based evaluation, and reassesses many tabular methods. Simple MLP-like models and GBDT perform best, while complex methods transfer less effectively and split choice changes method rankings.

  • Problem

    Existing tabular benchmarks underrepresent temporal drift and extensive feature engineering found in industrial applications, limiting evaluation under these conditions.

  • Method

    The paper introduces TabReD, eight real-world industrial datasets with time-based splits and production-oriented feature engineering, and evaluates numerous tabular methods.

  • Results

    Simple MLP architectures with embeddings and GBDT show top performance on TabReD, while complex methods are less effective and time-based splits change method rankings relative to random splits.

  • Takeaways & Limitations

    Benchmark design and evaluation splits materially affect conclusions about tabular method performance in deployment-like settings.

  • Takeaways & Limitations

    TabReD is biased toward large, feature-rich, temporally drifting industrial applications, omits domains such as medicine and science, and has imprecise feature information.

Abstract

from arXiv · show

Advances in machine learning research drive progress in real-world applications. To ensure this progress, it is important to understand the potential pitfalls on the way from a novel method's success on academic benchmarks to its practical deployment. In this work, we analyze existing tabular benchmarks and find two common characteristics of tabular data in typical industrial applications that are underrepresented in the datasets usually used for evaluation in the literature. First, in real-world deployment scenarios, distribution of data often changes over time. To account for this distribution drift, time-based train/test splits should be used in evaluation. However, popular tabular datasets often lack timestamp metadata to enable such evaluation. Second, a considerable portion of datasets in production settings stem from extensive data acquisition and feature engineering pipelines. This can have an impact on the absolute and relative number of predictive, uninformative, and correlated features compared to academic datasets. In this work, we aim to understand how recent research advances in tabular deep learning transfer to these underrepresented conditions. To this end, we introduce TabReD -- a collection of eight industry-grade tabular datasets. We reassess a large number of tabular ML models and techniques on TabReD. We demonstrate that evaluation on time-based data splits leads to different methods ranking, compared to evaluation on random splits, which are common in current benchmarks. Furthermore, simple MLP-like architectures and GBDT show the best results on the TabReD datasets, while other methods are less effective in the new setting.

1 INTRODUCTION

Existing tabular benchmarks underrepresent temporally evolving, feature-rich industrial data, limiting evidence about whether recent tabular methods transfer to deployment-like conditions. TabReD addresses this gap with eight industry-grade datasets and shows that evaluation protocols and model rankings change in this setting.

  • Motivation: Industrial tabular applications commonly involve temporal shifts and extensive feature engineering, but these characteristics are underrepresented in academic benchmarks.Existing benchmarks also include leakage, synthetic or untraceable data, and non-tabular datasets.
  • TabReD benchmark: TabReD introduces eight industry-grade datasets spanning domains from finance to food delivery, with time-based evaluation and industrial feature-engineering pipelines.The benchmark increases coverage of deployment-like tabular use cases in academic evaluation.
  • Main findings: Time-based evaluation produces different method rankings and relative performance than commonly used random splits, with XGBoost's performance margin diminishing under correct evaluation.This demonstrates that split choice materially affects benchmark conclusions.
  • Main findings: GBDT and MLPs with embeddings achieve the best average performance on TabReD, while more complex deep-learning methods are less effective.The result concerns the feature-rich, time-evolving setting represented by TabReD.

2 RELATED WORK

Prior tabular benchmarks draw mainly from public repositories and often omit industrial temporal shift and extensive feature engineering. Related benchmarks address distribution shift or temporal shift, but they do not provide the same broad evaluation of recent tabular methods in this setting.

  • Existing benchmarks: Common tabular benchmarks rely heavily on UCI and OpenML repositories, while industrial conditions such as temporal shift and extensive feature engineering remain underrepresented.Kaggle provides another source of real-world problem datasets but is underused in many benchmarks.
  • Existing benchmarks: Tabzilla and the Grinsztajn et al. (2022) benchmark primarily use OpenML datasets and therefore do not represent common practical conditions involving temporal shift and extensive feature engineering.These benchmarks have been adopted for evaluating recent tabular methods.
  • Distribution-shift benchmarks: TableShift and WildTab evaluate distribution shifts, but focus on out-of-distribution robustness and domain generalization rather than broad comparison of tabular-data methods.They also consider more extreme shifts than the gradual shifts emphasized here.
  • Temporal-shift benchmarks: Wild-Time reports an average 20% performance drop under temporal shift, but tabular data was not its primary focus.Temporal shift has also been studied in recommender systems, finance, health insurance, and general ML practice.

3 A CLOSER LOOK AT THE EXISTING TABULAR ML BENCHMARKS

The authors audit 100 datasets from existing tabular benchmarks and identify leakage, synthetic or untraceable data, non-tabular datasets, limited feature richness, and inadequate support for time-based evaluation. These findings motivate more deployment-relevant benchmark construction.

  • Audit methodology: The audit analyzes dataset sizes, feature counts, temporal shift, split treatment, data sources, and notable quality issues across existing tabular benchmarks.Table 1 summarizes 100 unique classification and regression datasets, with additional metadata collected in the appendix.
  • Data quality issues: Existing benchmarks contain 11 datasets with leakage, 13 synthetic or untraceable datasets, and 25 datasets that are not inherently tabular.The identified problems arise from preparation errors, near-duplicates, inappropriate test splits, undocumented synthetic generation, or raw non-tabular data stored in tables.
  • Dataset size and feature engineering: Most existing datasets have fewer than 60 features and fewer than 100,000 instances, whereas industrial applications often use extensive feature engineering and richer representations.Public datasets frequently expose only high-level statistics rather than production-style engineered features.
  • Temporal evaluation: Among 53 issue-free datasets that may contain temporal drift, only 15 provide timestamps for time-based splits.Most benchmarks do not specify canonical evaluation splits beyond random splitting or cross-validation.

4 CONSTRUCTING THE TABRED BENCHMARK

TabReD is constructed from industrial-like tabular datasets selected for industry-relevant data collection and feature-engineering practices. The benchmark supports evaluation under time-based, feature-rich conditions but is not a complete replacement for existing benchmarks.

  • Dataset selection: TabReD selects inherently tabular datasets using feature collection and engineering practices intended to resemble industry settings.
  • Benchmark documentation: Table 2 summarizes the included datasets, while additional appendices provide dataset descriptions, feature-collinearity analysis, and Kaggle-selection annotations.
  • Benchmark documentation: The benchmark's large datasets use random subsets to make extensive hyperparameter tuning feasible, while parenthesized values denote full data size.
  • Benchmark role: TabReD adds underrepresented time-based splits and feature-rich representations, enabling industrial-like evaluation that can identify limitations of tabular models and techniques.
  • Limitations: TabReD is biased toward large, industry-relevant applications with extensive feature engineering and temporal drift, and omits medicine, science, and social data.

5 HOW DO TABULAR DL TECHNIQUES TRANSFER TO TABRED CONDITIONS?

TabReD evaluates tabular methods in temporally evolving, feature-rich industrial conditions using time-based splits and extensive baselines. GBDT and MLPs with numerical embeddings perform best overall, while several methods successful on prior benchmarks transfer less effectively.

  • 5.2 RESULTS: GBDT and MLP with embeddings (MLP-PLR) are the overall best models on TabReD, while ensembles consistently improve MLP and MLP-PLR.Numerical embeddings retain their utility in the new evaluation scenario.
  • 5.2 RESULTS: FT-Transformer is a runner-up but can train more slowly because attention complexity scales quadratically with the number of features.This efficiency issue is relevant because TabReD datasets contain more features than average literature datasets.
  • 5.2 RESULTS: SNN, DCNv2, ResNet, and Trompt are no better than the MLP baseline, while Trompt is slower than both MLP and FT-Transformer.Trompt’s promising results on an earlier benchmark do not consistently transfer to TabReD.
  • 5.3 COMPARISON WITH PRIOR BENCHMARKS: Improved training recipes and retrieval-based models underperform on TabReD despite consistently high performance on previously available datasets.Ensembling and numerical feature embeddings benefit both benchmark settings, unlike the other two techniques.
  • 5.3 COMPARISON WITH PRIOR BENCHMARKS: The authors hypothesize that feature complexity and gradual temporal shift may impair retrieval-based methods and long training recipes through neighbor-selection, augmentation, or memorization effects.These explanations are presented as hypotheses and motivate future research.
  • 5.4 INFLUENCE OF DATA VALIDATION SPLITS ON MODEL RANKING: Temporal evaluation changes model rankings, score spreads, absolute performance, relative differences, and standard deviations compared with random splits.XGBoost’s performance margin over MLPs decreases on temporally shifted test sets, while random splits may produce overly optimistic estimates.

6 FUTURE WORK

The authors identify temporal drift and feature complexity as possible reasons why effective tabular techniques fail to transfer to TabReD, and propose several directions for further study.

  • Future work should investigate continual learning, gradual temporal-shift mitigation, missing-data imputation, and feature selection using TabReD.
  • The authors posit that temporal data drift and complex features, including multicollinear or noisy features, may explain failed transfer.

7 CONCLUSION

The paper analyzes limitations in existing tabular benchmarks, introduces TabReD to reflect industrial conditions, and evaluates recent tabular deep-learning methods. Simple MLPs with embeddings, deep ensembles, and GBDT methods perform best, while more complex methods fail to transfer previous gains.

  • TabReD evaluation finds that MLPs with embeddings, deep ensembles, and GBDT methods such as XGBoost, CatBoost, and LightGBM work best.
  • The authors identify two underrepresented deployment conditions: temporal shift and extensive feature engineering in feature-rich datasets.
  • More complicated tabular deep-learning methods fail to transfer their improved performance from previous academic benchmarks.
  • TabReD is proposed as a more representative evaluation benchmark and testing ground for future tabular deep-learning methods.

REPRODUCIBILITY STATEMENT

The paper describes reproducibility resources and implementation details for its experiments.

  • Code covers dataset downloading, preprocessing, experiment reproduction, and plot generation through the provided preprocessing folder and README.Newly introduced datasets were unavailable for anonymity at submission time.

A.1 EXPLORATION OF TABRED DATASETS: SHIFTS

The appendix examines temporal shift and feature complexity in TabReD using prediction variability, model errors, correlations, and mutual information. The datasets show diverse temporal patterns, including worsening errors, seasonality, and apparently improving performance under shift, while TabReD is more feature-complex than comparison benchmarks.

  • Temporal shift: TabReD shift analysis compares MLP-ensemble prediction standard deviations with model errors over time, using prediction spread as a shift proxy.Figure 3 reports MAE for regression and error rate for binary classification.
  • Temporal shift: Some datasets show increasing shift alongside increasing error rates, notably Homesite Insurance and Sberbank Housing.
  • Temporal shift: Cooking Time exhibits strong seasonal behavior, with performance dropping significantly at specific times of day.
  • Temporal shift: Homecredit Default can show apparently improving performance over time under shift, because decreasing target variance or irreducible noise may mask its detrimental effects.
  • Feature correlations: Feature analysis uses linear correlations and mutual information with the target, and finds TabReD datasets more complex than most comparison datasets in feature count and correlation-importance patterns.Microsoft is the only comparably complex dataset among the comparison datasets shown.

A.3 DISTRIBUTION SHIFT ROBUSTNESS METHODS

The study adapts DeepCORAL and Deep Feature Reweighting to temporal distribution shifts, but neither improves on the MLP baseline on TabReD.

  • A.3 DISTRIBUTION SHIFT ROBUSTNESS METHODS: DeepCORAL is adapted to temporal shift by bucketing timestamps into domains, while DFR fine-tunes the MLP representation on later training instances.
  • A.3 DISTRIBUTION SHIFT ROBUSTNESS METHODS: Neither DFR nor DeepCORAL improves upon the MLP baseline on TabReD.The result is consistent with recent findings for other distribution shifts.

B TABRED DATASET DETAILS

TabReD provides industrially derived tabular datasets and documents their tasks, preprocessing, evaluation, and reproducibility procedures.

  • B TABRED DATASET DETAILS: TabReD datasets cover industrial tasks including insurance, e-commerce, credit default, housing, cooking time, delivery ETA, routing, and weather forecasting.
  • B TABRED DATASET DETAILS: The datasets are preprocessed for production ML use, with deterministic anonymization applied to some data while feature-engineering code and internal logs remain proprietary.
  • B TABRED DATASET DETAILS: Models use temporal train, validation, and test splits by default, with validation-based selection and test results aggregated across 15 random seeds.
  • B TABRED DATASET DETAILS: Missing numerical values are mean-imputed, and unseen categorical validation or test values receive a special unknown category.
  • B TABRED DATASET DETAILS: Most models receive 100 hyperparameter-optimization iterations, whereas FT-Transformer receives 25 because it is less efficient on datasets with hundreds of features.

C.3 ADDITIONAL IMPLEMENTATION DETAILS

The implementation uses official model sources where available, evaluates distribution-shift baselines with specified temporal procedures, and checks ranking significance statistically.

  • C.3 ADDITIONAL IMPLEMENTATION DETAILS: Model implementations come from official code repositories except Trompt, which is reproduced from its paper using its default hyperparameters.
  • C.3 ADDITIONAL IMPLEMENTATION DETAILS: The DFR baseline fine-tunes its last layer on the final 20% of datapoints, while CORAL defines nine timestamp-based domains.
  • C.3 ADDITIONAL IMPLEMENTATION DETAILS: Tamhane’s T2 testing does not significantly alter the model ranking relative to the simple testing procedure.
  • C.3 ADDITIONAL IMPLEMENTATION DETAILS: The supplementary repository includes annotated Kaggle competitions used during dataset sourcing, including annotations for non-tabular datasets.

E DETAILED ACADEMIC DATASETS OVERVIEW

The academic-dataset review catalogs split, leakage, traceability, and representation problems that limit how well existing benchmarks reflect industrial tabular learning.

  • E DETAILED ACADEMIC DATASETS OVERVIEW: Many datasets lack usable time splits despite temporal dependence, while some have no timestamps or make time-based evaluation impossible.
  • E DETAILED ACADEMIC DATASETS OVERVIEW: Existing datasets are often shallow relative to industrial data, such as California housing with eight features versus industrial housing data with hundreds.
  • E DETAILED ACADEMIC DATASETS OVERVIEW: Some datasets have task-specific limitations, including only six time-and-location features for UK air pollution and only five extracted features for phoneme audio.
  • E DETAILED ACADEMIC DATASETS OVERVIEW: Several benchmarks are synthetic, untraceable, non-tabular, or better suited to other representations such as graphs or image-derived features.
Loading 2406.19380v4…