PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiTab: A Comprehensive Benchmark Suite for Multi-Dimensional Evaluation in Tabular Domains

Kyungeun Lee, Moonjung Eo, Hye-Seung Cho, Dongmin Kim, Ye Seul Sim, Seoyoon Kim, Min-Kook Suh, Woohyung Lim

arXiv:2505.14312v1cs.LGcs.AI

TL;DR

Average-case tabular benchmarks provide limited evidence about how model behavior changes across dataset regimes. MULTITAB evaluates 13 models on 196 datasets grouped by interpretable data characteristics, finding regime-dependent performance patterns that support condition-aware evaluation and model selection.

  • Problem

    Average-case benchmarks obscure how tabular model performance varies with dataset characteristics and provide limited insight into interactions between modeling assumptions and data regimes.

  • Method

    MULTITAB groups 196 public datasets by quantitative characteristics and evaluates 13 models with diverse inductive biases using standardized cross-validation and optimization.

  • Results

    No single model dominates: GBDTs excel in small or categorical-heavy settings, while modern neural networks are competitive on larger, more regular datasets.

  • Takeaways & Limitations

    Condition-aware model selection and reporting performance across data regimes are supported over relying only on global rankings.

  • Takeaways & Limitations

    The study is limited to supervised settings, with future work suggested for pretraining-based or AutoML-driven scenarios.

Abstract

from arXiv · show

Despite the widespread use of tabular data in real-world applications, most benchmarks rely on average-case metrics, which fail to reveal how model behavior varies across diverse data regimes. To address this, we propose MultiTab, a benchmark suite and evaluation framework for multi-dimensional, data-aware analysis of tabular learning algorithms. Rather than comparing models only in aggregate, MultiTab categorizes 196 publicly available datasets along key data characteristics, including sample size, label imbalance, and feature interaction, and evaluates 13 representative models spanning a range of inductive biases. Our analysis shows that model performance is highly sensitive to such regimes: for example, models using sample-level similarity excel on datasets with large sample sizes or high inter-feature correlation, while models encoding inter-feature dependencies perform best with weakly correlated features. These findings reveal that inductive biases do not always behave as intended, and that regime-aware evaluation is essential for understanding and improving model behavior. MultiTab enables more principled model design and offers practical guidance for selecting models tailored to specific data characteristics. All datasets, code, and optimization logs are publicly available at https://huggingface.co/datasets/LGAI-DILab/Multitab.

1 Introduction

MULTITAB argues that average-case tabular benchmarks obscure how model performance depends on dataset regimes. It introduces a structured evaluation framework to expose these conditional patterns across diverse datasets and inductive biases.

  • Average-case rankings obscure when and why tabular models succeed or fail across differing dataset characteristics.Global comparisons provide limited insight into interactions between model design and sample size, feature types, and class imbalance.
  • MULTITAB evaluates 196 public datasets and 13 representative models across regimes defined by sample size, imbalance, feature-to-sample ratio, and feature interaction.Models are trained with consistent cross-validation and extensive hyperparameter optimization.
  • Model performance varies substantially by regime: sample-level methods excel with large samples or numerical-dominated data, feature-dependency methods with weakly correlated features, and trees on regression or low-imbalance tasks.These conditional patterns are not visible in average-case evaluation.
  • MULTITAB functions as a diagnostic tool for identifying which modeling assumptions succeed or fail under specific data conditions.The benchmark is intended to support model selection and future architecture design.

2 Previous Tabular Benchmarks

Prior tabular benchmarks generally compare broad model families using average-case or rank-based metrics. MULTITAB addresses the resulting gaps by organizing datasets and models for conditional analysis of architectural assumptions.

  • Earlier evaluations commonly emphasize average metrics, narrow task coverage, or rank summaries that compress complex model behavior into single numbers.These designs provide limited conditional insight, especially beyond classification.
  • Existing benchmarks often compare broad model classes rather than isolating how specific architectural components behave across data regimes.This limits understanding of the mechanisms behind model performance.
  • MULTITAB organizes diverse datasets and models along axes such as sample size, label imbalance, and feature interaction for hypothesis-driven conditional evaluation.The framework also expands regression coverage and architectural analysis.

3 MULTITAB: Tabular Benchmark Suite for Multi-Dimensional Evaluation

MULTITAB constructs a multi-dimensional benchmark by grouping diverse tabular datasets into interpretable regimes and evaluating representative models under standardized procedures. Its normalized metric and regime definitions support stable comparisons of inductive biases across heterogeneous tasks.

  • Benchmark construction: MULTITAB comprises 196 public datasets with over 10 million samples and 13 models spanning diverse architectural assumptions.Datasets are selected for public availability, metadata quality, task diversity, and varied sizes and dimensionalities.
  • Evaluation protocol: The evaluation applies standardized stratified cross-validation, independent fold-level hyperparameter optimization, and equal optimization budgets across algorithms.Datasets below 50,000 samples use 10-fold cross-validation, while larger datasets use 3-fold cross-validation.
  • Evaluation metrics: Normalized predictive error rescales each model between the best and worst errors within each dataset-split pair, producing scores from 0 to 1.Classification uses log loss and regression uses RMSE, enabling relative comparisons across datasets with different scales and difficulties.
  • Sub-category construction: Datasets are partitioned into interpretable regimes using seven core axes, including task type, sample size, feature heterogeneity, and feature-to-sample ratio.The axes are based on quantitative dataset statistics and capture properties that influence learning and generalization.
  • Sub-category construction: Grouping datasets into discrete regimes and using complementary axis definitions enables reproducible, hypothesis-driven comparisons less sensitive to noise.Figure 1 reports average normalized error across 24 sub-categories for four model classes, with 95% confidence intervals.

4 Empirical Findings on MultiTab Benchmark

MultiTab shows that tabular model performance varies substantially across data regimes, with no universally reliable architecture. Sub-category analyses reveal how sample size, feature composition, feature-to-sample ratio, label imbalance, and feature correlation favor different inductive biases.

  • ModernNCA achieves the best average performance overall, while XGBoost, FT-Transformer, T2G-Former, and TabR remain competitive within the confidence interval.
  • Only ModernNCA consistently ranks among the top-performing models across binary classification, multiclass classification, and regression.Its metric-learning latent space reflects target similarity, supporting smooth generalization across discrete and continuous outputs.
  • FT-Transformer performs best in small-sample regimes, whereas XGBoost leads large-sample regimes while several modern neural models remain statistically comparable.These results indicate that neural models can remain competitive without large datasets and have narrowed previously reported gaps as sample size grows.
  • Categorical structure changes model rankings: ModernNCA favors few categorical features, while NN-Feature and GBDTs remain competitive when categorical features dominate.Embedding modules help MLP-C over MLP when categorical features dominate, with normalized errors of 0.290 versus 0.397, but offer little difference when categorical features are scarce.
  • NN-Feature is robust across feature-to-sample ratios, whereas NN-Sample degrades when features greatly outnumber samples because inter-sample similarity becomes unreliable.Attention over inter-feature dependencies is more robust to data scarcity than neighborhood-based modeling.
  • CatBoost retains an advantage under highly skewed regression targets, while most neural models degrade as skew increases.CatBoost and ModernNCA have similar low-skew errors, 0.250 versus 0.255, but diverge under high skew, 0.245 versus 0.302.
  • NN-Sample performs best with highly correlated features, whereas NN-Feature excels under low correlation; ModernNCA also degrades under categorical-feature dominance.Redundant correlated features can improve neighborhood consistency, while attention over features benefits from diverse interactions.

5 Conclusion

MULTITAB moves tabular evaluation beyond average-case rankings by relating model performance to dataset regimes. Its findings support condition-aware model selection while noting supervised-only scope and broader responsible-use concerns.

  • No single model dominates: GBDTs excel in small or categorical-heavy settings, while modern neural networks are competitive on larger, more regular datasets.
  • MULTITAB supports selecting models according to data regimes rather than relying on global rankings.
  • Architectural components such as attention and metric learning provide benefits only under specific dataset conditions.
  • The benchmark covers supervised settings, with future extensions suggested for pretraining-based or AutoML-driven scenarios.
  • Better benchmark performance does not by itself guarantee equitable or trustworthy systems, requiring attention to fairness, privacy, interpretability, accountability, and human oversight.

B Detailed Description of the Benchmark Suite

The benchmark suite contains 196 publicly accessible datasets spanning domains, task types, sample sizes, and feature dimensionalities. Standardized preprocessing and dataset filtering support consistent model evaluation.

  • The suite includes 196 datasets from OpenML or scikit-learn, distributed under the CC-BY license with public benchmark and reproduction resources.
  • Datasets cover medical, financial, environmental, social, and scientific domains, with balanced binary, multiclass, and regression tasks.
  • Figure 4 maps datasets by logarithmic sample size and feature dimensionality, with colors distinguishing task types.
  • Columns exceeding 50% missingness are dropped, remaining incomplete rows are removed, and no imputation is applied.
  • Feature types follow OpenML metadata, while mislabeled string features and categorical features exceeding 1,000 unique values are excluded.
  • Numerical features are transformed with scikit-learn’s QuantileTransformer to stabilize optimization and reduce outlier effects.
  • Categorical features are label-encoded before algorithms apply model-specific handling, including ordered encoding in CatBoost and learnable embeddings in neural networks.

B.2 Training Protocols: Cross-validation

MULTITAB uses stratified cross-validation, normalized predictive error, and standardized metric choices to compare models consistently across datasets with different scales and difficulties.

  • Each dataset-split pair uses an 8:1:1 train-validation-test partition within stratified cross-validation.
  • Datasets with fewer than 50,000 samples use 10 folds, whereas larger datasets use 3 folds to reduce computational cost.
  • Each fold selects hyperparameters on validation data, retrains on the training portion, and averages predictive error across folds.
  • Normalized error scores each model relative to the best and worst models on the same dataset-split pair, with 0 best and 1 worst.
  • Log loss is the default classification metric because it remains informative under imbalance, while RMSE is used for regression.
  • Relative scoring avoids penalizing models for intrinsic dataset difficulty and mitigates scale mismatches across datasets.
  • Accuracy, AUROC, and raw rank are reported as secondary metrics but are excluded from the main analysis.

C.1 Task types

The benchmark organizes datasets by task type and data characteristics, including categorical composition, cardinality, feature-to-sample ratio, and label imbalance. These axes expose how dataset structure can shape model behavior.

  • C.1 Task types: The benchmark separates 196 datasets into binary classification, multiclass classification, and regression categories based on OpenML metadata.
  • Feature composition: Feature heterogeneity is analyzed through categorical-feature proportion and average categorical cardinality.
  • Feature-to-sample ratio: The feature-to-sample ratio measures dimensionality relative to sample availability and is tied to generalization, overfitting risk, and the curse of dimensionality.
  • Label imbalance: Label imbalance is evaluated with entropy ratio for classification, skewness for regression, and imbalance factor for both settings.

D.5 NN-Sample: TabR, ModernNCA

NN-Sample models capture relationships among dataset samples through retrieval, metric learning, or inter-sample attention. TabR and ModernNCA rely on sample-level representations, while SAINT combines sample- and feature-level dependencies.

  • TabR: TabR augments a feed-forward backbone with a learnable retrieval set whose prototypes provide contextual signals for prediction.Each sample queries retrieval vectors, and retrieval entries query the sample, creating bidirectional interactions.
  • Evaluation setup: The benchmark uses TALENT implementations or author implementations with tuning ranges drawn from prior work for these models.TabR and ModernNCA use TALENT resources, while SAINT uses the original authors’ implementation.
  • ModernNCA: ModernNCA embeds samples so similar classes or target values occupy nearby latent positions, using a deep neural network inspired by NCA.Its inductive bias assumes a smooth, clusterable latent geometry.
  • ModernNCA: ModernNCA can be effective under class imbalance, weak supervision, or noisy labels, but may degrade when latent distances fail to represent target structure.The contrastive objective is intended to stabilize learning in these settings.
  • SAINT: SAINT jointly models feature interactions with column-wise attention and sample relationships with row-wise attention.This richer context modeling is useful when dependencies span both columns and rows, but inter-sample attention increases complexity and can destabilize training.

E.1 Results with secondary metrics

Secondary evaluations using accuracy, RMSE, log loss, and average rank largely reproduce the normalized-error trends. Across top algorithms, ModernNCA is strong under imbalance and inter-sample structure, FT-Transformer under high-cardinality and high-dimensional conditions, and XGBoost remains broadly robust.

  • Results with secondary metrics: Secondary metrics largely align with normalized-error results and reinforce the comparative strengths of the evaluated architectural paradigms.Accuracy and RMSE results are aggregated across splits and dataset sub-categories, while average rank summarizes performance ordinally.
  • Metric views: The figures report lower-is-better normalized error and average-rank summaries across model families and dataset sub-categories.Figure 6 uses 1 - accuracy for classification, while Figure 7 uses raw log loss and RMSE.
  • Top-performing algorithms: ModernNCA performs strongly under class imbalance and inter-sample-structured data, whereas FT-Transformer is strong in high-cardinality and high-dimensional regimes.The comparison covers XGBoost, FT-Transformer, T2G-Former, and ModernNCA across dataset sub-categories.
  • Top-performing algorithms: XGBoost remains a robust baseline across nearly all dataset settings.The algorithm is included alongside neural models representing feature- and sample-aware architectural paradigms.

E.3 Correlation Between Dataset Statistics and Model Performance

Spearman analysis relates dataset statistics to model error and complements subgroup evaluation. It indicates that tree models benefit from larger samples, while several neural models are more sensitive to imbalance, irregularity, and feature entanglement.

  • Scale-related patterns: GBDTs, particularly CatBoost and XGBoost, show strong negative correlations between sample size and predictive error.Their positive correlations with the feature-to-sample ratio also reflect effectiveness when split decisions are statistically reliable.
  • Sensitivity to label imbalance and structural irregularity: Several neural models show higher error as entropy ratio, function irregularity, and inter-feature correlation increase.The pattern includes MLP variants, ResNet, FT-Transformer, and T2G-Former, indicating sensitivity to class skew, irregular label spaces, or entangled features.
  • Complementary role of correlation analysis: Correlation analysis complements subgroup evaluation but is sensitive to outliers, collinearity, and metric design.Its trends align with some subgroup findings, so the paper uses it as an additional rather than standalone perspective.

E.4 Additional Analysis on Comparison with TabPFN

Within the 42 datasets eligible for both models, TabPFN beats ModernNCA more often, but its gains are usually small and its applicability is constrained. TabPFN’s relative advantage is associated with lower entropy ratio, while ModernNCA often achieves larger improvements when it wins.

  • Applicability constraints: TabPFN supports only classification datasets with at most 3,000 training samples, 1,000 features, and 10 classes, leaving 42 of 196 benchmark datasets eligible.The comparison therefore covers only a restricted portion of the benchmark.
  • Model comparison: TabPFN outperforms ModernNCA on 29 of 42 eligible datasets, compared with 13 wins for ModernNCA.The relative-error comparison uses TabPFN error minus ModernNCA error, with negative values indicating TabPFN wins; TabPFN’s margin is small in most cases.
  • Interpretation: TabPFN is competitive but not consistently superior, while ModernNCA often achieves larger improvements when it outperforms TabPFN.The findings motivate more generalizable pretrained tabular models beyond TabPFN’s limited operational domain.
  • Dataset characteristics: Datasets where TabPFN wins have a lower average entropy ratio than datasets where ModernNCA wins: 0.550 versus 0.877, with p < 0.05.No significant differences were observed for the other reported dataset characteristics.

E.5 Discussion on Computational Costs

MULTITAB compares computational costs across model classes and shows that training time, tuning time, and trial counts vary substantially by architecture. Hyperparameter analysis further indicates that optimization controls often matter more than structural choices.

  • Training time per model: SAINT (450.0s), T2G-Former (317.0s), and FT-Transformer (208.7s) have substantially higher per-trial training costs than CatBoost (23.3s) and LightGBM (49.5s).TabR (71.2s) and ModernNCA (79.6s) remain only moderately slower than classical and GBDT baselines under the standardized setting.
  • Time to best configuration: XGBoost (1887.5s) and LightGBM (208.4s) show a clear disparity in cumulative time to reach their best hyperparameter configuration under a 100-trial budget.The supplied passage establishes the comparison but does not provide the full range of values for all models.
  • Trials to best configuration: Most models require 50–55 trials to reach their best configuration, including MLP-C (53.6), FT-Transformer (53.8), ResNet (53.0), LightGBM (52.9), and CatBoost (49.1).The similar trial counts persist despite differences in architectural complexity and data regimes.
  • Most influential hyperparameters: RandomForest is dominated by min_samples_leaf at 87.9% top-1 frequency, whereas GBDTs distribute importance across tree-structure and gradient-related controls.No single parameter dominates across CatBoost, XGBoost, and LightGBM, reflecting stronger interactions among their tuning knobs.
  • Most influential hyperparameters: MLP variants prioritize normalization and depth, which together exceed 50% of top-1 frequency in most models.These parameters are associated with representational capacity and training stability.
  • Most influential hyperparameters: Optimizer and learning_rate dominate NN-Feature models, while lr and frequency_scale dominate NN-Sample models over structural depth or width parameters.SAINT similarly prioritizes learning_rate (57.6%) and optimizer (29.8%), with structural hyperparameters contributing minimally.

F Limitations of MULTITAB

MULTITAB’s conclusions are bounded by finite dataset coverage, heuristic regime definitions, incomplete coverage of relevant data characteristics, and limited algorithm and tuning scope. The authors identify broader datasets, theory-grounded axes, and expanded models or protocols as future priorities.

  • Scope limitations: Findings come from 196 datasets and may not generalize to all tabular datasets.The authors characterize this as finite dataset coverage despite efforts to ensure domain and property diversity.
  • Scope limitations: Dataset sub-categories use empirical histogram thresholds that may not partition regimes along theoretically grounded axes.These thresholds capture broad trends but remain heuristic.
  • Scope limitations: The benchmark covers eleven metrics across seven axes but omits factors such as temporal drift, feature sparsity, label noise, and task-specific priors.These factors may affect real-world tabular performance but are not explicitly captured.
  • Scope limitations: The 13-model selection, hyperparameter spaces, and tuning budgets do not exhaust the tabular algorithm ecosystem and may influence performance in model-specific ways.The authors propose adding models and evaluation protocols in future work.
Loading 2505.14312v1…