PaperlayerView exact revision on arXiv

Source-linked AI summary

Well-tuned Simple Nets Excel on Tabular Datasets

Arlind Kadra, Marius Lindauer, Frank Hutter, Josif Grabocka

arXiv:2106.11189v2cs.LG

TL;DR

Deep learning has not consistently surpassed traditional methods on important tabular datasets. This paper searches dataset-specific cocktails of 13 regularization techniques and their hyperparameters for plain MLPs, finding that these models surpass recent neural architectures and GBDT, including XGBoost, in large-scale experiments.

  • Problem

    Deep learning remains understudied on tabular data, where thoroughly tuned recent neural networks do not outperform GBDT across 40 datasets.

  • Method

    The paper jointly searches which of 13 regularizers to apply and their subsidiary hyperparameters for each MLP and dataset.

  • Results

    Regularization cocktails substantially and significantly improve simple neural networks, enabling them to outperform recent neural architectures and GBDT on tabular datasets.

  • Takeaways & Limitations

    Even simple neural networks can achieve competitive tabular-classification accuracy when dataset-specific regularization cocktails are found through standard hyperparameter optimization.

  • Takeaways & Limitations

    The evaluation covers classification on somewhat balanced datasets, not regression or several other tabular-data regimes and modalities.

Abstract

from arXiv · show

Tabular datasets are the last "unconquered castle" for deep learning, with traditional ML methods like Gradient-Boosted Decision Trees still performing strongly even against recent specialized neural architectures. In this paper, we hypothesize that the key to boosting the performance of neural networks lies in rethinking the joint and simultaneous application of a large set of modern regularization techniques. As a result, we propose regularizing plain Multilayer Perceptron (MLP) networks by searching for the optimal combination/cocktail of 13 regularization techniques for each dataset using a joint optimization over the decision on which regularizers to apply and their subsidiary hyperparameters. We empirically assess the impact of these regularization cocktails for MLPs in a large-scale empirical study comprising 40 tabular datasets and demonstrate that (i) well-regularized plain MLPs significantly outperform recent state-of-the-art specialized neural network architectures, and (ii) they even outperform strong traditional ML methods, such as XGBoost.

1 Introduction

Tabular data remains a challenging and important domain for deep learning, where thoroughly tuned GBDT and recent neural networks have produced mixed results. The paper argues that jointly optimized regularization cocktails can make plain MLPs highly competitive.

  • Tabular data is important across applications but has not shared deep learning’s success on raw data such as images, speech, and text.
  • Thorough tuning on 40 datasets shows recent neural networks do not outperform GBDT on tabular data.
  • The paper hypothesizes that modern regularization and learning-dynamics techniques can improve neural-network performance on tabular data.
  • Plain MLPs achieve state-of-the-art results when multiple modern regularization techniques are applied jointly and simultaneously.
  • The proposed method jointly searches which of 13 regularizers to apply and their subsidiary hyperparameters for each dataset.
  • Well-regularized neural networks surpass recent neural architectures and GBDT in tabular datasets, with pronounced and highly significant improvements.

2 Related Work on Deep Learning for Tabular Data

Recent neural architectures for tabular data use mechanisms including attention, hierarchical decisions, self-normalization, and learned regularization to improve neural-network performance.

  • Tabular-data architectures include TabNet’s sequential attention, NODE’s hierarchical decisions, and self-normalizing networks’ activation normalization.

3 An Overview of Regularization Methods for Deep Learning

Deep-learning regularization methods span parameter penalties, data augmentation, ensembles, structural changes, and implicit training effects. These techniques target overfitting and generalization through different mechanisms.

  • Weight decay: Weight decay regularizes models by penalizing parameter norms through L1, L2, or Elastic Net formulations.
  • Data Augmentation: Data augmentation regularizers create altered or mixed training examples, including masked inputs, interpolated pairs, and augmentation chains.
  • Ensemble methods: Ensemble methods reduce variance and act as regularizers, with neural examples including Dropout, Mix-Out, and snapshot ensembles.
  • Structural and Linearization: Structural and linearization methods regularize deep models using skip connections, parallel blocks, and aggregated representations.
  • Implicit: Implicit regularizers such as Early Stopping and Batch Normalization improve generalization through their training behavior.

4 Regularization Cocktails for Multilayer Perceptrons

The paper formulates regularization-cocktail selection as validation-based hyperparameter optimization for MLPs. It jointly tunes regularizer strengths and applicability, then refits the selected configuration on all training data.

  • Problem setup: A neural network approximates targets as ŷ = f(X; θ), with parameters trained by minimizing a differentiable loss.
  • Problem setup: Regularization controls the model parameters to avoid overfitting, while hyperparameters are tuned using a validation split.
  • Problem setup: After selecting a well-performing configuration, the model parameters are refit on the combined training and validation datasets.
  • Cocktail formulation: The method treats the optimal regularizer as a dataset-specific cocktail of many simultaneously applied methods with different strengths.
  • Cocktail search space: The search includes 13 regularization methods and 19 hyperparameters, while allowing each regularizer to be applied or skipped.
  • Cocktail search space: BOHB is used for multi-fidelity Bayesian optimization because it handles categorical activation choices and conditional search structures.

5 Experimental Protocol

The study evaluates regularization cocktails for a fixed simple MLP across 40 diverse tabular datasets, comparing them with neural and traditional baselines under shared resource constraints. It measures balanced accuracy and examines performance and anytime efficiency against GBDT methods, including XGBoost.

  • Regularization setup: The cocktail configuration space covers 13 regularization methods, with explicit hyperparameters and conditional constraints controlling whether methods are enabled.Table 1 lists techniques including Batch Normalization, Dropout, weight decay, ensembles, data augmentation, and adversarial learning.
  • Datasets: 40 tabular datasets span 452–416 188 instances and 4–2 001 features, with 60%/20%/20% train-validation-test splits.The collection includes numerical and categorical features and combines OpenML, UCI, and Kaggle datasets.
  • Optimization budget: BOHB searches cocktail configurations for up to 4 days or 40 × D configurations, using ten parallel workers with constrained CPU and memory allocations.Each worker receives 2 CPU cores and 12GB of memory, and configurations train for 105 epochs.
  • Model and training: A fixed 9-layer, 512-unit-per-layer MLP isolates the effects of regularization, while core training hyperparameters remain fixed across experiments.The setup uses AdamW, cosine annealing with restarts, and a learning rate of 10^-3.
  • Baselines: The evaluation compares cocktails with Node, TabNet, self-normalizing networks, AutoGluon neural variants, unregularized or Dropout MLPs, and three GBDT implementations.The GBDT baselines are scikit-learn optimized by Auto-sklearn, XGBoost, and CatBoost.
  • Evaluation: All baselines use identical data splits, seeds, HPO resources, and constraints, then are refitted on training plus validation data before test evaluation.The primary reported metric is test-set balanced accuracy over the 40-dataset collection.

6 Experimental Results

Across 40 tabular datasets, dataset-specific regularization cocktails for plain MLPs substantially outperform specialized neural architectures and GBDT baselines, with statistically significant advantages and strong anytime performance.

  • Benchmark comparisons: MLP+C is compared with ten baselines, including specialized neural architectures, three GBDT implementations, and reference MLPs.The baselines include NODE, TabNet, AutoGluon variants, XGBoost, auto-sklearn, CatBoost, and unregularized or singly regularized MLPs.
  • Benchmark comparisons: Across 40 datasets, the test-set metric is balanced accuracy, and Figure 1 shows substantial improvements for MLP+C.The reported baseline hyperparameters were carefully tuned on validation sets, except for the unregularized MLP and AutoGluon stacking variant.
  • Neural-network comparison: Wilcoxon critical-difference analysis shows that regularization cocktails significantly outperform the other neural methods, while TabNet and NODE remain weaker even with early stopping.Apart from both AutoGluon variants, the other neural architectures are not competitive against the cocktail approach or even a Dropout-regularized MLP.
  • GBDT comparison: The cocktails outperform XGBoost, auto-sklearn GBDT, and CatBoost with a statistically significant margin, validating the comparison against GBDT.Adding early stopping to XGBoost did not improve its performance.
  • Cumulative comparison: None of TabNet, NODE, or AutoGluon Tabular significantly outperforms GBDT, whereas this study reports neural networks beating GBDT under thorough hyperparameter optimization.This is presented as a large-scale result over the 40-dataset experimental protocol.
  • Cocktail composition and efficiency: Cocktails often include at least one ingredient from every regularization family, and they outperform XGBoost for most datasets within 15 minutes of HPO.After 30 minutes, the cocktails are statistically significantly better than XGBoost, with the performance gap increasing at larger budgets.

7 Conclusion

The paper concludes that dataset-specific regularization cocktails make simple MLPs competitive with, and able to surpass, strong tabular-data baselines. Its empirical scope is limited mainly to balanced classification datasets and excludes several important tabular regimes.

  • Regularization cocktails are per-dataset-optimized combinations of multiple regularization techniques for simple neural networks.
  • 13 regularization methods were evaluated across 40 tabular datasets.
  • Regularization cocktails significantly outperform recent neural network architectures and GBDT on tabular datasets.
  • The evaluation covers classification but not regression, and uses somewhat balanced datasets ranging from 452 to 416 188 data points.
  • The study excludes extreme data sizes, outliers, missing labels, semi-supervised data, streaming data, and other tabular modalities.

Societal Implications

The work connects improved tabular-data neural networks to applications where tabular data is prevalent, including socially consequential domains. The supplied passages also describe BOHB as the optimization method used in the broader study, while noting a simplified fidelity setup.

  • Improved neural networks could support applications in climate science, medicine, manufacturing, and recommender systems.
  • The proposed networks could serve as backbones for data-science-for-social-good applications such as fair machine learning.
  • BOHB combines Hyperband and Bayesian Optimization and can handle categorical hyperparameters for enabling or disabling regularizers.
  • The experiments used BOHB with a single fidelity level, leaving this choice for future reconsideration.

B.1 Method implicit search space

The method and architecture hyperparameters in Table 4 define a fixed configuration space for the experiments rather than variables searched during optimization.

  • Table 4 lists the training and model architecture hyperparameters used in all experiments.
  • Every hyperparameter in this configuration space has exactly one allowed value and is therefore fixed.

B.2 Benchmark search space

The benchmark compares the regularization cocktails with tuned traditional and neural baselines using specified library search spaces and shared experimental constraints. Pairwise results are evaluated on refitted test performance, with statistical significance assessed by the Wilcoxon signed-rank test.

  • Auto-Sklearn: Auto-Sklearn is restricted to GBDT, without preprocessing, and uses ensemble size 1 for comparison with the single-run cocktail method.
  • XGBoost: XGBoost uses a refined search space augmented with recommended hyperparameters and does not use one-hot encoding for categorical features.
  • TabNet: TabNet uses the authors’ default recommended hyperparameter ranges and is tuned with BOHB under the same parallel resources and limiting conditions as the cocktails.
  • AutoGluon: AutoGluon compares stacking and hyperparameter-optimization configurations, using neural networks as base learners with early stopping and feature preprocessing allowed.
  • NODE: NODE’s proposed search space was infeasible under cluster constraints, so experiments used the authors’ default example configuration.
  • Regularization cocktail: The regularization cocktail is the only compared method with statistically significant improvements over every individual regularizer, with p-value ≤0.001 in all cases.

C.2 Dataset-dependent optimal cocktails

The experiments show that optimal regularization choices vary by dataset, supporting dataset-specific cocktails rather than fixed top-five combinations. Dynamic learning rates improve more datasets than they worsen, but the difference is not statistically significant.

  • Dataset-specific selection: The cocktail’s best hyperparameter configuration selects regularization techniques separately for each dataset.Occurrences are measured from the best-found configuration for each of the 40 datasets.
  • Dataset-specific selection: The regularization cocktail outperforms both the top-5 most frequent and top-5 highest-ranked technique baselines.For both baselines, selected techniques are activated and their hyperparameters are tuned on the validation set.
  • Dataset-specific selection: Figure 6 reports how often each regularization technique appears in the best cocktail configuration across 40 datasets.The frequencies are based on whether BOHB activated each technique in the best-found configuration.
  • Learning-rate choice: Dynamic learning rates outperform fixed-rate cocktails on 21 of 40 datasets, tie on 1, and lose on 18.The result is not statistically significant, with p-value 0.7, and does not identify a clear region where dynamic learning rates help.

D Tables

The tables document the datasets, test-set evaluation procedures, detailed MLP and regularization results, cocktail variants, and remaining baselines. Performance is reported primarily as test-set balanced accuracy after refitting selected configurations.

  • Datasets: Table 9 lists the experimental datasets with descriptive statistics and identifiers usable for downloading them from OpenML.The task identifiers can be used with the OpenML-Python connector.
  • Cocktail comparisons: Table 10 compares the Regularization Cocktail with Top-5 Most Frequent and Top-5 Highest Ranks baselines on test-set performance.The comparison follows retraining on the best dataset-specific hyperparameter configuration.
  • Detailed results: Table 11 reports test-set performance for the plain MLP, individual regularization methods, and regularization cocktails.Results use the best-found hyperparameter configurations after retraining, with balanced accuracy as the performance metric.
  • Learning-rate variants: Table 12 compares fixed initial learning rates with learning rates optimized for each dataset.The dynamic-learning-rate variant is treated as a cocktail hyperparameterization choice.
  • Additional baselines: Table 13 presents the final test accuracies of remaining experimental and ablation baselines after refitting the incumbent configuration.Each reported performance represents the incumbent configuration after refitting.
Loading 2106.11189v2…