PaperlayerView exact revision on arXiv

Source-linked AI summary

On Embeddings for Numerical Features in Tabular Deep Learning

Yury Gorishniy, Ivan Rubachev, Artem Babenko

arXiv:2203.05556v4cs.LG

TL;DR

Tabular deep learning often lags GBDT, and numerical-feature embedding design remains relatively underexplored compared with backbone design. The paper introduces piecewise linear and periodic embedding schemes, finding that embeddings improve diverse backbones and can bring MLP-like models on par with attention-based models. It concludes that numerical-feature embeddings are an important design aspect for tabular DL, while noting unresolved optimization questions and a shared-transformation scope constraint.

  • Problem

    Numerical-feature embedding design is underexplored in tabular deep learning, whose models often struggle to outperform GBDT.

  • Method

    The paper develops piecewise linear and periodic numerical-feature embeddings and evaluates them with conventional differentiable layers across deep-learning backbones.

  • Results

    Embeddings improve different deep architectures, enable MLP-like models to perform on par with attention-based models, and support new state-of-the-art tabular-DL results.

  • Takeaways & Limitations

    Numerical-feature embeddings are an important tabular-DL design aspect with potential for further embedding-scheme improvements.

  • Takeaways & Limitations

    The study applies the same functional transformation to all features, which may be suboptimal, and does not explain how the modules help optimization fundamentally.

Abstract

from arXiv · show

Recently, Transformer-like deep architectures have shown strong performance on tabular data problems. Unlike traditional models, e.g., MLP, these architectures map scalar values of numerical features to high-dimensional embeddings before mixing them in the main backbone. In this work, we argue that embeddings for numerical features are an underexplored degree of freedom in tabular DL, which allows constructing more powerful DL models and competing with GBDT on some traditionally GBDT-friendly benchmarks. We start by describing two conceptually different approaches to building embedding modules: the first one is based on a piecewise linear encoding of scalar values, and the second one utilizes periodic activations. Then, we empirically demonstrate that these two approaches can lead to significant performance boosts compared to the embeddings based on conventional blocks such as linear layers and ReLU activations. Importantly, we also show that embedding numerical features is beneficial for many backbones, not only for Transformers. Specifically, after proper embeddings, simple MLP-like models can perform on par with the attention-based architectures. Overall, we highlight embeddings for numerical features as an important design aspect with good potential for further improvements in tabular DL.

1 Introduction

Tabular deep learning still often trails GBDT, while recent Transformer-like models narrow the gap by embedding numerical features before backbone mixing. This work shifts attention from backbone design to numerical-feature embeddings, introducing two schemes and showing broad performance benefits.

  • Tabular deep models often remain behind decision-tree ensembles such as GBDT.
  • Transformer-like tabular models map scalar numerical features to high-dimensional embeddings before self-attention mixes them.
  • The paper introduces piecewise linear encoding and periodic activations as two building blocks for numerical-feature embeddings.
  • Embedding numerical features benefits diverse deep architectures, with properly embedded MLP-like models often matching attention-based models.
  • The proposed embedding schemes improve performance over prior models and achieve new state-of-the-art results on public tabular benchmarks.

2 Related work

Prior tabular architectures do not consistently outperform GBDT or properly tuned simple models, and their numerical-feature mappings are often simple. The paper studies binning-based and periodic representations as more expressive alternatives.

  • Systematic evaluations find that proposed tabular deep models do not consistently outperform GBDT.
  • Several sophisticated tabular architectures are not superior to properly tuned MLP and ResNet models.
  • Transformer-based tabular models require mapping scalar numerical features to high-dimensional vectors for self-attention.
  • Existing numerical-feature mappings commonly use relatively simple computational blocks that can limit model expressiveness.
  • Feature binning discretizes numerical ranges into intervals, while this work uses bin edges to construct lossless piecewise linear representations.
  • Periodic activations are presented as embedding components for numerical features in tabular data.

3 Embeddings for numerical features

The framework independently transforms each numerical feature into an embedding, then feeds those embeddings to MLP-like or Transformer-based backbones. It develops piecewise linear and periodic constructions, alongside conventional differentiable layers, for comparison.

  • 3.1 General framework: Each numerical feature is independently mapped by an embedding function to a vector with feature-specific dimensionality.
  • 3.1 General framework: MLP-like backbones concatenate feature embeddings, whereas Transformer-based backbones receive them without an additional mixing step.
  • 3 Embeddings for numerical features: Changing scalar representations is motivated by evidence that changing the input space can alleviate optimization limitations of vanilla MLPs.
  • 3.2 Piecewise linear encoding: Piecewise linear encoding uses bins and interpolated terms to create alternative representations while preserving numerical ordering.
  • 3.2 Piecewise linear encoding: Bin construction uses either unsupervised classical binning or supervised target-aware recursive splitting.
  • 3.3 Periodic activations: Periodic embeddings concatenate sine and cosine features whose trainable frequency coefficients are tuned with validation data.
  • 3.4 Simple differentiable layers: Conventional linear and ReLU-based layers can be combined with the proposed components, and such combinations often improve results.

4 Experiments

The experiments evaluate numerical-feature embeddings across MLP, ResNet, and Transformer backbones on eleven mostly GBDT-friendly public datasets. Piecewise linear and periodic embeddings generally improve deep models, with embedded MLPs often matching Transformer-based models and narrowing the gap with GBDT.

  • 4.1 Datasets: The benchmark uses eleven public datasets, focuses on middle- and large-scale tasks, and is biased toward GBDT-friendly problems.The experiments compare deep-learning models with Gradient Boosted Decision Trees.
  • 4.4 Simple differentiable embedding modules: Simple embedding modules improve MLP performance, while linear-relu combinations provide modest but consistent gains.The linear embedding can be fused with MLP’s first linear layer after training, eliminating its inference overhead.
  • 4.5 Piecewise linear encoding: Piecewise linear encoding often benefits both MLP and Transformer architectures, with significant gains on some datasets.Adding differentiable components can further improve performance, but the most expensive variants are not worthwhile.
  • 4.6 Periodic activation functions: Periodic embeddings improve MLP on average, and adding differentiable components can meaningfully improve or repair the periodic-only variant.MLP-PLR and MLP-PL improve results on GE, CA, and HO, and fix cases where MLP-P underperforms MLP on OT and FB.
  • 4.7 Comparing DL models and GBDT: Across most datasets, numerical-feature embeddings improve MLP, ResNet, and Transformer models, with PLR achieving the best average performance.PLE is highlighted for simplicity, interpretability, and efficiency, while embedded MLP-like models perform on par with Transformer-based models.
  • 4.7 Comparing DL models and GBDT: On most backbone–dataset pairs, proper embeddings close the gap with GBDT, including parity on California Housing and Adult.The benchmark is initially GBDT-biased, and exceptions include MI plus several ResNet and Transformer pairs.

5 Analysis

The analysis examines model-size overhead, binning-based encoding choices, preprocessing sensitivity, and transfer of periodic embeddings beyond deep models.

  • 5.1 Comparing model sizes: Embedding modules can substantially increase parameter counts, but this overhead does not translate proportionally to training time or throughput.For MLP-LR on CH, an almost 2000-fold parameter increase resulted in only a 1.5-fold training-time increase.
  • 5.2 Ablation study: Making binning-based encoding piecewise linear is a good default strategy compared with thermometer and generalized one-blob alternatives.The comparison evaluates two alternative binning-based schemes against PLE under the same tuning and evaluation protocols.
  • 5.3 Piecewise linear encoding as a feature preprocessing technique: PLE representations contain values from [0, 1] and are invariant to shifting and scaling, making them a potential preprocessing technique for deep learning.This property can reduce dependence on traditional preprocessing choices.
  • 5.3 Piecewise linear encoding as a feature preprocessing technique: Models using PLE are less sensitive to initial preprocessing than vanilla MLPs, which can often become unusable without preprocessing.The authors report that preprocessing sensitivity is less pronounced for PLE-based models across the evaluated datasets.
  • 5.4 Periodic modules for XGBoost: The periodic module was also tested with XGBoost, extending the analysis beyond the deep architectures evaluated in the main experiments.The experiment fixes the random coefficients from Equation 2 when adopting the Periodic module for XGBoost.

6 Conclusion & Future work

The paper concludes that numerical-feature embeddings improve tabular deep-learning backbones and can bring MLP-like models to parity with attention-based models, while leaving important questions open.

  • 6 Conclusion & Future work: Numerical-feature embeddings are presented as an important design aspect that can improve existing deep-learning backbones and reduce their gap with GBDT.The paper illustrates this through piecewise linear encodings and periodic functions.
  • 6 Conclusion & Future work: Traditional MLP-like models coupled with embeddings can perform on par with attention-based models.
  • 6 Conclusion & Future work: The paper leaves open how embedding modules help optimization and whether applying the same transformation to all features is suboptimal.These points define directions for further investigation.

Checklist

The checklist records affirmative reproducibility and ethics-related responses, while noting that the work uses public datasets and includes no theoretical results or human-subject research.

  • Checklist: The authors state that they describe their contributions, scope, and limitations.
  • Checklist: The paper reports no potential negative societal impacts because it focuses on a generic aspect of deep-learning models.
  • Checklist: The authors state that theoretical-results questions are not applicable because the paper includes no theoretical results.
  • Checklist: The authors state that the supplementary material provides code, data, instructions, training details, and error bars for reproducing experiments.
  • Checklist: The work uses existing publicly available datasets, cites their creators, and refers to their licenses in the supplementary README.
  • Checklist: The authors report no crowdsourcing or human-subject research and therefore mark participant-risk and compensation questions as not applicable.

Supplementary material

The supplementary material visualizes how numerical-feature embeddings enter MLPs, formalizes their concatenation, and illustrates target-aware PLE bin construction.

  • Supplementary material: Equation 3 formalizes the MLP pipeline by concatenating embedding vectors z1 through zk before applying the MLP.The concatenated vector lies in R^(d1+...+dk).
  • Supplementary material: Figures 2 and 3 contrast a vanilla MLP with the same MLP augmented by numerical-feature embeddings.The vanilla model takes two numerical features as input.
  • Supplementary material: Figure 4 visualizes how target-aware piecewise linear encoding obtains bins from decision trees.

C Additional details on datasets

The experiments use a broad collection of tabular datasets spanning classification, regression, simulated, and real-world prediction tasks.

  • The dataset collection includes Gesture Phase Prediction, Churn Modeling, California Housing, House 16H, Adult, and Otto Group Product Classification.
  • It also includes Higgs, Santander Customer Transaction Prediction, Facebook Comments, and Covertype.
  • California Housing is identified as real-estate data, Adult as income estimation, and Higgs as simulated physical-particle data.

D.1 Testing quantile-based PLE on the synthetic GBDT-friendly dataset

The experiments evaluate quantile-based PLE embeddings for MLP and Transformer models on a synthetic task designed to be easy for GBDT but difficult for traditional deep-learning models. The appendix also specifies evaluation protocols, preprocessing, tuning, model configurations, and comparison approaches.

  • D.1 Testing quantile-based PLE on the synthetic GBDT-friendly dataset: The synthetic dataset contains 10,000 objects and 8 features, with targets generated by 16 randomly constructed decision trees of depth 6.
  • D.1 Testing quantile-based PLE on the synthetic GBDT-friendly dataset: Using PLE representations instead of scalar values improves both MLP and Transformer performance on the same synthetic task.
  • D.1 Testing quantile-based PLE on the synthetic GBDT-friendly dataset: Increasing the number of PLE bins does not always produce better results in practice.
  • D.1 Testing quantile-based PLE on the synthetic GBDT-friendly dataset: The appendix compares PLE with Fourier features and DICE embeddings, with original Fourier features performing poorly and DICE embeddings described as suboptimal for tabular problems.
  • D.1 Testing quantile-based PLE on the synthetic GBDT-friendly dataset: The study evaluates models over 15 random seeds, reports average test performance, and forms three ensembles from disjoint groups of five models.

F Extended tables with experimental results

The appendix provides extended single-model and ensemble results, alongside additional reporting on DICE embeddings and their performance in tabular data problems.

  • F Extended tables with experimental results: Extended scores with standard deviations are provided for both single models and ensembles in Tables 18 and 19.
  • F Extended tables with experimental results: The extended results include DICE embeddings, which are described as a suboptimal way to represent numbers in tabular data problems.
Loading 2203.05556v4…