Source-linked AI summary
Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data
Sergei Popov, Stanislav Morozov, Artem Babenko
TL;DR
Deep neural networks had not consistently surpassed strong GBDT baselines on heterogeneous tabular data. The paper introduces NODE, a differentiable, multi-layer generalization of oblivious-tree ensembles, and reports that it outperforms competitors on most tasks.
Problem
DNN approaches had not consistently outperformed state-of-the-art GBDT implementations on heterogeneous tabular data.
Method
NODE generalizes oblivious decision-tree ensembles with differentiable feature selection and routing, entmax-based sparse splits, and end-to-end training.
Results
NODE outperforms leading GBDT competitors on most tabular-data tasks in extensive experiments with default and tuned hyperparameters.
Takeaways & Limitations
NODE is presented as a practical framework for machine learning on tabular data and as a layer that could be incorporated into complex backpropagation-trained pipelines.
Abstract
from arXiv · showhide
Nowadays, deep neural networks (DNNs) have become the main instrument for machine learning tasks within a wide range of domains, including vision, NLP, and speech. Meanwhile, in an important case of heterogenous tabular data, the advantage of DNNs over shallow counterparts remains questionable. In particular, there is no sufficient evidence that deep learning machinery allows constructing methods that outperform gradient boosting decision trees (GBDT), which are often the top choice for tabular problems. In this paper, we introduce Neural Oblivious Decision Ensembles (NODE), a new deep learning architecture, designed to work with any tabular data. In a nutshell, the proposed NODE architecture generalizes ensembles of oblivious decision trees, but benefits from both end-to-end gradient-based optimization and the power of multi-layer hierarchical representation learning. With an extensive experimental comparison to the leading GBDT packages on a large number of tabular datasets, we demonstrate the advantage of the proposed NODE architecture, which outperforms the competitors on most of the tasks. We open-source the PyTorch implementation of NODE and believe that it will become a universal framework for machine learning on tabular data.
1 INTRODUCTION
Deep learning has transformed several domains, but its advantage over shallow models remains uncertain for heterogeneous tabular data. NODE addresses this gap with a differentiable architecture inspired by oblivious-tree ensembles and is evaluated against leading GBDT methods.
- Motivation: DNNs have produced major advances in vision, language, speech, and other machine learning tasks.The passage attributes these gains to backpropagation-based optimization and hierarchical representation learning.
- Motivation: On heterogeneous tabular data, shallow models such as GBDTs often achieve state-of-the-art performance.Existing DNN approaches have not consistently surpassed properly competitive shallow baselines by a notable margin.
- Approach: NODE is a DNN architecture for tabular problems that generalizes CatBoost-style oblivious decision-tree ensembles.It makes feature selection and tree routing differentiable, enabling end-to-end optimization within deep-learning computational graphs.
- Approach: NODE combines differentiable oblivious decision tables with multi-layer architectures trained end-to-end.The paper characterizes this as a form of deep GBDT trained through gradient-based optimization.
- Evaluation: The paper reports extensive evaluation against leading GBDT implementations on many tabular datasets.Its stated contributions include a PyTorch implementation and experiments comparing NODE with existing GBDT methods.
2 RELATED WORK
Prior work establishes tree ensembles as strong tabular baselines and explores differentiable or multi-layer tree models. NODE builds on these directions while targeting consistent gains over properly tuned GBDTs.
- The state-of-the-art for tabular data: GBDT and random-forest ensembles are widely used top choices for tabular data, with leading GBDT packages often performing similarly.The cited packages include XGBoost, LightGBM, and CatBoost.
- Oblivious Decision Trees: Oblivious decision trees use one splitting feature and threshold at each depth, representing the tree as a table with 2^d entries.Their parallel splits make inference efficient, while ensemble use can reduce overfitting despite weaker individual learners.
- Differentiable trees: Differentiable-tree research softens internal decision functions so tree outputs and routing can be optimized end-to-end.NODE advocates entmax as its softening transformation and reports advantages over previously proposed approaches.
- Entmax: Entmax maps real-valued scores to sparse probability distributions and is used as an inductive bias for differentiable split construction.Unlike softmax, entmax can assign exactly zero probability to many alternatives.
- Multi-layer non-differentiable architectures: Earlier multi-layer tree architectures stack separately trained random forests or construct multi-layer GBDTs from non-differentiable blocks.These approaches differ from NODE’s end-to-end differentiable design.
- Specific DNN for tabular data: Prior tabular DNN studies often lacked comparisons with properly tuned GBDTs, and NODE is presented as the first approach to consistently outperform them across many datasets.The paper contrasts this with earlier reports of only marginal gains over GBDTs using default parameters.
3 NEURAL OBLIVIOUS DECISION ENSEMBLES
NODE uses differentiable oblivious decision trees as its core layer and stacks densely connected layers to combine shallow and deep decision rules. Entmax-based feature selection and routing enable end-to-end gradient training, while tree outputs are combined into predictions.
- Differentiable oblivious decision trees: NODE layers contain m differentiable oblivious decision trees of equal depth that receive a shared numeric feature vector.Each oblivious tree shares its splitting feature and threshold across all internal nodes at the same depth.
- Differentiable oblivious decision trees: Entmax replaces discrete feature selection and Heaviside routing with continuous, sparse choices that can be optimized by standard gradient descent.The model uses α-entmax for feature selection and a two-class entmax relaxation for threshold comparisons.
- Differentiable oblivious decision trees: Each tree computes a weighted combination of leaf-response entries using a choice tensor formed from the relaxed routing decisions.The relaxation matches a classic oblivious decision tree when feature selection becomes one-hot and routing becomes exactly zero or one.
- NODE architecture: A NODE layer concatenates the outputs of its individual trees, and the architecture averages tree outputs across layers for the final prediction.For classification, tree outputs may be multidimensional to represent class probabilities.
- NODE architecture: NODE follows a DenseNet-like sequence in which each layer receives all preceding outputs, allowing trees to learn both shallow and deep decision rules.A tree on layer i can use chains of up to i −1 previous layer outputs as features, while the output dimensionality l can be chosen independently of the class count.
- Training and inference: The model is trained end-to-end with mini-batch SGD by jointly optimizing feature selectors, thresholds, and response tensors.The reported protocol uses quantile preprocessing, Quasi-Hyperbolic Adam, checkpoint averaging, and validation-based stopping.
4 EXPERIMENTS
The experiments compare NODE with leading GBDT implementations and other tabular architectures across datasets, hyperparameter regimes, ablations, and runtime settings. NODE achieves the highest performance on most tasks, while entmax performs best across choice-function experiments and inference time is comparable to optimized GBDT libraries.
- Comparison to the state-of-the-art: NODE consistently outperforms CatBoost and XGBoost with default hyperparameters across all datasets.The default NODE uses a single layer containing 2048 depth-six decision trees.
- Comparison to the state-of-the-art: With tuned hyperparameters, NODE outperforms competitors on most tasks, except Yahoo and Microsoft, where tuned XGBoost performs best.The authors speculate that oblivious trees may be an inappropriate inductive bias for Yahoo.
- Comparison to the state-of-the-art: Properly tuned GBDTs outperform the compared mGBDT and DeepForest implementations on the evaluated datasets.The comparison was partial because source code was unavailable for many prior architectures, and both implementations produced out-of-memory errors on many datasets.
- Ablative analysis: Entmax with α=1.5 outperforms softmax, Gumbel-Softmax, and sparsemax across the reported shallow and deep experiments.Gumbel-Softmax was unable to learn deep architectures, probably because stochastic responses from earlier layers were too noisy for later layers.
- Ablative analysis: Feature-importance analysis on Higgs examines original and learned features, tree contributions, and response dependence on feature importance.The analysis uses permutation feature importance on 10,000 objects and evaluates representations across NODE layers.
- Ablative analysis: NODE inference time is on par with heavily optimized GBDT libraries in a million-object YearPrediction benchmark.The comparison used ensembles of 1024 depth-six trees and a pure PyTorch implementation without custom kernels.
5 CONCLUSION
NODE is a differentiable deep GBDT architecture trained end-to-end via backpropagation. The experiments demonstrate advantages over existing competitors, while eight-layer NODE runtime is on par with comparable shallow ensembles.
- NODE is a differentiable deep GBDT architecture trained end-to-end via backpropagation.
- Both timings of eight-layer NODE are on par with shallow counterparts using the same total number of ensemble trees.The comparison uses 1024 trees of depth six on YearPrediction, averaged over five runs.
- The experiments demonstrate NODE’s advantages over existing competitors with default and tuned hyperparameters.
A.1 DESCRIPTION OF THE DATASETS
The experiments use six open-source tabular datasets spanning classification, regression, and ranking tasks. The datasets include Epsilon, YearPrediction, Higgs, Microsoft, Yahoo, and Click.
- The experiments use six tabular datasets: Epsilon, YearPrediction, Higgs, Microsoft, Yahoo, and Click.
- Epsilon is a high-dimensional binary-classification dataset from the PASCAL Large Scale Learning Challenge 2008.
- YearPrediction is a regression dataset using audio features to predict song release years from 1922 to 2011.
- Higgs is a binary-classification task predicting whether an event produces Higgs bosons.
- Microsoft and Yahoo are ranking datasets with query-url pairs labeled from 0 to 4.
A.2 OPTIMIZATION OF HYPERPARAMETERS
Hyperparameters were optimized using validation splits and Hyperopt’s Tree-structured Parzen Estimator procedure. The protocol used 50 optimization steps for each method.
- Training data were split into 80% training and 20% validation for five datasets, while Yahoo used the authors’ provided train/validation/test split.
- Hyperopt optimized CatBoost, XGBoost, and FCNN hyperparameters with 50 Tree-structured Parzen Estimator steps per method.
- The number of trees was selected using the validation set, with a maximum of 2048 trees.
A.2.1 CATBOOST AND XGBOOST
CatBoost and XGBoost were tuned over specified search spaces covering regularization, sampling, tree structure, and optimization parameters. CatBoost’s tree count was also selected using validation performance.
- CATBOOST: CatBoost tuning searched random strength, one-hot maximum size, L2 leaf regularization, bagging temperature, and leaf-estimation iterations.
- CATBOOST: CatBoost’s random strength was sampled uniformly from the discrete range [1, 20].
- XGBOOST: XGBoost tuning included eta, maximum depth, subsampling, and column-sampling parameters.
- XGBOOST: XGBoost maximum depth was sampled from the discrete range [2, 10].
- XGBOOST: XGBoost tuning also covered minimum child weight, alpha, lambda, and gamma search spaces.
A.2.2 FCNN
FCNNs use Dense-ReLU-Dropout blocks and are tuned with Hyperas over architecture, learning rate, and dropout choices.
- FCNNs are constructed from Dense-ReLU-Dropout blocks, with independently selected layer widths and one dropout value shared across the network.
- The networks are tuned using Hyperas, a Keras wrapper for Hyperopt.
- Training uses Adam, averages parameters over c=5 consecutive checkpoints, and applies early stopping on validation.
- The hyperparameter search varies layers from 2 to 7 and units from {128, 256, 512, 1024}.
- Learning rates range from 1e−4 to 1e−2, while dropout ranges from 0 to 0.5.
A.2.3 NODE
NODE models are tuned by grid search using repeated architecture choices across layers, GPU-fitting batch sizes, and a fixed learning rate.
- NODE models are tuned by grid search over layer count, total tree count, tree depth, and tree output dimension.
- The multi-layer NODE uses the same architecture in every layer, with the same number of trees at the same depth.
- For each dataset, batch size is set to the maximum that fits in GPU memory, and the learning rate is always 10−3.
- The search considers 2, 4, or 8 layers and total tree counts of 1024 or 2048.
- Tree depths are 6 or 8, and tree output dimensions are 2 or 3.