Source-linked AI summary
Deep Neural Decision Trees
Yongxin Yang, Irene Garcia Morillo, Timothy M. Hospedales
TL;DR
Tabular modeling needs both strong predictive tools and interpretable decision processes, but neural networks are typically black boxes while trees are naturally interpretable. DNDT realizes trees as neural networks, trains them with differentiable gradient-based methods, and evaluates their accuracy, pruning behavior, and relationship to conventional trees.
Problem
Neural networks perform strongly on perceptual data but lack interpretability, while tree-based models are preferred for tabular data because their predictions are naturally interpretable.
Method
DNDT uses a neural-network architecture with differentiable soft binning and trainable cut points to realize interpretable decision trees optimized with stochastic gradient descent.
Results
DNDT performs better than neural networks on certain tabular datasets, shows self-pruning at split and feature levels, and can differ from conventional trees without necessarily losing performance.
Takeaways & Limitations
DNDT combines interpretable tree structure with neural-network implementation, simultaneous structure-and-parameter optimization, and straightforward GPU acceleration.
Takeaways & Limitations
DNDT is not scalable with respect to feature count because its Kronecker-product design can require random-subspace forests for wide datasets, reducing interpretability.
Abstract
from arXiv · showhide
Deep neural networks have been proven powerful at processing perceptual data, such as images and audio. However for tabular data, tree-based models are more popular. A nice property of tree-based models is their natural interpretability. In this work, we present Deep Neural Decision Trees (DNDT) -- tree models realised by neural networks. A DNDT is intrinsically interpretable, as it is a tree. Yet as it is also a neural network (NN), it can be easily implemented in NN toolkits, and trained with gradient descent rather than greedy splitting. We evaluate DNDT on several tabular datasets, verify its efficacy, and investigate similarities and differences between DNDT and vanilla decision trees. Interestingly, DNDT self-prunes at both split and feature-level.
1. Introduction
DNDT combines the interpretability of decision trees with neural-network implementation and training, addressing the need for understandable predictions without giving up neural-network tooling.
- 1. Introduction: Interpretability matters in ethical and mission-critical applications because users may need to verify how a model reaches its prediction.In Business Intelligence, understanding how factors contribute to predictions can matter more than the conclusion itself.
- 1. Introduction: DNDT is a neural network architecture in which every weight setting corresponds to an interpretable decision tree.The correspondence makes the model intrinsically interpretable while retaining a neural-network realization.
- 1. Introduction: Unlike conventional decision trees, DNDT simultaneously optimizes parameters with stochastic gradient descent rather than greedy splitting.This allows the tree structure and parameters to be learned together.
- 1. Introduction: DNDT can be implemented in neural-network frameworks, trained with mini-batches and GPU acceleration, and integrated into larger neural networks.The architecture can serve as a building block for end-to-end learning with back-propagation.
2. Related Work
Related work establishes decision trees as interpretable rule-based models and situates DNDT among neural, alternative, and oblique tree inducers, while highlighting its arbitrary-cardinality splits.
- Tree models: Decision trees recursively partition input space and assign labels or scores, with interpretability arising from their rule-based predictions.Ensembles such as Random Forest and XGBoost improve performance at the expense of interpretability.
- Model interpretability: Interpretability research addresses the need for explainable predictions, especially where ethical or safety concerns require verification of model reasoning.Approaches include model-agnostic methods and methods associated with specific model types.
- Neural Networks and Decision Trees: Neural decision forests and Deep-NDF combine neural components with differentiable or stochastic decision trees, but DNDT is presented as a distinct approach.The supplied related-work passage identifies these models as prior attempts to unify neural networks and decision trees.
- Neural Networks and Decision Trees: Oblique neural decision trees use all features at each node, unlike conventional univariate trees, and are described as uninterpretable.This distinguishes their design from DNDT’s interpretable tree structure.
- Alternative Decision Tree Inducers: Conventional decision trees use recursive greedy splitting, whereas alternative inducers seek less myopic optimization through structured prediction or reinforcement learning.The related work frames greedy search as potentially sub-optimal despite its efficiency and feature-selection benefits.
- Alternative Decision Tree Inducers: DNDT supports splits of arbitrary cardinality rather than only binary splits, which can sometimes produce more interpretable trees.The passage contrasts this flexibility with binary-split designs adopted for simplicity.
3. Methodology
DNDT uses differentiable soft binning networks to route feature values to interpretable tree leaves, then trains cut points and leaf classifiers jointly with SGD. Multiple feature-wise binning outputs are combined via a Kronecker product, although this design becomes less scalable as feature count grows.
- 3.1. Soft binning function: Soft binning replaces nondifferentiable hard binning with a differentiable approximation for DNDT split decisions.The function maps a real scalar to soft bin assignments using a neural-network softmax.
- 3.1. Soft binning function: n trainable cut points β1, β2, …, βn divide a continuous variable into n + 1 intervals in monotonically increasing order.The cut points satisfy β1 < β2 < · · · < βn.
- 3.1. Soft binning function: Lower temperatures make the softmax output nearly one-hot, and temperature annealing can produce a more deterministic model during training.Straight-Through Gumbel-Softmax is an optional alternative when an actual one-hot vector is preferred.
- 3.2. Making Predictions: Feature-wise binning outputs are combined with a Kronecker product to enumerate final leaves, whose classifiers predict the labels of routed instances.The resulting vector z is almost one-hot and identifies the leaf reached by each instance.
- 3.3. Learning the Tree: Because the forward pass is differentiable, DNDT trains bin cut points and leaf classifiers simultaneously with stochastic gradient descent.The learned network can also be rendered as a conventional decision tree, as illustrated for the reduced two-feature Iris dataset.
- 3.3. Learning the Tree: The Kronecker-product design is not scalable with respect to feature count, so wide datasets are handled with random-subspace forests at the expense of interpretability.Exploiting sparsity in final binning could avoid an uninterpretable forest but would complicate the implementation.
4. Experiments
DNDT is evaluated against decision trees and neural networks on tabular datasets, with experiments examining accuracy, self-pruning, feature selection, ranking similarity, and GPU acceleration. Results show self-regularization at cut-point and feature levels, moderate ranking correlation with decision trees, and shorter GPU training times for larger models.
- Implementation: DNDT, decision trees, and neural networks are compared on 14 Kaggle and UCI datasets using test-set accuracy.DNDT uses one cut point per feature by default; datasets with more than 12 features use an ensemble of 10 trees over random feature subsets.
- Accuracy: DNDT is slightly better than the vanilla neural network, while no model has a dominant advantage across the evaluated tabular datasets.The authors describe this result as indicative because all models have tunable hyperparameters.
- Analysis of active cut-points: As cut points per feature increase from 1 to 5, active-cut-point utilization generally decreases, indicating that DNDT does not use all available parameters.A cut point is active when data instances fall on both sides; cut points outside the data range can therefore remain inactive.
- Analysis of active cut-points: DNDT accuracy initially increases with more cut points and then stabilizes, suggesting that larger DNDTs do not over-fit the training data without explicit regularization.The number of cut points per feature serves as the model-complexity parameter.
- Analysis of active features: DNDT can implicitly select features by making all cut points for some features inactive; features ignored more often are considered less important.Across runs, some features are consistently ignored, and the resulting selection rate is compared with decision-tree Gini importance.
- Comparison to decision tree: DNDT and decision trees sometimes favor the same features, but their rankings show only moderate overall Kendall’s Tau correlation.For Haberman’s, DNDT used only feature 2 while the decision tree ranked feature 0 highest; their test accuracies were 70.9% and 66.1%, respectively.
- GPU acceleration: GPU mode has significantly shorter training time than a 3.6GHz CPU for larger DNDT models created by increasing the number of cut points.The comparison averages training time over five runs.
5. Conclusion
DNDT is a neural-network-based tree model that combines interpretable decision-tree structure with neural-network training and implementation advantages. The paper also identifies self-regularisation as an avenue for future investigation.
- DNDT combines an interpretable decision tree with a neural-network implementation.
- Compared with conventional decision trees, DNDT is simpler to implement, jointly searches tree structure and parameters with SGD, and supports GPU acceleration.
- The authors propose investigating the source of DNDT’s observed self-regularisation in future work.