Source-linked AI summary
Multi-Layered Gradient Boosting Decision Trees
Ji Feng, Yang Yu, Zhi-Hua Zhou
TL;DR
The paper asks how to jointly train multi-layered models without back-propagation and how to build such models with forests. It proposes a multi-layered GBDT forest and reports adaptive model complexity and competitive performance across a broad range of tasks.
Problem
The paper asks how to jointly train multi-layered models without back-propagation and how to construct a multi-layered forest model.
Source: §1 · p. 2Method
The paper proposes a multi-layered GBDT forest with an explicit focus on representation learning.
Source: §1 · p. 2, §5 · p. 13Results
The model achieved adaptive model complexity and competitive performance across a broad range of tasks, with theoretical and experimental results supporting its effectiveness.
Source: §1 · pp. 2–3Takeaways & Limitations
The proposed representation objective is to learn feature re-representations that are as linearly separable as possible.
Source: §3 · p. 6The discussed approach is described as suitable only in a supervised-learning setting.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Multi-layered representation is believed to be the key ingredient of deep neural networks especially in cognitive tasks like computer vision. While non-differentiable models such as gradient boosting decision trees (GBDTs) are the dominant methods for modeling discrete or tabular data, they are hard to incorporate with such representation learning ability. In this work, we propose the multi-layered GBDT forest (mGBDTs), with an explicit emphasis on exploring the ability to learn hierarchical representations by stacking several layers of regression GBDTs as its building block. The model can be jointly trained by a variant of target propagation across layers, without the need to derive back-propagation nor differentiability. Experiments and visualizations confirmed the effectiveness of the model in terms of performance and representation learning ability.
1. Introduction
The introduction asks whether tree ensembles can learn hierarchical distributed representations and be jointly trained without back-propagation. It proposes a multi-layered GBDT model trained through a target-propagation variant.
1. Introduction
- The introduction identifies representation learning by forests as underexplored, while noting that GBDTs have been a dominant approach for discrete or tabular data.It also notes that the Deep Forest framework achieved competitive performance across a broad range of tasks with adaptive model complexity.Source: §1 · p. 2
- The paper frames two questions: whether non-differentiable components can form intermediate distributed representations, and how to train such models jointly without back-propagation.For tree ensembles, the chain rule cannot propagate errors.Source: §1 · p. 2
- The introduction notes that the Deep Forest approach is suitable only for supervised learning.The proposed work is presented as an exploration of forest representation learning beyond that limitation.Source: §1 · p. 2
- The proposed model stacks GBDTs across layers to explore hierarchical representation learning and uses a target-propagation variant for joint training.The approach is intended to avoid relying on back-propagation or differentiability.Source: §1 · pp. 2–3
- The authors report theoretical justifications and experimental results as evidence for the effectiveness of tree-based hierarchical distributed representations.They describe this as a demonstration that such representations can be learned using trees, rather than only neural networks or differentiable systems.Source: §1 · p. 3
2. Related Works
Related work discusses proposed explanations for deep models, alternatives to back-propagation, and the role of tree ensembles in discrete and mixed-type data. It also notes that existing alternative training approaches remain in the differentiable regime.
2. Related Works
- Prior accounts attribute deep-model effectiveness to learned hierarchical distributed representations, with proposals linking depth to faster-mixing Markov chains or successive refinement of relevant information.The section also notes that there is no universal theory explaining why deep models work better than shallow ones.Source: §2 · p. 3
- Target-propagation methods [4] [5] propagate targets through inverse mappings, while feedback-alignment methods [15] use asymmetric or disconnected feedback paths.The cited alternatives remain differentiable and their theoretical justifications depend on calculating activation-function Jacobians.Source: §2 · p. 3
- Decision-tree ensembles are established tools for discrete or mixed-type data; bagging and boosting underpin Random Forest and GBDTs, respectively.The section also notes that XGBoost [19] is used in industrial applications and that eForest [22] demonstrated near-perfect input-pattern reconstruction.Source: §2 · p. 4
3. The Proposed Method
The method stacks GBDTs to learn intermediate representations, addressing the difficulty of applying back-propagation when layer mappings are non-differentiable. It trains the layers with inverse mappings and propagated pseudo-labels, with task-specific choices for initialization, loss, and classification.
3. The Proposed Method
- The section gives mean squared error or regularized cross-entropy as loss choices, and describes an unsupervised autoencoder objective based on reconstructing the input.For the unsupervised setting, the desired output can be the training data itself, with reconstruction error between output and original input.Source: §3 · p. 4
- The method targets the back-propagation barrier for non-differentiable GBDTs while seeking hierarchical representations through stacked layers.The section describes intermediate-layer outputs as learned representations and focuses on the case where layer mappings are gradient boosting decision trees.Source: §3 · pp. 4–5
- Inverse mappings are trained as pseudo-inverses by minimizing expected reconstruction loss, with noise injection intended to improve robustness around the training data manifold.The proposed inverse loss reconstructs noise-perturbed layer inputs; the text also suggests this randomness could support a generative path as future work.Source: §3 · p. 5
- At each iteration, the top layer uses real labels to set its pseudo-label, and inverse mappings propagate pseudo-labels downward before forward mappings are updated.The text suggests bottom-up updates of the forward mappings, with each layer potentially taking several additive boosting rounds toward its current pseudo-label.Source: §3 · p. 5
- Because random tree-structure initialization is difficult, the method initializes intermediate outputs with Gaussian noise and fits tiny trees before iterative updates.The section describes this as an alternative to randomly initializing the tree structures.Source: §3 · p. 6
- For classification, a linear top-layer classifier is used to encourage linearly separable lower-layer representations and avoid learning an inverse mapping for that layer.The pseudo-label for the layer below can instead be calculated from the global-loss gradient with respect to the last hidden-layer output.Source: §3 · pp. 6–7
end
Under stated conditions, an intermediate-layer update toward its pseudo-label can reduce the loss above it and thereby the global loss, without requiring differentiable mappings. The section contrasts this result with prior analysis that depends on Jacobians and motivates non-differentiable components for tree-suited data and potential resistance to gradient-based attacks.
end
- Earlier target-propagation analysis relies on Jacobians of forward and inverse mappings, limiting its stated applicability to neural networks.The text says that proof relies heavily on computing the Jacobians of F_i and G_i.Source: end · pp. 7–8
- Under certain conditions, updating an intermediate layer toward its pseudo-label reduces the loss of the layer above and thus the global loss, without requiring differentiable forward or inverse mappings.The claim is presented as Theorem 1.Source: end · p. 8
- The authors motivate non-differentiable components because trees can make input-gradient attacks more difficult, while some datasets are best modeled by trees.The section identifies combining tree-ensemble performance with multi-layered representations as an area of interest.Source: end · p. 8
4. Experiments
The experiments test whether multi-layered GBDTs can be jointly trained and learn useful representations across synthetic and real-world tasks. Results report easier class separation, distributed encodings, and competitive performance as depth increases.
4. Experiments
- The experiments assess joint-training feasibility, convergence, learned features, depth effects, and comparisons with neural networks across synthetic and real-world applications.Source: §4 · p. 9
4.1. Synthetic Data
- Synthetic tests showed hidden-layer features made classes easier to separate, while autoencoder encodings distributed curvature and relative-distance information across dimensions.Source: §4.1 · p. 10
4.2. Income Prediction
- On income prediction, mGBDT achieved the highest accuracy against same-structure neural networks and single or stacked GBDTs, while its layers reached lower training loss without overfitting.NNTargetProp converged less well than NNBackProp, whereas the same structure using GBDT layers achieved lower training loss without overfitting.Source: Table 1 · p. 11, §4.2 · pp. 11–12
4.3. Protein Localization
- As intermediate layers increase, mGBDT remains more robust than NNTargetProp: neural-network target-prop performance drops from .5964 to .3654, while representation quality improves with model depth.Source: §4.3 · pp. 12–13
5. Conclusion and Future Explorations
The conclusion presents mGBDT as a jointly trainable tree-ensemble approach to multi-layered representation learning, and outlines structural extensions and hybrid integrations as future explorations.
5. Conclusion and Future Explorations
- mGBDT combines multi-layered representation learning with GBDT forests and can be jointly trained using a variant of target propagation.The authors state that multi-layered distributed representations are not tied to differentiable systems and report theoretical and experimental support for the approach.Source: §5 · p. 13
- The authors propose integrating mGBDT layers as feature extractors within the deep forest structure.Source: §5 · pp. 13–14
- The training method permits recurrent or convolutional mGBDT structures and mixed training methods across layers, including tree layers embedded in differentiable systems.Source: §5 · p. 14