Source-linked AI summary
Neural Random Forests
Gérard Biau, Erwan Scornet, Johannes Welbl
TL;DR
Neural networks are expressive but can overfit small data sets, while tree-based random forests use fewer tuning parameters yet can impose box-like decision boundaries. This paper turns regression trees into neural networks and proposes two hybrid forest procedures. The methods are proved consistent under regulated parameters, and experiments report improvements over random forests, with fully connected Method 1 strongest overall aside from BART.
Problem
Neural networks can overfit small data sets, whereas random forests have fewer tuning parameters but their orthogonal splits can create box-like boundaries that are suboptimal for correlated features.
Source: §1 · p. 2Method
The paper represents regression trees as neural networks and combines these tree-informed networks in two random-forest-style predictors.
Source: §1 · p. 3, §2.1 · p. 5Results
All NRF models consistently improve over the original random forest across the tested data sets, and fully connected Method 1 has the best overall performance aside from BART.
Source: §5.2 · pp. 28–29Takeaways & Limitations
Tree-informed initialization gives the networks a warm start at traditional forests’ prediction level while retaining fewer parameters to tune than standard networks and fewer boundary-geometry restrictions than trees.
Source: §1 · p. 3The tree-to-network interpretation does not readily extend to deeper networks, and the consistency proof does not characterize aggregation’s finite-sample or asymptotic benefits.
Source: §3 · p. 12, §4.2 · p. 20
Abstract
from arXiv · showhide
Given an ensemble of randomized regression trees, it is possible to restructure them as a collection of multilayered neural networks with particular connection weights. Following this principle, we reformulate the random forest method of Breiman (2001) into a neural network setting, and in turn propose two new hybrid procedures that we call neural random forests. Both predictors exploit prior knowledge of regression trees for their architecture, have less parameters to tune than standard networks, and less restrictions on the geometry of the decision boundaries than trees. Consistency results are proved, and substantial numerical evidence is provided on both synthetic and real data sets to assess the excellent performance of our methods in a large variety of prediction problems.
1 Introduction
Random forests offer strong performance with relatively few tuning parameters, but their axis-aligned splits can produce box-like decision boundaries; neural networks are more expressive but risk overfitting, especially on small data sets. The paper reformulates random forests as neural networks and introduces two hybrid predictors, with theoretical consistency guarantees and reported improvements over random forests in most situations.
1 Introduction
- Decision trees are valued for simplicity and transparency, including their ability to explain complex data sets.Source: §1 · p. 2
- Random forests combine trees grown on sampled data fractions, and have shown strong performance across practical problems despite incomplete theory.Source: §1 · p. 2
- Neural networks offer expressive modeling but can overfit small data sets, while trees use fewer tuning parameters but can yield box-like boundaries that may be suboptimal for correlated features.Source: §1 · p. 2
- The paper introduces two neural random forest predictors by restructuring ensembles of random trees as sparse multilayer neural networks with trainable, differentiable activations.The authors describe the connection between random forests and neural networks as largely unexamined theoretically.Source: §1 · p. 3
- The paper establishes that any regression tree can be represented as a neural network, then uses this equivalence to define two forest-style predictors initialized with tree knowledge.This initialization gives the networks a warm start at traditional forests’ prediction level.Source: §1 · p. 3
- The authors prove consistency guarantees for both methods and report that they outperform random forests in most tested situations.Experiments cover simulated and real data sets.Source: §1 · p. 3
2 Trees, forests, and networks
The section sets up nonparametric regression estimation, then explains how regression trees and randomized forests can be represented as neural networks. It describes the CART splitting procedure and how randomization constructs the forest.
2 Trees, forests, and networks
- The framework estimates the regression function from an independent training sample of input-response pairs.Consistency is defined through mean squared error tending to zero as sample size grows.Source: §2 · pp. 3–4
2.1 From one tree to a neural network
- A regression tree partitions the input space into axis-aligned regions and predicts in each leaf with the average response of training observations there.Its structure can be represented by a three-layer network: neurons encode split decisions, then leaf membership, and the output produces the leaf prediction.Source: §2.1 · pp. 4–7
2.2 The CART program
- CART recursively selects a feature direction and cut by evaluating the split criterion, then stops when the tree has the specified number of terminal nodes.The criterion measures the renormalized difference between empirical variance before and after a cut.Source: §2.2 · pp. 7–8
2.3 Random forests
- A random forest combines randomized CART trees, each grown on a without-replacement sample and using randomly selected coordinates when choosing splits.The number of leaves per tree is fixed at K_n; m_try is independent of n and is often set to d/3.Source: §2.3 · pp. 8–9
- Each forest tree can be reinterpreted as a neural network, with K_n − 1 first-layer neurons and K_n second-layer neurons.The architecture and coefficients depend on the tree's randomization and the training data.Source: §2.3 · p. 9
3 Neural forests
Neural forests convert randomized CART trees into sparse neural networks with smooth activations, then combine the networks through independent or joint training. Tree-derived structure initializes the networks, while differentiable activations allow backpropagation training.
3 Neural forests
- Smooth hyperbolic-tangent activations make the loss differentiable for backpropagation and allow smoother decision boundaries than crisp tree membership.The activation contrast parameters control transition sharpness; as they grow, the activations approach threshold functions.Source: §3 · p. 10
- For roughly balanced trees, fitting the sparse architecture takes O(Kn log Kn) parameters on average, independent of input dimension d.The corresponding fully connected network requires O(dKn+K^2n) parameters; an unbalanced tree can require O(K^2n) work.Source: §3 · pp. 10–11
- Tree-based initialization supplies a CART-like starting predictor and a stronger inductive bias than random initialization, while sparse connectivity retains some interpretability.The paper also considers relaxing sparsity by allowing full connectivity during training.Source: §3 · p. 11
- Tree-derived sparse architectures with smooth activations turn CART trees into neural networks, providing a principled way to specify network structure and weight priors.The paper proposes combining the resulting networks in two different ways.Source: §3 · p. 11
Method 1: Independent training
- Each tree-type network can be trained independently, with the resulting small networks averaged to form the predictor.The parameters are fitted network by network.Source: §3 · pp. 11–12
Method 2: Joint training
- Alternatively, the tree networks are concatenated into one large network whose shared output layer combines second-layer neuron outputs.The full network is optimized jointly in one run rather than network by network.Source: §3 · p. 12
3 Neural forests
- The tree-to-network interpretation does not immediately extend to deeper networks, and adding intact layers may be difficult to optimize because many connections play the same role.Source: §3 · p. 12
4 Some theory
The theory section defines two neural-forest training procedures and proves consistency under stated conditions. It also explains that consistency depends on controlling tree depth or subsample size, while the proof does not establish benefits from aggregating trees.
4.1 Empirical risk minimization
- Independent training fits each randomized tree-network estimate by empirical error minimization and aggregates the estimates; hyperbolic-tangent activations relax crisp leaves into fuzzy membership across branches and leaves.The parameter range is restricted for consistency, and the stated bounds leave room for optimization; original tree estimates satisfy the condition when Y is almost surely bounded.Source: §4.1 · pp. 15–17
- Joint training concatenates the small tree-networks into a “big” network and minimizes empirical error over its constrained function class.The concatenated network retains the individual trees’ network structure while optimizing jointly.Source: §4.1 · pp. 17–18
4.2 Consistency
- The consistency theorem applies to a class F of continuous functions with a stated rectangular constancy property, which includes additive continuous functions and certain same-sign polynomials and products.F is defined so that coordinate-wise constancy on a hyperrectangle implies constancy over that rectangle.Source: §4.2 · p. 19
- Both neural-forest estimates are consistent when the number of terminal nodes and parameters γ1 and γ2 are regulated as functions of sample size, under the theorem’s assumptions.The theorem assumes uniformly distributed inputs on [0, 1]^d, bounded responses, and a regression function in F.Source: §4.2 · p. 19
- Consistency requires controlling tree depth or subsample size: trees must be pruned or subsampled when fully grown trees would violate the theorem’s assumptions.Fully grown, subsampled trees are covered when the assumptions hold with the subsample size in place of the number of leaves.Source: §4.2 · p. 19
- The consistency proof relies on consistency of individual small networks, so it does not establish aggregation benefits in finite-sample analysis or asymptotic behavior.The authors identify room for improvement in this analysis.Source: §4.2 · p. 20
4.3 Proof of Theorem 4.1
- The proof separates estimation and approximation error, uses uniform-deviation and covering-number bounds for the former, and analyzes the tree pseudo-estimate’s cell geometry for the latter.The approximation step uses a piecewise-constant pseudo-estimate with true conditional expectations in each leaf; two terms are shown to vanish under conditions on γ1 and γ2 and on the cells’ asymptotic geometry.Source: §4.3 · pp. 20–21
5 Experiments
Across benchmark experiments, NRF variants improve on RF, and Method 1 achieves the best overall performance aside from BART. On synthetic data, fully connected Method 2's test error decreases faster than RF's as sample size grows.
5.1 Training procedure
- The training procedure initializes NRF networks from RF split directions and positions, then optimizes them using gradient-based training.Parameters are selected by minimum validation error across 100 epochs; cases where NRF validation loss remains worse than RF retain the RF predictions.Source: §5.1 · pp. 24–25
- Fully connected NRF relaxes the sparse connectivity constraint, retaining tree-based initialization while allowing more weights to be optimized.The sparse variant optimizes fewer weights and retains the forest structure throughout optimization; fully connected optimization uses it as a warm start.Source: §5.1 · p. 26
5.2 Benchmark comparison experiments
- All NRF models improve on RF across the benchmark data sets, for both sparse and fully connected networks.Table 2 reports test-set RMSE and standard deviations across models and data sets.Source: §5.2 · p. 28, Table 2 · p. 28
- Aside from BART, fully connected NRF Method 1 achieves the best overall performance, and it usually outperforms Method 2.BART ranks first in 6 of the 7 experiments; Method 1 averages individually trained per-tree networks, while Method 2 jointly trains one large network.Source: §5.2 · pp. 28–29
- In the Concrete Compressive Strength example, Method 2's validation RMSE falls below RF's during training and is lower than a same-size NN's on training and validation sets.Figure 5 compares RF with NRF across epochs and contrasts NRF with a NN having the same number of layers and neurons.Source: §5.2 · pp. 29–30, Figure 5 · p. 29
5.3 Asymptotic behavior
- On synthetic data, fully connected Method 2's test RMSE decreases faster than RF's as training-set size increases.The figure reports means over three randomly drawn data sets per size, with empirical standard deviation error bars.Source: §5.3 · p. 30
6 Some technical results
The technical results establish bounds for the constrained network classes and use them in proofs of the paper's propositions. The arguments include a result about when a regression function must be constant on a cell and a bound that tends to zero under stated conditions.
6.1 Proof of Proposition 4.1
- The proof of Proposition 4.1 considers a deterministic class of two-hidden-layer networks with bounded outputs, using covering-number bounds to control the class.The class has hidden-layer sizes K_n−1 and K_n and contains the networks constrained by (6.1).Source: §6.1 · pp. 31–33
6.2 Proof of Proposition 4.2
- For Proposition 4.2, the derived upper bound tends to zero under the conditions of the proposition.Source: §6.2 · p. 37
6.3 Proof of Proposition 4.3
- In the Proposition 4.3 argument, if the cut criterion is zero throughout a nonempty cell, the regression function is constant there; the proof then obtains a quantity tending to zero.The stated assumptions include uniform X on [0, 1]^d, bounded Y, and r in F.Source: §6.3 · pp. 37–39