PaperlayerView exact revision on arXiv

Source-linked AI summary

Local Linear Forests

Rina Friedberg, Julie Tibshirani, Susan Athey, Stefan Wager

arXiv:1807.11408v4stat.MLcs.LGecon.EMmath.STstat.TH

TL;DR

Random forests can fit smooth signals poorly, motivating a method that combines forest neighborhoods with local linear regression. Local linear forests use this adjustment to improve prediction and confidence-interval performance, with the paper identifying conditions under which accuracy gains may be largest.

  • Problem

    Random forests can fail to exploit smoothness in regression surfaces, limiting their fit to strong local trends.

    Source: §1 · p. 2

  • Method

    Local linear forests pair forest-based neighborhoods with a local regression adjustment to address asymmetries and imbalances in nearby points.

    Source: §1 · pp. 3, 5

  • Results

    Local linear forests improve predictive performance and confidence-interval coverage over regression forests, while producing shorter intervals.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The authors expect the largest accuracy gains when some features have strong effects with moderate curvature.

    Source: §1 · p. 3

    With small samples and strong signals, requiring honesty may limit forest expressive power and hurt predictive performance.

    Source: §2.3 · p. 16

Abstract

from arXiv · show

Random forests are a powerful method for non-parametric regression, but are limited in their ability to fit smooth signals, and can show poor predictive performance in the presence of strong, smooth effects. Taking the perspective of random forests as an adaptive kernel method, we pair the forest kernel with a local linear regression adjustment to better capture smoothness. The resulting procedure, local linear forests, enables us to improve on asymptotic rates of convergence for random forests with smooth signals, and provides substantial gains in accuracy on both real and simulated data. We prove a central limit theorem valid under regularity conditions on the forest and smoothness constraints, and propose a computationally efficient construction for confidence intervals. Moving to a causal inference application, we discuss the merits of local regression adjustments for heterogeneous treatment effect estimation, and give an example on a dataset exploring the effect word choice has on attitudes to the social safety net. Last, we include simulation results on real and generated data.

1 Introduction

The paper pairs forest-based neighborhoods with local regression adjustments to better fit smooth signals, then develops theory and empirical examples for local linear forests. It reports improved prediction and confidence-interval performance, including a wage-regression application, and distinguishes its approach from earlier local-linear and tree-based methods.

1 Introduction

  • Random forests can miss strong local trends, producing step-like fits where the target is smooth.The motivating example simulates a response that varies smoothly with one feature.Source: §1 · p. 2
  • Local linear forests improve predictive performance and confidence-interval coverage over regression forests, with intervals that are also shorter.The paper’s central limit theorem supports pointwise Gaussian intervals; Figure 1 illustrates the comparison.Source: Figure 1 · p. 3, §1 · p. 4
  • The method extends to quantile regression and generalized random forests, with causal forests highlighted as an application.The paper also describes applicability to forests targeting heterogeneous estimating equations.Source: §1 · p. 4
  • Regularization and tree-splitting modifications appear important for the method’s performance; an earlier approach reported roughly 5% lower mean-squared error on the Friedman function.The paper reports larger, systematic improvements and says regularization is needed for its smoothness-based convergence guarantees.Source: §1 · pp. 4–5
  • The method combines forest adaptivity with local linear regression to capture smoothness and address imbalances in neighboring observations.The authors expect the largest gains when some features have strong effects with moderate curvature.Source: §1 · pp. 2, 3, 5

1.1 Empirical Example: Wage Regressions

  • In CPS wage prediction, paired tests on squared errors provide evidence that local linear forests improve on OLS, lasso, boosting, BART, and random forests.The comparison covers overall and sparse covariate regions; the table reports mean squared error across methods.Source: §1.1 · pp. 6–7, Table 1 · p. 7

1.2 Related Work

  • Unlike Bloniarz, Talwalkar, Yu, and Wu [2016], this method adapts tree splitting and uses features such as ridge penalization; the earlier work establishes consistency but not rates exploiting smoothness.The paper also says the predecessor does not provide a central limit theorem or confidence intervals.Source: §1.2 · pp. 9–10
  • Local linear forests use leaves to identify neighbors for regression adjustment, rather than treating leaf nodes as local prediction models.This distinguishes the method from related model-based tree approaches.Source: §1.2 · p. 10

2 Local Linear Forests

Local linear forests treat forest weights as an adaptive kernel and use them in a local linear regression, while residual-based splits reserve tree structure for effects the regression step does not model. The section compares splitting rules, describes honest forest construction and tuning, and explains how these choices affect performance.

2 Local Linear Forests

  • Local linear forests use random-forest weights as a kernel for locally weighted least squares, producing regression weights modulated to align better with the test point.The estimator uses a centered regression matrix and can include a ridge penalty; without the penalty, the stated moment alignment is exact.Source: §2 · pp. 11–12

2.1 Splitting for Local Regression

  • Residual splitting fits a ridge regression in each parent node, then splits on residuals so trees model local effects while the prediction step restores global effects.The split rule can be implemented with trees that split on every variable with nonzero probability and balance observations in child nodes.Source: §2.1 · pp. 12–13
  • CART splits repeatedly target the strongest linear signal, while local linear forests rarely split on those signals and instead allocate splits to harder-to-model variables.Figure 3 shows split frequencies by variable and tree depth for forests trained on 600 observations.Source: §2.1 · p. 13, Figure 3 · p. 14

2.2 The Value of Local Linear Splitting

  • As dimension and the number of global linear signals increase, random-forest RMSE rises dramatically and residual splitting gains value, but its advantage levels off when the fixed sample cannot fit more complex linear functions.Figure 4 compares prediction RMSE across dimensions for random forests and local linear forests with CART or ridge-residual splits.Source: §2.2 · p. 15, Figure 4 · p. 15

2.3 Honest Forests

  • Honest forests grow tree structure on one subsample and define prediction neighborhoods with another, helping control overfitting; honesty can also hurt prediction with small samples and strong signals.The theory assumes symmetric, regular trees that split on each variable with nonzero probability and balance parent observations between children.Source: §2.3 · p. 16

2.4 Tuning a Local Linear Forest

  • Ridge penalties should be cross-validated and may differ between forest growth and final prediction; the paper's tuning heuristics are outside its theoretical guarantees.Small growth-stage penalties and parent-node regression estimates for very small leaves help avoid over-regularization and unstable child-leaf coefficients.Source: §2.4 · p. 17

3 Extension to Causal Forests

The paper extends local linear adjustment to causal forests to strengthen estimation when treatment effects are smooth. In a welfare-wording example, local linear causal forests improve estimated performance across subsample sizes and show an approximately linear pattern in predicted treatment-effect heterogeneity.

3 Extension to Causal Forests

  • Local linear correction extends to generalized random forests, including causal forests, to improve performance when the conditional average treatment effect is smooth.The causal version adjusts the orthogonalized causal forest; its setup assumes unconfoundedness and overlap.Source: §3 · p. 18
  • Local linear causal forests estimate nuisance components, then fit the treatment effect with an intercept retained for robustness and cross-validated penalties selected by the R-learning criterion.The leave-one-out superscript denotes predictions from nuisance models; large penalties recover causal-forest estimates.Source: §3 · pp. 18–19

3.1 Empirical Example: Attitudes to Welfare

  • Local linear causal forests improve estimated treatment-effect error across subsample sizes in the welfare survey example, and predictions suggest an approximately linear heterogeneity pattern across political views and income.The example uses subsamples of a 28,646-observation dataset; Table 2 reports estimated mean squared error averaged over 200 runs at each subsample size.Source: §3.1 · pp. 20–21, Figure 5 · p. 20, Table 2 · p. 21

4 Asymptotic Theory

Local linear forests have a different dependence structure from standard random forests, so existing asymptotic arguments do not directly apply. Under stated forest and smoothness conditions, the paper establishes asymptotic normality with an improved subsampling rate.

4 Asymptotic Theory

  • Conditioning on features separates outcomes from forest weights for standard forests, but not the adjusted responses and weights in local linear forests.This difference breaks a key component of the argument used by Wager and Athey [2018].Source: §4 · pp. 21–22

4.1 Main Result

  • The theory assumes regular, honest trees and a leaf-distribution condition that rules out degenerate feature variation within leaves.Honesty uses separate subsamples for split selection and parameter estimation.Source: §4.1 · pp. 22–23
  • Subsampling tunes the bias–variance trade-off: smaller tree subsamples lower bias but increase variance, while larger subsamples do the reverse.The paper identifies subsampling as central to its asymptotic-normality argument, building on Efron and Stein [1981].Source: §4.1 · p. 23
  • With smooth µ, local regression permits smaller tree subsamples without sacrificing asymptotic variance control, reducing bias; the main result establishes asymptotic normality at an improved subsampling rate.The result uses the condition ω ≤ 0.2 to leverage part of the theory of Wager and Athey [2018].Source: §4.1 · p. 23

Appendix B.

Under regularity and smoothness conditions, the paper derives an improved convergence rate for local linear forests. It also develops pointwise confidence intervals using a forest-based variance estimate, while noting that the required undersmoothing can be difficult to tune.

Appendix B.

  • The best attainable convergence rate improves over the traditional honest-forest rate when smoothness is assumed and exploited through local linear regression.The theorem assumes uniform features on the unit cube, smooth conditional moments, and trees satisfying the stated assumptions.Source: Appendix B · pp. 23–24
  • The subsampling condition enforces undersmoothing, making estimator error variance-dominated and facilitating confidence intervals, but cross-validation does not guarantee this tuning.The authors leave inference methods that do not rely on undersmoothing beyond the paper’s scope.Source: Appendix B · p. 24

4.2 Pointwise Confidence Intervals

  • Pointwise intervals use the random-forest delta method and estimate variance with a computationally efficient bootstrap-of-little-bags half-sampling procedure.The full computation is impractical, and naive Monte Carlo approximations suffer from bias; bias-corrected randomized algorithms are available but not detailed here.Source: §4.2 · pp. 25–27
  • For intervals on µ(x0), the variance calculation needs only the upper-left entry of the relevant estimating-equation matrix, which can be read from the forest.The paper calls this entry σ̂²_n.Source: §4.2 · p. 26

5 Simulation Study

The simulations compare local linear forests with forest and other regression baselines across smooth, linear-trend, and near-step signals. Local linear forests perform well across several settings, with confidence-interval behavior and relative performance varying by signal and sample size.

5.1 Methods

  • The study compares local linear forests with random forests, BART, gradient boosting, and a lasso-random-forest residual baseline, using cross-validation for tuning where specified.Forests use honest construction; local linear regression is omitted from the high-dimensional comparisons because loess and locfit fail for d > 6 in this simulation.Source: §5.1 · pp. 27–28

5.2 Simulation Design

  • The simulation designs include Friedman’s model across dimensions, sample sizes, and noise levels; a strong-linear-trend setup; and a near-step function intended to favor random forests.The near-step comparison uses out-of-bag RMSE, confidence-interval coverage, and length for d = 5 and 20 and n = 500, 2000, and 10000.Source: §5.2 · p. 28

5.3 Results

  • Local linear forests consistently perform well across dimensions and noise levels in the Friedman simulation; lasso-random forest is close in high noise, while BART and boosting fare worse there.At low noise, BART and random forests form the next tier; at high noise, honest random forests cluster with local linear forests.Source: §5.3 · p. 29, Figure 6 · p. 29
  • For equation 1’s strong linear trend, local linear forests model the signal across the feature space, while boosted trees’ performance sharply declines as noise increases.The full comparison is reported in Appendix A.Source: §5.3 · p. 30
  • On equation 1, local linear forest intervals are shorter and closer to 95% coverage than random-forest intervals, while both methods have coverage at or above 88%.On equation 7, neither method exceeds 75% coverage, and local linear forest intervals are longer; the authors note asymptotic intervals may perform poorly at relatively low n.Source: Table 3 · p. 30, §5.3 · p. 31
  • On the approximate step-function setup designed to favor random forests, local linear forests achieve equivalent or better coverage, with longer intervals in low dimensions.They also improve on random-forest RMSE, especially with small training datasets.Source: §5.3 · p. 31

5.4 Local Linear Causal Forests

  • In the heterogeneous-treatment-effect simulation, local linear forests significantly improve on causal forests’ RMSE across the reported evaluation.The comparison uses 100 repetitions and reports results in Table 4.Source: Table 4 · p. 32, §5.4 · p. 33

6 Discussion

The discussion presents local linear forests as a forest modification for smooth signals and boundary bias, with regression adjustments that can center predictions when neighboring observations are asymmetric. It highlights settings where the adjustment is useful and identifies extensions not yet studied.

6 Discussion

  • Local linear forests are proposed to model smooth signals and address boundary bias, with theory showing better convergence rates than generalized random forests under smoother-signal assumptions.The paper also reports modeling smooth heterogeneous causal effects on the welfare dataset and delta-method confidence intervals for regression.Source: §6 · p. 33
  • Regression adjustments are especially useful for strong global effects with moderate curvature and center predictions by correcting errors from asymmetric neighborhoods.The authors note that a hand-tuned polynomial basis may not suit mixtures of smooth and non-smooth signals.Source: §6 · p. 33
  • The paper does not explore estimating slope parameters, and extending pointwise theory is needed to obtain uniform confidence bands.The authors identify both directions as potentially important future work.Source: §6 · p. 33
  • Theoretical and empirical gains from assuming higher-order smoothness remain unexamined.Source: §6 · p. 34

A Remaining Simulation Results

The simulations compare local linear forests with several regression methods across Friedman and low-dimensional models. In the simple linear case lasso performs best, while local linear regression performs best in the polynomial case.

A Remaining Simulation Results

B Proof of Theorem 1

The proof decomposes prediction error into variance and bias components, controls curvature and regularization bias, and establishes that a centered Gaussian variance term dominates under the stated conditions.

B Proof of Theorem 1

  • The prediction error is decomposed into a variance term, a regularization-bias term, and a curvature-bias term.The curvature contribution is bounded on the order of the average squared leaf radius.Source: B Proof of Theorem 1 · pp. 43–44
  • The average squared leaf radius is O_p(r_s^2) under the proof’s regular random-split tree conditions.The bound follows after controlling typical leaf radii and applying Markov’s inequality twice.Source: B Proof of Theorem 1 · p. 44
  • Without regularization, the local linear correction removes slope bias, whereas as λ grows without bound the estimator approaches a regression forest with bias of order the average leaf radius.The proof requires positive regularization for other parts of the argument.Source: B Proof of Theorem 1 · pp. 44–45
  • The transformed-outcome regression-forest term is asymptotically centered Gaussian and dominates prediction error at the stated scale.The proof uses conditional independence of the transformed outcomes and forest weights; the remaining error has a polynomial gap from the dominant term.Source: B Proof of Theorem 1 · pp. 46–47

Related papers

Loading 1807.11408v4…