PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalized Random Forests

Susan Athey, Julie Tibshirani, Stefan Wager

arXiv:1610.01271v4stat.MEecon.EMstat.ML

TL;DR

The paper addresses how to estimate heterogeneous quantities defined by local moment equations without relying on kernel weights vulnerable to high-dimensional settings. It develops adaptive, gradient-split generalized random forests, proves consistency and asymptotic Gaussianity, and applies them to several statistical tasks.

  • Problem

    Kernel-based local weighting can be sensitive to the curse of dimensionality, while standard forests primarily target conditional mean estimation rather than general local moment parameters.

  • Method

    Generalized random forests learn forest-based adaptive weights and use gradient-based splitting to target heterogeneity in any parameter identified by local moment conditions.

  • Results

    The framework provides formal asymptotic guarantees and supports quantile regression, conditional average partial effect estimation, and heterogeneous treatment effect estimation with instrumental variables.

  • Takeaways & Limitations

    Generalized random forests offer a flexible non-parametric estimation and inference framework spanning multiple statistical models and applications.

  • Takeaways & Limitations

    Valid confidence-interval coverage relies on undersmoothing, and avoiding this requirement through principled bias correction remains an open challenge.

Abstract

from arXiv · show

We propose generalized random forests, a method for non-parametric statistical estimation based on random forests (Breiman, 2001) that can be used to fit any quantity of interest identified as the solution to a set of local moment equations. Following the literature on local maximum likelihood estimation, our method considers a weighted set of nearby training examples; however, instead of using classical kernel weighting functions that are prone to a strong curse of dimensionality, we use an adaptive weighting function derived from a forest designed to express heterogeneity in the specified quantity of interest. We propose a flexible, computationally efficient algorithm for growing generalized random forests, develop a large sample theory for our method showing that our estimates are consistent and asymptotically Gaussian, and provide an estimator for their asymptotic variance that enables valid confidence intervals. We use our approach to develop new methods for three statistical tasks: non-parametric quantile regression, conditional average partial effect estimation, and heterogeneous treatment effect estimation via instrumental variables. A software implementation, grf for R and C++, is available from CRAN.

1. Introduction.

Generalized random forests extend random forests from conditional mean estimation to any quantity identified by local moment equations. They replace fixed kernel weights with adaptive forest weights and provide problem-specific splitting, asymptotic guarantees, and broad applications.

  • Motivation and framework: The method estimates θ(x) defined by a local moment equation involving a scoring function and optional nuisance parameter.This setup covers conditional means, quantiles, average partial effects, robust regression, and instrumental-variables treatment effects.
  • Methodological contribution: The framework targets heterogeneity in the parameter of interest through unified, gradient-based problem-specific splitting rules.The procedure approximates the estimating equation with gradients and uses resulting pseudo-outcomes to guide tree splits.
  • Theory and applications: Generalized random forests provide a flexible non-parametric estimation and inference framework with formal asymptotic guarantees.The paper develops consistency, Gaussian asymptotics, and methodology for asymptotic confidence intervals.
  • Theory and applications: Applications include non-parametric quantile regression, conditional average partial effects, and heterogeneous treatment effects with instrumental variables.The framework is also presented as applicable to panel regression, robust regression, and consumer-choice models.
  • Motivation and framework: Kernel-based local estimators can be sensitive to the curse of dimensionality, motivating adaptive forest-based neighborhood weights.The weights are learned from recursive partitions rather than specified by a deterministic kernel.

2. Generalized Random Forests.

Generalized random forests construct adaptive neighborhoods by averaging tree-based leaf co-membership weights, then solve weighted local estimating equations. Gradient-based splitting is designed to expose parameter heterogeneity while preserving the forest framework and enabling formal split-quality analysis.

  • Forest-based local estimation: The estimator solves a weighted empirical estimating equation using observations selected by the forest-based adaptive neighborhood.This replaces deterministic kernel weighting with weights learned from recursive partitions.
  • Forest-based local estimation: For each target x, forest weights measure how frequently each training observation falls in the same leaf as x.Each tree assigns equal positive weight within the target’s leaf and zero weight elsewhere; averaging across trees yields weights summing to one.
  • Forest-based local estimation: In the least-squares special case, the weighting formulation is equivalent to the standard average-of-regression-trees perspective.The weighted estimating equation recovers the usual CART tree prediction when applied to conditional mean estimation.
  • Splitting to maximize heterogeneity: Splits are chosen to improve child-node estimates of the target parameter, with the criterion favoring increases in estimated heterogeneity.The theoretical split criterion decomposes prediction error into a parent-purity term minus expected between-child estimate differences, up to sampling terms.
  • Splitting to maximize heterogeneity: The approximate gradient-tree split criterion is asymptotically close to the exact criterion under consistency.The paper argues that using the approximate criterion should not create substantial inefficiency relative to the exact criterion.
  • Statistical construction: Subsampling and honesty are used to support the statistical behavior of generalized random forests.The construction adapts established forest techniques while tailoring splits to heterogeneous estimating equations.

3. Asymptotic Analysis.

The asymptotic analysis establishes consistency and asymptotic normality for generalized random forest estimates under regularity, convexity, and honest-tree conditions. These results rely on explicit assumptions governing the signal, scores, solutions, and tree construction.

  • Assumptions: The theory assumes Lipschitz variation of the population score in covariates and smooth, identifiable parameter dependence.The identification derivative must be invertible, while the score covariance structure and score representation satisfy regularity conditions.
  • Assumptions: Existence, convexity, and regularity assumptions ensure that weighted estimating equations have suitable solutions and stable population behavior.The framework covers settings such as instrumental-variables regression and quantile regression under these conditions.
  • Tree conditions: The required forests are honest, balanced, symmetric, randomized, and built with subsamples satisfying the specified growth conditions.Randomized feature selection supports the minimum split-probability condition used in the theory.
  • Examples: Quantile regression satisfies the framework when conditional distributions and densities meet the stated smoothness and positivity conditions.The paper verifies the assumptions using the quantile score and conditional distribution function.
  • Asymptotic results: Under Assumptions 1–6 and the specified forest construction, generalized random forest estimates converge in probability to the target parameter.Theorem 3 states consistency for both the target and nuisance parameter estimates.
  • Asymptotic results: The estimator and its coupled oracle analogue are asymptotically normal under the stated assumptions and subsample scaling.The coupling result separates consistency from the local approximation needed for the Gaussian limit.

4. Confidence Intervals via the Delta Method.

The paper estimates forest-score variance through a half-sampling target and a computationally feasible bootstrap of little bags, enabling asymptotically valid confidence intervals under consistent curvature estimation.

  • Variance estimation: The confidence-interval construction requires estimating the noise scale of the forest estimator, then combines it with a consistent curvature estimator.The curvature term is problem-specific; for quantile regression it equals f_x(θ(x)).
  • Variance estimation: The forest score’s inner variance is treated as a regression-forest variance problem with effective outcomes given by the forest scores.Because the score depends on unknown parameters, the paper uses plug-in estimates and a noisy bootstrap procedure.
  • Bootstrap of little bags: The ideal half-sampling variance estimator is approximated by grouping trees into little bags built from random half-samples.The resulting estimator uses between-group and within-group variance terms.
  • Bootstrap of little bags: With sufficiently many trees, the feasible bootstrap-of-little-bags estimator matches the ideal half-sampling estimator.This makes the variance calculation computationally practical without growing the very many forests required for direct half-sampling.
  • Validity: Under the theorem’s conditions, the half-sampling estimator consistently estimates the score variance, and Gaussian confidence intervals achieve nominal asymptotic coverage.The result requires a consistent estimator of the problem-specific curvature matrix.
  • Implementation caveat: When the number of trees is small, empirical variance estimates may be negative, so the software uses a Bayesian analysis-of-variance adjustment.The adjustment becomes immaterial when the number of trees is large enough.

5. Application: Quantile Regression Forests.

The quantile-regression application uses gradient-based forest splitting to target quantile heterogeneity directly. In the reported example, this preserves performance under scale shifts where regression-based splits fail, while honesty produces smoother paths.

  • Quantile-regression setup: Quantile regression identifies the conditional q-th quantile through a moment function based on whether observations exceed a candidate threshold.The application estimates q = 0.1, 0.5, and 0.9 quantiles.
  • Quantile-regression splitting: Gradient-based quantile trees label observations by whether they lie above or below the parent node’s q-th quantile.Scaling and recentering the pseudo-outcomes do not affect the subsequent regression split.
  • Figure 2 comparison: In the scale-shift example, generalized random forests perform well because their splitting rule targets quantile changes, whereas Meinshausen’s method breaks down.The conditional mean remains constant while the distributional scale changes at (X_i)_1 = 0.
  • Figure 2 comparison: In the mean-shift example, both generalized random forests and Meinshausen’s method detect the change.The comparison uses n = 2,000 observations and p = 40 covariates.
  • Figure 2 comparison: Honesty yields somewhat smoother generalized-random-forest sample paths, while removing honesty produces local oscillations similar to the baseline.Without honesty, the method still identifies the jumps at x = 0.
  • Multiple quantiles: For multiple quantiles, the paper proposes one forest using interval labels and a multiclass classification split rather than separate forests.This addresses computational cost and the risk that separately estimated quantiles cross in finite samples.

6. Application: Estimating Conditional Average Partial Effects.

The paper applies generalized random forests to estimate heterogeneous treatment effects under unconfoundedness, using forest weights and local centering to improve robustness to confounding. Simulations suggest centered generalized random forests perform well across settings combining treatment-effect heterogeneity and confounding.

  • Estimation framework: Generalized random forests estimate conditional treatment effects as local moment-equation solutions using adaptive forest weights.The framework targets θ(x) = ξ · β(x), where β(x) is the conditional average treatment effect for binary treatment assignment.
  • Local Centering: Local centering residualizes outcomes and treatments before forest fitting, potentially improving robustness when weights are not sharply concentrated around x.The procedure uses leave-one-out estimates of marginal expectations and applies the forest to centered outcomes and treatments.
  • Interpretation: Generalized random forests provide a common framework for reconciling sensitivity to treatment-effect heterogeneity with robustness to confounding.The paper motivates this flexibility by contrasting the hard-coded strengths of the two comparison procedures.
  • Simulation results: Centered generalized random forests perform well under both treatment heterogeneity and confounding, including settings combining both features.The simulation compares centered and uncentered generalized random forests with two procedures from Wager and Athey (2018).
  • Simulation results: Procedure 1 of Wager and Athey performs poorly under pure confounding, whereas Procedure 2 performs poorly under strong treatment heterogeneity.The reported pattern matches the respective design strengths of the two procedures.

7. Application: Instrumental Variables Regression.

The paper uses generalized random forests with instrumental variables to estimate heterogeneous causal effects when treatment and outcome are confounded. In the child-rearing application, effects are estimated across maternal and socioeconomic characteristics, with results suggesting stronger effects among mothers whose husbands have lower incomes.

  • Motivation: Instrumental variables regression addresses causal-effect estimation when treatment and outcome are linked through non-causal pathways.An auxiliary source of randomness identifies causal effects without relying on treatment exogeneity.
  • Heterogeneous IV effects: Generalized random forests extend instrumental variables estimation from global effects to heterogeneous conditional local average treatment effects.The application studies how the causal effect of child rearing varies with maternal age and socioeconomic status.
  • Estimation procedure: The instrumental variables forest constructs pseudo-outcomes for recursive partitioning and solves a weighted estimating equation to obtain personalized treatment-effect estimates.The procedure uses parent-node moments and forest weights, with local centering applied in the implementation.
  • Data and instrument: The application analyzes 334,535 married mothers with at least two children using 1980 census data and the same-sex instrument.The instrument exploits differences in subsequent family size between parents whose first two children have mixed versus same sexes.
  • Results: The estimated treatment effect appears driven by mothers whose husbands have lower incomes.Figure 3 varies maternal age at first birth and father’s income while holding other covariates at median values.
  • Caveats: The authors caution that census-measured father’s income may be endogenous, and Figure 3 confidence intervals are rather wide.They would prefer income measured when the second child was born, but that measurement is unavailable.

8. Discussion.

The discussion presents generalized random forests as versatile adaptive local estimators while identifying unresolved challenges for inference and boundary behavior.

  • Scope: Generalized random forests provide adaptive local estimation across a wide variety of statistical models.The paper discusses applications including quantile regression and heterogeneous treatment effects, with potential extensions to demand estimation and panel data.
  • Inference: Valid confidence-interval coverage relies on undersmoothing because the construction otherwise does not capture forest-estimation bias.The intervals account for sampling variability without undersmoothing, but not bias.
  • Open challenges: Principled bias correction could eliminate the need for undersmoothing in generalized-random-forest confidence intervals.The paper identifies this as an important conceptual and practical direction for future work.
  • Open challenges: Forest estimates can exhibit edge effects, with estimated slopes tapering near the boundary even when the true function continues changing.Addressing edge effects could improve the resulting confidence intervals.

APPENDIX A: PROOF OF MAIN RESULTS

The appendix develops technical ingredients for the paper’s main central limit theorem by controlling forest weights, moments, localization, and variance under honest subsampling.

  • Proof strategy: The appendix’s main theoretical target is the central limit theorem stated in Theorem 5.The proof sequence begins with technical lemmas and propositions supporting the main result.
  • Moment bounds: Lipschitz continuity of the M-function helps establish bounds on forest-weighted moment expressions.The stated lemma applies to weights obtained from forests under the relevant specification.
  • Localization: Forest localization follows from prior results for honest subsampled trees and supports control of local approximation error.The appendix derives localization of α_i(x) and uses it with the Lipschitz assumption.
  • Variance control: Honest subsampling and U-statistic arguments bound forest-score variance by s/n times the variance of a tree score.The variance relationship is stated as a consequence of subsampling and classical U-statistic results.
  • Sample-path control: A local discrepancy measure is introduced to control the sample paths of the forest score before proving the main results.This provides additional control over how the score varies locally.

A.1. Local Regularity of Forests.

This section establishes local concentration and approximation properties needed to analyze generalized random-forest estimates. Under stated regularity and entropy conditions, the estimates achieve consistency and support asymptotic Gaussian analysis.

  • Local concentration: Uniform concentration of the forest is established by controlling local fluctuations through bracketing entropy and empirical-process bounds.The argument combines concentration for a single point with uniform control over parameter neighborhoods.
  • Regularity conditions: The forest weights and estimating equations are analyzed using local smoothness, Lipschitz continuity, convexity, and strong-convexity conditions.These conditions control Taylor expansions and ensure approximate solutions remain close to one another.
  • Asymptotic behavior: The approximating estimator inherits a central limit theorem from regression-forest theory, while the coupling error decays faster than its stochastic scale.This establishes the asymptotic Gaussian behavior of the generalized random-forest estimator.
  • Inference: Half-sampling and related variance arguments extend the asymptotic analysis toward valid inference for the generalized estimator.The analysis separates oracle half-sampling behavior from negligible plug-in effects.

APPENDIX B: TECHNICAL RESULTS

These technical results establish concentration, influence-function approximations, and variance controls for generalized random forests. They also quantify errors from splitting and show when overfitting terms become negligible.

  • Leaf approximations: The actual leaf estimator is coupled to a gradient-based approximation and an average influence-function representation around the parent-node center.The leaf radius and sample size determine the approximation error and consistency rate.
  • Split evaluation: The split-error decomposition separates split-independent variation, sampling variance, and overfitting to random noise.The overfitting term is OP(1/nC1, 1/nC2) and can be ignored when nP ≫ r^-2.
  • Forest structure: Honest forests make the weights conditionally independent of outcomes, yielding mean-zero fluctuation terms and enabling U-statistic arguments.These properties support the variance and concentration bounds used in the technical proofs.
  • Concentration: Bracketing and Bernstein-type bounds control the empirical-process fluctuations of the forest at the required rate.The analysis uses entropy bounds for the kernel and an entropy integral to obtain concentration.
  • Simulation setup: Figure 5 uses n = 10, 000 observations and p = 20 covariates in both simulation panels.The simulations study instrumental-variables forests under these common design dimensions.

C.1. Evaluating the Instrumental Variables Splitting Rule.

The instrumental-variables splitting rule is evaluated in simulations where causal-effect jumps can be obscured or accompanied by non-causal correlation changes. IV forests identify the causal structure more reliably than exogeneity-based alternatives.

  • Endogeneity: IV forests identify a causal-effect jump while ignoring a spurious jump caused by changing correlation between treatment and outcome errors.A plain causal forest incorrectly detects the non-causal jump because it assumes the received treatment is exogenous.
  • Masked heterogeneity: IV forests detect a causal-effect jump even when treatment and outcome observables have a joint distribution that does not vary with covariates.A causal-tree split designed without instrumental variables fails to detect this masked jump.

C.2. Numerical Comparisons.

Numerical comparisons examine generalized random forests against nearest-neighbor and series instrumental-variables estimators across confounding, nuisance, sparsity, and dimensionality settings. Forest methods perform consistently well, while centering helps when nuisance structure is present and series methods excel in a narrow additive low-dimensional regime.

  • Simulation factors: The simulations vary confounding, signal sparsity, nuisance terms, ambient dimension p, and sample size n.Confounding is toggled using ω, while signal sparsity is represented by κτ ∈ {2, 4}.
  • Overall comparison: Forest-based methods achieve consistently good performance across varied simulation designs, including non-additive signals and fairly strong confounding.The comparison includes nearest-neighbor IV, series IV, plain generalized random forests, and centered generalized random forests.
  • Centering: When nuisance structure is absent, centered and uncentered forests perform comparably; with nuisance terms, centering substantially improves performance.Centering uses residualization to reduce the influence of nuisance components.
  • Series-method advantage: Series estimation substantially improves over generalized random forests only when the signal is additive, p = 10, and κτ = 4.These settings offer relatively little advantage to adaptively learning a sparse neighborhood function.
  • Confidence intervals: The study evaluates confidence-interval coverage using centered forests under confounding and nuisance terms.Coverage results are reported for a subset of the preceding simulation settings.

C.3. Evaluating Confidence Intervals.

The confidence-interval evaluation varies simulation complexity and examines both empirical coverage and whether intervals quantify uncertainty around forest predictions. Coverage is closer to nominal for sparser treatment-effect signals but declines for less sparse signals, with low coverage attributed mainly to forest bias.

  • Coverage improves with larger n, smaller p, sparser signals, and additive true signals.These are the main simulation conditions associated with better coverage results.
  • The simulations do not perfectly satisfy the theoretical Lipschitz assumptions, although this mismatch does not appear to hurt performance.The discrepancy arises after mapping features into the unit cube via a monotone transformation.
  • The study focuses on whether generalized random forests improve over non-adaptive baselines and leaves a thorough lasso-versus-forest comparison beyond its scope.The paper characterizes lasso- and forest-based methods as complementary in traditional regression settings.
  • Table 3 reports empirical coverage of 95% confidence intervals for instrumental variables forests across simulation replications.Coverage is evaluated for true treatment effects, while a second panel measures coverage of expected forest predictions.
  • When κτ = 2, confidence intervals achieve closer to nominal coverage; when κτ = 4, performance declines considerably at the investigated sample sizes.The paper identifies treatment-effect sparsity as the most important factor in Table 3.
  • Low coverage in the true-effect panel is attributed mainly to non-negligible forest bias rather than failures of Gaussianity or variance estimation.The comparison with coverage of expected forest predictions is used to assess the source of the discrepancy.
Loading 1610.01271v4…