Source-linked AI summary
Orthogonal Random Forest for Causal Inference
Miruna Oprescu, Vasilis Syrgkanis, Zhiwei Steven Wu
TL;DR
Prior approaches separate nonparametric target estimation from support for high-dimensional nuisance functions, despite the need to control confounders in observational HTE estimation. ORF combines Neyman orthogonality and random forests for nonparametric estimation with high-dimensional nuisance functions. The paper reports that it outperforms evaluated benchmarks under theory-guided parameter settings and provides asymptotic inference.
Problem
Prior work either restricts target functions to parametric forms when allowing high-dimensional nuisance functions or estimates targets nonparametrically without allowing high-dimensional nuisance functions.
Source: §1 · p. 1Method
ORF combines Neyman-orthogonal moments with random-forest estimation, and uses locally residualized moments for heterogeneous treatment effects.
Source: §1 · pp. 1–2Results
ORF consistently outperforms the evaluated benchmarks with theory-guided parameters, and bootstrap confidence intervals show good finite-sample coverage.
Source: §1 · p. 3Takeaways & Limitations
ORF permits nonparametric heterogeneous-effect estimation while controlling for high-dimensional variables, including when the target depends on only a small set of variables.
Source: §1 · p. 2The stated normality result’s special case assumes a locally parameterized nuisance function identified by local loss, and the procedure requires local orthogonality.
Source: §2 · p. 3
Abstract
from arXiv · showhide
We propose the orthogonal random forest, an algorithm that combines Neyman-orthogonality to reduce sensitivity with respect to estimation error of nuisance parameters with generalized random forests (Athey et al., 2017)--a flexible non-parametric method for statistical estimation of conditional moment models using random forests. We provide a consistency rate and establish asymptotic normality for our estimator. We show that under mild assumptions on the consistency rate of the nuisance estimator, we can achieve the same error rate as an oracle with a priori knowledge of these nuisance parameters. We show that when the nuisance functions have a locally sparse parametrization, then a local $\ell_1$-penalized regression achieves the required rate. We apply our method to estimate heterogeneous treatment effects from observational data with discrete treatments or continuous treatments, and we show that, unlike prior work, our method provably allows to control for a high-dimensional set of variables under standard sparsity conditions. We also provide a comprehensive empirical evaluation of our algorithm on both synthetic and real data.
1. Introduction
The paper addresses the gap between methods that allow high-dimensional nuisance functions but restrict the target and flexible nonparametric methods with low-dimensional assumptions. It proposes ORF for nonparametric heterogeneous-effect estimation with high-dimensional controls and reports favorable benchmark results and an orange-juice application.
1. Introduction
- Prior approaches either use parametric target functions with potentially high-dimensional nuisance functions or estimate targets nonparametrically without allowing high-dimensional nuisance functions.In observational HTE estimation, omitting potential confounders can bias direct regression.Source: §1 · pp. 1–2
- ORF combines Neyman orthogonality with random-forest estimation to estimate nonparametric targets while permitting high-dimensional nuisance functions.The paper also presents Forest Lasso for locally sparse nuisance parameters and asymptotic normality for valid confidence intervals.Source: §1 · p. 1
- For heterogeneous treatment effects, ORF supports nonparametric estimation with many controls when the target depends on only a small set of variables.The paper identifies contextual pricing and personalized medicine as examples of this setting.Source: §1 · p. 2
- The HTE approach uses locally residualized orthogonal moments, providing a theoretical foundation for local centering and differing from global residualization before forest estimation.The authors report that experiments reinforce the intuition for local residualization.Source: §1 · p. 2
- ORF consistently outperforms the evaluated benchmarks when parameters follow the theory, and bootstrap confidence intervals show good finite-sample coverage.On Dominick’s orange-juice data, the estimated demand elasticity indicates that lower-income consumers are more price-sensitive.Source: §1 · p. 3, Figure 1 · p. 3
2. Estimation via Local Orthogonal Moments
This section formulates estimation of a target parameter through local conditional moments with nuisance functions. Its two-stage procedure estimates nuisance functions on one sample and solves weighted moments near the target feature, relying on local orthogonality for robustness to nuisance estimation error.
2. Estimation via Local Orthogonal Moments
- The target θ0(x) is defined as the unique solution to a conditional moment equation at each feature value x.The moment is the conditional expectation of a score depending on the observation, target parameter, and nuisance function.Source: §2 · p. 3
- The two-stage procedure estimates the nuisance function on one half of the data, then uses feature-similarity weights and the nuisance estimate to solve a plug-in moment condition.The framework assumes fixed dimensions for p, ℓ, and d, while the dimension of W may grow with sample size.Source: §2 · p. 3
- For the normality result’s special case, the nuisance function is locally parameterized as h(x,w)=g(w;ν(x)), with ν(x) identified by minimizing a local loss.The framework permits any first-stage nuisance estimator; locally weighted penalized estimation is one option, with Forest Lasso guarantees when ν0(x) is sparse.Source: §2 · p. 3
- Local orthogonality requires the score’s derivative with respect to nuisance perturbations around their true values to vanish at each target x.This is the key technical condition for reliable two-stage estimation.Source: §2 · pp. 3–4
3. Orthogonal Random Forest
ORF builds forest-based similarity weights while incorporating nuisance estimation and orthogonalization into its tree-splitting criterion. Its trees use honest, balanced, randomized splits and minimum leaf sizes; the splitting criterion is presented as a practical heuristic.
3. Orthogonal Random Forest
- ORF averages tree-based weights from random subsamples to estimate similarity between observations and a target feature.The weights are used in the second stage of the two-stage estimator.Source: §3 · p. 4
- Trees use honesty, ρ-balanced splits, minimum leaf sizes, and π-random splitting across features.These properties govern how observations are used for split placement and estimation, and how splits distribute observations across features.Source: §3 · p. 4
- The tree learner incorporates orthogonal nuisance estimation into its splitting criterion, a practical heuristic that is not part of the theoretical analysis.At each node, nuisance estimation is performed using examples reaching that node.Source: §3 · p. 4
- Candidate splits are scored using a proxy for child-node heterogeneity, combining mean and maximum coordinate scores with a randomly chosen weight.The proxy uses a Newton step from the parent-node estimate rather than computing child estimates for every candidate split.Source: §3 · p. 4
- The leaves containing target feature x determine tree weights, which are then combined into ORF weights for each observation.The method relies on tree weights being concentrated near x under the stated forest structure.Source: §3 · pp. 4–5
4. Convergence and Asymptotic Analysis
The analysis gives error guarantees and asymptotic normality results for ORF under regularity conditions and conditions on nuisance estimation. The asymptotic-normality result is restricted to locally parametric nuisance functions and imposes additional conditions on their estimates.
4. Convergence and Asymptotic Analysis
- The analysis assumes local orthogonality, identification, smoothness, curvature, score regularity, boundedness, and full support for X.The paper describes all listed conditions except local orthogonality as standard boundedness and regularity conditions on the moments.Source: §4 · p. 5
- The paper states Lq and high-probability error bounds, with the latter requiring a convex-loss score and a bound on nuisance estimation error.The supplied theorem statements do not include the resulting bound expressions.Source: §4 · p. 5
- Asymptotic normality is established for locally parametric nuisance functions when their estimates and subsample size satisfy additional conditions.The result also assumes the conditional variance is Lipschitz and nonzero in the stated directions; the paper notes that valid confidence intervals can then be built using Bootstrap of Little Bags.Source: §4 · p. 5
5. Nuisance Estimation: Forest Lasso
The Forest Lasso estimates nuisance functions with locally sparse parameterizations using ORF weights and local ℓ1-penalized estimation. The paper provides parameter-recovery guarantees under sparsity, smoothness, and conditional restricted-eigenvalue assumptions.
5. Nuisance Estimation: Forest Lasso
- Forest Lasso uses ORF weights to fit a local ℓ1-penalized M-estimator for the nuisance parameters.The nuisance functions are represented as h(x, w) = g(w; ν(x)), with ν potentially growing in dimension.Source: §5 · p. 6
- The nuisance-estimation guarantees assume sparse, smooth local parameters and a conditional restricted eigenvalue condition.The parameter is k-sparse at each x, and the loss curvature is bounded below on directions in a cone around its local support.Source: §5 · p. 6
- Under these assumptions, the local penalized estimator has a parameter-recovery guarantee; examples include locally sparse linear and logistic regression.For linear regression, the conditional restricted eigenvalue condition becomes a conditional covariance condition.Source: §5 · p. 6
6. Heterogeneous Treatment Effects
The paper applies ORF to heterogeneous treatment effects by constructing orthogonalized moments and estimating nuisance functions. It treats continuous and discrete treatments, including a doubly robust moment for discrete treatments when treatment residuals may correlate with the outcome component.
6. Heterogeneous Treatment Effects
- The treatment-effect model distinguishes treatment T, outcome Y, controls W, and features X that capture heterogeneity; the target is CATE conditional on X.The model includes real-valued or discrete treatments and potential confounding controls.Source: §6 · p. 6
- Residualizing treatment and outcome yields an orthogonalized score for estimating θ(x) locally, under stated conditional independence and invertibility conditions.The score satisfies local orthogonality, while formal nuisance guarantees require restricting nuisance-function forms.Source: §6 · pp. 6–7
- For real-valued treatments, locally sparse linear nuisance functions can be estimated with Forest Lasso to obtain ORF error-rate and asymptotic-normality guarantees.The described construction represents q0 using degree-2 polynomial features of W.Source: §6 · p. 7
- For discrete treatments, the paper describes nuisance estimation using forest logistic Lasso and Forest Lasso under a restriction on μ0.The restriction makes μ0 a function of X alone, supporting the stated conditional independence between μ0 and η.Source: §6 · p. 7
- An alternative doubly robust moment identifies CATE for discrete treatments even when μ depends on W and η is correlated with μ.The approach constructs counterfactual outcome estimates for each potential treatment and can yield asymptotic normality under locally sparse linear or logistic nuisance parameterizations.Source: §6 · p. 7
- Figure 2 plots bias, variance, and RMSE against support size for a piecewise linear treatment-response function.Lines show means across Monte Carlo experiments and test points; shaded regions show standard deviations scaled down by 3.Source: Figure 2 · p. 8
7. Monte Carlo Experiments
The Monte Carlo results favor ORF-CV and ORF over GRF-Res methods, while the appendix passages state conditions for a real-valued-treatment guarantee and outline a U-statistic concentration proof.
7. Monte Carlo Experiments
- ORF-CV, ORF, and GRF-Res-Lasso best capture the piecewise-linear treatment effect, with GRF-Res-Lasso showing larger bias and variance.GRF-Res-RF has consistent bias, while HeteroDML methods do not capture the treatment-effect function’s complexity.Source: §7 · p. 9, Figure 3 · p. 9
- Bootstrap 1%-99% confidence intervals contain approximately 90% of sampled test points in the reported example.The intervals use 1,000 bootstrap iterations.Source: Figure 4 · p. 9
- ORF-CV and ORF outperform GRF-Res methods on all metrics in the example; ORF-CV’s bias, variance, and RMSE remain nearly constant as support size increases.The same performance pattern is reported for other support sizes, sample sizes, and treatment-effect functions.Source: §7 · p. 9
A. Two-forest ORF v.s. One-forest ORF
- ORF-NS bias, variance, and RMSE rise sharply with subsample ratio, unlike ORF-CV, consistent with sample collisions causing spill-over bias and error.ORF-NS uses one forest and shared importance weights; ORF-CV uses two forests.Source: A. Two-forest ORF v.s. One-forest ORF · p. 12
B. Formal Guarantee for Real-Valued Treatments
- The real-valued-treatment guarantee assumes Lipschitz nuisance components with bounded ℓ1 norms, full support for X, and conditional covariance matrices bounded away from singularity.The passage also states an ORF subsampling rate of s = Θ(n^αd/(1+αd)).Source: B. Formal Guarantee for Real-Valued… · p. 12
C. Uniform Convergence of Lipschitz U-Processes
- The U-statistic proof applies Bernstein concentration at fixed parameters, then uses Lipschitz control, an ε-net, and a union bound.The lemma setup assumes a bounded parameter space and a Lipschitz function with bounded second moment and supremum.Source: C. Uniform Convergence of Lipschitz… · pp. 12–13
D. Estimation Error and Asymptotic Normality
This section develops ORF error and limit theory, controlling nuisance, parameter, and forest-sampling errors alongside orthogonality-based remainders, honest-weight conditioning, and uniform concentration. These controls establish consistency and asymptotic normality under the stated conditions.
D. Estimation Error and Asymptotic Normality
- The error analysis separates a second-order nuisance remainder from kernel, sampling, and subsampling errors, then isolates a stochastic-equicontinuity term for the normal-limit argument.The decomposition distinguishes terms controlled for the mean absolute error from the finer separation needed for asymptotic normality.Source: D. Estimation Error and Asymptotic… · pp. 13–14
D.1. Consistency of ORF Estimate
- Under the theorem’s nuisance-estimation conditions and B ≥ n/s, s=o(n), and s→∞, the ORF estimate is consistent.The proof also derives convergence of the estimate’s qth moment error for any constant integer q ≥ 1.Source: §D.1 · pp. 15–16
D.2. Proof of Theorem 4.2: Mean Lq Estimation Error
- The mean Lq error analysis combines consistency with kernel, sampling, and subsampling error controls; for B>n/s, the subsampling term is negligible relative to the leading term.The proof integrates the high-probability bounds for sampling and subsampling errors to obtain the moment-error bound.Source: §D.2 · p. 16
D.3. Proof of Theorem 4.3: Finite Sample High Probability Error Bound for Gradients of Convex Losses
- For convex losses, the proof bounds parameter error through gradient deviations under curvature, while local orthogonality makes the nuisance contribution second-order.The argument uses a positive lower bound on the Jacobian eigenvalues and controls the nuisance term at order O(χ_n,δ^2).Source: §D.3 · p. 17
D.4. Proof of Theorem 4.4: Asymptotic Normality
- For any fixed projection, the asymptotic-normality argument uses a random-forest regression limit and requires the remaining error terms to be negligible relative to σ_n.The proof checks conditions including an honest, α-balanced, symmetric tree and the stated density and moment assumptions.Source: §D.4 · p. 18
- Markov bounds for parameter and nuisance errors combine by a union bound to give simultaneous control with probability at least 1−3δ.The proof then integrates a tail bound using the score’s boundedness.Source: §D.4 · p. 19
D.5. Omitted Proofs of Technical Lemmas
- A second-order Taylor expansion of the expected score cancels the nuisance first-order term by local orthogonality, while bounded, invertible Jacobian eigenvalues control the remainder.The argument also invokes a bounded-eigenvalue Hessian assumption.Source: §D.5 · p. 20
- Honesty makes ORF weights independent of the held-out treatment conditional on covariates and other conditioning variables, enabling a conditional expectation relation between scores and moments.Source: §D.5 · p. 20
- Conditional on the data and nuisance estimate, the forest moment estimator is an average of independent, identically distributed terms; boundedness yields Chernoff concentration and Lipschitzness extends it uniformly over parameters.The uniform argument uses an ε-cover of the constant-diameter parameter space.Source: §D.5 · p. 21
E. Omitted Proofs from Section 5
The omitted proofs establish technical conditions for local nuisance estimation and heterogeneous-treatment-effect moments, including orthogonality, identifiability, curvature, and smoothness.
E. Omitted Proofs from Section 5
- The Section 5 proof uses convexity of the weighted penalized loss and restricted strong convexity, with concentration arguments supporting the loss conditions.The proof also invokes concentration for a U-statistic matrix and a restricted-strong-convexity condition.Source: E. Omitted Proofs from Section 5 · pp. 21–22
F.1. Local Orthogonality
- Both residualized and doubly robust treatment-effect moments satisfy conditional orthogonality.The appendix states this result separately for the score functions defined for the two moment formulations.Source: §F.1 · pp. 23–24
F.2. Identifiability
- For residualized and doubly robust moments, conditional independence of μ(X, W) and η given X plus invertibility of E[ηη⊺|X=x] ensures θ(x) is uniquely identified.Source: §F.2 · p. 24
F.3. Smooth Signal
- Under standard boundedness conditions, the heterogeneous-treatment-effect moment functions are O(1)-Lipschitz in x.Source: §F.3 · p. 25
F.4. Curvature
- For residualized moments, a conditional covariance eigenvalue bound gives curvature; for doubly robust moments, the Jacobian is −I, with minimum eigenvalue 1.Source: §F.4 · p. 26
F.5. Smoothness of scores
- Bounded scales for θ, q, and g imply the score smoothness condition for both real-valued and discrete treatments.Source: §F.5 · p. 26
F.6. Accuracy for discrete treatments
The discrete-treatment accuracy corollaries give results for residualized and doubly robust scores under regularity conditions on the score functions, feature distribution, and conditional covariance matrices. Both require constant assignment probabilities for each discrete treatment, including the null treatment.
F.6. Accuracy for discrete treatments
- For both score functions, each discrete treatment, including the null treatment, must be assigned with constant probability.The stated accuracy results also impose Lipschitz, bounded-sparsity, density, and conditional covariance assumptions.Source: §F.6 · p. 27
- Corollary F.5 gives an accuracy result for residualized scores under Lipschitz and bounded-ℓ1-norm conditions on the score functions, plus density and conditional covariance assumptions.The supplied passage does not include the bound itself.Source: §F.6 · p. 27
- Corollary F.6 gives an accuracy result for doubly robust scores under corresponding Lipschitz, bounded-ℓ1-norm, density, and conditional covariance assumptions.The supplied passage does not include the bound itself.Source: §F.6 · p. 27
G. Orange Juice Experiment
The section reports an orange-juice elasticity application and synthetic experiments across treatment-response shapes and support sizes. In the reported parameter space, ORF-CV and ORF outperform the other estimators on all regimes.
G. Orange Juice Experiment
- In the orange-juice data, the estimated price elasticity becomes more inelastic as income rises.The dataset contains 28,947 store-week observations and 15 customer and brand variables; the analysis estimates elasticity as a function of income amid high-dimensional controls.Source: G. Orange Juice Experiment · p. 28
H. All Experimental Results
- ORF-CV and ORF outperform the other estimators on all regimes in the considered parameter space.The evaluation varies support size, feature dimension, and treatment-response form, and measures bias, variance, and RMSE.Source: H. All Experimental Results · p. 28
H.1. Experimental results for one-dimensional, piecewise linear θ0
- For n = 5000, p = 500, d = 1 and θ0, Figure 6 reports bias, variance, and RMSE by support size, with mean metrics across test points averaged over 100 experiments.Treatment-effect estimations are shown for k = 1, 5, 10, 15, and 20 under a piecewise linear treatment response; shaded regions show the mean and 5%-95% interval across 100 Monte Carlo experiments.Source: Figure 6 · p. 28, Figure 7 · p. 28, Figure 8 · p. 29
H.2. Experimental results for one-dimensional, piecewise constant θ0
- For a piecewise constant treatment function, Figure 14 reports bias, variance, and RMSE by support size, while Figures 15–19 show treatment-effect estimates for k = 1, 5, 10, 15, and 20.The figures use n = 5000, p = 500, and d = 1.Source: Figure 14 · p. 31, Figure 15 · p. 31, Figure 16 · p. 31
H.3. Experimental results for one-dimensional, piecewise polynomial θ0
This subsection presents one-dimensional experiments for a piecewise polynomial treatment response, with treatment-effect estimates across support sizes and bias, variance, and RMSE summarized as support size varies.
H.3. Experimental results for one-dimensional, piecewise polynomial θ0
- Bias, variance, and RMSE are plotted as functions of support size for the piecewise polynomial treatment function.The plotted lines give means across test points averaged over Monte Carlo experiments, with standard deviations scaled down by a factor of 3 in the filled regions.Source: Figure 22 · p. 34
- Figures 23–28 report treatment-effect estimations for the piecewise polynomial treatment response across support sizes k = 1, 5, 10, 15, 20, and 25.Each figure summarizes 100 Monte Carlo experiments with n = 5000, p = 500, and d = 1; shaded regions depict the mean and 5%-95% interval.Source: Figure 23 · p. 34, Figure 24 · p. 34, Figure 25 · p. 35
H.4. Experimental results for larger control support
For a one-dimensional piecewise linear treatment response with n = 5000, ORF-CV behaves consistently with the theoretical parameter choices through support sizes as large as 200. It performs well on the selected metrics and outperforms other estimators for larger support sizes.
H.4. Experimental results for larger control support
- ORF-CV behaves consistently with its theoretical parameter choices through support sizes up to 200 and outperforms other estimators for larger support sizes.The evaluation uses n = 5000, a piecewise linear response, and measures bias, variance, and RMSE.Source: §H.4 · p. 37, Figure 30 · p. 37
- The treatment-effect estimation figures cover support sizes k = 50, 75, 100, 150, and 200.Each figure summarizes 100 Monte Carlo experiments with n = 5000, p = 500, and d = 1, showing the mean and 5%-95% interval.Source: Figure 31 · p. 37, Figure 32 · p. 37, Figure 33 · p. 38
H.5. Experimental results for two-dimensional heterogeneity
For two-dimensional heterogeneity, ORF-CV performs better than the ORF and alternative methods across all tested support sizes and evaluation metrics. The experiments examine overall performance and treatment-effect estimates across slices of the feature space.
H.5. Experimental results for two-dimensional heterogeneity
- ORF-CV yields better performance than ORF and alternative methods for every tested support size and evaluation metric.The comparison uses n = 5000 and support sizes k ∈ {1, 5, 10, 15, 20, 25, 30}.Source: §H.5 · p. 39, Figure 36 · p. 39
- The feature coordinates are generated with x1 ∼ U[0, 1] and x2 ∼ Bern(0.5).Source: §H.5 · p. 39
- Figures 36–37 show bias, variance, and RMSE across support sizes, overall and for slices x2 = 0 and x2 = 1.The experiments use n = 5000, p = 500, and d = 2.Source: Figure 36 · p. 39, Figure 37 · p. 39
- For support sizes k = 1, 5, 10, 15, 20, and 25, Figures 38–43 show treatment-effect estimates across slices x2 = 0 and x2 = 1.Each figure summarizes 100 Monte Carlo experiments with shaded means and 5%–95% intervals.Source: Figure 38 · p. 40, Figure 39 · p. 40, Figure 40 · p. 41
- Figure 44 shows treatment-effect estimates for k = 30 across slices x2 = 0 and x2 = 1.It summarizes 100 Monte Carlo experiments with shaded means and 5%–95% intervals.Source: Figure 44 · p. 43