Source-linked AI summary
Recursive Partitioning for Heterogeneous Causal Effects
Susan Athey, Guido Imbens
TL;DR
The paper addresses how to estimate heterogeneous causal effects and conduct valid inference when individual treatment effects are unobserved. It develops honest causal-tree methods with separate samples and treatment-effect-specific cross-validation criteria. The methods support confidence intervals for discovered subpopulations without concern about multiple testing in randomized experiments.
Problem
Individual causal effects are unobserved, so standard prediction-based cross-validation cannot directly evaluate treatment-effect predictions.
Method
The paper develops honest causal trees that separate partition construction from leaf-effect estimation and use treatment-effect-specific splitting and cross-validation criteria.
Results
Honest methods achieve nominal coverage rates for 90% confidence intervals in all simulated designs, unlike adaptive methods with substantially below-nominal coverage.
Takeaways & Limitations
The method can discover subpopulations with lower- or higher-than-average treatment effects and report confidence intervals without concern about multiple testing.
Takeaways & Limitations
Squared T-statistic trees place no value on splits that improve fit, although such splits still have some value.
Abstract
from arXiv · showhide
In this paper we study the problems of estimating heterogeneity in causal effects in experimental or observational studies and conducting inference about the magnitude of the differences in treatment effects across subsets of the population. In applications, our method provides a data-driven approach to determine which subpopulations have large or small treatment effects and to test hypotheses about the differences in these effects. For experiments, our method allows researchers to identify heterogeneity in treatment effects that was not specified in a pre-analysis plan, without concern about invalidating inference due to multiple testing. In most of the literature on supervised machine learning (e.g. regression trees, random forests, LASSO, etc.), the goal is to build a model of the relationship between a unit's attributes and an observed outcome. A prominent role in these methods is played by cross-validation which compares predictions to actual outcomes in test samples, in order to select the level of complexity of the model that provides the best predictive power. Our method is closely related, but it differs in that it is tailored for predicting causal effects of a treatment rather than a unit's outcome. The challenge is that the "ground truth" for a causal effect is not observed for any individual unit: we observe the unit with the treatment, or without the treatment, but not both at the same time. Thus, it is not obvious how to use cross-validation to determine whether a causal effect has been accurately predicted. We propose several novel cross-validation criteria for this problem and demonstrate through simulations the conditions under which they perform better than standard methods for the problem of causal effects. We then apply the method to a large-scale field experiment re-ranking results on a search engine.
1 The Problem
The paper targets conditional average treatment effects by partitioning covariate space, under randomized or unconfounded treatment assignment. It formalizes potential outcomes, observed data, and the assumptions supporting causal interpretation.
- The unit-level causal effect is defined as the difference between potential outcomes under treatment and control.
- Observed outcomes reveal only the potential outcome corresponding to each unit’s received treatment.
- The analysis assumes exchangeable observations and no interference between units.
- Unconfoundedness requires treatment assignment to be conditionally independent of potential outcomes given observed covariates.
- The paper estimates conditional average treatment effects τ(x) using treatment-effect estimates that remain constant within feature-space partitions.
2 Honest Inference for Population Averages
The honest approach separates partition construction from leaf estimation to reduce adaptive bias. Its criteria account for independent estimation samples and the greater variance created by finer partitions.
- Honest estimation uses separate samples to construct the partition and estimate conditional means, unlike adaptive CART.
- A partition divides feature space into leaves, and a tree-building algorithm selects among candidate partitions using sample-based criteria.
- Adaptive estimation can bias selected leaf means because the algorithm groups observations with spurious extreme outcomes.
- The honest target evaluates expected mean-squared error using independent test and estimation samples, with criteria typically maximizing negative MSE.
- The honest criterion differs from conventional CART because the latter uses the same training sample for tree construction and estimation.
- Conventional in-sample fit improves with additional splits even when small leaves increase expected error through higher-variance estimates.
- 2.5 Honest Splitting: Honest splitting uses an independent sample for leaf means and adjusts criteria for unbiased estimation while accounting for finer partitions’ greater variance.
- 2.5 Honest Splitting: For prediction, variance adjustment changes little because gains under adjusted and unadjusted criteria are proportional up to a leaf-size-dependent constant.
3 Honest Inference for Treatment Effects
Causal-tree criteria adapt honest partitioning to treatment effects, whose individual values are unobserved. The resulting estimators use feasible criteria that reward detected heterogeneity while penalizing noisy leaf estimates.
- Causal trees estimate conditional average treatment effects rather than conditional population means.
- The treatment-effect MSE compares estimated leaf effects with unit-level effects, making the direct criterion infeasible from observed data.
- Treatment-effect criteria are difficult because individual causal effects are unobserved, but the paper shows they can be estimated for splitting and cross-validation.
- The proposed treatment-effect criterion is an unbiasedly estimated analogue of the infeasible in-sample goodness-of-fit criterion.
- For cross-validation, treatment effects are estimated on a cross-validation sample rather than the training sample to reflect later independent leaf estimation.
- The honest criteria reward partitions that find treatment-effect heterogeneity and penalize partitions that increase variance in leaf estimates.
4 Four Partitioning Estimators for Causal Effects
The paper compares four tree-based estimators for discovering heterogeneous treatment effects, including transformed-outcome, fit-based, squared t-statistic, and causal-tree criteria. It distinguishes adaptive and honest versions and shows that the proposed criterion combines treatment-effect heterogeneity with fit improvements.
- Four Partitioning Estimators for Causal Effects: The section compares causal trees with transformed-outcome, fit-based, and squared t-statistic tree estimators.Each approach has adaptive and honest variants, yielding eight estimators considered in the paper.
- Transformed Outcome Trees (TOT): Transformed-outcome trees use Y_i* to let standard regression trees target treatment effects, but generally lose efficiency by discarding treatment-indicator information.Their leaf estimates can be interpreted as treatment effects, while the method is primarily used as a benchmark.
- Fit-based Trees (F): Fit-based trees split according to outcome-model fit, which can favor covariates affecting mean outcomes even when they do not generate treatment-effect heterogeneity.Their main advantage is straightforward extension of conventional CART methods.
- Squared T-statistic Trees (TS): Squared t-statistic trees choose splits maximizing T^2 for testing equality of treatment effects between leaves, but assign no value to splits that improve fit without increasing heterogeneity.Using a goodness-of-fit criterion for pruning can also undermine their focus on heterogeneous treatment effects.
- Causal Trees: The honest causal-tree criterion focuses primarily on T^2 while incorporating benefits from improved fit, unlike the squared t-statistic approach.In the simple comparison, its change in criterion is expressed as a combination of the F and TS criteria.
5 Inference
After an honest tree is constructed, inference reduces to estimating treatment effects within a partition using a distinct sample. Standard inference approaches are then valid without assumptions about model complexity.
- Inference: Using a distinct inference sample makes treatment-effect estimation within the constructed partition amenable to standard valid inference.The honest procedure requires no assumptions about model complexity for these estimates.
6 A Simulation Study
The simulations compare adaptive and honest estimators across three designs, examining tree size, treatment-effect MSE, and confidence-interval coverage. Honest methods achieve nominal coverage while accepting costs in fit and sample size, and estimator performance varies by design.
- A Simulation Study: The simulation study uses three designs with 500 or 1000 observations in training and estimation samples and compares adaptive with honest estimators.The designs vary covariates affecting treatment effects, mean outcomes, or neither.
- Tree Size: Adaptive estimators generally build shallower trees than honest estimators, while fit-based estimators build the deepest trees.Fit-based trees reward splits on covariates affecting mean outcomes as well as treatment effects.
- Estimator Performance: Alternative honest estimators perform differently across designs: TOT-H performs well in Designs 2 and 3, F-H suffers in all designs, and TS-H performs well in Design 1.TS-H performs worse in the more complex Design 3 because its preferred splits differ from those minimizing overall treatment-effect MSE.
- Costs and Benefits of Honest Estimation: Honest estimation has a cost in treatment-effect MSE that varies by design and estimator.The comparison gives adaptive methods more observations at each tree-building and estimation step.
- Inference: 90% confidence intervals from honest methods achieve nominal coverage in all designs, whereas adaptive methods fall substantially below nominal coverage.The simulations therefore show a tradeoff between valid inference and treatment-effect fit.
7 Observational Studies with Unconfoundedness
The proposed methods can be adapted to observational studies under unconfoundedness by modifying within-leaf treatment-effect estimates to remove bias from simple treated-control comparisons. Propensity-score weighting, within-leaf weight normalization, and trimming are discussed as examples.
- Observational Studies with Unconfoundedness: For observational studies, the methods require unconfoundedness and modified within-leaf estimates to address bias in treated-control comparisons.The paper gives propensity-score weighting as one example.
- Observational Studies with Unconfoundedness: Renormalizing propensity-score weights within each leaf and treatment group can improve efficiency, while trimming extreme scores can improve robustness.The paper notes that additional conditions are required for asymptotic normality.
8 The Literature
The paper situates its contribution within a growing literature applying supervised machine learning to heterogeneous treatment effects, while distinguishing its direct, single-stage treatment-effect approach from related strategies.
- A growing literature applies supervised machine learning techniques to estimating heterogeneous treatment effects.
- Related approaches transform features, estimate potential-outcome functions, or use separate penalties for treatment effects and other attributes.
- The paper differs by applying machine learning methods directly to treatment effects in a single-stage procedure.
9 Conclusion
The paper introduces causal-effect trees for valid inference in randomized experiments and observational studies satisfying unconfoundedness. The method partitions feature spaces, reports treatment effects with confidence intervals, and supports discovering heterogeneous effects without concern about multiple testing.
- The method enables valid causal-effect inference in randomized experiments and observational studies satisfying unconfoundedness, without restricting covariate count or data-generating-process complexity.
- Its output partitions the feature space into subspaces and reports treatment effects with confidence intervals for each subspace.
- Researchers can use the method to discover subpopulations with lower-than-average or higher-than-average treatment effects.
- These estimates can be reported with confidence intervals without concern about multiple testing.