PaperlayerView exact revision on arXiv

Source-linked AI summary

Machine Learning Estimation of Heterogeneous Causal Effects: Empirical Monte Carlo Evidence

Michael C. Knaus, Michael Lechner, Anthony Strittmatter

arXiv:1810.13237v2econ.EM

TL;DR

The paper asks which causal machine learning estimators reliably estimate heterogeneous treatment effects across data-generating settings and aggregation levels. It uses an Empirical Monte Carlo study with realistic, data-informed designs and compares estimators under the paper’s identification framework. The results identify a group with competitive performance across DGPs, while no single estimator consistently outperforms all others.

  • Problem

    The paper addresses limited practitioner guidance on which methods are well suited for estimating heterogeneous causal effects.

  • Method

    The study compares causal machine learning estimators across realistic Empirical Monte Carlo DGPs and multiple effect aggregation levels.

  • Results

    No single estimator consistently outperforms the others, but four estimators show competitive performance across all DGPs.

  • Takeaways & Limitations

    Estimators using multiple steps to account for treatment selection and the outcome process perform competitively across the study’s settings.

  • Takeaways & Limitations

    Despite using arguably realistic DGPs, the external validity of simulation studies remains uncertain.

Abstract

from arXiv · show

We investigate the finite sample performance of causal machine learning estimators for heterogeneous causal effects at different aggregation levels. We employ an Empirical Monte Carlo Study that relies on arguably realistic data generation processes (DGPs) based on actual data. We consider 24 different DGPs, eleven different causal machine learning estimators, and three aggregation levels of the estimated effects. In the main DGPs, we allow for selection into treatment based on a rich set of observable covariates. We provide evidence that the estimators can be categorized into three groups. The first group performs consistently well across all DGPs and aggregation levels. These estimators have multiple steps to account for the selection into the treatment and the outcome process. The second group shows competitive performance only for particular DGPs. The third group is clearly outperformed by the other estimators.

1 Introduction

The study addresses limited practitioner guidance on estimating heterogeneous causal effects by comparing causal machine learning estimators under realistic, data-informed DGPs and multiple aggregation levels. No estimator dominates universally, but four estimators perform relatively well across settings.

  • The literature lacks practical guidance on which methods estimate heterogeneous causal effects well for intended applications.
  • The comparison covers eleven causal machine learning estimators and considers individualized, intermediate, and population-level aggregation of effects.The intermediate level is presented as relevant for feasible practitioner action rules.
  • The study uses an Empirical Monte Carlo approach that minimizes synthetic DGP components by informing them with real data.
  • The design varies six heterogeneous-effect specifications, two sample sizes, and DGPs with and without selection into treatment.
  • No estimator is superior across all DGPs and aggregation levels, while four estimators show relatively good performance across settings.These include Random Forests with doubly robust outcome modification, Causal Forest with local centering, covariate-modified efficiency-augmented Lasso, and Lasso with R-learning.
  • The better-performing estimators use multiple estimation steps to account for treatment selection and the outcome process, whereas other estimators can be unstable across DGPs.Lasso estimators tend to be more unstable than Random Forests and may fail to achieve a normal distribution.

2 Notation and estimation targets

The paper defines treatment-effect targets at individualized, group, and population aggregation levels using the potential-outcomes framework. Identification from observational data relies on assumptions including conditional independence, common support, exogenous covariates, and SUTVA.

  • Only one of each individual’s two potential outcomes is observed, so the treatment effect itself is never directly observed.
  • CATEs are expectations of individual treatment effects conditional on exogenous pre-treatment covariates.
  • IATEs use all available covariates, while GATEs average effects over predefined groups and ATEs average effects over the population.Intermediate GATEs can summarize individualized effects for communication or action.
  • Non-random treatment assignment complicates identification, but observed covariates can identify IATEs and coarser effects when they contain all confounders.
  • Under the stated assumptions, IATEs, GATEs, and ATE are identified from observable data.The assumptions include conditional independence, common support, exogenous covariates, and no spillovers under SUTVA.

3 Causal machine learning of effect heterogeneity

IATE estimation is a non-standard prediction problem because individual treatment-effect differences are never observed. The paper organizes methods into generic multi-step approaches and a specialized Causal Forest approach.

  • Individual treatment effects are unobserved, so IATEs must be estimated from treated and non-treated subpopulations.
  • Generic approaches: Generic approaches decompose causal estimation into several standard prediction problems that can use supervised machine-learning estimators.
  • Conditional mean regressions: Conditional mean regression estimates treated and non-treated outcome means separately and takes their difference.
  • Modified outcome methods: Modified outcome methods transform outcomes using inverse-probability weighting or doubly robust constructions before predicting IATEs.
  • Modified covariate and R-learning methods: Modified covariate and R-learning approaches estimate IATEs through weighted minimization with modified covariates and nuisance-parameter estimates.
  • Common representation: The methods can be represented as weighted minimization problems with modified outcomes, weights, or covariates, clarifying their differences.

4 Implementation of estimators

The study implements eleven estimators by combining generic causal approaches with Random Forest or Lasso prediction methods. Cross-fitting separates nuisance-parameter estimation from IATE estimation to reduce overfitting.

  • The study restricts implementation to two representative machine-learning methods because evaluating every possible combination is computationally infeasible.
  • Random Forests represent local conditional-mean approximations, whereas Lasso represents global functional approximations.
  • Random Forests and Causal Forests: Causal Forests split samples using treatment-effect heterogeneity rather than observed outcomes, producing weights for local mean comparisons.
  • Random Forests and Causal Forests: Local centering replaces treatment and outcome variables with residualized versions to remove confounding before building the Causal Forest.
  • Cross-fitting: Cross-fitting estimates nuisance models in one sample half and predicts nuisance parameters in the other half, avoiding same-sample overfitting.

5 Simulation set-up

The Empirical Monte Carlo Study builds realistic treatment-effect settings from Swiss administrative data and evaluates out-of-sample estimation across varied samples and DGPs. Its design preserves observable selection and complex outcome features while specifying true ITEs for validation.

  • The EMCS uses Swiss administrative social-security data to mimic evaluation of job-search programs and heterogeneous employment effects over 33 months.
  • A 10,000-observation validation sample compares estimated ITEs with true ITEs to focus evaluation on out-of-sample predictive power.
  • The study draws estimation samples of 1,000 and 4,000 observations from a 78,844-observation population, with 2,000 and 500 replications respectively.
  • The main design removes observations with propensity scores below 5% or above 95%, leaving unbalanced treatment ratios and insufficient common support for future work.
  • The simulated ITEs depend on observables and unobservables, while their functional form is unknown to the estimators.
  • The evaluation targets how well estimators recover ITEs and thereby approximate unknown IATEs, with aggregation to GATEs and ATEs.

6 Results

Estimator performance varies across sample sizes, DGPs, and aggregation levels: several methods perform consistently well, while others are volatile or consistently weak. Random Forest-based and efficiency-augmented procedures are often competitive, but no estimator is uniformly best.

  • Estimator comparisons: Random Forest conditional mean regression consistently outperforms its Lasso version, with more than three times lower mean MSE for ITE0 in smaller samples.The gap is attributed mainly to Random Forest’s substantially lower mean SD.
  • Estimator comparisons: The MOM DR estimator with Lasso is competitive in larger samples but performs poorly in smaller samples, making it a risky application choice.Its critical sample size for good performance may depend on the DGP.
  • Estimator comparisons: Efficiency augmentation substantially improves MCM, more than halving its mean SD across baseline DGPs, while augmented MCM and R-learning perform similarly.The results characterize the additional computational effort for MCM as fruitful.
  • Overall comparison: No estimator is uniformly superior across sample sizes, and the best-performing choice depends on the DGP and aggregation level.Conditional mean regression, MOM IPW with Lasso, and Causal Forest are discouraged when minimizing MSE is a high priority, whereas Random Forest conditional mean regression and Causal Forest remain attractive under computational constraints.
  • Overall comparison: Four estimators—Random Forest MOM DR, efficiency-augmented MCM, R-learning, and locally centered Causal Forest—perform consistently within the top five.These estimators are presented as reasonable choices for IATE estimation across DGPs and sample sizes.
  • GATE and ATE estimation: Aggregation improves estimation: GATE and ATE estimators converge faster than IATE estimators, while averaging reduces excess kurtosis and benefits locally centered Causal Forests.The passages attribute faster convergence to averaging noisily estimated IATEs.

7 Conclusion

This study evaluates the finite-sample performance of causal machine-learning estimators using empirically grounded DGPs, treatment-selection settings, and three aggregation levels. Four estimators perform competitively across all DGPs, while reliability generally transfers from individualized to more aggregated effects, subject to important scope limitations.

  • This is the first comprehensive economics simulation study of finite-sample performance across many causal machine-learning estimators and aggregation levels.The study examines individualized, intermediate, and population-level effects.
  • The evaluation uses arguably realistic DGPs, including settings with and without selection into treatment, to assess individualized and aggregated treatment effects.The DGPs are intended to have more external validity than mostly synthetic designs used in earlier simulation literature.
  • The study does not identify a single estimator that consistently outperforms all others, and its external validity remains uncertain across empirical settings.The analysis focuses on finite-sample point estimates and does not investigate inference procedures.
  • Four estimators show competitive performance across all DGPs: locally centered Causal Forest, Random Forest-based MOM DR, MCM with efficiency augmentation, and R-learning.These estimators explicitly use both outcome and treatment equations in a multiple-step procedure.
  • The best estimators for individualized average treatment effects are generally also the most reliable at higher aggregation levels.In some settings, noisy but low-bias individualized estimates become competitive for higher aggregates because aggregation averages out noise.

Appendices

The study uses linked Swiss administrative and caseworker data to construct its simulation sample, retaining 88,844 observations after propensity-score trimming and reserving validation and simulation subsets.

  • Data sources and covariates: The data combine unemployment-insurance, social-security, caseworker, and regional labor-market information for unemployed individuals.Observed characteristics include residence status, education, language skills, employment history, occupation, and local labor-market measures.
  • Sample construction: The initial population contains 238,902 people registered as unemployed in 2003, with the first registration per individual retained.Each person is assigned to a caseworker, and the first caseworker is used when assignments change.
  • Sample construction: The final analysis sample contains 96,298 observations after excluding alternative treatments and applying the study’s eligibility restrictions.The exclusions cover age, benefit status, disability-benefit applications, unclear caseworker assignment, missing questionnaires, residence permits, and incomparable agencies.
  • Simulation sample: After dropping observations outside propensity scores of 5% to 95% and treated observations, 88,844 observations remain for simulation.The sample is split into 10,000 validation observations and 78,844 observations used to draw the simulation.

A.2 Descriptive statistics

The descriptive statistics document rich covariates, substantial treated-control imbalance, and bounded, discrete employment outcomes used in the empirical Monte Carlo study.

  • Covariates and imbalance: Standardized differences normalize the absolute mean difference between treated and control groups by the square root of their mean variance.The dataset contains 7,545 treated and 88,844 comparison observations in the descriptive-statistics table.
  • Covariates and imbalance: The estimated propensity score has a standardized difference of 77, indicating high selectivity because values above 20 signal large imbalance.The imbalance is mainly associated with language regions, previous labor-market history, and employability.
  • Outcome distribution: Nearly 30% of unemployed people find no job during the 33-month period, while roughly 10% find a job immediately and remain employed throughout.The observed non-treated outcome therefore has mass points at both bounds.
  • Covariates and imbalance: The dataset provides 105 covariates for heterogeneity analysis and 1,749 covariates after additional variables and transformations are included.The reported standardized differences quantify imbalance between treated and control groups.

B.1 Propensity score

The propensity-score appendix characterizes strong treatment selection without overlap failures and describes simulated effect heterogeneity, group aggregation, and predictive difficulty across components.

  • Propensity-score setting: Many covariates have sizeable and statistically significant propensity-score coefficients, creating substantial selection into treatment.The appendix reports coefficients and average marginal effects for caseworker and regional labor-market characteristics.
  • Propensity-score setting: The manipulated propensity scores do not create problems from extreme values or lack of overlap.This assessment is based on the treated and non-treated propensity-score distributions after the study’s manipulations.
  • Treatment-effect heterogeneity: Participants with propensity scores above 50% benefit on average, whereas those below 50% have negative IATEs on average.The simulated assignment is favorable for most unemployed people but incorporates cream-skimming, with high-potential individuals more likely to participate.
  • Group aggregation: The validation sample is divided into 64 groups of sizes 32 to 420 using employability, gender, age, qualification, foreigner status, and language-region characteristics.These groups provide the basis for summarizing heterogeneous effects as GATEs.

C Implementation

The implementation reuses estimated nuisance components across estimators, with IATE estimation requiring at most one additional step after estimating outcome or propensity components.

  • Estimator inputs: Figure C.1 maps which inputs enter each estimator as sole inputs, necessary nuisance parameters, or optional nuisance parameters.Solid, dashed, and dotted lines encode these three input roles.
  • Estimator inputs: Estimating IATEs requires at most one additional step after obtaining estimated outcome regressions or propensity components.The relevant estimated inputs are ˆµd(x), ˆp(x), or ˆµ(x).
  • Computational design: Reusing inputs across estimators helps keep computational time under control.The implementation omits repeated tuning-parameter and cross-fitting descriptions because they follow procedures introduced earlier.
  • Implementation variants: The compared estimators can be implemented either by modifying covariates or by modifying outcomes.The appendix presents separate implementation tables for conditional mean regression, causal forests, MOM, MCM, and R-learning variants.

D More results

The appendix reports additional performance measures for IATEs and GATEs, including bias, distributional diagnostics, replication-level accuracy, variance ratios, and simulation noise.

  • Standard errors assess whether differences in mean MSE reflect estimator performance rather than simulation noise.
  • Median MSE is also reported because it can produce different rankings from mean MSE when outliers matter.
  • The appendix reports mean bias and distributional diagnostics alongside mean absolute bias and rejected JB tests.
  • Replication-level performance is summarized by correlations between estimated and true ITEs and by estimated-to-true variance ratios.

D.1 Results for IATE estimation

IATE estimates generally recover systematic variation but often overshoot the dispersion of true effects. Performance depends on the DGP, noise, assignment mechanism, sample size, and aggregation of noisy estimates.

  • Estimated IATEs are mostly positively correlated with true ITEs, with larger correlations without noise and at larger sample sizes.
  • Variance ratios above one show that estimators often produce IATEs varying more than the true ITEs, especially for ITE0 and ITE1.
  • Inverse-propensity-only MOM IPW estimators and MCM are prone to highly variable IATEs when ITE variation is small relative to outcome variation.
  • Positive mean bias remains in selective settings because estimators do not completely remove selection bias.
  • Some competitive results are driven by outliers: Random Forest MOM IPW has low mean MSE in selected settings, while its median MSE reveals the distinction.
  • Randomized assignment makes mean biases close to zero, while substantial remaining absolute bias indicates roles for approximation errors and irreducible noise.
  • For GATE estimation, averaging can make noisy IATE estimators competitive, but averaging noisy estimates is not always successful.

D.3 Results for ATE estimation

ATE results resemble the GATE findings: selection-bias correction matters in selective settings, while averaging heterogeneous-effect estimates can yield competitive ATEs.

  • Across ATE estimators, standard-deviation differences are minor, making bias the decisive performance component.
  • Estimators that best account for selection bias perform best when treatment is selective.
  • Averaging noisy IATE estimates can produce competitive ATE estimators, and MCM shows consistently small ATE bias.
  • Randomized settings leave little difference between estimators because ATE bias is close to zero.
  • Computation-time comparisons between Random Forests and Lasso are difficult because tuning parameters strongly affect runtime.
Loading 1810.13237v2…