PaperlayerView exact revision on arXiv

Source-linked AI summary

Double Machine Learning based Program Evaluation under Unconfoundedness

Michael C. Knaus

arXiv:2003.03191v5econ.EM

TL;DR

Program evaluation under unconfoundedness requires methods for adjusting for confounders while addressing average effects, heterogeneity, and treatment assignment. This paper reviews and applies DML methods to Swiss labour-market programs, introduces the NDR-learner to stabilize individualized effects, and reports differing program and subgroup effects.

  • Problem

    The paper addresses how to evaluate programs under unconfoundedness across average effects, heterogeneous effects, and treatment assignment when individual counterfactual outcomes are unobserved.

    Source: §2.2 · p. 6, §2.1 · p. 6

  • Method

    The paper reviews and applies DML methods for program evaluation and proposes the NDR-learner, which normalizes inverse probability weights individually to stabilize DR-learner estimates.

    Source: §1 · p. 4, §3.1 · p. 8

  • Results

    Computer, vocational, and language courses increase employment in the 31 months after program start, while job-search training effects are mostly negative; effects also vary across observed subgroups.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The application presents DML as a promising program-evaluation toolbox, with the NDR-learner needed to obtain stable results for all individualized treatment effects in this application.

    Source: §7 · p. 28

    The extent to which individualized heterogeneity can be explored, and whether the NDR-learner is useful beyond this application, remains unclear; individualized methods warrant caution.

    Source: §7 · p. 29

Abstract

from arXiv · show

This paper reviews, applies and extends recently proposed methods based on Double Machine Learning (DML) with a focus on program evaluation under unconfoundedness. DML based methods leverage flexible prediction models to adjust for confounding variables in the estimation of (i) standard average effects, (ii) different forms of heterogeneous effects, and (iii) optimal treatment assignment rules. An evaluation of multiple programs of the Swiss Active Labour Market Policy illustrates how DML based methods enable a comprehensive program evaluation. Motivated by extreme individualised treatment effect estimates of the DR-learner, we propose the normalised DR-learner (NDR-learner) to address this issue. The NDR-learner acknowledges that individualised effect estimates can be stabilised by an individualised normalisation of inverse probability weights.

1 Introduction

The paper presents Double Machine Learning as a framework for comprehensive program evaluation under unconfoundedness, combining average and heterogeneous effect estimation with treatment assignment rules. It applies the methods to Swiss labour-market programs and introduces the NDR-learner to stabilize individualized effect estimates.

1 Introduction

  • DML supports flexible estimation of average effects, heterogeneous effects, and optimal treatment assignment rules after adjustment for confounding.The framework can combine standard statistical tools and supervised machine-learning methods.Source: §1 · p. 2
  • A shared doubly robust score can be reused to estimate multiple parameters, providing a comprehensive pipeline within one framework.Constructing the score can be computationally expensive because it requires machine-learning estimates of outcomes and treatment probabilities.Source: §1 · p. 3
  • The application evaluates four Swiss Active Labour Market Policy programs and illustrates DML methods as a potential blueprint for similar evaluations.The authors describe the evaluation as computationally convenient.Source: §1 · p. 3
  • The proposed NDR-learner builds on the DR-learner and stabilizes individualized effect estimates through individualized normalization of inverse probability weights.This stability comes with a restriction to prediction methods that form predictions as convex combinations of outcomes, such as Random Forests.Source: §1 · p. 4
  • Computer, vocational, and language courses increase employment in the 31 months after program start, while job-search training effects are mostly negative.The analysis also finds substantial heterogeneity by gender, nationality, prior labour-market success, and qualification, reflected in estimated optimal assignment rules.Source: §1 · p. 4

2 Estimands of interest

The paper defines average and heterogeneous treatment-effect estimands and policy values for multiple programs, then states assumptions under which these quantities can be identified. A doubly robust representation of conditional average potential outcomes supports the subsequent estimands.

2.1 Definition

  • The average potential outcome is the population outcome under assigning everyone to a program; ATE, ATET, and CATE compare potential outcomes across programs for different target populations or covariate groups.The CATE conditions on researcher-chosen pre-treatment variables, whereas the ATET conditions on observed assignment to program w.Source: §2.1 · p. 5
  • A policy rule maps observed characteristics to a program, and its value is the average population outcome if that rule were used.The optimal rule maximizes policy value over candidate rules, which may be restricted for interpretability, cost, or fairness considerations.Source: §2.1 · p. 6

2.2 Identification

  • Because each individual is observed in only one program, the other potential outcomes remain latent, requiring assumptions to identify the estimands.The paper considers unconfoundedness using pre-treatment variables that include the variables used to define the CATE.Source: §2.2 · pp. 6–7

Assumption 1

  • Identification assumes unconfoundedness, common support for every program, and SUTVA; together these require observed confounders, program overlap, and no interference.Source: Assumption 1 · p. 7
  • The doubly robust representation Γ(w, x) identifies conditional average potential outcomes and supports CATEs, policy values, and optimal policy rules.Its residual-augmented form remains valid when either the conditional outcome mean or treatment probability in that representation is replaced by an arbitrary function of x.Source: Assumption 1 · pp. 7–8

3 Estimation based on Double Machine Learning

The section builds DML estimators from cross-fitted doubly robust scores for average and heterogeneous effects, then extends them to individualized treatment effects and policy rules. It also proposes normalizing DR-learner residual weights separately for each individual prediction to stabilize estimates.

3.1 The doubly robust scores

  • Cross-fitted nuisance estimates feed doubly robust scores for average potential outcomes and treatment effects, whose means provide estimates and support standard one-sample inference.The estimators achieve the stated asymptotic properties when nuisance estimates are consistent and their convergence-rate product is faster than n^1/2.Source: §3.1 · pp. 8–9, §3.2 · pp. 9–10

3.2 Average potential outcomes and treatment effects

  • DML average-effect estimators use rate double robustness to attain the parametric rate even when machine-learning nuisance estimates converge more slowly, subject to the required rate conditions.Neyman orthogonality makes the scores resistant to small nuisance-estimation errors.Source: §3.2 · p. 10

3.3.1 DR-learner

  • DR-learner CATE estimates regress a doubly robust pseudo-outcome on heterogeneity variables, covering prespecified-group GATEs and covariate-level IATEs.Full-sample IATE prediction is computationally convenient but prone to overfitting; the out-of-sample variant uses four-fold cross-fitting.Source: §3.3.1 · pp. 11–12

3.3.2 Normalised DR-learner

  • The NDR-learner normalizes residual weights for each x, addressing individualized weight imbalances that can produce implausibly large DR-learner estimates.The normalization is more computationally demanding and ensures residual weights sum to one when prediction weights are non-negative.Source: §3.3.2 · pp. 13–14

3.4 Optimal treatment assignment

  • Optimal treatment assignment searches candidate rules for the one maximizing estimated value, with rule quality assessed by regret relative to the true optimum.The number of values to compare grows substantially with multiple treatments and vector-valued, non-binary covariates.Source: §3.4 · pp. 14–15

4 Application: Swiss Active Labour Market Policy

The application evaluates four Swiss Active Labour Market Policy programs using an observational sample of unemployed people in the German-speaking region. It constructs a final sample of 62,497 and studies employment over the maximum available 31-month period.

4 Application: Swiss Active Labour Market Policy

  • The analysis starts from 100,120 unemployed people aged 24–55 and compares non-participants with participants in job search, vocational, computer, and language programs.The analysis focuses on the German-speaking region because assignment policies differ across language regions and could create common-support problems.Source: §4 · p. 16
  • After assigning pseudo-start dates to non-participants and retaining those still unemployed at that point, the final sample contains 62,497 observations.This addresses the concern that non-participants who quickly return to work could make non-participation look overly effective.Source: §4 · p. 16
  • The outcome is cumulative months employed during the 31 months after program start, while 45 confounders include socioeconomic, caseworker, assignment, prior-job, and regional indicators.Program groups differ in observed characteristics, including female and foreigner shares and past income, so raw outcome differences may reflect selection.Source: §4 · pp. 16–17

5 Implementation

The implementation combines cross-fitted Random Forest nuisance estimation with several approaches to heterogeneous effects and policy learning. The paper also describes a modular workflow and an R package that implements the required steps.

5 Implementation

  • Random Forest nuisance models use honest splitting, five-fold cross-fitting, out-of-bag tuning, and the full confounder set.Outcome regressions are fitted separately by treatment group, and propensity scores are separately estimated and normalized within individuals.Source: §5 · p. 17
  • Heterogeneity analyses include subgroup and continuous-variable GATEs, a joint linear model, and DR-learner and NDR-learner IATEs using all 45 confounders.The IATE procedures are run both in-sample and out-of-sample.Source: §5 · pp. 17–18
  • Policy rules are estimated with decision trees of depths one, two, and three, using either five handpicked variables or 16 objective education and labor-market-history measures.The latter variable set excludes gender and foreigner status, which the paper notes may be too sensitive for practical recommendations.Source: §5 · p. 18
  • The evaluation steps can be assembled modularly, and the causalDML R package implements them, including fast C++ individual normalization for the NDR-learner.A replication notebook accompanies the package.Source: §5 · pp. 18–19

6 Results

Across four Swiss ALMP programs, hard-skill courses increase employment while job-search training lowers it; further analyses identify substantial effect heterogeneity and derive treatment rules.

6.1 Average effects

  • Job-search participation lowers employment by about one month, whereas hard-skill programs add roughly three months on average.Figure 2 presents ATE and ATET estimates for participation versus non-participation.Source: §6.1 · p. 19, Figure 2 · p. 20

6.2.1 Subgroup GATEs

  • Women benefit more or suffer less from job-search and computer programs but less from language courses; language-course effects are much smaller for foreigners, while low-employability groups fare better in three programs.The foreigner language-course GATE is reported as 0.72 (3.56 −2.84, standard error: 0.69).Source: §6.2.1 · pp. 20–21

6.2.2 Nonparametric GATEs for continuous heterogeneity variables

  • Effects decrease with higher past income for all programs except language courses, whose small positive effect at low income increases with income; age shows no notable heterogeneity.Kernel and series regression estimates agree by and large.Source: §6.2.2 · pp. 21, 23

6.2.3 Best linear prediction of GATEs

  • The multivariate OLS model summarizes GATE heterogeneity as a best linear predictor, with coefficients representing partial effects that hold included covariates constant.This helps explain why some female coefficients are smaller than in subgroup regressions: those subgroup estimates partly reflect women's lower past income.Source: §6.2.3 · pp. 23–24

6.2.4 Individualised average treatment effects

  • Out-of-sample DR-learner IATEs remain impossible, motivating the NDR-learner; individualised weight normalisation stabilizes estimates, especially for the three smaller programs.The DR- and NDR-learner distributions otherwise correlate at least 0.90; full-sample DR-learner predictions range from -265 to 161 despite the outcome being bounded between zero and 31.Source: Figure 4 · p. 24, §6.2.4 · pp. 24–25

6.3 Optimal treatment assignment

  • Depth-two rules split on past income and gender, assigning low-earning men to vocational training, low-earning women to computer training, and earners above CHF 55k to language training.The depth-one tree without gender splits on past income near where computer and language GATEs intersect.Source: Figure 5 · p. 26, §6.3 · p. 26
  • Estimated policy rules have positive value but do not significantly outperform sending everyone to vocational or computer programs; costs and capacity constraints were unavailable.Cross-validated trees differ from full-sample trees, and evaluating a specific tree's value on a held-out test set would be viable for an application with bigger programs.Source: Table 7 · p. 27, §6.3 · pp. 27–28

7 Discussion and conclusions

DML offers a convenient framework for comprehensive program evaluation, and the application produces plausible results; the DR-learner required the NDR-learner modification for stable individualised effects.

7 Discussion and conclusions

  • DML provides a convenient toolbox for comprehensive evaluation, estimating different parameters within one framework using standard statistical software.The Active Labour Market Policy application shows plausible results, with the DR-learner as the exception requiring modification.Source: §7 · p. 28
  • The NDR-learner produced stable results for all individualised treatment effects where the plain DR-learner did not.The paper says further research is needed to assess whether NDR-learner adjustments are useful beyond this application.Source: §7 · pp. 28–29
  • Choosing nuisance estimators, implementing cross-fitting, and deciding how to address common support remain open DML questions.The authors note that established trimming methods may not transfer straightforwardly to DML estimands beyond average effects.Source: §7 · p. 29
  • Individualised heterogeneous-effect methods are unstable and should be used with caution; the usefulness of NDR adjustments beyond this application needs study.The paper describes OLS, kernel, and series regressions as valuable, easy-to-use options for summarising predefined heterogeneity.Source: §7 · p. 29
  • Optimal treatment assignment rules remain mostly unexplored in practice, with open questions about inference, constraints, rule flexibility, and policy variables.The authors identify these topics as directions for future research.Source: §7 · p. 29
  • Comparisons with specialised causal machine-learning methods could help clarify which application choices are critical.The authors propose investigating both DML-specific questions and comparisons across estimands.Source: §7 · p. 29

Appendices

The appendices revisit doubly robust identification and Neyman orthogonality, then describe cross-fitted DR- and NDR-learner procedures for estimating out-of-sample individualised effects.

A.1 Doubly robust identification

  • The conditional average potential outcome is identified when the stated remainder is zero under correct propensity and outcome models, or when either model is correct and the other is replaced by a function.The appendix presents these as three scenarios for identification.Source: §A.1 · pp. 38–39

A.2 Neyman orthogonality

  • Neyman orthogonality requires the expected score’s derivative with respect to nuisance parameters to be zero at their true values.The appendix demonstrates this by perturbing the nuisance parameters, taking a conditional expectation, differentiating, and evaluating at the true values.Source: §A.2 · pp. 40–41

Algorithm 1 (DR-learner) Let (SN

Algorithm 2 (NDR-learner) Let (SN

C Results

The appendix reports program-specific control-variable means and notes that past labour-market success, particularly past income, is associated with participation.

C.1 Descriptives

  • Table C.1 reports means of control variables by program, with past labour-market success measures such as past income associated with participation.Source: §C.1 · p. 44, Table C.1 · p. 44

C.2 Nuisance parameters

The nuisance-parameter analyses identify intuitive predictors of program selection and future employment; propensity-score distributions show overlap for the low-dimensional parameters considered.

C.2 Nuisance parameters

  • Citizenship, qualification, and previous labour-market success predict program selection; citizenship appears to drive much of selection into language courses.These patterns come from the classification analysis of propensity-score distributions.Source: §C.2 · p. 45, Table C.2 · p. 46
  • Citizenship, qualification, and previous labour-market success also predict future employment in intuitive directions.Swiss citizens, degree-holders, and people with high past income are overrepresented in the upper outcome-prediction quintile.Source: §C.2 · p. 45, Table C.3 · p. 47
  • The smallest reported propensity score is 0.003 for a computer-training participant, whose inverse-probability weight is about 333, or 0.5% of total weights.With imbalanced treatment-group sizes, the authors warn that focusing on the minimum propensity score alone can be misleading.Source: §C.2 · p. 45, Table C.4 · p. 47
  • All treatment groups contain individuals with similarly low propensity scores, so overlap does not seem to be a major issue for the low-dimensional parameters studied.The propensity scores are variable, indicating that selection into programs is not negligible.Source: §C.2 · pp. 45–46
  • Figure C.1 shows the distribution of propensity scores across programs.Source: Figure C.1 · p. 48

C.3 Average treatment effects

Employment effects show an initial lock-in across programs, followed by sustained gains for participants in hard-skill programs.

C.3 Average treatment effects

  • Up to 10 percentage points, participants in hard-skill programs show a sustained increase in employment rates after an initial lock-in.The lock-in occurs within the first months after program start.Source: §C.3 · p. 49
  • All programs show a lock-in effect in the first months after program start.Source: §C.3 · p. 49, Figure C.3 · p. 50
  • Figure C.2 shows average potential outcomes, while Figure C.3 traces employment-probability effects over 31 months, with ATEs, ATETs, and 95% confidence intervals.Source: §C.3 · pp. 49–50, Figure C.2 · p. 49, Figure C.3 · p. 50

C.4 GATEs

Age-related heterogeneity is generally limited in the kernel and spline regressions, with an uncertain positive pattern for language courses among job seekers in their late twenties.

C.4 GATEs

  • Kernel and spline regressions detect no substantial age heterogeneity for most programs; language courses show a larger positive effect in the late twenties.The language-course pattern appears in both regressions, but erratic curves prevent strong conclusions.Source: §C.4 · p. 52, Figure C.4 · p. 53
  • For programs other than language courses, curves are basically flat or their 95% confidence intervals include the ATE.Source: §C.4 · pp. 52–53

C.5 IATEs

The appendix compares DR- and NDR-learner IATEs and examines their classification and optimal treatment assignment rules.

C.5 IATEs

  • From -265 to 161, full-sample DR-learner IATEs for vocational training are extreme; NDR-learner estimates mitigate this but still range from -19 to 20.The paper prefers out-of-sample IATE predictions for discussion.Source: §C.5 · p. 54, Figure C.5 · p. 54
  • At least 0.90 correlation between DR- and NDR-learner IATEs coexists with differences driven mainly by a few outliers and larger DR-learner kurtosis.Source: §C.5 · pp. 54, 55, 57, Table C.6 · p. 55, Figure C.6 · p. 57
  • Caseworker characteristics play a negligible role in explaining IATE variation, in line with previous results of Knaus et al. (2020).All caseworker-related variables appear in the lower half of the classification table.Source: §C.5 · pp. 54, 56, Table C.7 · p. 56

C.6 Optimal treatment assignment

Related papers

Loading 2003.03191v5…