PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition

Vincent Dorie, Jennifer Hill, Uri Shalit, Marc Scott, Dan Cervone

arXiv:1707.02641v5stat.MEstat.ML

TL;DR

Applied researchers face many causal-inference methods, while comparisons often cover only a few competitors. This paper evaluates methods in a large-scale competition across diverse data-generating processes and finds that methods flexibly modeling the response surface routinely distinguished themselves as high performers.

  • Problem

    Applied researchers face many causal-inference methods, while method papers typically compare only a few competitors, leaving broad comparative evidence limited.

    Source: Few methods compared and unfair… · p. 2, §2 · p. 2

  • Method

    The paper evaluates causal-effect methods through a large-scale competition across a broad range of data-generating processes, using crowdsourced implementations.

    Source: §8 · p. 25

  • Results

    Methods able to flexibly model the response surface routinely distinguished themselves as high performers.

    Source: §8 · pp. 24–25

  • Takeaways & Limitations

    Several methods provide good options for estimating causal effects, particularly when reducing bias is the primary goal.

    Source: §8 · p. 25

    Computational-time comparisons are rough because shared-cluster usage varied widely from week to week.

    Source: Computational time · p. 22

Abstract

from arXiv · show

Statisticians have made great progress in creating methods that reduce our reliance on parametric assumptions. However this explosion in research has resulted in a breadth of inferential strategies that both create opportunities for more reliable inference as well as complicate the choices that an applied researcher has to make and defend. Relatedly, researchers advocating for new methods typically compare their method to at best 2 or 3 other causal inference strategies and test using simulations that may or may not be designed to equally tease out flaws in all the competing methods. The causal inference data analysis challenge, "Is Your SATT Where It's At?", launched as part of the 2016 Atlantic Causal Inference Conference, sought to make progress with respect to both of these issues. The researchers creating the data testing grounds were distinct from the researchers submitting methods whose efficacy would be evaluated. Results from 30 competitors across the two versions of the competition (black box algorithms and do-it-yourself analyses) are presented along with post-hoc analyses that reveal information about the characteristics of causal inference strategies and settings that affect performance. The most consistent conclusion was that methods that flexibly model the response surface perform better overall than methods that fail to do so. Finally new methods are proposed that combine features of several of the top-performing submitted methods.

1. INTRODUCTION … Testing grounds not calibrated to “real life.”

The paper introduces a competition to compare causal inference methods across a broad range of testing grounds and finds strong evidence that methods flexibly modeling the response surface perform best. It argues that published comparisons and testing grounds often provide incomplete or biased guidance for choosing methods.

1. INTRODUCTION

  • The competition found strong evidence that methods flexibly modeling the response surface outperform other methods overall.It was designed to compare a wide variety of methods across a broad range of testing grounds.Source: §1 · p. 2

2. MOTIVATION FOR CAUSAL INFERENCE COMPETITION

  • The richness of causal inference methods gives researchers many options but makes it difficult to identify the most useful approach for a given task.Conditioning on many pre-treatment covariates can require stronger parametric assumptions or a sufficiently flexible model, contributing to the growth of semi-parametric and nonparametric methods.Source: §2 · p. 2, §1 · p. 2

2.1 Shortcomings of existing literature that compares performance of causal inference methods

  • Method papers often compare a new method with few competitors, and authors’ greater familiarity with their own method can make empirical comparisons favor it.Comparisons may use naïve implementations or off-the-shelf models without careful tuning, or evaluation metrics that emphasize bias while omitting root mean squared error or confidence-interval coverage.Source: Few methods compared and unfair… · pp. 2–3, §2.1 · p. 2

Few methods compared and unfair comparisons.

  • Researchers disagree about whether adding pre-treatment covariates is generally preferable, since covariates can amplify residual bias when ignorability fails.Some caution against conditioning on such variables, while others argue it is rarely preferable to omit an additional pre-treatment covariate.Source: Few methods compared and unfair… · p. 2
  • Simulations may test only a few data-generating mechanisms and may not reflect practice’s varied variables and complex joint distributions.Observed data may not follow the multivariate normal distribution often used to create simulated testing grounds.Source: Testing grounds not calibrated to… · p. 3

Testing grounds not calibrated to “real life.”

  • Highly specialized simulations may reflect real data closely but offer limited guidance beyond their target setting, while theoretical results may not map to small samples or violated assumptions.In practice, sample sizes may fall below those required by theory, and regularity conditions or distributional assumptions may not hold.Source: Testing grounds not calibrated to… · p. 3
  • Constructed observational studies offer real-world calibration but represent only one data-generating process and cannot establish whether ignorability holds.When an observational estimate differs from the experimental benchmark, the cause may be failed ignorability or poor model fit, and the acceptable degree of closeness is unclear. Examples cited include LaLonde and Maynard, 1987; Hill, Reiter and Zanutto, 2004; and Shadish, Clark and Steiner, 2008.Source: Testing grounds not calibrated to… · pp. 3–4
  • The file-drawer effect can hide inconclusive method comparisons, leaving researchers without results that might inform their choice.Such comparisons are unlikely to be published.Source: File drawer effect · p. 4

File drawer effect. · 3. NOTATION AND ASSUMPTIONS

The competition counters the file-drawer problem by publishing its simulation and evaluation code, while the paper defines sample causal estimands and the assumptions used to identify them from observed data.

2.2 Attempts to address these shortcomings through our competition

  • The competition sought broad, fair comparisons by evaluating 30 submitted methods across varied performance metrics and data features.Submitters either implemented their method or provided a black-box version they considered competent.Source: §2.2 · p. 4
  • Its simulations used real-data covariates and varied key causal-inference features; the authors describe it as the first statistical competition focused on estimating effects of causes in observational studies.The varied features included nonlinearity, overlap or imbalance, treatment prevalence, alignment of assignment and response surfaces, and treatment-effect heterogeneity.Source: §2.2 · p. 5

File drawer effect.

  • Publishing the simulation and evaluation code on GitHub lets anyone test a method using an evaluation metric of their choice.The authors identify this as an antidote to the file-drawer problem.Source: §2.2 · p. 5

3. NOTATION AND ASSUMPTIONS

  • For binary treatment, individual causal effects are differences between potential outcomes, and the observed outcome is the potential outcome corresponding to assigned treatment.The passage defines Y as (1 − Zi)Yi(0) + ZiYi(1).Source: §3 · p. 5

3.1 Estimands

  • The sample average treatment effect (SATE) averages effects over the analysis sample, while SATT and SATC average over treated and control groups, respectively.The competition does not address analogous full-population estimands.Source: §3.1 · p. 5

3.2 Structural Assumptions

  • Because each person’s counterfactual outcome is unobserved, identification requires assumptions; the paper constructs observed covariates as a pre-treatment back-door adjustment set and assumes ignorability and overlap.Overlap requires treatment probability strictly between zero and one at every observed covariate value; without it, stronger response-surface modeling assumptions may be needed.Source: §3.2 · pp. 5–6

3.3 Parametric assumptions

  • Even under ignorability, unbiased estimation requires conditional outcome expectations, which can be difficult to estimate in high dimensions without strong parametric assumptions.The competition focuses on reliable, unbiased causal inference in observational settings where those assumptions may be difficult to satisfy.Source: §3.3 · p. 6

4. TESTING GROUNDS: DATA AND GENERATIVE MODELS … Simulating treatment and outcome.

The section builds realistic but controlled testing grounds to compare causal-inference methods, using real-study covariates and simulated treatment and outcomes. Its design fixes key assumptions and varies features of the data-generating processes across scenarios.

4. TESTING GROUNDS: DATA AND GENERATIVE MODELS

  • The testing grounds were designed to distinguish methods while retaining features typical of real studies.The authors imposed a small set of assumptions to keep the competition practical and not overly complex.Source: §4 · p. 7, §4.1 · p. 7

4.1 Embedded assumptions and design choices

  • All data-generating processes assumed ignorability, avoiding the need to invent or privilege a scientific theory of treatment assignment.The joint distribution factors into potential outcomes conditional on covariates and treatment assignment conditional on covariates.Source: Ignorability · p. 7, §4.3 · p. 8

Estimand.

  • The target estimand was the sample average treatment effect for the treated (SATT), chosen because many methods can target it and some lack natural variance estimators for population estimands.Source: Estimand · p. 7

Overlap for the inferential group.

4.2 Calibration to “real data”

  • Covariates came from the Collaborative Perinatal Project, preserving plausible variable types and natural associations in a hypothetical study of birth weight and child IQ.The complete-case sample had 4,802 observations and 58 covariates: 3 categorical, 5 binary, 27 count, and 23 continuous.Source: §4.2 · p. 8

4.3 Simulation procedure and “knobs”

  • The treatment-assignment and response models used generalized additive functions, combining transformed covariate contributions additively or multiplicatively and allowing link functions to produce bounded or highly nonlinear outputs.Source: §4.3 · p. 8
  • The simulation framework defined 216 knob combinations, narrowed to 77 nontrivial scenarios; each black-box scenario had 100 replications, yielding 7,700 realizations.The knobs varied nonlinearity, treated percentage, overlap, alignment, treatment-effect heterogeneity, and treatment-effect magnitude; participants were told only 20 datasets would be used for the do-it-yourself competition.Source: §4.3 · pp. 8–9
  • Nonlinearities included higher-order moments, interactions, and non-additive functions; the outcome-on-covariate Pearson R2 ranged from 0.02 to 0.93 across realizations.The authors also created oracle and non-oracle variables describing the datasets, distinguished by whether they would be available to researchers using real-world data.Source: Degree of nonlinearity · p. 9, §4.3 · p. 9

Degree of nonlinearity. · Percentage of treated. · Overlap for the treatment group.

The simulations varied the expected percentage treated and treatment-group overlap, while also producing imbalance and examining alignment between the treatment-assignment mechanism and response surface.

Degree of nonlinearity.

  • 95% of simulations had treated shares of 0.20–0.38 in the low-treatment setting and 0.41–0.67 in the high-treatment setting.The two settings had expected treated shares of 35% and 65%, respectively.Source: Percentage of treated · p. 9
  • Because the estimand was the effect on the treated, a low percentage of controls relative to treated units could challenge estimation approaches.The passage attributes this concern to Abadie and Imbens (2006).Source: Percentage of treated · p. 9

Percentage of treated.

  • Lack of overlap was designed to test whether controls dissimilar from treated units on confounders would challenge model-based methods that extrapolate beyond common support.Source: Overlap for the treatment group · p. 9

Overlap for the treatment group.

  • Low-overlap settings forced treatment propensity to zero in a selected covariate-space corner, including for observations with high propensity scores.More complicated neighborhood definitions made it harder for methods to identify the region and avoid unwarranted extrapolation; covariate interactions were included in the response surface for alignment.Source: Overlap for the treatment group · pp. 9–10
  • Oracle mean Mahalanobis-distance quartiles for opposite-treatment nearest neighbors were [3.73, 4.31, 4.99] when overlap was set to 1 and [4.16, 4.75, 5.50] when set to 0.The metric uses the ground-truth covariate space after the polynomial expansions in the true assignment mechanism and response surface.Source: Overlap for the treatment group · p. 10
  • Imbalance was not targeted as a simulation knob but arose from the treatment-assignment specification; unlike lack of overlap, imbalance can occur even with perfect overlap.The oracle Euclidean-distance quartiles between control and treated covariate means were [0.78, 1.30, 2.68].Source: Overlap for the treatment group · p. 10
  • Following Kern et al. (2016), alignment denotes correspondence between the assignment mechanism and response surface, determining which covariate terms must be conditioned on to remove bias.A covariate affecting only one mechanism, or a functional-form term appearing in only one, need not be conditioned on for bias removal; shared terms do.Source: Alignment · p. 10

Alignment. · Overall magnitude of the treatment effect.

The competition varied alignment and treatment-effect heterogeneity, while SATT magnitude varied implicitly; it also left several real-world data and treatment settings for future competitions to address.

Alignment.

  • Alignment can alter bias potential; when true confounders are few, methods may struggle to identify them, and privileging predictors of only treatment or outcome may disadvantage a method.Methods able to target true confounders may avoid that disadvantage.Source: Alignment · pp. 10–11
  • Absolute correlation between the logit of the true assignment score and outcome ranged from near 0 to about .94, with a median of approximately .29.Alignment was varied by changing how frequently terms appeared in both models, allowing complex models with only a fraction of shared confounding terms.Source: Alignment · p. 11
  • Treatment-effect heterogeneity was created by allowing selected response-surface terms to have different coefficients for treated and untreated potential outcomes.Its normalized standard deviation ranged from 0 to 2.06, with quartiles [0.47, 0.73, 1.01]; it was exactly 0 in 200 realizations with the heterogeneity knob set to 0.Source: Alignment · p. 11
  • 0.68 was the median SATT across 7,700 realizations, with an interquartile range of 0.57–0.79 outcome standard deviation units.The competition did not directly manipulate treatment-effect magnitude; other simulation knobs created its variation.Source: Overall magnitude of the treatment… · p. 11

4.4 Issues not addressed

  • The competition limited its scope, leaving nonbinary treatments and non-continuous responses unaddressed; without a linear treatment–outcome relationship, treatment effects can be hard to characterize simply.Binary treatments offer weaker, more transparent ignorability and overlap assumptions, while non-continuous response models can be non-collapsible, so marginal and conditional expectations differ.Source: §4.4 · pp. 11–12
  • Future competitions could address non-IID data, varying sample or covariate sizes, covariate measurement error, and violations of ignorability or overlap.Heterogeneous errors or correlated responses challenge IID-dependent methods; covariate measurement error can bias treatment-effect estimates.Source: §4.4 · p. 12

4.5 Competition logistics

  • The challenge was announced on April 21, 2016, to an Atlantic Causal Inference Conference mailing list of about 800 people and through conference, machine-learning, and statistics-blog channels.Source: §4.5 · p. 12

5. CAUSAL INFERENCE SUBMISSIONS AND KEY FEATURES · Stratification, matching, weighting. · Modeling of the assignment mechanism.

The competition received 15 DIY and 15 black-box submissions, and the paper classifies diverse causal-inference approaches by features including preprocessing and modeling the assignment mechanism. It explains how these features aim to balance covariates or model outcomes and assignment, with TMLE among the submitted approaches.

5. CAUSAL INFERENCE SUBMISSIONS AND KEY FEATURES

  • 15 DIY and 15 black-box submissions were received; two DIY entries were omitted, and a simple main-effects linear model served as the black-box baseline.The organizers could not submit methods; the baseline was distinct from the later “created” methods.Source: §5 · p. 12, Stratification, matching, weighting · p. 12

5.1 Features of causal inference methods

  • The diverse submitted methods are classified using features that distinguish approaches to causal inference in observational studies.The paper presents this taxonomy in Table 1 and provides additional details in Appendix A.2.Source: §5.1 · p. 12

Stratification, matching, weighting.

  • Stratification, matching, and weighting preprocess data to create treatment and control groups with similar covariate distributions, reducing reliance on parametric assumptions.With sufficient balance, effects can be estimated without a model or with a model whose estimate should be fairly robust to misspecification.Source: Stratification, matching, weighting · pp. 12–13
  • Stratification compares treated and control outcomes within covariate-defined cells; variants instead stratify within regression-tree leaves.The cited examples are Rosenbaum and Rubin (1984), Athey and Imbens (2015), and Wager and Athey (2015).Source: Stratification, matching, weighting · p. 13
  • Matching selects the closest controls by a distance metric and drops controls insufficiently similar to treated units; the propensity score is the most popular metric.Other distance metrics also exist.Source: Stratification, matching, weighting · p. 13
  • Weighting reweights controls to resemble treated units for the ATT, treated units to resemble controls for the ATC, or both groups to represent the full sample for the ATE.This creates a pseudo-population with a covariate distribution like that of the inferential group.Source: Stratification, matching, weighting · p. 13

Modeling of the assignment mechanism.

  • The propensity score, Pr(Z | X), is used in stratification, matching, and weighting, and is an important component of submitted TMLE methods.TMLE aims to estimate causal effects more efficiently and can help with high-dimensional nuisance parameters; propensity-score methods often require accurate assignment-mechanism models.Source: Modeling of the assignment mechanism · p. 13
  • With ignorability and sufficient balance, the difference in group mean outcomes is unbiased; alternatively, a correctly modeled response surface can make preprocessing unnecessary.Many approaches model both the assignment mechanism and response surface; the passage notes some submitted approaches fall formally or informally under “double robust,” but does not complete the description.Source: Modeling of the response surface · p. 13

Modeling of the response surface. … Ensemble methods.

The methods overview finds that most submissions modeled the response surface, while black-box methods favored sophisticated nonparametric modeling; it also describes approaches to flexible modeling, variable selection, and combining methods.

Modeling of the response surface.

  • The authors classify methods by whether they model the assignment mechanism or response surface, rather than decide whether each formally qualifies as doubly robust.Source: Modeling of the response surface · p. 14

Nonparametric modeling of the assignment mechanism or response surface.

  • Flexible propensity-score modeling gained attention in recent years, after propensity scores were often treated mainly as tools for achieving balance through iterative model adjustments.Earlier adjustments included adding or removing interactions and quadratic terms or transforming inputs until balance thresholds were met (Lee, Lessler and Stuart, 2010; Westreich, Lessler and Funk, 2010).Source: Nonparametric modeling of the… · p. 14
  • Nonparametric response-surface modeling offers an alternative to balancing samples so estimates remain robust to response-surface misspecification (Hill, 2011; Wager and Athey, 2015; Taddy et al., 2016).Source: Nonparametric modeling of the… · p. 14

Variable selection.

  • LASSO and elastic net can narrow estimation to true confounders, making it feasible to apply more complicated algorithms to the terms that matter.The motivation is that researchers often have more potential confounders than they can realistically include.Source: Variable selection · p. 14

Ensemble methods.

  • Ensemble methods run several methods on the same data, assess relative performance by cross-validation or model averaging, then select the best estimate or combine estimates using performance-based weights.Source: Ensemble methods · p. 14

5.2 Overview of submissions

  • Most submissions fit response-surface models, more than half used weighting, matching was nearly absent from black-box methods, and black-box entrants favored sophisticated nonparametric techniques.Source: §5.2 · p. 15

5.3 Top performers

  • BART fits a joint function of covariates and treatment, then estimates effects by contrasting posterior predictions under treatment and control for each person.Its sum of small regression trees uses priors over tree structure and predictions to avoid overfitting (Chipman, George and McCulloch, 2010); the causal approach is described by Hill (2011).Source: Bayesian Additive Regression Trees… · p. 16

Bayesian Additive Regression Trees (BART). · Super Learner plus Targeted Maximum Likelihood Estimation (SL+TMLE). · DR w/GBM + MDIA 1 and 2

The submissions and post-competition variants combine flexible response-surface modeling with approaches to modeling treatment assignment, weighting, calibration, and ensemble fitting. The methods also vary in whether they use TMLE corrections, joint or separate response models, and alternative BART configurations.

Super Learner plus Targeted Maximum Likelihood Estimation (SL+TMLE).

  • Super Learner combined cross-validated out-of-sample predictions using squared-error-minimizing weights, then applied a TMLE correction to its estimates.Its library modeled both treatment assignment and the response surface using glm, gbm, gam, glmnet, and splines; assignment-model fits entered through propensity-score weights.Source: Super Learner plus Targeted Maximum… · p. 16
  • calCause selected between random forests and a Gaussian process by cross-validation to predict controls’ outcomes, then averaged treated outcomes minus imputed counterfactuals.Uncertainty was measured by bootstrap resampling.Source: Super Learner plus Targeted Maximum… · p. 16
  • The h2o submission separately super-learned assignment and control response models, predicted treated outcomes from observed values, and averaged counterfactual differences using IPTW ATT weights.Its library included glm, random forest, deep learning, LASSO, and ridge regression.Source: Super Learner plus Targeted Maximum… · pp. 16–17

DR w/GBM + MDIA 1 and 2

  • DR w/GBM + MDIA used boosted models for treatment assignment and response, then calibrated control weights to balance covariates and estimated response surfaces.DR w/GBM + MDIA 1 selected both models by 5-fold cross-validation, while version 2 used 50,000 trees for each; bootstrap standard errors supplied confidence intervals.Source: DR w/GBM + MDIA 1 and 2 · p. 17, §5.4 · p. 17

5.4 Post-competition methods

  • BART IPTW added an assignment model to BART’s response-surface estimate, using cross-validation to select assignment-model prior parameters and IPTW weights.This variant tested modeling both mechanisms rather than only the response surface.Source: §5.4 · p. 17
  • The post-competition methods also removed TMLE from Super Learner and added TMLE to BART with a cross-validated BART propensity-score fit.Source: §5.4 · p. 17

Bayesian Additive Regression Trees (BART).

  • BART variants explored cross-validated hyperparameters, multiple chains, symmetric intervals, and propensity-score adjustment as a covariate.The propensity-score-covariate approach was inspired by Hahn, Murray and Carvalho (2017).Source: §5.4 · p. 18

5.4 Post-competition methods

  • SL+TMLE Joint changed Super Learner to fit treated and control response surfaces simultaneously, using an ensemble that included BART.This change addressed the difference between Super Learner’s separate response models and BART’s joint model with treatment as a covariate.Source: §5.4 · p. 18

6. EVALUATION OF PERFORMANCE OF SUBMITTED AND CONSTRUCTED METHODS · RMSE and bias.

Across the black-box competition, most methods performed well on bias and RMSE, but interval coverage and length varied widely. In the DIY comparison, the strongest performers used flexible modeling of the response surface or assignment mechanism.

6. EVALUATION OF PERFORMANCE OF SUBMITTED AND CONSTRUCTED METHODS

  • Performance summaries used RMSE and average bias across data sets; black-box bias additionally included its interquartile range across 77 settings and 100 replications.The evaluation also considered interval coverage, average interval length, PEHE, and black-box computational time; coverage and PEHE for DIY methods were reported only in Appendix A.3.Source: §6 · p. 18

6.1 Comparison of all methods in the 20 DIY data sets

  • In the 20 DIY data sets, the two DR w/GBM+MDIA submissions performed strongest on bias and RMSE, while the next-best DIY methods flexibly fit the assignment mechanism, response surface, or both.The authors caution against strong conclusions about relative performance because the DIY comparison covered only 20 data sets.Source: §6.1 · p. 19, Figure 1 · p. 19

6.2 Comparison of black box submissions

  • Across 7,700 black-box data sets, BART, SL + TMLE, calCause, and h2o led original submissions on bias and RMSE; most methods had both measures at or below about .05 standardized units.Post-deadline methods generally performed particularly well, except SL+TMLE joint, whose ensemble library often could not fit non-parallel response surfaces and thus could not estimate heterogeneous effects.Source: Figure 2 · p. 20, RMSE and bias · p. 20

RMSE and bias.

  • Average bias was negative for every method, reflecting the positive expected treatment effect and the tendency of most methods to shrink estimates toward zero.The black-box results report bias averaged across data sets, with interquartile ranges across settings and replications shown as well.Source: RMSE and bias · p. 20, Figure 2 · p. 20
  • Adding BART to SL+TMLE improved performance but did not beat standalone BART; BART+TMLE, BART MChains, and BART Xval slightly improved bias and RMSE, while BART IPTW did not.Source: RMSE and bias · p. 21
  • Interval performance varied widely: all originally submitted methods were disappointing, the last nine methods had coverage below 75%, and CBPS and h2o Ensemble had average interval lengths of 0.78 and 6.1, respectively.The comparison assessed coverage against the intended 95% level alongside interval length.Source: Interval coverage and length · p. 21, Figure 3 · p. 21

Interval coverage and length. … 7. PREDICTING PERFORMANCE

The competition found flexible response-surface modeling to be the clearest general recommendation, while data features offered little ability to predict relative performance and no important method-by-data interactions emerged.

Interval coverage and length.

  • BART + TMLE raised average coverage from about 82% to nearly nominal, with approximately 50% longer intervals; adding BART to SuperLearner + TMLE raised coverage from about 83% to about 92% without noticeable length increase.BART with propensity scores raised coverage to a little over 90% without increasing average interval length, while MBART SymInt improved coverage with only a small length increase. A point on the 95% line represented interval length, not coverage; one possible explanation for BART’s performance is that standard BART-for-binary implementations may perform poorly without adjustment.Source: Interval coverage and length · pp. 21–22

Precision in Estimation of Heterogeneous Effects.

  • Among methods reporting individual treatment effects, BART methods and calCause performed noticeably better on PEHE than two simpler linear-model competitors.The authors recommend that future competitions encourage submissions of individual-level effects and continue evaluating PEHE, given its relevance to targeting treatment by covariate profiles.Source: Precision in Estimation of… · p. 22

Computational time.

  • h2o took 24.8 seconds, close to BART’s 29.4 seconds, while calCause took 27.7 minutes, SuperLearner 46.9 minutes, and SL+TMLE Joint 46.8 minutes.h2o often failed and required restarts, offloaded computation to background processes, and its timing may be unreliable; shared-cluster variation also makes all runtime estimates rough.Source: Computational time · p. 22

7. PREDICTING PERFORMANCE

  • Relative performance was largely unpredictable across settings, beyond methods’ average performance; flexible non-parametric response-surface modeling was the general recommendation.The authors found that under almost no condition could they predict which black-box method would outperform others.Source: §7 · p. 23

7.1 Measures used to predict performance

  • The analysis used 25 metrics across 77 experimental settings, including oracle metrics requiring knowledge of the true DGP and non-oracle metrics available from observed data.For example, oracle metrics included the correlation between true propensity scores and outcomes; a non-oracle metric was the pseudo-R2 from logistic-regression propensity-score estimation.Source: §7.1 · p. 23

7.2 Performance variance explained

  • Non-oracle predictors rarely achieved R2 above 0.10, while oracle predictors reached 0.40–0.50 for just over a third of methods, mostly weaker-performing ones.Even with oracle knowledge, R2 rarely exceeded 0.10 for more successful methods; performance prediction was poor for most methods and roughly inversely proportional to overall performance. The DIY competition lacked enough data for similar analyses.Source: §7.2 · pp. 23–24

7.3 Cross-method analysis about features of data and models

  • 37% of performance variation was attributable to average method differences, and method features explained 76% of those differences, mostly through non-parametric response-surface fitting.Settings accounted for 5% of variation; poor alignment between assignment and outcome mechanisms and response-surface nonlinearity were almost completely predictive of poor performance, but no important method-by-data interactions emerged. Over half of total variation remained unexplained.Source: §7.3 · p. 24

8. DISCUSSION

Flexible response-surface modeling performed well, while coverage remained difficult even when bias was low. The findings also point to promising methods within a restricted set of data settings and motivate broader competitions.

DISCUSSION

  • Methods that flexibly modeled the response surface performed well amid nonlinear response surfaces and treatment effect heterogeneity, outperforming methods focused only on flexible assignment modeling.This included BART, which modeled only the response surface, and helps explain the performance of ensemble methods combining multiple models.Source: §8 · pp. 24–25
  • When assignment and response surfaces were misaligned, selecting or reweighting inputs appeared more beneficial for response-surface modeling than assignment modeling, consistent with prior advice (Austin, Grootendorst and Anderson, 2007; Hill, 2008; Pearl, 2010).Researchers may have many covariates, while only a subset—or particular transformations—are true confounders; methods that fail to privilege them may be disadvantaged. Misalignment has been discussed only rarely in the literature (see, for instance, Kern et al., 2016).Source: §8 · p. 25
  • Good coverage was difficult for most methods even when bias was low; TMLE adjustment often helped, but did not uniformly improve coverage.The authors did not offer strong advice for optimizing coverage.Source: §8 · p. 25
  • Several methods, many with readily available R packages, offered accurate causal-effect estimates, particularly when reducing bias was the primary goal, but the tested settings assumed ignorability, overlap, and i.i.d. data.Those properties may not hold in practice, and further work is needed to understand method performance under additional complications.Source: §8 · p. 25
  • The competition compared a wide range of methods across broader assignment-mechanism and response-surface DGPs than typically examined in one methodological paper, with implementations crowdsourced from competitors.The authors hope similar competitions will examine other challenges and help develop methods that perform reliably across settings.Source: §8 · p. 25

APPENDIX A: APPENDIX SECTION · A.1 Details of Simulation Framework · A.2 Glossary of Submitted and Created Methods

Appendix A details the simulation settings and data-generation process, then catalogs competition-submitted and organizer-created causal inference methods. Together, the sections document how the evaluation settings were constructed and what approaches were considered.

A.1 Details of Simulation Framework

  • The simulation framework enumerates knob settings for 77 scenarios and provides an R package to recreate the simulated data.The appendix specifies the settings used for the black-box and DIY data sets.Source: §A.1 · p. 28
  • Six knobs vary treatment-model form, treatment prevalence, overlap, response-model form, treatment-response alignment, and treatment-effect heterogeneity.The treatment prevalence ranges from 35% to 65%; low and high alignment correspond to approximately 25% and 75% term-copying chances, respectively.Source: §A.1 · p. 30
  • Generated coefficients and treatment effects use heavy-tailed distributions, while rescaling targets plausible propensity scores and response outcomes.With covariates scaled approximately to [−1, 1], propensity scores are almost exclusively 0.1–0.9, response means are 0, and expected response standard deviations are 1.Source: §A.1 · p. 31

A.2 Glossary of Submitted and Created Methods

  • The method glossary covers competition submissions and organizer-created methods, while excluding methods whose submitters did not provide descriptions; those exclusions were low performers.Descriptions were largely contributed by submitters, lightly edited, and listed alphabetically.Source: §A.2 · p. 31
  • DIY methods range from ad hoc and Bayesian linear approaches to weighted estimators, LASSO/GAM, matching, imputation, tree-based, and Gaussian-process methods.Examples include propensity-score stratification with regression adjustment, random-forest proximity matching, and weighted Gaussian processes for covariate shifting.Source: §A.2 · pp. 31–32
  • Other black-box methods include balancing-score weighting with outcome adjustment, response ensembles, LASSO-selected CBPS, Super Learner/TMLE, Stata estimators, and tree stratification.The glossary also describes a calCause response-surface approach and an adjusted tree-stratification variant.Source: §A.2 · pp. 33–34
  • Black-box BART entries include response-surface fits, joint assignment and response models, propensity-score augmentation, TMLE correction, and cross-validated hyperparameter selection.The variants also differ in whether they combine multiple chains and how they calculate intervals.Source: §A.2 · p. 33

A.3 Extra DIY Results

The supplementary DIY results report coverage, average interval length, and precision in estimation of heterogeneous effects (PEHE), with results shown in Figures 4 and 5.

A.3 Extra DIY Results

  • Figure 4 compares coverage and average interval length for DIY and original black-box methods across 20 DIY data sets; gray points mark very poor coverage or very large intervals.Methods are ordered by decreasing coverage, and newly created methods are shown with bold or filled points.Source: Figure 4 · p. 34, §A.3 · p. 34
  • Figure 5 displays PEHE for DIY and black-box methods that supplied individual-level treatment-effect estimates.Source: §A.3 · p. 34, Figure 5 · p. 35

A.4 Explaining variance: modeling results

The models explain much of the variation between methods and settings, but little of the variation among realizations within a setting. Flexible response-surface modeling is the strongest method feature, while observable setting metrics help predict average performance more than they guide method choice.

A.4 Explaining variance: modeling results

  • Methods, settings, and their interaction account for 46% of total variation (1.135), while realizations account for the remaining 54%.The setting-by-method interaction is 0.091, suggesting methods are not dramatically better or worse relative to one another across settings.Source: §A.4 · p. 36, Table 6 · p. 38
  • Method-feature indicators explain 76% of between-method main-effect variation, and non-parametric response-surface modeling is the most important feature, improving the outcome by a factor of -2 on the log scale.Only a few method features are significant.Source: §A.4 · pp. 36–37, Table 6 · p. 38
  • Non-oracle metrics explain 81% of between-setting main-effect variation, while oracle metrics explain nearly all of it.Observable quantities can reveal likely performance in a setting, but the small setting-by-method interaction means this knowledge should not substantially change method choice; a good method flexibly models the response surface.Source: §A.4 · p. 37, Table 6 · p. 38
  • Non-oracle metric interactions explain 45% of setting-by-method variation; adding oracle metrics raises this to 70%, while the final model explains only 6% of realization variation.Realizations within a setting are difficult to explain, although average performance across realizations can be predicted well.Source: §A.4 · p. 37, Table 6 · p. 38
  • Ignoring between-method variation, the models explain about 18% of total variation on average, consistent with Table 5’s 0.06–0.50 R2 range for the full settings-plus-metrics model.In the method-specific regressions, non-oracle metrics explain little variance; adding setting indicators, then realization-level metrics, and finally both settings and metrics explains progressively more.Source: §A.4 · pp. 35, 38, Table 5 · p. 36

A.5 Full list of metrics used to describe experimental settings · treatment effect heterogeneity. · A.6 Submissions and Acknowledgements

The appendix distinguishes oracle metrics from experiment-setting knobs, then lists measures of alignment, overlap, balance, nonlinearity, and treatment-effect heterogeneity; it also acknowledges submitters and notes a caveat about how some methods were submitted.

A.5 Full list of metrics used to describe experimental settings

  • Oracle metrics are unavailable in ordinary observational studies, while knobs are oracle metrics tied to explicit experimental settings; non-knob metrics name the aspect they measure.Source: §A.5 · p. 38
  • The experimental knobs encode outcome and treatment-assignment nonlinearity, treated proportion, overlap, alignment, and treatment-effect heterogeneity.Nonlinearity and alignment use 0–2 scales; overlap is binary, and treated proportion is easily estimated despite its oracle label.Source: §A.5 · p. 38

treatment effect heterogeneity.

  • Oracle measures of alignment, overlap, and balance include propensity-score–outcome correlation, nearest-counterfactual Mahalanobis distance, and mean-distance and Wasserstein comparisons.These measures use the ground-truth design matrix.Source: treatment effect heterogeneity · pp. 38–39
  • Nonlinearity metrics include R2 measures for true propensity scores, outcomes, and potential outcomes in observable or ground-truth design matrices, plus observed-outcome and logistic-propensity fits.Source: treatment effect heterogeneity · p. 39
  • Treatment-effect heterogeneity is measured by the ground-truth treatment-effect standard deviation and by R2 of the true treatment effect regressed on the observable design matrix.Source: treatment effect heterogeneity · p. 39
  • Observable-design measures include Mahalanobis distance to counterfactual neighbors, treated-control mean distance, Wasserstein distance, and alignment between BART effects and logistic propensity scores.Source: treatment effect heterogeneity · p. 39

A.6 Submissions and Acknowledgements

  • The authors thank submitters, state that affiliations refer to each first author at submission, and note that representative submissions may not reflect submitters’ best-practice beliefs.Source: §A.6 · p. 39

Related papers

Loading 1707.02641v5…