Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition

Vincent Dorie; Jennifer Hill; Uri Shalit; Marc Scott; Dan Cervone

Source-linked AI summary

Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition

Vincent Dorie, Jennifer Hill, Uri Shalit, Marc Scott, Dan Cervone

arXiv:1707.02641v5Jul 9, 2017stat.MEstat.ML

TL;DR

The expanding causal-inference literature offers many methods, but limited comparisons make it difficult to identify useful strategies for applied research. The authors ran a large-scale competition evaluating methods across 77 simulated scenarios spanning varied assignment mechanisms and response surfaces. Methods that flexibly modeled the response surface routinely performed best, especially under nonlinear response surfaces and treatment-effect heterogeneity.

Problem
The expanding causal-inference literature offers many methods, but limited comparisons make it difficult to identify useful strategies for applied research.
Method
The authors ran a large-scale competition evaluating methods across 77 simulated scenarios spanning varied assignment mechanisms and response surfaces.
Results
Methods that flexibly modeled the response surface routinely performed best, especially under nonlinear response surfaces and treatment-effect heterogeneity.
Takeaways & Limitations
Several readily available methods can accurately estimate causal effects, particularly when reducing bias is the primary goal.
Takeaways & Limitations
Runtime comparisons were rough because shared-cluster usage varied across weeks and was not equivalent to performance in a controlled environment.

Abstract

from arXiv · show

Statisticians have made great progress in creating methods that reduce our reliance on parametric assumptions. However this explosion in research has resulted in a breadth of inferential strategies that both create opportunities for more reliable inference as well as complicate the choices that an applied researcher has to make and defend. Relatedly, researchers advocating for new methods typically compare their method to at best 2 or 3 other causal inference strategies and test using simulations that may or may not be designed to equally tease out flaws in all the competing methods. The causal inference data analysis challenge, "Is Your SATT Where It's At?", launched as part of the 2016 Atlantic Causal Inference Conference, sought to make progress with respect to both of these issues. The researchers creating the data testing grounds were distinct from the researchers submitting methods whose efficacy would be evaluated. Results from 30 competitors across the two versions of the competition (black box algorithms and do-it-yourself analyses) are presented along with post-hoc analyses that reveal information about the characteristics of causal inference strategies and settings that affect performance. The most consistent conclusion was that methods that flexibly model the response surface perform better overall than methods that fail to do so. Finally new methods are proposed that combine features of several of the top-performing submitted methods.

1. INTRODUCTION … Testing grounds not calibrated to “real life.”

Causal inference offers diverse methods but difficult choices, while existing evaluations often provide incomplete, biased, or poorly calibrated evidence for selecting among them. These limitations motivate broader, fairer testing grounds for comparing causal inference strategies.

1. INTRODUCTION: Without randomized or natural experiments, causal inference requires fair treatment-control comparisons despite substantive and non-obvious group differences.Researchers are therefore incentivized to control for many pre-treatment covariates, which can require stronger parametric assumptions or create other concerns.
2. MOTIVATION FOR CAUSAL INFERENCE COMPETITION: The breadth of causal-inference methodology creates many options while making it difficult for applied researchers to identify the most useful approach.Several additional issues further complicate methodological choice.
2.1 Shortcomings of existing literature that compares performance of causal inference methods: Method papers commonly compare a new method with only a few competitors, often favoring traditional approaches over more sophisticated alternatives.Even broader comparisons may favor the introduced method because its authors understand it better and may implement or evaluate competitors less effectively.
Few methods compared and unfair comparisons.: Comparisons can be biased by naive or untuned competitor implementations and by evaluation metrics that emphasize bias while ignoring root mean squared error or interval coverage.These choices can inadvertently favor the proposed method.
Few methods compared and unfair comparisons.: Simulation studies often examine few data-generating mechanisms and may not reflect practice, where variables mix continuous, categorical, and binary types and arise from complex joint distributions.Real observational datasets cannot identify a winner when methods produce disparate findings.
Testing grounds not calibrated to “real life.”: Highly specialized simulations can mimic collaborators’ real-life data yet provide limited guidance for general researchers, while asymptotic theory may not apply at smaller sample sizes or under violated assumptions.Theoretical regularity conditions and distributional assumptions may fail in practical datasets.
Testing grounds not calibrated to “real life.”: Constructed observational studies offer another testing ground, but unknown ignorability makes poor recovery ambiguous between model failure and violated assumptions.Because comparisons involve two estimates, it is also unclear how close an observational estimate must be to qualify as adequate.
Testing grounds not calibrated to “real life.”: The file-drawer effect further distorts knowledge of relative method performance because inconclusive comparisons are unlikely to be published.Researchers may therefore lack access to important negative or ambiguous evidence.

File drawer effect. · 3. NOTATION AND ASSUMPTIONS

The competition addressed limited and potentially unfair method comparisons through broad submissions and diverse, realistically calibrated simulations. The paper defines causal effects using potential outcomes and states ignorability, overlap, admissible adjustment, and conditional-expectation assumptions needed for identification and estimation.

2.2 Attempts to address these shortcomings through our competition: The competition sought to address few and unfair comparisons by evaluating 30 methods submitted by researchers or teams knowledgeable about them.Participants either implemented methods themselves or submitted black-box versions intended to work across settings.
2.2 Attempts to address these shortcomings through our competition: The competition was designed across diverse fields and data-structure norms, varying effect sizes, nonlinearities, interactions, covariate structures, and bias levels.Its creators describe it as the first competition focused on estimating effects of causes in observational studies across complications in data features.
2.2 Attempts to address these shortcomings through our competition: Simulations used covariates selected from real data to mimic plausible observational-study correlations, addressing concerns that testing grounds may not reflect real life.The motivating example involved studying birth weight’s effect on IQ.
File drawer effect.: Publishing simulation and evaluation code on GitHub provides an antidote to the file-drawer effect by enabling researchers to test additional methods with their preferred metric.The passage identifies public code as the primary mechanism for addressing this potential problem.
3. NOTATION AND ASSUMPTIONS: For binary treatment Z, potential outcomes Yi(0) and Yi(1) define individual causal effects as Yi(1) − Yi(0), while the observed outcome combines the potential outcome corresponding to Zi.Z = 0 denotes control and Z = 1 denotes treatment.
3.1 Estimands: Average treatment effects are expectations of Y(1) − Y(0), with sample versions averaging over the analysis sample or restricted treatment and control populations.These variants formalize causal effects for subpopulations of convenience or interest.
3.2 Structural Assumptions: Because each person’s untreated or treated potential outcome is unobserved, causal effects require ignorability, which in observational studies conditions independence of potential outcomes on covariates X.Under this assumption, conditional observed outcomes identify the corresponding conditional potential-outcome means.
3.3 Parametric assumptions: Identification additionally requires overlap and an admissible pre-treatment back-door adjustment set X, while unbiased estimation requires modeling conditional expectations such as E[Y(1) | X] and E[Y(0) | X].Without overlap, some observations lack empirical counterfactuals; estimating these expectations can be difficult in high dimensions without strong parametric assumptions.

4. TESTING GROUNDS: DATA AND GENERATIVE MODELS … Simulating treatment and outcome.

The competition used realistic, tunable simulations to distinguish causal inference methods while imposing ignorability and targeting the SATT. Data were calibrated to a real study and varied nonlinear response surfaces, assignment, overlap, and treatment-effect features.

4. TESTING GROUNDS: DATA AND GENERATIVE MODELS: The testing grounds were designed to distinguish methods while exhibiting features typical of real-study data.The competition combined practical design choices with realistic data characteristics.
4.1 Embedded assumptions and design choices: All data-generating processes imposed a small set of assumptions to keep the competition practical and not overly complex.
Ignorability.: Treatment assignment was assumed ignorable, avoiding the need to invent or map an underlying scientific theory linking treatment, outcomes, and covariates.
Estimand.: The estimand was the sample average treatment effect for the treated because many causal methods naturally target it and lack variance estimators for population effects.
Overlap for the inferential group.: For SATT, overlap required empirical counterfactuals only for treated units, allowing inference to focus on treated observations with sufficient overlap.
4.2 Calibration to “real data”: Covariates from the Collaborative Perinatal Project provided plausible variable types and natural associations for calibrating the simulations to real data.The hypothetical study examined birth weight effects on child IQ using 4,802 complete-case observations and 58 covariates.
4.3 Simulation procedure and “knobs”: The simulation factorized potential outcomes and treatment assignment into a response surface and assignment mechanism conditional on covariates, reflecting ignorability.Both components used generalized additive functions with tunable transformations and interactions.
Simulating treatment and outcome.: Across 77 black-box scenarios, 100 replications each varied nonlinearity, treatment prevalence, overlap, alignment, effect heterogeneity, and treatment-effect magnitude.The framework deliberately included nonlinear response surfaces and assignment mechanisms because simple methods can fail when those nonlinearities are present.

Degree of nonlinearity. … Alignment.

The simulations varied treatment prevalence, overlap, imbalance, alignment, and treatment-effect heterogeneity to create distinct challenges for causal inference methods. These factors affected potential bias, extrapolation, variable prioritization, and the complexity of estimating heterogeneous effects.

Degree of nonlinearity.: Treatment prevalence was varied between settings with expected treated proportions of 35% and 65%, creating potential challenges for estimating the treatment effect on the treated.In the low-treatment setting, 95% of simulations had treatment proportions between 0.20 and 0.38.
Overlap for the treatment group.: Low overlap was constructed by preventing observations in a covariate-space corner from receiving treatment, forcing their propensity scores to zero and challenging models that extrapolate beyond common support.More complicated definitions of the excluded neighborhood made it harder to identify as fundamentally different from regions with overlap.
Overlap for the treatment group.: The simulations also produced imbalance even when overlap could be perfect, with the Euclidean distance between treated and control means varying across settings and having quartiles of [0.78, 1.30, 2.68].Lack of overlap always implies lack of balance, but imbalance can occur without lack of overlap.
Overlap for the treatment group.: Only covariates affecting both treatment assignment and the response surface can cause bias, while covariate functional form also matters.Covariates affecting only assignment or only the response may improve efficiency when included but should not affect bias.
Alignment.: When true confounders are few among many available covariates, methods that prioritize predictors of only treatment or only response may underperform methods that target predictors of both.Alignment therefore creates potential for bias and makes it harder to determine which variables deserve priority.
Alignment.: Alignment was varied by changing how often terms appeared in both the assignment and response models, allowing complex settings with different fractions of true confounders.The resulting scenarios varied widely in the correlation between the logit of the true assignment score p(Z | X) and the outcome Y.
Alignment.: Treatment-effect heterogeneity increased computational and statistical difficulty because nonparallel response surfaces are harder to fit than parallel ones, with normalized heterogeneity ranging from 0 to 2.06 and quartiles of [0.47, 0.73, 1.01].Across 7700 realizations, the median SATT was 0.68 and its interquartile range was 0.57 to 0.79, in outcome-standard-deviation units.

Overall magnitude of the treatment effect. · 5. CAUSAL INFERENCE SUBMISSIONS AND KEY FEATURES · Stratification, matching, weighting.

The competition organized diverse causal-inference submissions around features that distinguish methods, especially how they preprocess data to reduce parametric assumptions. Its scope was limited to binary treatments, continuous responses, IID data, fixed data dimensions, measured covariates, and ignorability and overlap.

4.4 Issues not addressed: The competition did not address non-binary treatments, non-continuous responses, non-IID data, varying sample or covariate dimensions, covariate measurement error, or violations of ignorability and overlap.These issues were identified as potential topics for future competitions, and the organizers described the competition as limited in scope.
5. CAUSAL INFERENCE SUBMISSIONS AND KEY FEATURES: The competition received 15 DIY and 15 black-box submissions, but two DIY submissions were omitted because they were inadequately described.Organizers could not submit methods; a simple main-effects linear model was included as a black-box baseline.
5. CAUSAL INFERENCE SUBMISSIONS AND KEY FEATURES: The competition classified substantially different submitted approaches using a taxonomy of features that distinguish causal-inference methods.The taxonomy was summarized in Table 1, with additional details in Appendix A.2.
Stratification, matching, weighting.: Methods that preprocess data to balance treatment and control covariates can support model-free estimation or make model-based estimates more robust to misspecification.Stratification, matching, and weighting pursue this balance by constructing comparable groups or pseudo-populations.
Stratification, matching, weighting.: Stratification compares treated and control outcomes within covariate-defined cells, with variants using regression-tree leaves.This approach is also called subclassification.
Stratification, matching, weighting.: Matching selects controls closest to treated units under a distance metric and drops controls judged insufficiently similar, most commonly using the propensity score.Other distance metrics are also possible.
Stratification, matching, weighting.: Weighting reweights controls to create a pseudo-population whose covariate distribution resembles that of the inferential group when estimating the ATT, or reverses the roles for other targets.This approach is closely related to survey-sampling weighting.
Stratification, matching, weighting.: Accurate treatment-assignment modeling is often required by methods that reduce parametric reliance because they incorporate the propensity score as a balancing score.Conditional on the same propensity score, treatment assignment is ignorable.

Modeling of the assignment mechanism. … Ensemble methods.

The paper organizes causal inference methods by whether they model treatment assignment or the response surface, while highlighting variable selection and ensembles as strategies for improving flexibility and robustness. Across submissions, response-surface modeling and sophisticated nonparametric methods were especially prominent.

Modeling of the assignment mechanism.: Propensity scores can enter causal analyses through stratification, matching, weighting, or TMLE, while balanced groups permit unbiased treatment-effect estimation under ignorability.The parallel strategy is to model the response surface correctly rather than preprocess observations for balance.
Modeling of the response surface.: Methods are categorized by whether they model the assignment mechanism or response surface because determining formal double robustness was outside the paper’s scope.This taxonomy separates the principal modeling targets without requiring every approach to receive a formal double-robustness classification.
Nonparametric modeling of the assignment mechanism or response surface.: Flexible assignment-mechanism modeling was historically underused because propensity-score estimation was viewed mainly as a tool for achieving covariate balance.Nonparametric response-surface modeling emerged as an alternative to balancing approaches intended to protect against response-surface misspecification.
Variable selection.: Variable-selection methods such as LASSO and the elastic net can focus estimation on true confounders when candidate covariates are numerous.Reducing the estimation problem can make it feasible to apply more complicated algorithms to the variables that matter.
Ensemble methods.: Ensemble methods address variation across settings by fitting multiple methods, evaluating relative performance with cross-validation or model averaging, and selecting or combining estimates.Combined estimates use weighted averages whose weights reflect relative performance.
5.2 Overview of submissions.: Among submissions, most methods fit response-surface models, more than half used weighting, matching was nearly absent from black-box methods, and black-box entrants favored sophisticated nonparametric techniques.Table 1 summarizes the submitted methods and distinguishes do-it-yourself from black-box approaches.
5.3 Top performers.: BART fits arbitrary functions as sums of small regression trees, uses priors to avoid overfitting, and models the joint function f(x, z) for both potential outcomes.The causal BART procedure draws from posterior predictive distributions for y(1) = f(x, 1) and y(0) = f(x, 0) (Chipman, George and McCulloch, 2010).

Bayesian Additive Regression Trees (BART). · Super Learner plus Targeted Maximum Likelihood Estimation (SL+TMLE). · DR w/GBM + MDIA 1 and 2

The methods combine flexible response-surface modeling with ensemble learning, propensity-score adjustment, TMLE, and calibrated weighting. Post-competition variants tested BART, joint versus separate outcome modeling, additional assignment-mechanism modeling, and alternative BART fitting strategies.

Super Learner plus Targeted Maximum Likelihood Estimation (SL+TMLE).: SL+TMLE used cross-validated Super Learner predictions, combined library fits by squared-error-minimizing weights, and applied a TMLE correction.Its library included glm, gbm, gam, glmnet, and splines for modeling assignment and response surfaces.
Super Learner plus Targeted Maximum Likelihood Estimation (SL+TMLE).: The SL+TMLE ensemble separately modeled the assignment mechanism and control response surface, incorporating propensity-score weights to expand ATT into individual conditional treatment effects.The implementation also used ensemble libraries including glm, random forest, deep learning, LASSO, and ridge regression.
Bayesian Additive Regression Trees (BART).: The calCause ensemble selected between random forests and Gaussian processes by out-of-sample prediction, then imputed treated units’ control responses for treatment-effect estimation.This method focused on fitting the control response surface.
DR w/GBM + MDIA 1 and 2: The DR w/GBM + MDIA methods estimated assignment and response surfaces with generalized boosted regression, allowed up to three-way interactions, and calibrated control treatment-on-treated weights using MDIA.MDIA exactly balanced covariate means and estimated response values while minimally perturbing the base weights.
Bayesian Additive Regression Trees (BART).: Post-competition analyses added BART to the Super Learner library and created BART IPTW to test whether modeling the assignment mechanism improved on response-surface modeling alone.The two top-performing ensemble submissions modeled both mechanisms.
Super Learner plus Targeted Maximum Likelihood Estimation (SL+TMLE).: Removing TMLE from Super Learner and adding IPTW plus TMLE to BART isolated the contributions of targeted correction and propensity-score modeling.The BART propensity score was fit with cross-validation.
Bayesian Additive Regression Trees (BART).: BART provided the main stand-alone flexible response-surface approach and was the only stand-alone method reported to rival ensemble performance.Post-competition variants included cross-validated hyperparameters and multiple chains.
Super Learner plus Targeted Maximum Likelihood Estimation (SL+TMLE).: Because BART jointly modeled treatment and response surfaces while the original Super Learner fit separate treatment- and control-condition models, a joint-response-surface SL+TMLE variant was created.The joint implementation included BART in its ensemble.

6. EVALUATION OF PERFORMANCE OF SUBMITTED AND CONSTRUCTED METHODS · RMSE and bias.

Across 20 DIY and 7,700 black-box data sets, performance was assessed using RMSE, bias, interval coverage, interval length, and PEHE. Flexible methods generally performed best on RMSE and bias, while interval coverage and length varied widely.

6. EVALUATION OF PERFORMANCE OF SUBMITTED AND CONSTRUCTED METHODS: The evaluation compared submitted and post-competition methods across 20 DIY data sets and 7,700 black-box methods using global performance summaries.RMSE measured average closeness of treatment-effect estimates to the true estimand, and bias measured average distance from the SATT.
6.1 Comparison of all methods in the 20 DIY data sets: Across the 20 DIY data sets, the two DR w/GBM+MDIA submissions performed best on bias and RMSE, followed by the top black-box performers.Because the DIY submissions were evaluated on only 20 data sets, the authors were reluctant to draw strong conclusions about relative performance.
6.2 Comparison of black box submissions: Across 7,700 data sets, BART, SL + TMLE, calCause, and h2o were the strongest originally submitted methods for bias and RMSE.Tree Strat, BalanceBoost, Adjusted Tree Strat, and LASSO + CBPS formed the next superior group.
RMSE and bias.: All methods had negative average bias because the treatment-effect distribution had a positive expected value and estimates were generally shrunk toward zero.For black-box methods, the reported bias summaries also included interquartile ranges across 77 settings and 100 replications.
RMSE and bias.: Adding BART to the SL+TMLE library improved performance over SL+TMLE without BART but did not outperform BART alone.BART+TMLE, BART MChains, and BART Xval slightly improved bias and RMSE, whereas BART IPTW did not.
RMSE and bias.: Many automated algorithms performed well on RMSE and bias, but interval coverage and average interval length varied widely, with originally submitted methods somewhat disappointing.Coverage measured the percentage of data sets whose interval covered the true SATT, while interval length captured the trade-off between coverage and precision.

Interval coverage and length. … 7. PREDICTING PERFORMANCE

Across methods, flexible response-surface modeling generally performed best, while relative performance was difficult to predict from observed data or simulation settings. Coverage could improve substantially with BART augmentation, but at the cost of longer intervals and limited interpretability of method-specific performance differences.

Interval coverage and length.: The interval-length display should not be interpreted as coverage: a triangle on the 95% line may be misleading because it represents interval length.The passage explicitly distinguishes interval length from coverage.
Interval coverage and length.: BART + TMLE achieved nearly nominal coverage versus approximately 82% for BART alone, with about 50% longer average intervals.The augmented intervals remained slightly shorter than those of other top-performing methods.
Precision in Estimation of Heterogeneous Effects.: Among methods reporting individual treatment effects, BART and calCause performed noticeably better on PEHE than the other, simpler linear-model options.Only a subset of methods produced the estimates required to calculate PEHE.
Computational time.: h2o required 24.8 seconds versus 29.4 seconds for BART, but h2o often failed, needed restarts, and offloaded computation, making its timing unreliable.Computational usage varied across runs because the cluster was shared and background processes complicated resource measurement.
7. PREDICTING PERFORMANCE: Relative performance was largely non-contextual: beyond average performance, the analysis could rarely predict which black-box method would outperform others across settings.The strongest general recommendation was to use flexible non-parametric response-surface modeling.
7.2 Performance variance explained: Predictive models using only non-oracle measures rarely exceeded R2 = 0.10, while oracle measures reached 0.40–0.50 for just over one-third of methods, mostly weaker performers.For more successful methods, oracle-inclusive R2 rarely exceeded 0.10.
7. PREDICTING PERFORMANCE: Flexible non-parametric response-surface modeling explained most between-method performance differences, but over half of dataset-level variation remained unexplained.Method features explained 76% of between-method average-performance differences, while unexplained across-dataset variation exceeded half of total variation.
7.3 Cross-method analysis about features of data and models: The settings accounted for only 5% of total performance variation, and their average differences were explained by response-surface nonlinearity and assignment-response alignment.Both predictors were non-oracle data features.

8. DISCUSSION

The competition revealed that flexible ensemble methods can perform well across complex observational-study settings, while coverage remained difficult even when bias was low. Several accurate, readily available methods exist, but conclusions are restricted to testing grounds with ignorability, overlap, and i.i.d. data.

8. DISCUSSION: Flexible ensemble methods performed strongly by combining models’ relative strengths to accommodate complexity in the data.The competition’s results emphasized flexibility as a reason for the superior performance of most ensemble methods.
8. DISCUSSION: Lack of alignment between the assignment mechanism and response surface was among the most challenging data features and has rarely been discussed in causal-inference literature.The difficulty arises when many available covariates include only a subset of true confounders or relevant transformations.
8. DISCUSSION: Good coverage was difficult for most methods even when bias was low, and TMLE adjustment improved coverage inconsistently.The authors did not offer strong advice for optimizing coverage.
8. DISCUSSION: Several methods accurately estimate causal effects, particularly when reducing bias is the primary goal, and many have readily available R packages.These conclusions apply only to settings with ignorability, adequate overlap for the inferential group, and i.i.d. data, which may not hold in practice.
8. DISCUSSION: The competition evaluated a broad range of methods across more assignment-mechanism and response-surface data-generating processes than typical methodological papers, using crowdsourced implementations.The authors hope this approach will inspire similar competitions addressing other challenges.

APPENDIX A: APPENDIX SECTION · A.1 Details of Simulation Framework · A.2 Glossary of Submitted and Created Methods

Appendix A details the 77-setting simulation framework and documents the submitted and organizer-created methods. The framework varies treatment and response-model structure, while the glossary records methods spanning propensity weighting, flexible response modeling, matching, regression, and targeted estimation.

A.1 Details of Simulation Framework: The simulation framework comprises 77 settings, with an R package reproducing each black-box and DIY dataset from enumerated parameter choices.The appendix specifies the levels of each manipulated simulation knob and provides the recreation package.
A.1 Details of Simulation Framework: Treatment-model complexity ranges from linear and polynomial terms to step functions with jumps and kinks in the assignment mechanism P(Z = 1 | X).These alternatives define the base function library used to generate treatment assignment mechanisms.
A.1 Details of Simulation Framework: Generated coefficients and sub-function locations are scaled to produce plausible outcomes and propensity scores, with coefficients drawn from Student-t or beta-prime distributions.Covariates are approximately scaled to [−1, 1], and combined functions are rescaled after functional terms are selected.
A.2 Glossary of Submitted and Created Methods: The glossary covers every competition submission and organizer-created method, using contributor-provided descriptions where available and excluding methods lacking a description.The methods are organized in separate tables for do-it-yourself and black-box approaches.
A.2 Glossary of Submitted and Created Methods: Several documented methods explicitly combine flexible outcome modeling with weighting or targeting, including GBM-plus-MDIA, BART-IPTW, and Super Learner/TMLE approaches.These descriptions specify cross-validation, propensity-score weighting, covariate balancing, or targeted corrections as components of the estimators.
A.2 Glossary of Submitted and Created Methods: Do-it-yourself methods include propensity-score weighting, matching, stratification, regression adjustment, generalized additive models, neural networks, and Gaussian processes.Examples combine variable selection with GenMatch and GAM, estimate propensity scores with trees or neural networks, or address covariate shifting with weighted Gaussian processes.
A.2 Glossary of Submitted and Created Methods: Black-box methods span BART, cross-validated response-surface models, covariate-balancing propensity scores, ensemble learners, LASSO, and TMLE or Super Learner combinations.The glossary also includes conventional linear models, Stata treatment-effect estimators, and tree-based stratification variants.

A.3 Extra DIY Results

Supplementary DIY results assess coverage, interval length, and precision in estimating heterogeneous effects across the competition’s data sets. Figures 4 and 5 report these comparisons for DIY and black-box methods.

A.3 Extra DIY Results: The supplementary DIY analysis examines coverage, average interval length, and PEHE for estimating heterogeneous effects.These results are reported in Figures 4 and 5.
A.3 Extra DIY Results: Figure 4 compares coverage and average interval length for all DIY and original black-box methods across 20 DIY data sets.Methods are ordered by decreasing coverage; gray points indicate very poor coverage or very large intervals beyond the plotting region.
A.3 Extra DIY Results: Figure 5 displays PEHE for DIY and black-box methods that supplied individual-level treatment-effect estimates.

A.4 Explaining variance: modeling results

Modeling explains substantial method- and setting-level variation in log absolute bias, especially through flexible response-surface features and observable setting metrics. Trial-level realizations remain difficult to explain, and observables predict contextual performance more readily than method choice.

Multilevel variance decomposition: Methods, settings, and their interaction accounted for 1.135 variance units, or 46% of total variation, while trial-level realizations accounted for the remaining 54%.The realization component reflects idiosyncratic error and predictor variability within trials.
Multilevel variance decomposition: Non-parametric response surfaces were the most important method feature, improving outcomes dramatically by a factor of -2 on the log scale.Method indicators explained 76% of the between-method main-effect variation, but few method features were significant.
Multilevel variance decomposition: Non-oracle metrics explained 81% of the between-setting main-effect variation, but the limited setting-by-method interaction suggests they offer little guidance for choosing a method.Observable quantities can provide substantial knowledge about the research setting without substantially influencing method selection.
Multilevel variance decomposition: Method-feature interactions explained 45% of the setting-by-method variance component, while making little progress on idiosyncratic realization variance.These interactions were intended to target setting-by-method variation but could also explain realization variance.
Model-specific explanatory power: The full setting-indicator and metric model produced R2 values ranging from 0.06 to 0.50 across methods, consistent with explaining about 18% of total variation after excluding between-method variation.The variance calculation leaves 1.26 of 1.53 units unexplained, averaged across methods.

A.5 Full list of metrics used to describe experimental settings · treatment effect heterogeneity. · A.6 Submissions and Acknowledgements

The paper defines oracle and knob metrics for characterizing experimental settings, including nonlinearity, overlap, balance, alignment, and treatment-effect heterogeneity. It also acknowledges submitters and notes that submitted methods were not necessarily endorsements of best practices.

A.5 Full list of metrics used to describe experimental settings: The competition distinguishes oracle metrics, unavailable in ordinary observational studies, from knob metrics representing explicitly created experimental settings.These labels clarify which setting characteristics are known only to the data creators and which correspond to experimental controls.
A.5 Full list of metrics used to describe experimental settings: Experimental-setting metrics include outcome and treatment-assignment nonlinearity, each scored 0, 1, or 2 for linear, nonlinear, or step-function models.The percentage treated is also tracked as a setting metric, despite being readily estimable from observed data.
treatment effect heterogeneity.: Oracle metrics assess alignment, overlap, and balance using propensity scores and distances between treated and control units.They include propensity-score–outcome correlation, Mahalanobis nearest-neighbor distance, mean-design-matrix distance, and Wasserstein distance.
treatment effect heterogeneity.: Additional metrics quantify treatment-assignment nonlinearity, treatment-effect heterogeneity, and outcome predictability through R2 values based on true or observable design information.The listed regressions include true propensity scores, true treatment effects, and outcomes against design matrices.
A.6 Submissions and Acknowledgements: The acknowledgements thank people who submitted methods and report affiliations as those of each first author at submission.The methods are listed in no particular order.
A.6 Submissions and Acknowledgements: Some submitted methods were representative of their fields and therefore may not reflect the submitters’ own beliefs about best practices.This qualification limits how the submissions should be interpreted as recommendations.

Do-It-Yourself Methods · Black Box Methods · calCause

The paper catalogs do-it-yourself, black box, and calCause submissions, spanning weighting, regression, boosting, tree-based, ensemble, and targeted-learning approaches. The listed methods were contributed by researchers from universities, government research centers, and private organizations.

Do-It-Yourself Methods: Do-it-yourself submissions included IPTW, Bayes LM, regression trees, calibrated IPW, DR w/GBM + MDIA and Ad Hoc, LAS Gen GAM, weighted GP, GLM-Boost, manual RBD, TwoStepLM, ProxMatch, and VarSel NN.Contributors included researchers from Harvard, MIT, Mount Sinai, Helmholtz Zentrum München, the University of Florida, Seoul National University, the University of Alberta, Acumen, UC Berkeley, and Columbia University.
Do-It-Yourself Methods: The do-it-yourself roster combined inverse-probability weighting and doubly robust estimators with Bayesian linear models, generalized additive models, Gaussian processes, boosting, matching, and neural-network variable selection.These entries included IPTW, Bayes LM, calibrated IPW, DR w/GBM + MDIA, LAS Gen GAM, weighted GP, GLM-Boost, ProxMatch, and VarSel NN.
Black Box Methods: Black box submissions included teffects methods and LASSO+CBPS.The listed contributors were affiliated with the University of Maryland, the University of Wisconsin-Milwaukee, the University of Mississippi Medical Center, and Brandeis University.
calCause: calCause submissions included BalanceBoost, Tree Strat and Adj. Tree Strat, h2o Ensemble, CBPS, and SL+TMLE.The listed contributors included researchers from Stanford University and the University of Wisconsin Madison.
calCause: The calCause roster represented boosting, tree stratification, ensemble learning, covariate balancing, and targeted maximum likelihood estimation.These approaches correspond to BalanceBoost, Tree Strat and Adj. Tree Strat, h2o Ensemble, CBPS, and SL+TMLE.