Source-linked AI summary

Causal inference methods for combining randomized trials and observational studies: a review

Bénédicte Colnet, Imke Mayer, Guanhua Chen, Awa Dieng, Ruohong Li, Gaël Varoquaux, Jean-Philippe Vert, Julie Josse, Shu Yang

arXiv:2011.08047v4stat.ME

TL;DR

Combining randomized trials and observational data raises the challenge of identifying causal effects from data with different strengths. This review synthesizes methods for combining them and finds that, in its tranexamic acid application, generalized estimates agree with RCT findings for at least half of estimators, while observational results also support the RCT findings.

  • Problem

    Identifying an average treatment effect from RCT and observational covariate data requires assumptions because only one potential outcome is observed per individual.

    Source: §3.1 · p. 7
  • Method

    The paper reviews methods for generalizing RCT findings with observational data and for addressing hidden confounding or improving estimation when observational treatment and outcome data are available.

    Source: §1 · p. 3
  • Results

    In the tranexamic acid application, generalized ATE estimates agreed with RCT findings for at least half the estimators, and the purely observational analysis also supported the RCT results.

    Source: §8 · p. 31
  • Takeaways & Limitations

    Which analysis to trust depends on the assumptions made about transportability or unconfoundedness and on the suitability of the selected variables.

    Source: §8 · p. 31

    All methods assume that, given covariates, conditional treatment effects are the same in observational data and the RCT, an assumption that may fail when shifted effect modifiers are unmeasured.

    Source: §3.2.5 · p. 15

Abstract

from arXiv · show

With increasing data availability, causal effects can be evaluated across different data sets, both randomized controlled trials (RCTs) and observational studies. RCTs isolate the effect of the treatment from that of unwanted (confounding) co-occurring effects but they may suffer from unrepresentativeness, and thus lack external validity. On the other hand, large observational samples are often more representative of the target population but can conflate confounding effects with the treatment of interest. In this paper, we review the growing literature on methods for causal inference on combined RCTs and observational studies, striving for the best of both worlds. We first discuss identification and estimation methods that improve generalizability of RCTs using the representativeness of observational data. Classical estimators include weighting, difference between conditional outcome models, and doubly robust estimators. We then discuss methods that combine RCTs and observational data to either ensure uncounfoundedness of the observational analysis or to improve (conditional) average treatment effect estimation. We also connect and contrast works developed in both the potential outcomes literature and the structural causal model literature. Finally, we compare the main methods using a simulation study and real world data to analyze the effect of tranexamic acid on the mortality rate in major trauma patients. A review of available codes and new implementations is also provided.

1 Introduction

The review examines how combining randomized and observational data can address the limitations of each source while supporting causal inference. It surveys the terminology, frameworks, assumptions, methods, and applications involved.

1 Introduction

  • RCTs support causal estimation through controlled allocation, but costs, time, and restrictive eligibility criteria can limit how well findings generalize to target populations.Balanced treatment allocation permits simple estimators such as differences in mean outcomes.Source: §1 · p. 2
  • Observational data are often large, representative, readily available, and less costly, but uncontrolled treatment allocation can introduce confounding and threaten internal validity.Under unconfoundedness, matching, inverse propensity weighting, and augmented inverse propensity weighting can estimate causal effects; additional assumptions can address unobserved confounding.Source: §1 · p. 2
  • Combining the sources can improve RCT representativeness, ground observational analyses against confounding, and increase efficiency for heterogeneous effects when trials are underpowered.Observational data can provide a representative target-population sample; RCTs can help detect confounding bias; and combined data can aid estimation of heterogeneous treatment effects.Source: §1 · p. 2
  • The review connects terminology for combining data—including generalizability, external validity, transportability, recoverability, and data fusion—and contrasts potential-outcomes and structural-causal-model approaches.It also relates the problem to non-probability survey samples and covariate shift.Source: §1 · pp. 2–3
  • The paper covers generalization with baseline covariates, observational data with treatment and outcomes, structural causal models, software, simulations, and a tranexamic-acid mortality application using trial and registry data.The application also reviews methods for addressing data-quality issues such as missing values.Source: §1 · p. 3

2 Problem setting

The section defines the potential-outcomes notation and estimands for combining separately sampled randomized and observational data, then distinguishes the study designs and terminology used to describe extending trial results to a target population.

2.1 Notations in the PO framework

  • RCT units are sampled from the trial-eligible population and observed with covariates, treatment, and outcome; observational units may have covariates alone or also treatment and outcome.The samples are assumed independent, defining the review’s non-nested setup.Source: §2.1 · pp. 3–4
  • The review uses absolute treatment-effect differences, defining conditional and average effects separately for the observational population and the RCT population.The population ATE can differ from the RCT ATE.Source: §2.1 · p. 4

2.2.1 Nested and non-nested study designs

  • Nested designs use two-stage sampling, whereas non-nested designs obtain trial and observational samples separately; this review focuses on the non-nested design.An embedded trial in a broader health system is an example of a nested design.Source: §2.2.1 · pp. 5–6, Figure 1 · p. 6

2.2.2 Transportability, generalizability, and recoverability

  • The review calls extending trial results to a target population generalizability when it is within the trial-eligible population, and transportability when it includes people ineligible for trial participation.The review notes that terminology varies; under common-support or nonzero-sampling assumptions, recovering from sampling bias and transportability use the same estimators and procedures.Source: §2.2.2 · pp. 6–7

3 When observational data have no treatment and outcome information

This section reviews assumptions and estimation methods for generalizing trial findings to a target population using observational covariate data without treatment or outcome information. It also considers how limited overlap and unobserved treatment-effect modifiers can threaten this generalization.

3.1 Assumptions needed to identify the ATE on the target population

  • Generalization requires a valid trial and assumptions linking trial participants to the target population, including participation ignorability, mean exchangeability, or effect exchangeability, plus overlap.Trial consistency and randomization provide internal validity; participation assumptions concern covariates that differ between populations and predict outcomes or modify treatment effects.Source: §3.1.1 · p. 7, §3.1.2 · p. 8

3.1.3 Identifications formulas

  • Under the identifying assumptions, the target ATE can be expressed through reweighting or by averaging the trial’s conditional treatment effect over the target covariate distribution.The two identification formulations motivate distinct estimation strategies.Source: §3.1.3 · p. 9

3.2.1 IPSW and stratification: modeling the probability of trial participation

  • IPSW weights trial outcomes by inverse participation odds to address covariate shifts, but extreme weights can destabilize estimates and correct weight-model specification is required.IPSW is consistent when the participation odds are consistently estimated; normalized weights can help with instability.Source: §3.2.1 · pp. 9–11

Stratification.

  • Stratification seeks to reduce the risk of extreme IPSW weights by grouping units on estimated participation odds and aggregating trial effects using target-population stratum proportions.The review notes that five strata are commonly used.Source: Stratification · p. 11

3.2.2 Plug-in g-formula estimators: modeling the conditional outcome in the trial

  • The plug-in g-formula models conditional outcome means among trial participants, then averages predicted effects over target covariates; it is consistent when the response model is correctly specified.The response model may use parametric or machine-learning methods, but consistency requires the stated model condition.Source: §3.2.2 · pp. 11–12

3.2.3 Calibration weighting: balancing covariates

  • Calibration weighting balances selected covariate functions between the trial and target samples while discouraging highly variable weights, and is consistent if either its log-linear selection model or linear CATE condition holds.Balance can target means or higher-order moments; the estimator is doubly robust under the stated alternative conditions.Source: §3.2.3 · pp. 12–13

3.2.4 Doubly-robust estimators

  • AIPSW combines participation-odds weighting and trial outcome modeling, and is consistent if either nuisance model is correctly specified, with cross-fitting required for the stated result.It is also asymptotically normal when both models are consistently estimated at least at rate n^1/4; machine-learning nuisance estimation requires care to limit overfitting.Source: §3.2.4 · pp. 13–14
  • Augmented calibration weighting is doubly robust and locally efficient when both calibration weights and the outcome model are correctly specified, with product-rate convergence supporting machine-learning estimation.The review states that n^1/4 convergence rates for both nuisance models preserve √n-consistency and notes extensions to survival outcomes.Source: §3.2.4 · p. 14

3.2.5 Practical issues: non-parametric estimation, overlap and unobserved covariates

  • When Assumption 7 fails, generalize to eligible target-population members or use a generalizability score to select subpopulations under limited overlap.Tipton (2013) suggests targeting people who meet trial eligibility criteria; Chen et al. (2021) propose a score combining participation probability and propensity score.Source: §3.2.5 · p. 15
  • All methods require conditional treatment effects to match across RCT and observational data given X; missing shifted modifiers can violate this assumption.Egami and Hartman (2021b) propose selecting sufficient covariates when the RCT has richer covariate data, and sensitivity analyses have been proposed for missing covariates in either or both data sets (Nguyen et al., 2017; Andrews and Oster, 2019; Nguyen et al., 2018; Dahabreh et al., 2019c; Colnet et al., 2022a; Nie et al., 2021; Huang, 2022).Source: §3.2.5 · p. 15

4 When observational data contain treatment and outcome information

When observational data include treatment and outcome information, RCTs can help make causal conclusions from observational analyses more trustworthy by addressing confounding or improving estimation efficiency. The section also surveys other ways to combine experimental and observational data.

4.1 Dealing with unmeasured confounders in observational data

  • Dagan et al. (2021) use prior RCT results as negative controls to assess observational-study confounding; sensitivity analyses offer another approach but require difficult-to-set parameters.The review describes combining experimental data with confounded observational data as a way to ground and debias the observational analysis.Source: §4.1 · p. 16
  • Athey et al. (2020a,b) use a secondary-outcome assumption to recover effects on a long-term outcome when RCT and observational samples measure different outcomes.Their latent-unconfoundedness assumption links confounding in the primary and secondary outcomes; proposed estimators impute missing primary outcomes, weight observational units, or use control functions.Source: §4.1 · p. 16
  • Kallus et al. (2018b) use an RCT to estimate and correct the observational CATE’s bias, yielding a consistent estimate under parametric identification of the bias function.Their approach assumes a high-internal-validity RCT with covariate support contained in the observational sample and a low-complexity bias function; a linear form allows extrapolation beyond experimental overlap.Source: §4.1 · p. 17
  • Yang et al. (2020b) show that an integrative CATE estimator can be strictly more efficient than an RCT-only estimator and generalize ATEs without covariate-overlap assumptions.Wu and Yang (2022a) extend the framework with an integrative R-learner that accommodates flexible machine-learning methods.Source: §4.1 · p. 18

4.2 Toward more efficient estimation

  • Yang et al. (2022) propose a preliminary test of observational-data comparability and an elastic integrative estimator designed to perform at least as well as the RCT-only estimator in mean squared error.They also provide data-adaptive, uniformly valid confidence intervals and characterize the estimator’s distribution under local alternatives.Source: §4.2 · p. 18

4.3 Other use cases

  • Hybrid control arms combine randomized patients with usual-care patients, potentially reducing trial costs and ethical constraints on control groups.The review notes that the FDA detailed their regulatory use.Source: §4.3 · p. 19
  • For case-control observational data, marginal structural models can address outcome-driven selection bias, while control variates can identify and estimate a correlated estimand (Guo et al., 2021).The review describes case-control samples as preferentially selected on the outcome, unlike the RCT sample.Source: §4.3 · p. 19
  • Rudolph and van der Laan (2017) provide a semiparametric efficiency score for transporting an ATE from an encouragement-design study to another population with treatment-allocation information.The review characterizes this setup as a variant of generalization from Section 3.Source: §4.3 · p. 19

5 Structural causal models (SCM) and transportability

The SCM framework uses causal diagrams and do-calculus to express causal assumptions and determine whether effects are identifiable from observational data. For transportability, it combines RCT results with target-population data when the assumptions encoded in a selection diagram permit it.

5 Structural causal models (SCM) and transportability

  • SCM effects are defined through interventions and mirror potential-outcome expressions, while causal diagrams and complete do-calculus support identification, including with some unmeasured confounding.The front-door formula is one example of an identification formula unavailable in the PO framework without additional structural and modeling assumptions.Source: §5 · pp. 19–20

5.1 Formulating transportability in the SCM framework

  • Selection diagrams distinguish sample selection from transportability and encode population differences that determine whether causal effects can be transported.A selection arrow to Y can prevent transportability; in Figure 3(b), the absence of an arrow from S to A encodes an independence condition implying transportability.Source: §5.1 · pp. 20–21, Figure 3 · p. 21
  • SCM generalization covers both selection bias and transport across populations, including post-treatment covariate scenarios not known to have a similar PO-literature identification scenario.For the post-treatment example, X is S-admissible but not S-ignorable, and the transport formula uses a conditional target-population weighting.Source: §5.1 · pp. 20, 22, Figure 4 · p. 22
  • The transport formula combines RCT intervention results with the target population’s covariate distribution to estimate its causal effect.It is also called re-calibration, re-weighting, or post-stratification.Source: §5.1 · p. 21
  • Transporting an RCT effect is one instance of data fusion, which also addresses observational confounding, sample selection, and extrapolation across heterogeneous environments.Causal and selection diagrams can make assumptions explicit, guide variable selection, and support algorithmic do-calculus checks for nonparametric identifiability.Source: §5.1 · p. 22
  • Practical SCM estimators with public implementations and detailed theoretical guarantees remain scarce, although weighting- and DML-based methods have been extended to identifiable effects.When effects are not identifiable, partial-identification methods can bound transported effects; the cited Vitamin A example reports bounds from one trial and a point estimate from two trials.Source: §5.1 · p. 23

6 Software for combining RCT and observational data

The section reviews available software for identifying and estimating effects with combined RCT and observational data, then illustrates generalization estimators’ behavior under well-specified and misspecified models. The simulations highlight robustness differences across estimators.

6.1 Review of available implementations

  • Table 3 reviews implementations for identifiability and estimation, which users must select with attention to sampling design and nested versus non-nested frameworks.The section situates this review within broader software surveys and an R implementation task view.Source: §6.1 · p. 23

6.2 Simulation study ofgeneralization estimators

  • The simulations implement most reviewed estimators in R, using genRCT for CW and ACW, and make code to reproduce the results available on GitHub.The authors implement their own versions of IPSW, normalized IPSW, stratification, plug-in g-formula, and AIPSW to match the review’s formulas.Source: §6.2 · p. 23
  • The RCT-only baseline averaged 14.24 against a true ATE of 27.4, while all generalization estimators were unbiased and IPSW variants had greater variability.The passage attributes the downward RCT-only bias to differences between the trial and target covariate distributions, especially for treatment-effect modifiers.Source: Figure 5 · p. 24, §6.2 · p. 24
  • When both models were misspecified, all estimators were biased except CW and ACW; with either model alone misspecified, AIPSW remained unbiased.IPSW was biased under sampling-score misspecification, and the plug-in g-estimator under outcome-model misspecification.Source: Figure 6 · p. 25, §6.2 · p. 25

7 Application: Effect of Tranexamic Acid

This application uses CRASH-3 trial and Traumabase observational data to assess whether tranexamic acid reduces mortality in traumatic brain injury with intracranial bleeding and to demonstrate generalization to the Traumabase population.

7.1.1 Context

  • The Traumabase analysis used 8,270 TBI patients from a registry with over 30,000 records, and 98% of records were incomplete.TXA treatment was administration during pre-hospital care or on hospital admission within three hours of trauma; the missing data require additional causal-inference assumptions.Source: §7.1.1 · p. 26

7.1.2 Purely-observational results from two different estimation strategies

  • In the Traumabase observational analysis, AIPW found no evidence of a TXA mortality effect, whereas IPW suggested a possible deleterious effect.The AIPW findings were consistent with CRASH-3’s main conclusion; the observational study estimated direct effects on TBI-related and all-cause intra-hospital mortality using 17 confounders and 21 outcome predictors.Source: §7.1.2 · p. 26, Table 4 · p. 27

7.2.1 Context

  • CRASH-3 found 28-day head-injury death risks of 18.5% with TXA versus 19.8% with placebo, with a nonsignificant RR = 0.94 [95% CI 0.86 - 1.02].The trial reported a positive effect only in mild and moderate cases.Source: §7.2.1 · p. 27

7.2.2 RCT selection

  • CRASH-3 included patients with GCS scores of 12 or lower or intracranial bleeding on CT, and no major extracranial bleeding; six covariates were measured at baseline.Baseline covariates were age, sex, time since injury, systolic blood pressure, GCS score, and pupil reaction.Source: §7.2.2 · p. 27

7.3 Transporting the ATE on the observational data

  • Before combining studies, the analysis emphasizes checking that baseline covariates, treatment, and outcome are defined consistently across the RCT and observational data.The target population for generalization is the Traumabase registry.Source: §7.3 · p. 27

7.3.1 Descriptive analyses

  • Because the RCT has almost no missing values but observational eligibility covariates have 0.27 to 29% missingness, the analysis uses strategies that accommodate incomplete data.These include logistic regression with an expectation-maximization algorithm and generalized regression forests using missing incorporated in attributes.Source: Missing values · p. 28

Missing values.

  • Comparisons of covariate means show treatment-assignment bias in the observational study, unlike the balanced treatment groups in the RCT.The analysis also explored distribution shift using univariate histograms.Source: Missing values · p. 28, Figure 8 · p. 29

7.3.2 Analyses

  • Half of transported estimators support CRASH-3’s no-significant-effect conclusion, while others suggest harm; findings require caution because missingness and target-population overlap may affect estimates.GRF weight estimates have large confidence intervals, likely due to differing missing-data proportions, whereas the GRF plug-in g-formula has lower variance; handling missingness in generalization remains an open question.Source: §7.3.2 · pp. 29–30, Figure 9 · p. 30

8 Conclusion · A Randomized controlled trial

The review finds that combining randomized and observational data can strengthen causal inference, while emphasizing that valid conclusions depend on clearly defined estimands, suitable variables, and assumptions. Its tranexamic acid analysis and discussion of single-RCT estimation illustrate both the potential and the limits of these methods.

8 Conclusion

  • Combining observational data and RCTs can increase statistical power and improve external validity, but methods remain scattered and lack practitioner-developed real-world benchmarks.The review highlights generalizability and transportability across populations as a practical concern when treatment effects vary between populations.Source: §8 · pp. 30–31
  • In the tranexamic acid analysis, the generalized ATE agreed with the RCT findings for at least half the estimators, and the observational study also supported the RCT.Which analysis to trust depends on transportability or unconfoundedness assumptions, variable suitability, and the target population; violated positivity prevented transportability for mild and moderate patients.Source: §8 · p. 31
  • Causal analyses combining data sets should state the causal question, estimands, and identifiability requirements, and assess whether outcomes, treatments, and covariates are comparable.Source: §8 · p. 31
  • Domain-informed causal graphs can guide covariate selection and identification, while adding shifted covariates that are not effect modifiers can inflate variance.The review notes that covariate-selection methods have not yet been developed for generalization or data fusion.Source: §8 · p. 31
  • Missing values often differ across data sources, and available approaches such as weighting or multiple imputation rely on untestable missingness assumptions (Carpenter and Kenward, 2007; National Research Council, 2012; Kenward, 2013; O’Kelly and Ratitch, 2014; Li and Stuart, 2019; Cro et al., 2020).Further work is needed on identifiability conditions and estimators for different missing-data patterns (Mayer et al., 2021).Source: §8 · pp. 31–32

A Randomized controlled trial

  • For a single RCT, SUTVA, consistency, and random treatment assignment identify the average treatment effect, and the difference-in-means estimator is unbiased and √n-consistent when the trial randomly samples the target population.Without that random sampling, the estimator is biased for the population average treatment effect.Source: A Randomized controlled trial · p. 46

B Estimation of ATE in observational data … E Nested study design

The review summarizes how observational data can identify treatment effects under assumptions, how non-nested trial and observational samples support target-population identification, and how design affects estimators and identifiable effects.

B Estimation of ATE in observational data

  • Under unconfoundedness—all confounders are measured—and overlap, observational data identify ATE and CATE; available approaches include IPW, regression, doubly robust estimators, and CATE learners.IPW reweights observations according to their propensity scores to balance treated and untreated groups.Source: B Estimation of ATE in observational… · pp. 46–47

C Identification formula

  • In the non-nested design, the g-formula identifies target-population mean potential outcomes by averaging trial conditional outcome means over the target population’s covariate distribution.The derivation invokes the law of total expectation and assumptions linking the trial to the target population and observed outcomes.Source: C Identification formula · p. 47

Identification by weighting

  • Because trial participation is unobserved in the observational sample, its probability and marginal trial-sampling probability cannot be estimated there, but conditional odds can be estimated by discriminating the samples.The passage describes fitting a logistic regression model to distinguish randomized trial from observational samples.Source: Identification by weighting · p. 48

D Sources of formal statements of estimators described in Section 3.2

D.1 Inverse Propensity of Sampling Weighting

  • For IPSW, cited work reports consistency under consistent parametric or nonparametric selection-model estimation, while other results cover oracle properties and parametric-model conditions (Egami and Hartman, 2021a; Buchanan et al., 2018; Dahabreh et al., 2020a; Colnet et al., 2022a).Buchanan et al. (2018) also note that estimating weights can reduce variance.Source: §D.1 · p. 49
  • Colnet et al. (2022b) characterize bias and variance regimes by the relative RCT and observational sample sizes and recommend estimating treatment propensity in the trial to reduce asymptotic variance.Source: §D.1 · p. 49

D.2 Stratification; D.3 Calibration Weighting

  • Stratification theory includes asymptotic normality and variance formulas, while Lee et al. (2021) give consistency, asymptotic normality, and inference procedures for calibration-weighting estimators.Source: §D.2 · p. 50, §D.3 · p. 50

E Nested study design

  • In a nested design, trial participation is observed within the cohort, making the sampling probability identifiable and allowing identification of both the overall ATE and the effect among nonparticipants.The nonparticipant-specific effect can help characterize cohort heterogeneity and variables influencing selection or treatment effects.Source: E Nested study design · p. 50

E.1 When observational data have no outcome and treatment information … F.1.1 Confounding bias

The block presents estimators for nested trial designs, methods for combining trial and observational estimates, implementation examples, and SCM concepts for identifying causal effects and addressing confounding.

E.1 When observational data have no outcome and treatment information

  • For nested trial designs, the review presents IPSW, g-formula, and doubly robust estimators.These methods address settings where observational data lack treatment and outcome information.Source: §E.1 · p. 50

E.1.1 IPSW

  • In nested-design IPSW, the selection probability πS is directly estimable, bringing the weighting expression closer to classic IPW without odds weighting.The passage says πS can be estimated from randomized and non-randomized data.Source: §E.1.1 · pp. 50–51

E.1.2 G-formula

  • For the target population, nested-design g-formula identification matches the non-nested case, but its estimator differs because covariate distributions differ.For a non-randomized target, the causal effect averages RCT conditional outcome contrasts over the covariate distribution in S = 0.Source: §E.1.2 · p. 51

E.1.3 Doubly-robust estimator

  • The nested-case AIPSW estimator combines g-formula and IPSW methods, extending the doubly robust approach to this design.The text describes this as analogous to the doubly robust estimator in the non-nested case.Source: §E.1.3 · p. 52

E.2 Combining treatment-effect estimates from both sources of data

  • For comprehensive cohort studies, Lu et al. (2019) distinguish the combined-sample ATE τ2 from the trial-population ATE τ1 and derive optimal influence functions for both.τ2 is defined over the combined RCT and observational sample, whereas τ in the paper’s setting refers to the observational sample.Source: §E.2 · p. 52

E.3.1 R package causaleffect

  • The causaleffect and dosearch packages support causal-effect identification; dosearch also handles transportability, selection bias, and missingness and returns a formula or an unidentifiability decision.The package applies do-calculus rules, though its general procedure is not necessarily complete for arbitrary queries and input distributions.Source: §E.3 · p. 52, §E.3.1 · p. 53

E.3.2 Beta version of causalfusion

  • The examples cover a no-code causalfusion interface for transportability and nested-design IPSW and g-formula implementations with empirical sandwich variance estimation.The nested IPSW example estimates sampling and treatment propensity scores; both estimator examples use geex through m_estimate for variance.Source: §E.3.2 · p. 53, §E.3.3 · p. 53, §E.3.4 · p. 54

F.1.1 Confounding bias

  • In SCMs, interventions replace the treatment mechanism; the backdoor criterion identifies adjustment sets that imply conditional unconfoundedness, while frontdoor or z-identifiability can apply when backdoor adjustment is unavailable.A backdoor-admissible set supports identification from observational data; the passage also notes surrogate RCTs in the z-identifiability case.Source: §F.1 · p. 56, §F.1.1 · pp. 57–58, Figure 12 · p. 58

F.1.2 Sample selection bias · F.2 Proof of the transport formula (6)

The section explains how causal effects can be recovered or transported when data are selected, by using graphical conditions to choose covariates and combine selected-sample outcomes with target-population information. It also outlines the assumptions behind the transport formula and warns that post-treatment covariates can invalidate the basic reweighting formula.

F.1.2 Sample selection bias

  • In Figure 13(b), the experimental distribution is recoverable when selection is d-separated from Y by A and A and Y are unconfounded.Then P(y | a, S = 1) = P(y | a) = P(y | do(a)).Source: §F.1.2 · p. 58, Figure 13 · p. 59
  • Figure 13(a) does not permit recovery without further assumptions, while the confounded, selected-data case in Figure 13(c) requires choosing an adjustment set usable despite selection.Among {W1, W2}, {W1, X}, {W2, X}, and X, only X is separated from S.Source: §F.1.2 · pp. 58–59, Figure 13 · p. 59
  • With confounding and selection bias, choosing X yields P(y | do(a)) = Σ_x P(y | a, x, S = 1)P(x | S = 1).The framework’s ability to select covariates relevant to identifiability is presented as an advantage of the SCM approach.Source: §F.1.2 · p. 59
  • For combining selected and population-level data, Bareinboim and Pearl (2016) give the sufficient, but not necessary, S-backdoor criterion: X must be backdoor admissible, block paths from S to Y given A, and be measured in both data sources.The criterion requires Y ⊥⊥ S | A, X.Source: §F.1.2 · p. 59
  • Under that criterion, the post-stratification formula weights selected-sample causal outcomes by target-population P(X = x); Pearl (2015) notes it is generally wrong when X includes post-treatment covariates.Pearl (2015) also describes another weighting strategy for generalizing with post-treatment variables.Source: §F.1.2 · p. 59

F.2 Proof of the transport formula (6)

  • The transport formula proof moves from conditioning to the S-admissibility assumption for X, then uses that X is pre-treatment to replace P(X = x | do(A = a)) with P(X = x).The resulting expression combines P(Y | do(A = a), X = x, S = 1) with P(X = x).Source: §F.2 · pp. 59–60

G Additional simulation results

The section presents additional simulation results following Section 6.2.

G.1 Distributional shift between RCT and observational samples · G.2 Stratification

The simulations examine covariate distributional shift between RCT and observational samples, including its effects on estimator variance, and assess how the number of strata affects prediction.

G.1 Distributional shift between RCT and observational samples

  • The simulated RCT covariates tend to have lower values than those in the observational sample.The setting uses approximately 1,000 RCT observations and 10,000 observational observations; Figure 14 illustrates the distributions and X1 overlap.Source: §G.1 · p. 60, Figure 14 · p. 60
  • Despite this distributional shift, the overlap assumption holds because every target-sample observation has a nonzero probability of inclusion in the experimental sample.Source: §G.1 · p. 60
  • Under stronger covariate shift, the weighted and CW estimators have increased variance.Figure 15 compares weak and strong shifts across 100 simulations.Source: §G.1 · p. 60, Figure 15 · p. 61
  • The simulated sampling propensity model produces weak covariate shift; replacing the −0.5X1 coefficient with −1.5X1 strengthens the shift at least for X1.Source: §G.1 · p. 60

G.2 Stratification

  • The stratification simulations vary the number of strata from 3 to 15, and higher numbers of strata yield better prediction.The results are reported in Figure 16.Source: §G.2 · p. 61

G.3 Impact of a hidden treatment effect modifier … H.2 Common covariates description between CRASH-3 and Traumabase

The simulations show when trial estimates can or cannot generalize under treatment-effect heterogeneity, while the Traumabase and CRASH-3 sections address missing data, confounding, and alignment of study definitions and populations.

G.3 Impact of a hidden treatment effect modifier

  • IPSW using only X1 remains unbiased, whereas omitting this shifted treatment-effect modifier produces strong bias.Including variables that affect both sampling and outcome can ensure identifiability; with a homogeneous treatment effect, the RCT estimate alone is consistent for the population ATE.Source: §G.3 · pp. 61–62, Figure 17 · p. 63

G.4 Homogeneous treatment effect

  • With a homogeneous treatment effect, the RCT estimate τ1 is consistent for the population ATE, so transport methods using observational data are unnecessary.The supplied simulation specifies Y(a) | X = −100 + X1 + 13.7X2 + 13.7X3 + 13.7X4 + 27.4a + ϵ.Source: §G.4 · p. 62, §G.3 · p. 62, Figure 18 · p. 63

H.1.1 Missing values

  • Missingness ranges from 0 to nearly 60% and may be MCAR or informative MNAR, so causal identification requires assumptions about both complete-case effects and missingness.Mayer et al. (2020) describe imputation with IPW or doubly robust estimators under M(C)AR and unconfoundedness, missing-aware doubly robust methods under adapted unconfoundedness, and latent-variable models for noisy proxies.Source: §H.1.1 · pp. 64–65, Figure 19 · p. 64

H.1.2 Covariate adjustment

  • Six critical-care specialists used a two-stage Delphi process to classify covariates and construct a causal graph, qualitatively assessing unconfoundedness despite untestable unobserved confounding.The graph guides variable selection and identifiability assessment, while treatment-effect estimation uses methods from the potential-outcomes framework.Source: §H.1.2 · p. 65, Figure 20 · p. 66

H.2 Common covariates description between CRASH-3 and Traumabase

  • CRASH-3 restricts TXA to administration within 3 hours, while Traumabase lacks timing and administration details; experts consider the within-3-hour assumption plausible, and outcome definitions differ.Both studies assess in-hospital death within 28 days, but CRASH-3 specifies head-injury-related death, whereas Traumabase includes a wider range of causes.Source: §H.2 · pp. 65–66
  • CRASH-3 spans countries and continents, whereas Traumabase covers more than 20 French trauma centers; this heterogeneity motivates estimating effects for populations with French high-level care.More than 65% of CRASH-3 patients came from developing countries, where pre-hospital care, access to brain imaging and neurosurgery, and specialist trauma care differ from developed-country settings.Source: §H.2 · pp. 66–67
  • Four CRASH-3 eligibility criteria depend on five variables, and exact variables or close proxies in Traumabase allow estimation of the trial-inclusion model on combined data.The criteria concern adult age, TBI, absence of major extracranial bleeding, and injury-to-randomization delay; the delay threshold was later reduced from 8 to 3 hours.Source: §H.2 · p. 67
  • The combined covariate set also includes sex, systolic blood pressure, and pupils reactivity because they may moderate treatment effects and are identified as outcome-related baselines.Pupils reactivity is categorical and ranges from 0 to 2, representing the number of active pupils.Source: §H.2 · p. 67

H.3 Additional analysis · H.3.1 Distributional shift between CRASH-3 and Traumabase

The additional analysis proposes examining distributional shift and other diagnostics, while the CRASH-3 and Traumabase histograms show differences in patient distributions and suggest overlap holds.

H.3 Additional analysis

  • The additional analysis proposes a principal component analysis of the combined database.Source: §H.3 · p. 67
  • Propensity scores estimated with logistic regression or a forest are to be examined using histograms and scatter plots.Source: §H.3 · p. 67
  • The proposed additional analyses also include a focus on patient strata defined by injury severity.Source: §H.3 · p. 67

H.3.1 Distributional shift between CRASH-3 and Traumabase

H.3.2 Principal component analysis … H.4.3 Generalizing treatment effect on patient strata

Across the combined CRASH-3 and Traumabase analyses, results vary by injury severity, and generalization estimates can be affected by missing-data patterns. For mild-to-moderate TBI with major extracranial bleeding, the presented methods cannot support transport under the necessary assumptions.

H.3.2 Principal component analysis

  • PCA of the combined data relates Glasgow Coma Scale to pupil reactivity and also links age with systolic blood pressure.The text explains the latter association by the relationship of atherosclerosis with both age and increased blood pressure.Source: §H.3.2 · p. 70, Figure 26 · p. 71

H.3.3 Conditional odds

  • Generalization from CRASH-3 to Traumabase yields extreme conditional odds, strengthened by the forest method, which uses missing systolic blood-pressure values to predict study membership.The differing missing-value patterns influence propensity-score estimates and warrant attention when combining the two data sets.Source: §H.3.3 · p. 71, Figure 27 · p. 72, Figure 28 · p. 73

H.4 Evidence on other patient strata

  • The analysis focuses on patient subgroups defined by brain-trauma severity measured using Glasgow Coma Scale or pupil reactivity.Source: §H.4 · p. 74

H.4.1 Traumabase: evidence on different strata

  • Traumabase shows no average treatment effect in any severity stratum, whereas CRASH-3 finds a beneficial effect for mild forms of TBI.The strata use pupil reactivity and Glasgow Coma Scale severity criteria.Source: §H.4.1 · p. 74, Table 7 · p. 77

H.4.2 CRASH-3: evidence on different strata

  • CRASH-3 shows significant treatment effects only in some severity strata, with the strongest effect and statistical evidence among mild-to-moderate injuries.The analysis attributes the attenuation toward the null to the poor prognosis of patients with GCS 3 or bilateral non-reactive pupils regardless of treatment; Table 8 presents risk ratios and ATEs.Source: §H.4.2 · p. 74, Table 8 · p. 77

H.4.3 Generalizing treatment effect on patient strata

  • For severe TBI, generalization supports either no effect or a deleterious effect, while CRASH-3 and Traumabase estimators support no effect.The comparison concerns severe Traumabase patients; the supplied analysis does not support transport for mild-to-moderate TBI patients with major extracranial bleeding because the necessary assumptions are not met. In the mild-to-moderate stratum, CRASH-3 has no patients with major extracranial bleeding, while Traumabase has only four treated patients without it, versus 1,867 untreated.Source: §H.4.3 · p. 74, Figure 30 · p. 75
Loading 2011.08047v4…