Source-linked AI summary
Combining Observational and Experimental Datasets Using Shrinkage Estimators
Evan Rosenman, Guillaume Basse, Art Owen, Michael Baiocchi
TL;DR
Observational data are abundant but can produce biased causal estimates when confounders are unmeasured, while experiments are often costly and imprecise. The paper develops shrinkage estimators to combine the sources and reports lower risk than experimental-only estimation under tested conditions, with theoretical risk guarantees under stated conditions.
Problem
Combining observational data, which may retain unmeasured-confounding bias, with costly experiments that can have imprecise estimates is a central causal-inference challenge.
Source: §1 · p. 2Method
The paper derives shrinkage estimators by minimizing a generalized unbiased risk estimate to trade off biased observational and unbiased experimental estimates.
Source: §7 · pp. 19, 22Results
The proposed estimators have testable finite-K conditions for lower risk than the unbiased estimator and achieve a notion of optimality as K grows; simulations report lower risk than experimental-only estimation under all tested conditions.
Source: §7 · p. 22Takeaways & Limitations
The estimators provide a template for managing the bias-variance tradeoff when combining observational and experimental data to estimate causal effects.
Source: §7 · p. 22The theory assumes well-defined shared strata and common average treatment effects across datasets, and risk reduction requires at least four strata, potentially more with substantial heteroscedasticity.
Source: §3.2 · p. 5, §7 · p. 22
Abstract
from arXiv · showhide
We consider the problem of combining data from observational and experimental sources to make causal conclusions. This problem is increasingly relevant, as the modern era has yielded passive collection of massive observational datasets in areas such as e-commerce and electronic health. These data may be used to supplement experimental data, which is frequently expensive to obtain. In Rosenman et al. (2018), we considered this problem under the assumption that all confounders were measured. Here, we relax the assumption of unconfoundedness. To derive combined estimators with desirable properties, we make use of results from the Stein Shrinkage literature. Our contributions are threefold. First, we propose a generic procedure for deriving shrinkage estimators in this setting, making use of a generalized unbiased risk estimate. Second, we develop two new estimators, prove finite sample conditions under which they have lower risk than an estimator using only experimental data, and show that each achieves a notion of asymptotic optimality. Third, we draw connections between our approach and results in sensitivity analysis, including proposing a method for evaluating the feasibility of our estimators.
1 Introduction
Observational data offer scale but can yield biased causal estimates when confounders remain unmeasured, while experiments support unbiased estimates but are often small and imprecise. This paper combines the sources using shrinkage, developing estimators and conditions for lower risk than using experimental data alone.
1 Introduction
- Massive observational datasets may inform causal effects, but unmeasured confounding can undermine estimates based solely on them.Analysts are cautioned against trusting those estimates without qualification.Source: §1 · p. 2
- Experiments can yield unbiased causal estimates without problematic statistical assumptions, but their expense and smaller samples can make subgroup estimates imprecise.The paper frames observational data as having a bias problem and experimental data as having a variance problem.Source: §1 · p. 2
- The paper relaxes the earlier all-confounders-measured assumption, allowing residual bias of unknown magnitude after propensity-score stratification.The earlier approach could quantify residual bias from imperfect stratification under its unconfoundedness assumption.Source: §1 · p. 2
- The authors propose a generalized-unbiased-risk procedure, two shrinkage estimators with finite-sample lower-risk conditions and asymptotic optimality, and a sensitivity-analysis-based feasibility method.The paper also applies its methods to a simulated dataset.Source: §1 · p. 2
2 Related Literature
Related work spans transportability from experiments to target populations and methods that incorporate observational causal estimates. Prior approaches commonly assume unconfoundedness or impose alternative structures on hidden confounding.
2 Related Literature
- Transportability concerns extending causal findings across domains, and observational data have been used to assess generalization of experimental effects to target populations.Prior work also derives assumptions and placebo tests for identifying population treatment effects from randomized trials.Source: §2 · p. 3
- Relatively little prior work incorporates observational causal estimates, reflecting concerns that doing so may introduce bias.Some work assumes all treatment and outcome causes are measured, while other approaches impose parametric hidden-confounding structure or preserved unit-level bias ranks.Source: §2 · p. 3
- Green and Strawderman study combining biased and unbiased estimators; one proposed estimator dominates the unbiased estimator and has bounded risk as bias grows.When bias is very small, it can underperform a precision-weighted estimator.Source: §2 · p. 3
- Green et al. propose estimators for heteroscedastic settings, with dominance results depending on the loss and estimation of the experimental covariance.The shrinkage parameter defaults to K −2 when information about ξ is absent.Source: §2 · pp. 3–4
3 Notation, Assumptions, and Set-Up
The setup defines stratum-specific causal effects in observational and randomized studies, then combines their estimates under allocation and shared-effect assumptions. The analysis uses weighted loss and approximate estimator distributions while allowing observational estimates to be biased.
3.1 Setup
- The setup assigns units to observational or randomized samples and defines potential outcomes, measured covariates, an unmeasured covariate, treatment, and observed outcomes.For observational units, the propensity score is unknown and depends on observed and unobserved covariates.Source: §3.1 · p. 4
3.2 Assumptions and Loss Function
- The analysis assumes observational treatment follows unit-specific Bernoulli probabilities and experimental treatment is randomly allocated within strata.A known stratification scheme and stratum weights define the target structure; the common-treatment-effect assumption equates average effects across populations within each stratum.Source: §3.2 · pp. 4–5
- Shared stratum effects provide a mathematically convenient target but may not hold precisely for finite-sample averages; the paper leaves approximate or adjusted alternatives for future work.The analysis targets individual stratum effects rather than an overall average treatment effect.Source: §3.2 · p. 5
- Stratum weights represent the importance of accuracy in each stratum and may approximate target-population weights or use observational frequencies as a surrogate.The diagonal loss-weight matrix uses entries dk/K.Source: §3.2 · pp. 5–6
3.3 Estimator Distributions
- The randomized-study estimator is approximately normal under sufficient sample sizes and regularity conditions, while the observational estimator is allowed an unspecified distribution and bias parameter.That bias arises from within-stratum correlation between potential outcomes and propensity scores; the covariance matrices are diagonal under the assumptions.Source: §3.3 · p. 6
4 Proposed Estimators
The section derives shared and component-wise shrinkage estimators by optimizing a generalized unbiased risk estimate, then addresses their risk properties and practical variance and propensity-score adjustments.
4.1 Preliminaries
- The procedure chooses a shrinkage structure, minimizes a generalized unbiased risk estimate to derive its factor, and can optionally correct for estimating that factor from data.The risk result is generalized to weighted loss with a diagonal matrix encoding the relative importance of components.Source: §4.1 · p. 7
4.2 κ1, Common Shrinkage Factor
- For a shared shrinkage factor, the stated testable condition guarantees lower risk than the RCT-only estimator, requiring at least four strata and possibly more when variances or weights vary substantially.The estimator's positive-part version has the lowest risk among shared-factor estimators as K approaches infinity, under mild conditions.Source: §4.2 · pp. 8–9
- The corrected shared-factor estimator does not necessarily outperform the uncorrected versions, but provably beats κ1 when stratum variances and weights are sufficiently concentrated.The correction accounts for added risk from estimating the shrinkage factor; the procedure stops after one correction iteration.Source: §4.2 · pp. 9–10
4.3 κ2, Variance-Weighted Shrinkage Factors
- Variance-weighted shrinkage relies more on the RCT estimate for components with small RCT variance and more on the observational estimate when that variance is large.Its functional form is derived by minimizing the unbiased risk estimate.Source: §4.3 · p. 11
- Under mild conditions, κ2 has the lowest risk as K approaches infinity among estimators using variance-weighted shrinkage factors across components.The estimators also have finite-sample properties analogous to those established for the preceding estimator.Source: §4.3 · p. 11
4.4.1 Variance Estimation
- When within-stratum treatment effects are heterogeneous, estimated RCT variances tend to be biased upward, which can shift or distort shrinkage toward observational estimates.The resulting distortion differs by estimator: κ1+ shifts toward the observational estimate in expectation, while κ2+ may over-shrink some components and under-shrink others.Source: §4.4.1 · p. 12
- Possible mitigations are to use smaller strata or to calculate a menu of estimators under different assumed within-stratum potential-outcome correlations.A correlation of 1 corresponds to the stated variance-estimation assumption; lower correlations represent greater treatment-effect heterogeneity.Source: §4.4.1 · p. 12
4.4.2 Propensity Score Adjustment
- Propensity-score adjustment can reduce, but not eliminate, observational selection bias; stabilized inverse probability weighting is advocated because it supports a relatively straightforward sensitivity analysis.Because the observational study is assumed much larger than the RCT, an increase in variance may be compensated by reduced bias.Source: §4.4.2 · pp. 12–13
5 Sensitivity Analysis
The sensitivity analysis uses assumptions about unmeasured confounding to estimate a conservative shrinkage tradeoff and assess whether the data-driven estimator implies a plausible degree of confounding.
5.1 Set-Up
- Sensitivity analysis can estimate the weighted-MSE-optimal tradeoff by positing stratum-specific violations of unconfoundedness and plugging worst-case bias and variance into the shrinkage formula.Unlike estimating the shrinkage factor from parallel causal-effect estimates, this approach translates untestable confounding assumptions into a conservative tradeoff.Source: §5.1 · p. 13
- In the marginal sensitivity model, Γ bounds the odds ratio between true and estimated treatment probabilities; Γ = 1 means no unmeasured confounding, and larger values mean greater deviations.For stabilized inverse-probability weighting, worst-case bias is obtained through a linear fractional program and variance is estimated by bootstrap.Source: §5.1 · pp. 13–14
5.2 Estimating Implied Γ
- For each proposed Γ, the method estimates stratum-level worst-case bias and bootstrap variance, plugs them into λopt, and uses binary search to find Γimp matching λ1+ within tolerance.The bias estimates come from the extrema of estimates consistent with the stratum's observational data and sensitivity model.Source: §5.2 · p. 14
- Γimp gives an interpretable indication of the bias implicit in the shrinkage estimate; if it is within the analyst’s intuitive range, the paper says this licenses proceeding with the analysis.Bootstrap variance makes the algorithm random, but with enough replicates and a reasonable tolerance it should converge quickly.Source: §5.2 · p. 14
6 Simulations
The simulations compare shrinkage estimators across settings with differing bias, strata, and covariate distributions. Risk reductions are larger when stabilized weighting reduces observational selection bias.
6.1 Simulation Set-Up
- Simulations use 10,000 observational and 1,000 RCT units, with a correlated unmeasured covariate in treatment assignment so IPW cannot fully remove selection bias, and vary strata count and sizes.The simulations use 6 or 20 strata and equal- or variable-sized strata.Source: §6.1 · pp. 15–16
6.2 Identical Observational and RCT Covariate Distributions
- Without adjustment, δ1 leads in the 6-strata similar-size condition, κ⋆1+ in the 6-strata variable-size condition, and κ1+ in the 20-strata similar-size condition; single-factor estimators generally outperform component-level ones, while δ2 lags.The 20-strata variable-size winner is κ⋆1+.Source: §6.2 · pp. 16–17, Figure 1 · p. 17
- Without adjustment, all estimators improve on ˆτr, typically by about 1% (6 strata) and 5% (20 strata), while ˆτo risk is about 50 and 15 times higher, respectively.Risk reductions are about half the oracle's with few strata and move closer to oracle performance with more strata.Source: §6.2 · pp. 16–17, Figure 1 · p. 17
- With stabilized IPW, our estimators' risk reductions are about 25% (6 strata) and 40% (20 strata) versus ˆτr, with performance quite close to the oracle especially at 20 strata.The observational estimator's risk is about 40% higher than ˆτr with 6 strata and almost identical with 20 strata.Source: Figure 2 · p. 18, §6.2 · p. 19
- With stabilized IPW, δ1 leads in the 6-strata similar-size condition, κ⋆1+ in the 20-strata similar-size condition, and κ1+ in both variable-size conditions.With more strata, single-factor estimators tend to outperform their component-level counterparts.Source: Figure 2 · p. 18, §6.2 · p. 19
6.3 Differing Observational and RCT Covariate Distributions
- Without stabilized IPW adjustment for τ̂o, all estimators generally degrade relative to the oracle; δ1 leads in the first three conditions, while κ⋆2+ leads in the 20-strata, variable-size condition.With stabilized IPW estimation for τ̂o, δ1 modestly edges κ⋆1+ in each condition.Source: §6.3 · p. 19
7 Conclusion
The authors develop URE-based shrinkage estimators to combine observational and experimental estimates despite residual confounding. Simulations report lower risk than experimental-only estimation across tested conditions, with theoretical and practical qualifications.
7 Conclusion
- The procedure derives shrinkage estimators by minimizing a generalized unbiased risk estimate, with optional further optimization to address reuse of data for shrinkage and estimation.It first generalizes a result from Strawderman, then specifies and minimizes the tradeoff estimator’s structure.Source: §7 · p. 19
- Figures 3 and 4 compare risk reductions relative to ˆτr under four conditions, contrasting unadjusted ˆτo with ˆτo computed using stabilized IPW.Both figures also vary covariate distributions between observational and experimental units; their captions do not state which estimator wins.Source: Figure 3 · p. 20, Figure 4 · p. 21
- Both new estimators have testable finite-K conditions for lower risk than the unbiased estimator and achieve a notion of optimality as K grows.The conclusion identifies the unbiased estimator as one using experimental data alone.Source: §7 · p. 22
- Across all tested conditions, the proposed estimators outperform experimental-only estimation; κ1 typically has lower risk than Green and Strawderman’s estimators when covariate distributions are similar, while δ1 is slightly more robust when they differ.The simulations use a small experiment and a larger observational study with unmeasured confounding.Source: §7 · p. 22
- The general procedure could support alternative shrinkage structures, including using auxiliary information to shrink less in strata thought to have more observational bias.Another suggested extension is thresholding, relying only on the experimental estimate when the two estimates differ beyond a threshold.Source: §7 · p. 22
- The theoretical results assume well-defined strata and shared average treatment effects across datasets; guaranteed risk reduction requires at least four strata and potentially more under substantial heteroscedasticity or differential weights.The authors identify weakening these assumptions as future work.Source: §7 · p. 22
8 Proof of Theorem 2
The proof establishes asymptotic optimality for the constant-shrinkage estimator by showing that its unbiased risk estimate tracks its actual loss and is minimized over the shrinkage choice.
8 Proof of Theorem 2
- As K grows, the constant-shrinkage estimator selected by minimizing the unbiased risk estimate is asymptotically optimal among the considered constant-shrinkage choices.The proof bounds its excess loss by twice the maximum discrepancy between estimated risk and actual loss, which vanishes in expectation.Source: §8 · pp. 25, 27
- The argument uses uniform closeness of the unbiased risk estimate to actual loss, then compares the selected shrinkage factor with any fixed choice of λ.The proof establishes the convergence using separate term bounds, including a martingale and an Lp maximal inequality.Source: §8 · pp. 26–27
- The stated minimization result is under Conditions 1–3 and holds for any choice of λ in the comparison.Source: §8 · p. 27
9 Proof of Theorem 3
The proof of Theorem 3 establishes uniform convergence of the unbiased risk estimate to actual loss for the variance-weighted shrinkage family, using regularity conditions and an argument analogous to Theorem 2.
9 Proof of Theorem 3
- The proof is substantively similar to Theorem 2 and shows L2 convergence by analogous computations and regularity-condition bounds.Source: §9 · p. 28
- The supremum over λ of the discrepancy between estimated risk and actual loss for variance-weighted shrinkage converges to zero.The proof states that the first three regularity conditions guarantee the result.Source: §9 · pp. 28–29
- For the remaining term, the argument uses a martingale and the Lp maximal inequality.Source: §9 · p. 28