Source-linked AI summary
Removing Hidden Confounding by Experimental Grounding
Nathan Kallus, Aahlad Manas Puli, Uri Shalit
TL;DR
Observational data offers scale and covariate coverage but may contain hidden confounding, while experiments provide unconfounded evidence with narrower scope. The paper introduces a parametric correction learned by comparing both sources and proves consistency under identification conditions, with empirical support from simulation and educational data.
Problem
Hidden confounding can invalidate causal estimates from large observational datasets, while experimental data is often limited in scale or population coverage.
Method
The method learns a parametric hidden-confounding correction by combining a large observational sample with a smaller unconfounded experimental sample requiring only support overlap.
Results
The method consistently estimates CATE under parametric identification conditions and produces more accurate estimates than baselines in the reported experiments.
Takeaways & Limitations
The approach can estimate CATE for populations lacking experimental data under weaker conditions than methods assuming observational unconfoundedness.
Takeaways & Limitations
Existing reweighting alternatives require observational unconfoundedness and experimental support containing the observational support; the paper also identifies non-comparable outcomes or interventions as future work.
Abstract
from arXiv · showhide
Observational data is increasingly used as a means for making individual-level causal predictions and intervention recommendations. The foremost challenge of causal inference from observational data is hidden confounding, whose presence cannot be tested in data and can invalidate any causal conclusion. Experimental data does not suffer from confounding but is usually limited in both scope and scale. We introduce a novel method of using limited experimental data to correct the hidden confounding in causal effect models trained on larger observational data, even if the observational data does not fully overlap with the experimental data. Our method makes strictly weaker assumptions than existing approaches, and we prove conditions under which it yields a consistent estimator. We demonstrate our method's efficacy using real-world data from a large educational experiment.
1 Introduction
Observational data offers large-scale, fine-grained information for individual-level causal inference, but hidden confounding can invalidate causal estimates. The paper studies combining such data with smaller experiments that may differ in population and scope.
- Observational datasets provide many samples and covariates, enabling fine-grained effects applicable to diverse target populations.
- Hidden confounders affect both treatment assignment and outcomes while remaining unmeasured, creating a central obstacle to causal inference.
- Hidden confounding can produce bias that does not vanish even with infinitely many observations.
- Experiments can help address hidden confounding, but their participants may not represent the target population, creating an external-validity problem.
- The paper targets fine-grained causal effects using large observational samples and possibly much smaller experimental samples with limited population overlap.
- Its approach models hidden confounding parametrically rather than imposing a parametric structure on potentially complex treatment effects.
2 Setup
The setup combines confounded observational data with unconfounded experimental data to estimate individual-level treatment effects. The experiment provides identification within its domain, while observational data supplies broader scale or scope for extending those estimates.
- The framework studies binary treatments using observational data that may be confounded and unconfounded data typically obtained from an experiment.
- The unconfounded sample may come from an RCT or an internally recognized sub-experiment where treatment depends only on observed covariates.
- The unconfoundedness assumption implies that CATE can be identified by regression for covariates within the experiment’s support.
- Experimental identification is limited because the unconfounded sample may cover a restricted domain and contain relatively little data.
- The observational data may have confounded study selection and treatment assignment with potential outcomes.
- The observational conditional outcome difference ω(x) differs from the causal effect τ(x), with their difference represented as the confounding effect η(x).
3 Method
The method estimates a confounded observational CATE function, learns a parametric hidden-confounding correction from unconfounded data, and adds that correction to obtain CATE estimates over the observational population. It extrapolates the correction rather than the full CATE function, requiring only overlap of experimental support with observational support.
- Method: The method first fits an observational CATE estimate ω̂, then learns a correction η̂ from unconfounded samples and combines them as τ̂(x) = ω̂(x) + θ̂ᵀx.The correction is identified using unconfounded outcomes and propensity scores, while the observational model supplies the broader population coverage.
- Method: The correction term adjusts observational CATE estimates for hidden confounding and is extrapolated across the confounded sample.This uses the identity τ(X) = ω(X) + η(X).
- Method: Unconfounded samples support identification through propensity-score-based signed reweighting, with constant propensity q in a randomized controlled trial.The paper defines e_Unc(x) as the unconfounded-study propensity score and notes that RCTs typically have e_Unc(x) = q, often q = 0.5.
- Method: Unlike direct CATE extrapolation, the approach extrapolates only the hidden-confounding correction; if CATE extrapolates well, the learned correction is approximately zero.This avoids relying on a parametric structure for the entire CATE function.
- Comparison: RCT reweighting alternatives assume no observational unmeasured confounding and experimental support containing the observational support, assumptions this method does not require.Those alternatives can require inverse weights close to or identical to zero when experimental and observational populations differ.
4 Theoretical guarantee
Under parametric identification of the hidden-confounding correction and stated regularity conditions, Algorithm 1 consistently estimates CATE over the confounded population. The assumptions are weaker than observational unconfoundedness, with strong overlap required only in the unconfounded sample.
- Guarantee: Under parametric identification of η, Algorithm 1 consistently estimates τ(x) over the confounded population at a rate governed by estimation of ω̂.For the linear specification, identification requires a nonsingular design matrix.
- Assumptions: The base observational CATE estimator must be consistent, while covariates, outcomes, and its predictions require finite fourth moments in the experimental data.These conditions support the theorem’s rate and regularity requirements.
- Assumptions: The covariates in the confounded data must cover those in the unconfounded data, although the overlap condition may be relaxed when sufficient intersection remains for learning η.This formalizes the observational dataset’s broader scope relative to the experimental dataset.
- Assumptions: The correction specification may use any finitely parameterized form whose parameters are identifiable from the experimental dataset.The linear specification is therefore sufficient but not exclusive.
- Interpretation: Because the correction model includes the zero function, its assumption is strictly weaker than assuming the observational data is unconfounded.When the observational sample is unconfounded, the correction is η(x) = 0 with θ₀ = 0.
- Assumptions: Strong treatment overlap is required only in the unconfounded sample, where it holds trivially for randomized trials with constant propensities.The paper contrasts this with the stronger difficulty of requiring overlap in high-dimensional confounded data.
5 Experiments
The experiments evaluate the method in simulated partially overlapping data and in Tennessee STAR data, using randomized information to assess CATE estimation under hidden confounding. Across these settings, the two-step method uses unconfounded data to improve estimates beyond baselines.
- 5.1 Simulation study: The simulation combines unconfounded and confounded samples with only partial covariate overlap and evaluates recovery of the true CATE τ(X).The unconfounded covariate is sampled from [−1, 1], while the confounded sample extends beyond that range.
- 5.1 Simulation study: Even with unconfounded data restricted to [−1, 1], the method reasonably estimates τ despite limited access to the full X^2 effect and strong unobserved confounding.The simulated outcome includes nonlinear treatment–covariate interactions, while the true CATE is τ(X) = 0.75X^2 + 2X + 1.
- 5.2 Real-world data: The Tennessee STAR experiment supplies randomized class-size assignments and standardized test outcomes for evaluating CATE predictions against an unbiased ground truth.After excluding missing outcomes, the randomized sample contains 4,218 students: 1,805 treated and 2,413 controls.
- 5.2 Real-world data: The evaluation creates a smaller unconfounded subset and a larger confounded subset with partial population overlap by splitting students according to rural or inner-city versus urban or suburban status.The construction selectively removes higher-scoring treated students, producing downward-biased naive treatment-effect estimates.
- 5.2 Real-world data: Performance is measured by RMSE on a held-out unconfounded version of ALL \ UNC, rather than directly on the confounded subset.The comparison includes regression and difference-of-regressions baselines using Random Forest or Ridge Regression, alongside the two-step method.
- 5.2 Real-world data: The two-step Random Forest and Ridge variants consistently outperform the baselines and improve as the unconfounded subset grows.The result is reported for the Tennessee STAR evaluation shown in Figure 2.
6 Discussion
The discussion frames the method as addressing causal inference with large observational datasets aided by experiments on different populations. It identifies extensions for causal-graph information and for settings where outcomes or interventions are not directly comparable.
- 6 Discussion: The paper targets users who combine large observational datasets with unconfounded experiments conducted on different populations.This is the scenario the authors identify as increasingly common.
- 6 Discussion: Future work includes combining the method with analyses of causal graphs containing measured and unmeasured covariates.
- 6 Discussion: Another extension concerns settings where experimental and observational outcomes or interventions are not directly comparable but their difference can be modeled.The paper gives short-term experimental outcomes versus longer-term observational outcomes as an example.
A Proofs
The proofs analyze estimation-error terms under the paper’s conditions and derive an asymptotic bound for the estimated confounding correction and CATE. The conclusion is stated in probabilistic order notation rather than as an empirical result.
- A Proofs: The proof treats the experimental design matrix and associated vectors as intermediate objects in decomposing the estimation error.
- A Proofs: The proof bounds an error term using Cauchy–Schwarz together with a finite second-moment condition.
- A Proofs: The argument uses the outcome-weighting formula to establish that a centered term has expectation zero.
- A Proofs: The final bound applies to the estimated CATE error under the stated conditions, combining the correction-estimation rate with the 1/n term.