PaperlayerView exact revision on arXiv

Source-linked AI summary

Flexible sensitivity analysis for observational studies without observable implications

Alexander Franks, Alexander D'Amour, Avi Feller

arXiv:1809.00399v3stat.ME

TL;DR

Data do not reveal assumptions about missing potential outcomes, making sensitivity analysis necessary; existing approaches can entangle it with observed-data modeling. The paper applies Tukey’s factorization to separate identified from unidentified components, and demonstrates the framework with flexible observed-data models.

  • Problem

    Data can inform models of observed potential outcomes but are uninformative about assumptions for missing potential outcomes, which must be probed through sensitivity analysis.

    Source: §1 · p. 2

  • Method

    Tukey’s factorization separates identified and unidentified parts of the potential-outcome distribution, representing unidentified factors with interpretable selection functions.

    Source: §1 · p. 2

  • Results

    The framework is demonstrated with BART response-surface models, Dirichlet-process mixture residuals, and zero-inflated models for semi-continuous data.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The framework requires fitting the observed-data model only once, supporting post hoc sensitivity analysis and parameter calibration to selection on observed covariates.

    Source: §7 · p. 29

    The approach requires outcome overlap, and inferred missing-outcome distributions can be sensitive to estimated density tails when observed and missing outcomes are far apart (Franks et al., 2016).

    Source: §7 · p. 30

Abstract

from arXiv · show

A fundamental challenge in observational causal inference is that assumptions about unconfoundedness are not testable from data. Assessing sensitivity to such assumptions is therefore important in practice. Unfortunately, some existing sensitivity analysis approaches inadvertently impose restrictions that are at odds with modern causal inference methods, which emphasize flexible models for observed data. To address this issue, we propose a framework that allows (1) flexible models for the observed data and (2) clean separation of the identified and unidentified parts of the sensitivity model. Our framework extends an approach from the missing data literature, known as Tukey's factorization, to the causal inference setting. Under this factorization, we can represent the distributions of unobserved potential outcomes in terms of unidentified selection functions that posit an unidentified relationship between the treatment assignment indicator and the observed potential outcomes. The sensitivity parameters in this framework are easily interpreted, and we provide heuristics for calibrating these parameters against observable quantities. We demonstrate the flexibility of this approach in two examples, where we estimate both average treatment effects and quantile treatment effects using Bayesian nonparametric models for the observed data.

1 Introduction

Unconfoundedness assumptions cannot be tested from observed data, while some sensitivity methods constrain the observed-data model. The paper adapts Tukey’s factorization to separate model checking from sensitivity analysis while retaining flexible observed-data models.

1 Introduction

  • Tukey’s factorization separates identified observed-data factors from unidentified factors, representing the latter through interpretable selection functions and a copula for potential-outcome dependence.For a broad class of common estimands, the copula can be ignored, and missing potential outcomes can be interpreted as extrapolations from observed data.Source: §1 · p. 2
  • The framework extends Tukey’s factorization from missing-data methods to observational causal inference through a practical workflow.The approach adapts Franks et al. (2016); related methods have largely been applied to missing-data settings, including clinical trials with dropout.Source: §1 · pp. 2–3
  • The workflow accommodates flexible observed-data models, requires fitting them only once, and supports interpretable sensitivity parameters calibrated against covariate variation in a propensity score model.Examples include BART, Dirichlet process mixtures, and zero-inflated models for semicontinuous data.Source: §1 · p. 3
  • The paper applies the approach to two examples, estimating average and quantile treatment effects with Bayesian nonparametric models for observed data.The paper’s roadmap describes these applications and also covers formal setup, factorization, model specification, and calibration heuristics.Source: §1 · p. 3

2 Setup and Overview of Sensitivity Analysis

The section defines causal estimands using observed and missing potential outcomes, then outlines a factorization-based sensitivity analysis. It also contrasts this approach with prior methods and illustrates how parametric latent-confounder models can blur sensitivity analysis with model checking.

2.1 Setup and Notation

  • Average treatment effects for the population, treated, and control groups, as well as quantile treatment effects, contrast marginal distributions of the complete potential outcomes.Because each complete-data distribution mixes observed and missing outcomes, identifying these estimands requires assumptions about the missing outcomes, which the data do not inform.Source: §2.1 · p. 4

2.2 Summary of Approach

  • Tukey’s factorization represents missing potential-outcome distributions as tilts of observed distributions, using selection functions while leaving the observed-data distributions unchanged.The resulting weights transform observed outcomes into missing-outcome distributions, separating model checking from sensitivity analysis.Source: §2.2 · p. 5

2.3 Related Methods

  • The method builds on NPI models and turns their suggested use for unobserved confounding in observational studies into a practical sensitivity-analysis approach.Prior NPI applications focused mainly on longitudinal missing-data problems and clinical trials with dropout; related Bayesian methods also largely remained in missing-data settings.Source: §2.3 · p. 6

2.4 Separating Sensitivity Analysis from Model Checking

  • In a binary-latent-confounder example, most sensitivity settings misfit observed outcomes, so model fit can reflect the parametric specification rather than test unconfoundedness in full generality.Figure 1 shows observed control densities are correctly inferred only when assumed sensitivity parameters equal their true values; this example uses a continuous outcome and no covariates.Source: §2.4 · pp. 7–8, Figure 1 · p. 8
  • Mixing model checking with sensitivity analysis can require refitting for each parameter setting, enable data snooping, and complicate interpretation of Bayesian posterior uncertainty.These concerns arise when sensitivity parameters are informed by observed data.Source: §2.4 · p. 9

3 Tukey’s Factorization for Causal Inference

The paper extends Tukey’s factorization to observational causal inference by separating identified observed-data components from unidentified selection functions. For marginal contrasts such as ATEs and QTEs, the sensitivity analysis does not require specifying the copula between potential outcomes.

3 Tukey’s Factorization for Causal Inference

  • Causal inference can be treated as two missing-data problems, one for each potential outcome.The factorization includes a copula for dependence between potential outcomes, but sensitivity for a broad class of estimands does not depend on it.Source: §3 · pp. 9–10

3.1 Tukey’s Factorization with One Potential Outcome

  • For one potential outcome, the observed-data density is identified, while an unidentified selection function determines how observed outcomes relate to missing outcomes.The factorization requires integral constraints on selection functions and an outcome-overlap condition that cannot be enforced by model specification.Source: §3.1 · pp. 10–11

3.2.1 General factorization

  • The joint distribution of treatment and both potential outcomes is specified by two treatment-given-potential-outcome models and a conditional copula.Apart from the selection factors and copula, the other terms are identifiable.Source: §3.2.1 · pp. 11–12

3.2.2 Marginal Contrast Estimands

  • Marginal contrasts, including ATEs and QTEs, are invariant to the conditional copula, whereas estimands depending on the joint potential-outcome distribution are excluded.In Bayesian inference, posterior invariance holds when copula parameters are a priori independent of the other factor parameters; sensitivity analysis for unit-specific effects is left to future work.Source: §3.2.2 · pp. 12–13

4 Logistic Selection with Mixtures of Exponential Family Models

The proposed logistic-mixture exponential-family models combine logistic selection functions with flexible observed-data models. Their analytic tractability supports sensitivity analysis with both single exponential-family distributions and mixtures.

4 Logistic Selection with Mixtures of Exponential Family Models

  • Logistic-mEF models pair logistic selection functions with exponential-family mixtures, accommodate flexible observed-data models including DPM and BART, and make missing-outcome distributions analytically tractable.The framework also applies to DPMs of exponential-family distributions and structured semi-continuous data.Source: §4 · p. 13, §4.2.2 · p. 18

4.1 Logistic selection specifications

  • Logistic selection makes treatment-assignment log-odds linear in sufficient statistics of potential outcomes, with sensitivity parameters γ_t and an identified arm-specific parameter α_t(X).The main practical difficulty is computing a normalizing constant, which can require expensive calculation or restrictions on the observed-data model.Source: §4.1 · pp. 14–15

4.2 Mixtures of exponential family models

  • When observed-data densities are exponential-family mixtures, the normalizing constant is analytically tractable under logistic selection.The paper leaves specifications outside the logistic-mEF family to future work.Source: §4 · p. 14, §4.2 · p. 15

4.2.1 Single exponential family models

  • With a single exponential-family observed-data model, the missing potential-outcome distribution remains in that family when the shifted natural parameter lies in its natural parameter space.For Bernoulli outcomes, logistic selection implies an additive shift in the missing outcomes’ log-odds relative to the observed distribution.Source: §4.2.1 · pp. 15–16

4.2.2 Extension to mixtures

  • For a K-component exponential-family mixture with common sufficient statistics, the missing potential-outcome distribution is a mixture of the same family, provided the shifted parameters remain in the natural parameter space.The sensitivity parameters can affect mixture weights and component means.Source: §4.2.2 · pp. 17–18

5 Calibrating Sensitivity Parameters

The paper interprets sensitivity parameters through how potential outcomes relate to treatment assignment and proposes calibrating their magnitude using variation explained. It maps that calibration target to the sensitivity parameters under stated conditions.

5.1 Interpreting the sign of sensitivity parameters

  • Sensitivity parameters describe how potential outcomes are over- or under-represented among observed treated and control units, and their signs indicate the direction of that selection.When both parameters have the same sign, group means are biased in opposite directions; with opposite signs, they are biased in the same direction.Source: §5.1 · pp. 18–19

5.2.1 Calibrating against variation explained

  • The proposed magnitude calibration targets the variation in treatment assignment explained by a potential outcome beyond observed covariates, using observed predictors as analogues for plausible targets.This uses an implicit R2 measure and is described as robust to multicollinearity among predictors.Source: §5.2.1 · pp. 20–21

5 Calibrating Sensitivity Parameters

  • Directly calibrating γ_t against regression coefficients from observed covariates is discouraged because collinearity can make those coefficients difficult to interpret.The proposed variance-explained calibration is robust to multicollinearity.Source: §5.2.1 · p. 20

5.2.2 Mapping between variation explained and sensitivity parameters

  • Under the stated logistic model, the partial variance explained by a potential outcome is monotone in γ_t, giving a one-to-one mapping for calibration.The mapping uses the variance of the propensity-score logit and the residual standard deviation of the potential outcome.Source: §5.2.2 · pp. 21–22

6 Applications of the Logistic-mEF Model

The applications use flexible observed-outcome models with Tukey’s factorization to examine average and quantile treatment effects under different sensitivity settings. The NHANES and JTPA examples show how those settings relate to estimated effects and counterfactual outcome distributions.

6.1 Analysis of NHANES data

  • The NHANES analysis uses BART to model observed potential-outcome means and separate residual variances by treatment arm, without latent confounders in the BART model.ATEs and QTEs are estimated from marginal complete-data potential-outcome distributions averaged over covariates.Source: §6.1 · p. 23
  • In NHANES, calibrated sensitivity magnitudes are limited to about |γ0| = 0.046 and |γ1| = 0.048.These values are obtained using the estimated residual standard deviations for the two treatment arms.Source: §6.1 · p. 24
  • The NHANES ATE becomes no longer significantly different from zero when γ = 0.01 along the diagonal where γ = γ0 = γ1.The passage defines “not significant” as a 95% posterior credible interval containing zero.Source: §6.1 · p. 24
  • For NHANES, τq increases with q across the explored sensitivity settings, consistent with greater variance in treatment than control potential outcomes.The reported residual variance for treated potential outcomes is slightly larger than for control potential outcomes.Source: §6.1 · p. 25
  • The NHANES QTEs remain sensitive to assuming normally distributed residuals despite flexible response-surface modeling.The appendix compares alternative observed-data models while holding treatment-selection specifications fixed.Source: §6.1 · p. 25

6.2 Analysis of Job Training Data

  • For JTPA’s zero-inflated income outcomes, the analysis models employed log income with a Dirichlet process mixture and specifies separate selection functions for employed and unemployed individuals.Participation among employed individuals is logistic in log income; for unemployed individuals, odds of participation shift multiplicatively by exp(ωt).Source: §6.2 · p. 26
  • JTPA counterfactual employment and income distributions differ across sensitivity settings, with the lower tail of employed income especially sensitive to selection.Among missing control outcomes assigned to treatment, the unemployed fraction is roughly twice as large for ω = −0.1 as for ω = 0.1.Source: §6.2 · p. 28
  • In the JTPA analysis, QTEs are zero at the lowest quantiles, then their direction and significance vary with employment shares and sensitivity settings.With very large selection effects, such as (γ, ω) = (0.1, 0.4), effects lose significance and shift slightly negative.Source: §6.2 · pp. 28–29

7 Discussion

The framework separates identified data modeling from sensitivity analysis, supports post hoc use with flexible models, and offers interpretable parameters that can be calibrated against observed covariates. Its scope is limited by covariate-independent sensitivity specifications and outcome-overlap concerns, among other extensions left for future work.

7 Discussion

  • The framework separates identified and unidentified components, so sensitivity parameters remain unidentified and model checking is decoupled from sensitivity analysis.A single data fit reduces computational burden and supports post hoc sensitivity analysis across models that previously assumed unconfoundedness.Source: §7 · p. 29
  • A sensitivity analysis could fit modern two-stage workflows by inserting a sensitivity-parameterized weighting step between observed-data modeling and estimation.The proposed weighting step would not constrain the flexible first-stage model.Source: §7 · p. 29
  • The paper restricts sensitivity specifications to be covariate-independent; allowing γt(X) would add parameters and create calibration and reporting challenges.Multiple or multilevel treatments and observational studies with missing data are also left for future work.Source: §7 · p. 30
  • Outcome overlap requires missing potential-outcome support to lie within observed-outcome support, but inferred distributions can remain sensitive to estimated tails when the densities are far apart.When covariates explain most outcome variance, even small unobserved-confounder variation may violate overlap.Source: §7 · p. 30
  • The authors conclude that the framework is general, interpretable, implementable, and extendable to a range of models, including complex data-generating models.Source: §7 · p. 30

A Theory

The appendix proof relates treatment-effect estimands to the marginal potential-outcome distributions and argues that the copula parameters do not affect them. It also connects the logistic selection specification to calibration through a residual coefficient of determination.

A Theory

  • The proof states that the considered treatment-effect estimands depend on potential-outcome marginals, then shows the cross-arm distributions are independent of copula parameters.The argument uses the extrapolation factorization, which models the observed-arm potential-outcome distribution directly.Source: A Theory · p. 31
  • The calibration argument seeks γt values that achieve a target residual R², relating γt to the residual-scaled parameter ˜γt.Source: A Theory · pp. 31–32
  • The logistic selection specification sets logit e(X,Y(t)) to αt(X) + γtY(t), with αt(X) identified once γt is specified.Source: A Theory · p. 32

B Additional Results from Section 5.1

The additional results show that ATE estimates vary with both observed-data model specification and confounding strength. The pooled and Bayesian Causal Forest models yield different uncertainty under unconfoundedness, while the sensitivity parameter is calibrated against BMI-related quantities.

B Additional Results from Section 5.1

Related papers

Loading 1809.00399v3…