PaperlayerView exact revision on arXiv

Source-linked AI summary

Demystifying Double Robustness: A Comparison of Alternative Strategies for Estimating a Population Mean from Incomplete Data

Joseph D. Y. Kang, Joseph L. Schafer

arXiv:0804.2958v1stat.ME

TL;DR

Estimating a population mean with incomplete outcomes can require adjusting for covariates related to both outcomes and missingness. This paper reviews and compares doubly robust and other estimators, finding in its simulations that none of the tested doubly robust methods improved on simple regression-based prediction when both models were moderately misspecified.

  • Problem

    The practical performance of doubly robust estimators when both adjustment models are moderately misspecified remains an important question for estimating population means under uncontrolled nonresponse.

    Source: §1.1 · pp. 1–2

  • Method

    The paper reviews incomplete-data estimation strategies and how double robustness arises, then compares doubly robust and non-doubly-robust estimators under correct and moderately misspecified models.

    Source: §1.1 · p. 2

  • Results

    In the simulations, no tested doubly robust procedure outperformed a y-model-only procedure, and propensity-based corrections to OLS tended to move estimates in the wrong direction.

    Source: §4 · p. 15

  • Takeaways & Limitations

    In settings like the simulation, two approximate models should not be assumed to improve on a reasonably predictive outcome model.

    Source: §4 · p. 15

    The discussion's overlap conclusion is bounded to cases where estimated propensity distributions overlap enough to form five strata and true propensities are nonzero.

    Source: §4 · p. 16

Abstract

from arXiv · show

When outcomes are missing for reasons beyond an investigator's control, there are two different ways to adjust a parameter estimate for covariates that may be related both to the outcome and to missingness. One approach is to model the relationships between the covariates and the outcome and use those relationships to predict the missing values. Another is to model the probabilities of missingness given the covariates and incorporate them into a weighted or stratified estimate. Doubly robust (DR) procedures apply both types of model simultaneously and produce a consistent estimate of the parameter if either of the two models has been correctly specified. In this article, we show that DR estimates can be constructed in many ways. We compare the performance of various DR and non-DR estimates of a population mean in a simulated example where both models are incorrect but neither is grossly misspecified. Methods that use inverse-probabilities as weights, whether they are DR or not, are sensitive to misspecification of the propensity model when some estimated propensities are small. Many DR methods perform better than simple inverse-probability weighting. None of the DR methods we tried, however, improved upon the performance of simple regression-based prediction of the missing values. This study does not represent every missing-data problem that will arise in practice. But it does demonstrate that, in at least some settings, two wrong models are not better than one.

1. INTRODUCTION

The article surveys multiple ways to construct doubly robust estimates and compares them with non-DR methods. It frames their practical performance under model misspecification as an open question, despite consistency when either model is correct.

1.1 Purpose

  • The article presents several distinct constructions of doubly robust estimates, including generalized regression and parametric methods, and investigates their behavior when both underlying models are moderately misspecified.It also compares strategies that use an outcome model, a response-probability model, or both; the pooling method can affect performance.Source: §1.1 · p. 2, §1.4 · p. 5

1.2 Description of the Problem

  • For population-mean estimation, the observed respondent mean estimates the respondent mean, but observed data alone generally cannot identify the nonrespondent or population mean.The article also connects this missing-outcome problem to estimating average causal effects by separately estimating means for each potential outcome.Source: §1.2 · pp. 2–3

1.3 Assumptions

  • Strong ignorability assumes outcomes and response are conditionally independent given covariates, implying missing at random and providing the benchmark on which doubly robust procedures rest.The article notes that MAR may be unrealistic in applications, but uses it as a starting point for sensitivity analyses.Source: §1.3 · p. 3
  • Propensity-model misspecification can be problematic, while outcome-model misspecification can cause extrapolation for nonrespondents, especially when covariate distributions differ or covariates are high-dimensional.The article notes the difficulty of specifying flexible outcome models without inflating prediction variance.Source: §1.3 · pp. 3–4

1.4 A Simulated Example

  • The simulation uses covariates that make both the outcome and propensity models incorrect but apparently plausible, with fitted values under each misspecified model correlating about 0.9 with its true counterpart.In the illustrated sample, outcome prediction has R2 = 0.81, and a check of the propensity-model link gives p = 0.20.Source: §1.4 · pp. 4–5
  • With a 50% response rate, the simulated population, respondent, and nonrespondent means are 210.0, 200.0, and 220.0, respectively, making naive respondent-mean inference problematic despite modest selection bias.The passage says the respondent-population difference is one-quarter of a population standard deviation, yet large enough to harm confidence intervals and tests based on the naive estimate.Source: §1.4 · p. 4

2. WEIGHTING, STRATIFICATION AND REGRESSION

The section compares weighting, propensity stratification, regression prediction, and combined stratification strategies for estimating a population mean. In the simulated example, combined methods can improve on propensity-based methods, but not necessarily on regression prediction when both models are wrong.

2.1 Inverse-Propensity Weighting

  • IPW is sensitive to propensity-model misspecification and extreme weights; its bias and RMSE worsen as sample size grows, while NR weighting outperforms POP weighting in this example.Even when propensities are correctly modeled, IPW has higher variance than other examined procedures; with misspecification it breaks down and may require ad hoc weight modifications.Source: §2.1 · p. 7, Table 1 · p. 7
  • Nearly 50% lower bias and RMSE result from using a robit link with ν = 4 instead of logit for misspecified IPW models at n = 1000.Robit models offer minor improvement with correct covariates and major improvement when covariates are wrong.Source: §2.1 · p. 8

2.2 Propensity-Based Stratification

  • Propensity stratification has higher RMSE than IPW when the propensity model is correct, but under misspecification it is more biased at n = 200 and less biased at n = 1000.Coarsening propensities can stabilize the largest weights, offsetting the added bias with greater efficiency; the stratified bias does not worsen as sample size increases.Source: §2.2 · p. 8, Table 2 · p. 9

2.3 Regression Estimation

  • When the outcome model is incorrect, regression estimates are more stable and efficient than estimates using an incorrect propensity model, though residual bias can impair inference at n = 1000.With a correct outcome model, bias is removed; with a misspecified model, remaining bias exceeds 50% of a standard error at n = 1000.Source: §2.3 · p. 9, Table 3 · p. 9

2.4 Stratification by Propensity Scores and Predicted Values

  • With both models incorrect, dual stratification improves on propensity-only estimates but performs worse than OLS using the incorrect outcome model.The authors describe the gains over a propensity model alone as modest and conclude that two incorrect models are not necessarily better than one.Source: §2.4 · p. 10

3. CONSTRUCTING DOUBLY ROBUST ESTIMATES

The section presents residual-bias correction, inverse-propensity weighted regression coefficients, and propensity-based covariates as ways to construct doubly robust estimates. In the simulations, their performance depended on model misspecification and extreme weights; the five-category propensity-covariate method was the strongest DR method when both models were wrong, but simple OLS remained better in that condition.

3.1 Regression Estimation with Residual Bias Correction

  • Residual-bias correction estimates the population mean residual using a π-model to correct OLS bias, and is doubly robust because either a correct y-model or π-model makes the correction valid.With a correct y-model, the residual correction has expectation zero; with a correct π-model, it estimates the first term’s bias.Source: §3.1 · p. 10
  • When both models were misspecified, bias-corrected regression performed worse than IPW and OLS; occasional enormous weights caused erratic estimates, and trimming did not reverse the comparison.Its local semiparametric efficiency property assumes both models are correct; normalizing weights, using nonrespondent weights, or stratifying the correction improved this estimate in the example.Source: Table 5 · p. 11, §3.1 · p. 12

3.2 Regression Estimation with Inverse-Propensity Weighted Coefficients

  • Inverse-propensity weighted least squares estimates regression coefficients that yield a consistent mean estimate if either the π-model or the y-model is correct.The weighted coefficients target the full-population linear regression even if it does not accurately represent the conditional outcome mean.Source: §3.2 · pp. 12–13

3.3 Regression Estimation with Propensity-Based Covariates

  • Adding functions of estimated propensities to the outcome model can address lack of fit along the propensity-score direction, with consistency if either model is correct under the stated smoothness and approximation conditions.A single inverse-propensity covariate is sufficient for double robustness, but the resulting regression performed disastrously when some estimated propensities were small and the π-model was misspecified.Source: §3.3 · pp. 13–15

3.2 Regression Estimation with Inverse-Propensity Weighted Coefficients

  • When both models were misspecified, inverse-propensity weighted regression coefficients performed much better than bias-corrected OLS, but a misspecified π-model made matters worse than OLS.The WLS estimate was sometimes inferior to bias-corrected OLS when one model was true.Source: Table 6 · p. 13, §3.2 · p. 13

3.3 Regression Estimation with Propensity-Based Covariates

  • With both models incorrect, five-category propensity-covariate regression outperformed the other DR methods and methods based on a π-model alone, but simple OLS remained better under an incorrect y-model.Polynomial splines sometimes produced erratic predictions for low-propensity nonrespondents and increased variance.Source: Table 7 · p. 14, §3.3 · p. 15

4. DISCUSSION

Double robustness does not guarantee good performance when both models are misspecified: in this simulation, no DR method improved on regression-based prediction. When propensities must be estimated, analysts should choose methods that are not overly sensitive to propensity-model misspecification.

Discussion

  • Double robustness can arise in many ways, but it does not ensure good performance under dual misspecification; when propensities are estimated, choose an estimator not overly sensitive to propensity-model misspecification.In survey settings, propensities may be known from the sample design, and related estimators have long been viewed as robust.Source: §4 · p. 15
  • The simulation approximated a high-school dieting study with strong outcome prediction and overlapping estimated propensities, though some nondieters had very small propensities and were useful proxies for dieters.The goal was to estimate the mean BMI dieters would have had if they had not dieted.Source: §4 · p. 15
  • With moderate selection bias, good outcome predictors, approximate models, and some near-zero estimated propensities, no DR method improved on regression-based prediction; bias corrections moved its estimate in the wrong direction.The fitted propensity model did not reduce the bias from the approximate outcome model.Source: §4 · p. 15
  • Across all 1,000 samples at n = 200 and n = 1,000, estimated propensities overlapped enough for five-quintile stratification, and none of the true propensities was zero; routine use of inverse-propensity weighting is unwarranted.The authors considered complete empirical support coverage too stringent, particularly because estimated ranges were sensitive to minor propensity-model changes.Source: §4 · p. 16
  • Schafer and Kang (2005) found a DR procedure performed slightly better than parametric multiple imputation when outcome-model R2 was about 0.5 and the smallest fitted propensities were 0.14 and 0.07.The authors suggest AIPW can yield modest gains when outcome prediction is not strong and estimated propensities are not near zero.Source: §4 · p. 16
  • Across Bang and Robins (2005)'s examples, DR under dual misspecification was clearly better than both incorrect-model competitors only in the fourth example, and its advantage there was slight.Their examples kept estimated propensities away from zero, yet did not show that two wrong models were necessarily better than one.Source: §4 · p. 16

Related papers

Loading 0804.2958v1…