PaperlayerView exact revision on arXiv

Source-linked AI summary

How many imputations do you need? A two-stage calculation using a quadratic rule

Paul T. von Hippel

arXiv:1608.05406v6stat.ME

TL;DR

The paper addresses how many imputations are needed when users want standard-error estimates that remain replicable after re-imputation, rather than only efficient point estimates. It proposes a two-stage procedure using a small pilot to estimate missing information conservatively and calculate the final number of imputations. The procedure uses a quadratic rule and achieved the targeted SE replicability in the reported example, while five pilot imputations produced highly variable recommendations.

  • Problem

    The traditional recommendation of M=2 to 10 imputations addresses efficient point estimates but not necessarily efficient and replicable variance or standard-error estimates.

  • Method

    The paper estimates the fraction of missing information from a small pilot, uses the upper bound of its 95% confidence interval, and applies a quadratic rule to determine stage 2 imputations.

  • Results

    Across 100 replications, the SD of stage 2 SE estimates was .001, while the reported M=5 example produced final estimates and SEs differing by .001 across repetitions.

  • Takeaways & Limitations

    Reliable SE estimates require choosing imputations based on the fraction of missing information and the desired replicability, not relying only on the traditional 2–10 rule.

  • Takeaways & Limitations

    Estimated degrees of freedom is an unreliable guide because estimation error can make df > 100 appear sufficient when it is not.

Abstract

from arXiv · show

When using multiple imputation, users often want to know how many imputations they need. An old answer is that 2 to 10 imputations usually suffice, but this recommendation only addresses the efficiency of point estimates. You may need more imputations if, in addition to efficient point estimates, you also want standard error (SE) estimates that would not change (much) if you imputed the data again. For replicable SE estimates, the required number of imputations increases quadratically with the fraction of missing information (not linearly, as previous studies have suggested). I recommend a two-stage procedure in which you conduct a pilot analysis using a small-to-moderate number of imputations, then use the results to calculate the number of imputations that are needed for a final analysis whose SE estimates will have the desired level of replicability. I implement the two-stage procedure using a new Stata command called how_many_imputations (available from SSC) and a new SAS macro called %mi_combine (available from the website missingdata.org).

1 OVERVIEW

Multiple imputation estimates are affected by imputation variation, so the number of imputations must be chosen differently for efficient point estimates versus replicable standard errors. The paper recommends a conservative two-stage calculation based on a quadratic rule and a pilot estimate of missing information.

  • Overview: Multiple imputation fills missing values repeatedly, analyzes each completed dataset, and combines the results into point and standard-error estimates.Imputed values are drawn from a posterior predictive distribution conditioned on observed values.
  • Why the number matters: M = 2 to 10 imputations usually suffice for efficient point estimates, but replicable standard errors may require M > 200 in some situations.The older recommendation addresses point-estimate efficiency rather than standard-error replicability.
  • A quadratic rule: For replicable standard errors, the required number of imputations follows a quadratic rather than linear relationship with the fraction of missing information.The linear rule M = 100γ_mis understates requirements when γ_mis is large and overstates them when it is small.
  • A two-stage procedure: γ_mis is the fraction of information missing about a parameter, not simply the fraction of values missing, and it is typically estimated using multiple imputation.Because γ_mis is unknown beforehand, estimating it requires an initial choice of M.
  • A two-stage procedure: The first stage uses a small-M pilot to estimate γ_mis conservatively and derive a target coefficient of variation from the pilot standard error and desired standard-error variability.The recommended conservative estimate uses the upper bound of a 95% confidence interval for γ_mis.
  • A two-stage procedure: The second stage uses the pilot estimates in the quadratic rule to determine M for the final analysis, unless the pilot already used enough imputations.A blanket M = 200 policy may be practical for small, fast analyses but inefficient for large or slow ones.

2 APPLIED EXAMPLE

The applied example uses ECLS-K body-mass-index data to illustrate the two-stage procedure. The analysis simplifies the survey design by treating the complex sample as a simple random sample, although accounting for that complexity changes the standard errors and recommended imputations.

  • Missingness: 76% of round-3 BMI measurements were missing: 70% by design and 6% because children were unavailable or refused weighing.The design-related missingness came from measuring only a random 30% subsample of participating schools.
  • Assumption: Treating the ECLS-K as a simple random sample omits clusters, strata, and sampling weights from the analysis and imputation models.An analysis accounting for sample complexity produced SEs 50% larger and recommended twice as many imputations.
  • Baseline: Listwise deletion estimated mean BMI as 16.625 with SE .037 while using only 24% of the sample.The observed values were almost a random sample because over 90% of missing values were missing completely at random.

2.3 Multiple imputation

The multiple-imputation analysis generates completed datasets from a multivariate normal model, analyzes each as complete, and combines the results. It also estimates missing information and the degrees of freedom of the standard-error estimate for the two-stage procedure.

  • Imputation model: Each analysis obtains M imputations of missing BMI values from a multivariate normal model for BMI measurements in rounds 1–4.The model imputes missing round-3 BMI using normal linear regression on observed BMI measurements from other rounds for the same child.
  • Analysis and combination: Each completed dataset is analyzed as complete to estimate its weighted sample mean and standard error, which are then combined into MI estimates.The combined outputs include a point estimate and a standard-error estimate.
  • Diagnostic quantities: The procedure also calculates a point estimate and 95% confidence interval for γ_mis, plus an estimated degrees of freedom for the standard-error estimate.These quantities support the two-stage recommendation for a final number of imputations.

2.5 Results with M=5 pilot imputations

With five pilot imputations, the two-stage procedure achieved the targeted replicability of final SE estimates, but the pilot produced highly variable recommendations for the final number of imputations.

  • Pilot and stage 2 results: M=5 pilot imputations yielded an estimated missing-information fraction of .39, with a 95% CI of (.15, .69) implying 125 stage 2 imputations.The stage 2 analysis returned an MI point estimate of 16.650 and an SE estimate of .021.
  • Pilot and stage 2 results: A replication recommended M=219 rather than 125 stage 2 imputations because the random imputation process changed the pilot estimates.
  • Replicated final estimates: The two final point estimates differed by .001, and the final SE estimates differed by .001, matching the targeted SE replicability.The corresponding final estimates and SEs were 16.650 (.021) and 16.651 (.022).
  • Replicated final estimates: Across 100 replications, the SD of the stage 2 SE estimates was .001, exactly matching the goal.
  • Variability in recommendations: The recommended number of imputations had a mean of 97 and an SD of 61, ranging from 4 to 266 across pilots.This variation arose because the pilot CI for the missing-information fraction varied substantially with only M=5 imputations.

2.6 Results with M=20 pilot imputations

Using 20 rather than 5 pilot imputations made stage 2 recommendations less variable and lower on average, while preserving the targeted replicability of SE estimates.

  • Why recommendations change: Using M=20 pilot imputations yields a narrower, more replicable confidence interval for the missing-information fraction and a more consistent final recommendation.
  • Results with M=20 pilot imputations: M=20 pilot imputations produced stage 2 recommendations with a mean of 62 and an SD of 26.The recommendations ranged from 22 to 167, covering one order of magnitude rather than two with M=5.
  • Comparison with M=5: Increasing pilot imputations from M=5 to M=20 reduced the average recommended final imputations from 97 to 62, a reduction of 35.
  • Why recommendations change: With fewer pilot imputations, high missing-information estimates are more likely and lead to high final recommendations.Because the recommendation increases with the square of the missing-information estimate, it is more sensitive when that estimate is high.

2.7 How many pilot imputations do you need?

The procedure generally protects SE replicability even with few pilot imputations, but larger pilots can reduce unnecessary size and variability in the final analysis.

  • General guidance: The two-stage procedure almost always ensures enough stage 2 imputations to achieve the desired SE replicability, regardless of pilot size.
  • General guidance: Few pilot imputations may produce unnecessarily high and variable stage 2 recommendations, especially when the true missing-information fraction is about 38%.
  • Choosing pilot size: When a large missing-information fraction is expected, using more pilot imputations is recommended; an example had 76% of values missing.
  • Choosing pilot size: When a low missing-information fraction is expected, a few pilot imputations may suffice, and any inadequate recommendation will not vary as much.

2.8 Why the estimated df is an unreliable guide

Estimated degrees of freedom can falsely suggest that five imputations are sufficient, because estimation error makes it an unreliable guide to the true degrees of freedom and SE replicability.

  • Estimated versus true df: About a quarter of pilot analyses with M=5 had estimated df > 100, despite M=5 being insufficient for the desired SE replicability.
  • Estimated versus true df: The true df is (M−1)γ_mis^-2, whereas the estimate substitutes the estimated missing-information fraction and is sensitive to its error.
  • Estimated versus true df: An estimated df above 100 does not guarantee that the true df exceeds 100 or that enough imputations were used.

3 VERIFYING THE QUADRATIC RULE

Bodner’s simulation supports the quadratic rule for selecting imputations to obtain replicable SE estimates. The linear rule overstates requirements at low missing information and understates them at high missing information.

  • M=200γ_mis^2 is the quadratic rule when the target CV is .05, whereas the comparison rule is M=100γ_mis.
  • The quadratic rule fits Bodner’s simulation results better than the linear rule.The simulation varied missing information and estimated the imputations needed for a criterion similar to CV(SÊ_MI|Y_obs)=.05.
  • At γ_mis=.5, the linear and quadratic rules agree.
  • For γ_mis<.5, the linear rule overstates the required imputations; for γ_mis>.5, it substantially understates them.
  • The quadratic rule slightly under-predicts Bodner’s required imputations, partly because his criterion differed from the target CV for SÊ_MI.Bodner’s criterion concerned confidence-interval half-width rather than SE directly, and the criteria are not equivalent when M is small.

4 FORMULAS

The formulas distinguish imputation requirements for point estimates, variance or SE estimates, and degrees of freedom. The resulting quadratic rules depend on missing information, which is estimated through a pilot analysis because its true value is unknown.

  • 4 FORMULAS: Relatively few imputations are needed for an efficient point estimate, while replicable SE estimates and precise missing-information estimates require more.
  • 4.1 Point estimates: MI creates M completed datasets, analyzes each as complete, and averages the resulting point estimates.
  • 4.1 Point estimates: Imputation variation decreases as M increases, but the infinite-imputation variance remains above the complete-data variance.
  • 4.2 Variance estimates: M=2 to 10 is adequate for efficient point estimates, but replicable variance or SE estimates require more imputations because the between-imputation variance is volatile when M is small.
  • 4.1 Point estimates: With M imputations, the point-estimate variance is (1+γ_mis/M) times its infinite-imputation value; at γ_mis=80% and M=10, the variance is 8% larger and the SE 4% larger.
  • 4.2 Variance estimates: The quadratic rule is obtained by approximating the coefficient of variation for the MI variance estimator and relating it to the SE estimator through CV(V̂_MI)≈2CV(SÊ_MI).
  • 4.3 Degrees of freedom: Degrees of freedom and SE-estimate CV are linked by CV=√(2/df); therefore, CV(SÊ_MI|Y_obs)=.05 corresponds to df=200.
  • 4.3 Degrees of freedom: Because γ_mis is unknown, a pilot analysis uses a conservatively large estimate, such as the upper limit of a 95% CI, to select M for the final analysis.Estimated degrees of freedom can be substantially inaccurate, so they are an unreliable indicator of whether enough imputations were used.

5 CONCLUSION

The paper replaces linear rules for replicable SE estimates with a quadratic rule and addresses the unknown missing-information fraction through a two-stage procedure. Software implementations make the procedure available in Stata and SAS.

  • Earlier guidance of M=2 to 10 addressed point-estimate variability, while newer linear rules for SE variability required γ_mis known in advance.
  • The proposed quadratic rule makes the required imputations increase with γ_mis^2.
  • The two-stage procedure estimates γ_mis in a small-M pilot, then uses the upper limit of a 95% CI to choose a conservative stage-2 number of imputations.
  • The procedure is implemented in Stata as how_many_imputations and in SAS as the %MI_COMBINE macro.

ENDNOTES

The endnotes clarify notation, pilot-based planning, simulation coverage, modeling assumptions, and degrees-of-freedom limitations.

  • SÊ_MI is a plug-in estimate for E(SÊ_MI|Y_obs).
  • The two-stage imputation procedure parallels pilot-based sample-size planning using an upper confidence-limit estimate of an unknown variance parameter.
  • The derivation of the confidence interval for γ_mis is asymptotic, but simulations showed close-to-nominal coverage with n=100, 40% missing information, and M=5.
  • Adding model rounds 5–7 increases runtime but produces little improvement in the imputation model’s R2.
  • The normal imputation model can yield approximately unbiased mean estimates for skewed BMI because regression imputations preserve the observed-BMI pattern.
  • The target CV estimate also varies because the estimated mean SE is itself estimated.
  • For vector estimates, V_MI may be defined as a covariance matrix, but only its diagonal squared SEs are used here.
  • The small-sample degrees-of-freedom formula supports calibrated inference but should not be used to choose M.
Loading 1608.05406v6…