Multiple imputation of multilevel missing data: An introduction to the R package pan

• Multilevel models account for dependency among persons nested within groups and allow predictor effects to vary across groups.Source: Multilevel Modeling: An Empirical… · pp. 5–6
• Approximately 99.3% of students had available reading-achievement and cognitive-ability scores, while disciplinary-problem ratings in mathematics were missing for half the sample by design.Source: Multilevel Modeling: An Empirical… · p. 6

Example 1: Random-Intercept Model

• The first model estimates between- and within-group effects of disciplinary problems and SES on mathematics achievement using group-mean centering.Source: Example 1: Random-Intercept Model · p. 7
• The random-intercept model separates class-level residual variation from student-level residual variation.Source: Example 1: Random-Intercept Model · p. 7

Example 2: Random-Slope Model

• The second model allows the association between cognitive ability and mathematics achievement to vary across classes through a random slope.Source: Example 2: Random-Slope Model · p. 8

• Listwise deletion can reduce power and produce biased results, motivating multiple imputation for incomplete data.Source: Multiple Imputation of Incomplete… · p. 8
• Multiple imputation draws replacements from the missing-data distribution conditional on observed data, analyzes each completed data set, and combines results.Source: Multiple Imputation of Incomplete… · pp. 8–9
• The article assumes MCAR for some planned missingness and MAR for remaining missingness conditional on observed auxiliary variables.Source: General Aspects of MI · pp. 9–10
• An imputation model should be at least as complex as the analysis model, including interactions, clustered structure, and random slopes when relevant.Source: General Aspects of MI · p. 10
• Single-level dummy-indicator methods can distort parameter estimates and standard errors in multilevel analyses.Source: General Aspects of MI · p. 10
• pan follows the joint-modeling approach, imputing incomplete variables simultaneously with one statistical model.Source: General Aspects of MI · p. 11

• The pan model is a multivariate extension of regular multilevel models and represents multiple dependent variables simultaneously.Source: The Multivariate Linear Mixed-Effects… · p. 11
• The model expresses each response vector as fixed-effects contributions, random-effects contributions, and residuals: yij = xijβ + zijbj + eij.Source: The Multivariate Linear Mixed-Effects… · p. 11
• Random effects are assumed normally distributed across groups, while individual residuals are normally distributed across persons.Source: The Multivariate Linear Mixed-Effects… · pp. 11–12
• The MLMM imputes left-hand-side variables conditional on right-hand-side variables containing fixed and random effects.Source: The Multivariate Linear Mixed-Effects… · p. 12
• Only response variables may contain missing values; predictor variables must be completely observed.Source: The Multivariate Linear Mixed-Effects… · p. 12

• The multivariate empty model uses only an intercept on the predictor side while placing all variables of interest on the response side.Source: Multivariate empty model · p. 12
• For each response variable, the model separates variance and covariance into between-group Ψ and within-group Σ portions.Source: Multivariate empty model · p. 12
• The model is a multivariate version of the null or intercept-only multilevel model, with dependent variables decomposed across levels.Source: Multivariate empty model · p. 12
• Its covariance matrices allow relationships among dependent variables at both group and person levels.Source: Multivariate empty model · p. 12
• The empty model is useful for estimating individual- and group-level relationships, including settings with group-level predictors.Source: Multivariate empty model · p. 12

Full mixed-effects model

• The model estimates fixed and random effects for variables included as predictors in the multivariate linear mixed-effects model.Dependent variables may contain missing data, whereas right-hand-side predictors must be completely observed.Source: Full mixed-effects model · p. 13
• The full mixed-effects model includes both fixed and random effects for the intercept and observed predictors.It can represent random intercepts and slopes in the imputation model.Source: Full mixed-effects model · p. 13
• Variables are not decomposed in the full model, so fixed and random effects represent their overall effects on the dependent variables.Separate within- and between-group effects require decomposing the predictor into between- and within-group portions before MI.Source: Full mixed-effects model · p. 13
• Random slopes in the analysis model should be represented in the imputation model through corresponding slope variances.This is why the full mixed-effects model is particularly useful for models with random slopes.Source: Full mixed-effects model · p. 13

Software Alternatives

• Mplus implements a two-level procedure similar to pan’s empty model, denoted H1, and a more complex procedure, denoted H0.H0 is intended for models such as random-slope models.Source: Software Alternatives · p. 13
• Joint modeling approaches for multilevel multiple imputation are available in SAS, REALCOM, and the R package jomo.Source: Software Alternatives · p. 13
• The R package mice provides a fully conditional specification approach to multiple imputation.Source: Software Alternatives · p. 13
• Some alternative packages can support analysis steps similar to those outlined for pan.The article indicates that this possibility is discussed later.Source: Software Alternatives · pp. 13–14

Example Applications with Multilevel Missing Data

• mitml provides a convenient interface to pan and tools for managing and analyzing multiply imputed data sets.The examples also compare pan with Mplus, single-level MI, and listwise deletion.Source: Example Applications with Multilevel… · pp. 14–15
• The examples used lme4 to fit analysis models to the imputed data sets and Rubin’s rules to combine parameter estimates.mitml also provides functions for extracting completed data sets and performing the analysis workflow.Source: Example Applications with Multilevel… · p. 14, Example 1: Random-Intercept Model · p. 17
• Multilevel missing-data analysis specifies an imputation model, generates multiple imputed data sets, analyzes each separately, and combines the estimates.The imputation model should include variables and auxiliary variables relevant to the analysis model.Source: Example Applications with Multilevel… · pp. 14–15
• pan uses MCMC to simulate parameters and missing-value replacements from the posterior predictive distribution.Burn-in precedes the imputation phase, and saved imputations are separated by iterations to reduce autocorrelation.Source: Example 1: Random-Intercept Model · p. 16

Intraclass correlations

• Multilevel MI produced larger ICC estimates than single-level MI, especially for variables with substantial missing data.Listwise-deletion estimates were closer to multilevel MI without an obvious systematic pattern.Source: Intraclass correlations · p. 20
• Higher SES was associated with higher math achievement, whereas disciplinary problems were associated with lower math achievement.Class-level estimates were roughly twice as large as student-level estimates.Source: Intraclass correlations · p. 22
• Single-level MI yielded similar within-group estimates but consistently larger between-group estimates than multilevel MI.Source: Intraclass correlations · p. 22
• Listwise deletion produced larger standard errors, especially at the student level, and smaller class-level effect estimates.Source: Intraclass correlations · p. 22
• The contextual effect of SES was statistically significant at p < .001 (F = 15.297, df 1 = 1, df 2 = 1293.0).Classes with higher SES tended to have higher math achievement after controlling for student-level SES.Source: Intraclass correlations · p. 23
• The latent-group-mean and manifest-group-mean analysis models led to essentially the same conclusions in the example.Source: Intraclass correlations · p. 24

• The largest ˆR was 1.001, indicating that the MCMC chain had become stationary for all parameters.Source: Convergence diagnostics · p. 26
• Diagnostic plots showed that parameters related to the variables of interest converged quickly and were not greatly affected by autocorrelation.Source: Convergence diagnostics · p. 26
• Autocorrelation was quite persistent for some parameters, especially the group-level variance components.Source: Convergence diagnostics · p. 26
• Autocorrelation vanished for all parameters with a lag of 15,000 to 20,000 iterations.Source: Convergence diagnostics · p. 26

• The model examined student-level variables and included predictors centered around their group means using within.Source: Model of interest · p. 26
• Students with higher cognitive ability than their class average tended to score higher on the math achievement test after controlling for SES.Source: Model of interest · p. 27
• The relation between cognitive ability and math achievement appeared to vary substantially across groups.Source: Model of interest · p. 27
• Compared with single-level MI, the multilevel estimates differed slightly, whereas Mplus H0 imputation produced results similar to pan4.Source: Model of interest · p. 27
• The pooled LRT was statistically significant at p = .006 (F = 5.119, df 1 = 2, df 2 = 10386.2), indicating that slope variance differed from zero.Source: Model of interest · p. 28
• Students with different cognitive ability may differ more or less strongly in their math achievement scores depending on their class.Source: Model of interest · p. 28
• The LRT approach may have low statistical power for testing variance components, and few alternatives are available for multiply imputed data.Source: Model of interest · p. 28

• Researchers may prefer jomo, mice, or Mplus when the analysis includes categorical variables with missing data.Source: Analyzing Imputations Generated by… · pp. 28–29
• The analysis steps presented can be carried out on multiply imputed data sets irrespective of their origin.Source: Analyzing Imputations Generated by… · p. 29
• mitml’s analysis functions require multiply imputed data sets to be represented as a list of data sets in R.Source: Analyzing Imputations Generated by… · p. 29
• mitml includes wrapper functions for pan and jomo, plus conversion functions for data sets generated by mice.Source: Analyzing Imputations Generated by… · p. 29
• For other software packages, imputed data must be converted manually, using functions such as long2mitml.list or as.mitml.list.Source: Analyzing Imputations Generated by… · p. 29
• Using wrapper functions is recommended because they allow access to mitml’s convergence diagnostics.Source: Analyzing Imputations Generated by… · p. 29

Motivation

• Multilevel missing-data methods remain underused in applied research, with listwise deletion, single-level MI, and ad hoc approaches still prevalent.The paper attributes this partly to the limited accessibility of state-of-the-art procedures and software.Source: Discussion · pp. 29, 31

Empirical illustrations

• The two examples illustrate pan and mitml for random-intercept models, between- and within-group effects, random slopes, and slope-variance estimation and testing.Example 1 focuses on random-intercept models; Example 2 addresses random slopes.Source: Discussion · pp. 29–30

Practical contribution

• pan preserves multilevel features, uses available and auxiliary information, and supports reliable estimation without altering the model of interest.The discussion contrasts these capabilities with simpler methods that may distort or lose multilevel-model features.Source: Discussion · p. 30

Empirical illustrations

• Multilevel MI using pan produced substantially different parameter estimates from single-level MI, while listwise-deletion estimates were sometimes closer to the multilevel results.These findings illustrate that ignoring clustering can distort estimates in subsequent multilevel analyses.Source: Discussion · pp. 29–30

Open challenges

• Important limitations remain for missing predictors in models with random slopes or interactions, and for estimating multilevel-model goodness-of-fit measures after MI.The paper notes that clear recommendations are still lacking and that some advanced methods are not available in standard software.Source: Discussion · pp. 30–31

Practical contribution

• mitml provides tools for combining parameter estimates, model comparisons, and model constraints across multiply imputed data sets, including Rubin’s rules and D1–D3 procedures.The package addresses the challenge of analyzing multiply imputed data when software lacks tools for combining results.Source: Discussion · p. 31