Source-linked AI summary
robumeta: An R-package for robust variance estimation in meta-analysis
Zachary Fisher, Elizabeth Tipton
TL;DR
Traditional meta-regression struggles with dependent effect sizes because within-study correlations are often unknown. The paper presents robumeta, an R package implementing robust variance meta-regression with alternative weighting and large- or small-sample procedures. RVE supports valid inference without specifying the exact dependence structure, while small-sample adjustments improve finite-sample performance.
Problem
Traditional meta-analysis assumes independent effect sizes, although dependence is common and within-study covariance information is rarely reported.
Method
robumeta implements robust variance meta-regression using residual-based variance estimation, working covariance structures, alternative weights, and large- or small-sample procedures.
Results
RVE provides valid standard errors, point estimates, confidence intervals, and significance tests for non-independent effect sizes without modeling their exact dependence.
Takeaways & Limitations
robumeta makes RVE meta-regression available for analyses with unknown dependence and extends its practical utility through small-sample adjustments.
Takeaways & Limitations
Original RVE performance was limited in small samples, with slope-interval coverage only nominal when studies numbered 40 or fewer.
Abstract
from arXiv · showhide
Meta-regression models are commonly used to synthesize and compare effect sizes. Unfortunately, traditional meta-regression methods are ill-equipped to handle the complex and often unknown correlations among non-independent effect sizes. Robust variance estimation (RVE) is a recently proposed meta-analytic method for dealing with dependent effect sizes. The robumeta package provides functions for performing robust variance meta-regression using both large and small sample RVE estimators under various weighting schemes. These methods are distribution free and provide valid point estimates, standard errors and hypothesis tests even when the degree and structure of dependence between effect sizes is unknown.
1. Introduction
Non-independent effect sizes are common in meta-analysis, but traditional methods assume independence or require covariance information that is rarely reported. RVE addresses this problem, with small-sample adjustments extending its use to meta-analyses containing fewer studies.
- Sources of dependence: Non-independent effect sizes arise from shared units, repeated measurements, common controls, and hierarchical relationships among studies.Dependence may involve correlated estimation errors or correlated effect-size parameters.
- Limitations of traditional methods: Traditional meta-analysis assumes independent effect sizes, while study averaging loses information and multivariate methods require within-study covariances that are rarely reported.Multivariate meta-analysis is optimal when the covariance structure is known.
- Robust variance estimation: RVE provides consistent coefficients and valid inference for dependent effect sizes without modeling their exact covariance structure.The method accommodates non-normality and does not require fixed predictors.
- Small-sample limitation: Slope-interval coverage was only nominal above 40 studies, while more than 50% of psychology and education meta-analyses have fewer than 40 studies.Intercept coverage was adequate with as few as 10 studies.
- Package contribution: The robumeta package implements both original large-sample and small-sample adjusted RVE procedures for meta-regression.The small-sample adjustments are especially important below 40 studies and with unbalanced or highly skewed covariates.
2. RVE
RVE estimates meta-regression coefficients with weighted least squares while using residual cross-products to estimate unknown within-study covariance. Its weighting schemes target efficiency, and working covariance models represent common dependence structures.
- Model formulation: RVE formulates study-level effect estimates, residuals, design matrices, and weights within a weighted linear meta-regression model.The stacked model uses study-specific blocks and unknown regression coefficients.
- Variance estimation: Unknown within-study covariance is approximated with cross-products of residuals, yielding a robust variance estimator whose accuracy converges as the number of studies increases.The resulting robust standard errors support inference on regression coefficients.
- Weighting: RVE weights affect efficiency rather than validity, because asymptotically accurate standard errors and valid inference hold for any set of weights.More precise studies receive greater weight, while inverse-covariance weights are the efficiency target.
- Correlated effects: Correlated-effects weights use a working common-correlation model and assign equal within-study weights after partitioning total study variance across effect sizes.The general formulation permits unequal estimation-error variances within studies and uses average study variance.
- Parameter estimation: The weighting parameters are estimated through preliminary weighted regressions, residual sums of squares, and method-of-moments estimators.Correlated-effects results depend on the common correlation, although they vary little across correlations between 0 and 1.
- Hierarchical effects: Hierarchical-effects weights model within- and between-study variation, assigning effect-size weights based on estimation variance, between-study variance, and within-study variation.Weights are constant within a study only when effect-size error variances are equal.
3. Small-sample adjustments
Small-sample RVE adjustments correct both the variance estimator and coefficient-specific inference, extending robust meta-regression to settings where asymptotic approximations may be unreliable.
- Motivation: The original variance estimator converges to V(b) only asymptotically, motivating corrections for small-sample limitations.The paper notes that linearization estimators such as VR have documented small-sample limitations.
- Correction strategy: Two correction types adjust the RVE estimator itself and the degrees of freedom used for coefficient inference.These adjustments are attributed to Hedges et al. (2010) and extensions by Tipton.
- Variance adjustment: Tipton’s BRL-based solution specifies adjustment matrices separately for hierarchical and correlated-effects weighting models.The adjustment depends on weighting matrices and working covariance matrices.
- Variance adjustment: The adjustment framework also supports non-efficient user-specified weights and uses matrix transformations based on eigen decomposition.For non-efficient weights, robumeta specifies a corresponding adjustment matrix; the paper gives the general B^-1/2 construction.
- Inference adjustment: Tipton recommends Satterthwaite degrees of freedom for each coefficient, with simulations indicating validity when df > 4.The degrees of freedom are based on the coefficient of variation and are calculated separately across regression coefficients.
4. Testing and confidence intervals
RVE tests coefficient null hypotheses with robust t statistics and constructs confidence intervals using degrees of freedom that depend on the selected variance-estimation approach.
- Hypothesis tests: Robust tests evaluate H0 : βk = 0 using a t statistic based on the kth diagonal of the adjusted variance matrix.The null is rejected when the absolute test statistic reaches the relevant t critical value.
- Confidence intervals: A 95% confidence interval for each regression coefficient uses the corresponding degrees of freedom and adjusted variance estimate.The interval is formed separately for the kth coefficient.
- Estimator choice: The original approach uses equation 5 with df = m −p, whereas small-sample corrections use adjusted variance estimates.The corrected approach changes both the variance calculation and the inference degrees of freedom.
- Estimator choice: Under small-sample corrections, adjusted variance estimates differ across correlated-effects, hierarchical-effects, and non-efficient weighting models.The degrees of freedom dfSk vary from covariate to covariate.
5. The robumeta package
robumeta’s robu() function fits robust meta-regression models with selectable dependence structures, weighting options, and large- or small-sample RVE procedures.
- Core function: The robu() function is robumeta’s main fitting function and implements both large- and small-sample RVE estimators.The estimators correspond to Hedges et al. (2010) and Tipton (in press).
- Model specification: The formula specifies effect sizes as the outcome and user-selected covariates, with options for intercepts and interactions.Intercept-only and no-intercept models can be written using y ~ 1 and y ~ -1.
- Required data: The data frame includes effect sizes, within-study effect-size variances, and unique study identifiers.These correspond to effect.size, var.eff.size, and studynum.
- Weighting: Optional userweights allow non-efficient weighting when those weights are of interest.The option is distinct from the model’s default weighting procedures.
- Model options: Users select hierarchical or correlated-effects weighting through modelweights, specify rho for correlated effects, and choose small-sample corrections with small.The correlated-effects model is the default, rho defaults to 0.8, and small = TRUE is the default.
6. Examples
Examples demonstrate robumeta’s intercept-only, sensitivity, forest-plot, and hierarchical meta-regression workflows, including diagnostics for dependence, covariate variation, and small-sample inference.
- Intercept-only example: The Oswald example contains 308 effect sizes from 46 studies, with a subset containing 32 effect sizes from 9 studies.The subset examines neurological activity or response latency criteria and uses Fisher’s Zr transformations.
- Intercept-only example: The intercept-only example fits a correlated-effects model with rho = 0.8 and small-sample corrections.The model is specified with effect.size ~ 1 and study-level identifiers and variances.
- Intercept-only example: The fitted intercept is 0.277 with StdErr 0.181, t-value 1.53, df 7.84, and 95% CI [-0.141, 0.695].The output reports 9 studies and 32 outcomes, with I.sq = 82.05272 and Tau.Sq = 0.1849812.
- Sensitivity analysis: Sensitivity analysis varies rho over (0, 1) to recompute τ 2, the average effect size, and its standard error.For the Oswald intercept model, τ 2 and the average effect size appear relatively robust to different rho values.
- Forest plots: RVE forest plots display individual effect sizes with confidence intervals, study-aware weights, and an overall-effect diamond.Individual boxes have areas proportional to assigned weights, while combined study weight is divided evenly across study effect sizes.
- Hierarchical meta-regression: Hierarchical RVE examples require separating within- and between-level-2 covariate variation before model fitting.Group means and group-centered variables are created with group.mean() and group.center().
7. Conclusions
The robumeta package supports meta-regression with both large-sample and adjusted robust variance estimators, plus RVE-consistent forest plots.
- robumeta provides functions for meta-regression using large-sample and adjusted robust variance estimators.The adjusted estimators incorporate small-sample corrections that improve finite-sample properties.
- The package also produces forest plots consistent with robust variance estimation methodology.