Source-linked AI summary
Estimates of heterogeneity (I2) can be biased in small meta-analyses
Paul T. von Hippel
TL;DR
Small meta-analyses make heterogeneity difficult to assess, and the usual I² estimator is biased. The paper calculates that bias across homogeneity and heterogeneity settings; it is greatest with few studies and little true heterogeneity. The authors advise caution with small-study point estimates and suggest reporting confidence intervals.
Problem
In small meta-analyses, Cochran’s Q has little power to reject homogeneity even when substantial heterogeneity is present.
Source: §1 · p. 2Method
The paper calculates the expectation and bias of the commonly used I² estimator under homogeneity and heterogeneity.
Source: §3.1 · p. 5, §3.2 · p. 6Results
With K=7 studies and no true heterogeneity, the average I² estimate is about 12%, and bias is greatest near zero heterogeneity.
Source: §1 · p. 3, DISCUSSION · p. 6Takeaways & Limitations
In small meta-analyses, report a 95% confidence interval for I² in addition to, or instead of, the point estimate.
Source: DISCUSSION · pp. 6–7I² should be interpreted cautiously in small meta-analyses, especially when homogeneity has not been rejected, because estimates are biased near zero and confidence intervals may be wide.
Source: DISCUSSION · pp. 6–7
Abstract
from arXiv · showhide
In meta-analysis, the fraction of variance that is due to heterogeneity is known as I2. We show that the usual estimator of I2 is biased. The bias is largest when a meta-analysis has few studies and little heterogeneity. For example, with 7 studies and the true value of I2 at 0, the average estimate of I2 is .124. Estimates of I2 should be interpreted cautiously when the meta-analysis is small and the null hypothesis of homogeneity (I2=0) has not been rejected. In small meta-analyses, confidence intervals may be preferable to point estimates for I2.
1 INTRODUCTION
The section motivates I² as an interpretable way to describe heterogeneity, while noting that small meta-analyses leave substantial uncertainty. It introduces the central finding that the commonly used I² point estimator is biased, especially with few studies and little heterogeneity.
1 INTRODUCTION
- Heterogeneity matters because high between-study variation makes treatment effects in a new population difficult to predict when its causes are not understood.When studies are homogeneous or only slightly heterogeneous, there is some assurance that treatment effects will be similar in a new population.Source: §1 · p. 2
- Cochran’s Q is uninformative in many small meta-analyses because its value depends on both study count and heterogeneity.With few studies, Q has little power to reject homogeneity even when heterogeneity is substantial; with many studies, it may reject for trivial heterogeneity.Source: §1 · p. 2
- The I² statistic describes the proportion of variance between study estimates attributed to heterogeneity and is more interpretable than Q.It ranges from 0 to 1 regardless of the number of studies, but does not eliminate uncertainty in small meta-analyses.Source: §1 · p. 2
- With K=7 studies and no true heterogeneity, the average I² estimate is about 12%.The passage notes this is not trivial relative to observed values: half of Cochrane Library I² estimates are below 21%.Source: §1 · p. 3
2 BACKGROUND
This section establishes the notation for variation across study effects and explains how heterogeneity contributes to that variation. It also describes how Cochran’s Q behaves under homogeneity and heterogeneity.
2 BACKGROUND
- The precision-weighted average of estimated study effects estimates the simple average of the studies’ true effects.The section represents each study with a true effect, its estimate, and a true standard error estimated from the data.Source: §2 · p. 3
- The variance of estimated study effects includes both variation in true effects and sampling variation from study standard errors.The paper denotes the variance of true effects by τ² and focuses on the simplifying case of equal standard errors when defining variance fractions.Source: §2 · p. 3
- Under the homogeneity null, Cochran’s Q is approximately central chi-square with K−1 degrees of freedom, and conventionally rejects at p<.1 or p<.05.Source: §2 · pp. 3–4
- Under heterogeneity, Q has an asymptotic noncentral chi-square distribution; with equal standard errors, its noncentrality increases with I² and is zero when I² is zero.As I² approaches zero, Q converges toward the central chi-square distribution it has under homogeneity.Source: §2 · p. 4
3 BIAS OF THE ESTIMATOR ࡵ
The paper evaluates the bias of the widely used estimator that truncates negative I² estimates at zero. Its bias varies with the true heterogeneity and number of studies, and is positive under homogeneity.
3 BIAS OF THE ESTIMATOR ࡵ
- The raw I² estimate can be negative although the target cannot, so the commonly used estimator rounds negative estimates up to zero.Under the homogeneity null, negative raw estimates occur whenever Q is below its degrees of freedom and have probability above 50%.Source: §3 · pp. 4–5
3.1 Under homogeneity
- Under homogeneity, the estimator has positive bias because it can take only positive or zero values, whose average exceeds the true value of zero.Source: §3.1 · p. 5
- Bias was .135 at K=5, .11 at K=10, and .06 at K=50 under homogeneity.The positive bias shrinks at a decreasing rate as K grows.Source: §3.1 · p. 5
3.2 Under heterogeneity
- Bias is generally larger for small K; it is near zero around I²=.2 and .8, but changes direction across other heterogeneity levels.At I²=.1 it is positive but smaller than at .05 or 0; above .2 it becomes negative, then turns positive again at .9.Source: §3.2 · p. 6
DISCUSSION
The widely used I2 estimator is biased in small meta-analyses, especially when true heterogeneity is near zero. The discussion recommends cautious interpretation and reporting confidence intervals to convey uncertainty.
DISCUSSION
- The bias is negligible around true I2 values of .2 or .8, larger around .5, and greatest near 0.Any reasonable estimator restricted to nonnegative values will have a positive average when true I2 is close to zero.Source: DISCUSSION · p. 6
- In small meta-analyses, interpret I2 cautiously, especially when homogeneity has not been rejected, because estimates are positively biased if true I2 is zero or nearly zero.The estimator can still be useful, including in small meta-analyses, and is less dependent on study count than Q.Source: DISCUSSION · p. 6
- A 95% confidence interval for I2 can be reported alongside or instead of the point estimate, giving the range of possible values without highlighting a potentially biased, imprecise point.The cited interval formulas are described as complicated but having good coverage.Source: DISCUSSION · p. 6
- Wide confidence intervals in small meta-analyses reveal limited information about heterogeneity; often, only the average effect size can be estimated with much confidence.The discussion says reported statistics should make these limits clear.Source: DISCUSSION · p. 7
FIGURES
The figures show the expectation of the estimated I2 under a true I2 of 0, .05, and values from .1 to .9.
FIGURES
- Figure 1 shows the expected estimated I2 when true I2 is 0.Source: Figure 1 · p. 8
- Figure 2 shows the expected estimated I2 when true I2 is .05.Source: Figure 2 · p. 9
- Figure 3 shows the expected estimated I2 when true I2 ranges from .1 to .9.Source: Figure 3 · p. 10