Source-linked AI summary
FMRI Clustering and False Positive Rates
Robert W. Cox, Gang Chen, Daniel R. Glen, Richard C. Reynolds, Paul A. Taylor
TL;DR
This paper reassesses claims that FMRI clustering methods produce severely inflated false-positive rates and that AFNI is especially problematic. Reanalysis indicates more modest inflation and a minor effect from the AFNI bug, while nonparametric results remain within nominal confidence intervals.
Problem
The paper examines whether claims of severe false-positive inflation and unusually high AFNI error rates are supported by the reported results.
Method
The authors characterize false-positive-rate results using typical ranges and medians or percentile ranges rather than maxima.
Results
False-positive inflation was often ≤10% for Beijing and ≤5% for Cambridge, while the AFNI bug changed rates by only 3–5% or 1–2%.
Takeaways & Limitations
The reported results do not support characterizing AFNI as having particularly high false-positive rates or the conclusions as alarmist.
Takeaways & Limitations
Nonparametric permutations may not generalize feasibly to complicated covariate structures and models such as complex ANOVA, ANCOVA, or LME.
Abstract
from arXiv · showhide
Recently, Eklund et al. (2016) analyzed clustering methods in standard FMRI packages: AFNI (which we maintain), FSL, and SPM [1]. They claimed: 1) false positive rates (FPRs) in traditional approaches are greatly inflated, questioning the validity of "countless published fMRI studies"; 2) nonparametric methods produce valid, but slightly conservative, FPRs; 3) a common flawed assumption is that the spatial autocorrelation function (ACF) of FMRI noise is Gaussian-shaped; and 4) a 15-year-old bug in AFNI's 3dClustSim significantly contributed to producing "particularly high" FPRs compared to other software. We repeated simulations from [1] (Beijing-Zang data [2], see [3]), and comment on each point briefly.
AFNI and 3dClustSim
The AFNI 3dClustSim bug had only a minor effect on false positive rates, and the reported data do not support claims that AFNI produced particularly high FPRs.
- AFNI and 3dClustSim: ΔFPR was typically ≤3–5% at per-voxel p = 0.01 and ≤1–2% at p = 0.001 between buggy and fixed 3dClustSim, indicating only a minor bug effect.Figure 1 compares buggy and fixed 3dClustSim across simulations; the reported typical differences were small.
Smoothness
Using an empirical mixed ACF with longer tails reduced all false positive rates, although block designs remained above 5%. The results indicate that spatial-smoothness heavy tails significantly affect clustering, potentially through temporal-frequency dependence.
- Smoothness: All FPRs decreased with the empirical mixed ACF, but block designs remained > 5%.The mixed ACF allowed for longer tails and was computed from residuals.
- Smoothness: The block-design FPR elevation likely reflects dependence of noise spatial smoothness on temporal frequency.
- Smoothness: Heavy tails in spatial smoothness have significant consequences for clustering.
Nonparametric approach
AFNI’s spatial model-free nonparametric randomization approach yielded false positive rates within the nominal confidence interval. However, extending nonparametric permutations to complex covariate structures and models may be infeasible.
- Nonparametric approach: All false positive rates were within the nominal confidence interval using AFNI’s nonparametric approach.The approach was implemented in AFNI’s group-level GLM program, 3dttest++.
- Nonparametric approach: Generalizing nonparametric permutations may be infeasible for complex ANOVA, ANCOVA, or LME covariate structures and models.The limitation concerns applying the approach beyond simpler model settings.
- Nonparametric approach: The approach shows promise, consistent with prior findings.This promise is tempered by potential limitations in handling complicated covariate structures and models.
Inflated FPRs
The section argues that false-positive-rate inflation was variable rather than uniformly extreme, with smaller inflation under stricter voxelwise thresholds and event-related designs. It also contends that emphasizing maximum FPRs made Eklund et al.’s conclusions unnecessarily alarmist.
- Inflated FPRs: Although several cases showed significant inflation, deviations from nominal FPR depended strongly on multiple factors and were not uniformly large.The authors therefore reject a blanket characterization of existing FMRI software as producing extreme false-positive rates.
- Inflated FPRs: FPR inflation was often ≲10% in Beijing and ≲5% in Cambridge under stricter voxelwise p-values and event-related stimuli.These effects primarily involved clusters with marginally significant volume.
- Inflated FPRs: The authors dispute the claim of FPRs up to 70%, noting that the nonparametric method itself reached up to 40%.They argue that medians or percentile ranges are more informative than maxima for characterizing results.
- Inflated FPRs: The authors conclude that focusing on the highest observed FPRs was unnecessarily alarmist and recommend lower voxelwise p-values and event-related paradigms.These recommendations are presented as useful guidance for experimental design and analysis.
AFNI and 3dClustSim
The supplied passages only acknowledge NIH support and use of the NIH HPC Biowulf cluster; they provide no substantive findings about AFNI or 3dClustSim.
- AFNI and 3dClustSim: The research was supported by NIMH and NINDS Intramural Research Programs and used NIH HPC Biowulf computational resources.The support is identified as NIH/DHHS funding, with computations performed on the Biowulf cluster.