Source-linked AI summary
How Reliable Is Individualized Functional-Network Attribution in Resting-State fMRI Cognitive Prediction in ADNI?
Sheethal Bhat, Mahfuzur Rahman Chowdhury, Imam Ahmed Shaik, Rose Dawn Bharath, Siming Bayer, Andreas Maier
TL;DR
The test–retest reliability of individualized functional-network attribution for resting-state fMRI cognitive prediction is rarely quantified, despite attribution being intended to identify networks driving an individual prediction. The study evaluates network-ablation signatures from a frozen FC predictor with split-half, scan–rescan, and permutation-null checks. Split-half reliability is higher than scan–rescan reliability, while matched-subject results exceed a substantial mismatched-subject baseline.
Problem
Test–retest reliability of per-subject functional-network attribution for FC-based cognitive prediction has not been quantified over independent repeat scans and multiyear aging-cohort intervals.
Source: §1 · pp. 1–2Method
Results
Split-half reliability was 0.948/0.950, versus scan–rescan reliability of 0.609 (MMSE) and 0.616 (ADAS-Cog13); matched results exceeded mismatched-subject correlations of 0.463/0.472 (z=6.89/6.63, p≤10−4).
Source: §3.3 · p. 6Takeaways & Limitations
In ADNI, attribution contains a modest person-specific excess atop strong population-shared structure, so the authors recommend reporting scan–rescan reliability with a permutation-null control.
Source: §4 · p. 7The repeat-visit sample is modest (74 subjects; 205 session pairs), and robustness to other cohorts, preprocessing, parcellations, attribution operators, and models remains untested.
Source: §4 · p. 7
Abstract
from arXiv · showhide
Individualized attribution estimates which functional networks drive a model prediction for a given subject and is often proposed as an interpretable resting-state fMRI (rs-fMRI) biomarker, but its test-retest reliability is rarely quantified. We evaluate per-subject network-ablation attribution for a frozen resting-state functional connectivity (FC) predictor (Schaefer-200 FC predicting MMSE/ADAS-Cog13) trained on N=875 subjects from the Alzheimer's Disease Neuroimaging Initiative (ADNI), using, and (iii) a subject-level permutation null. Split-half reliability is near-ceiling (Spearman p=0.948/0.950) but drops on scan-rescan (p=0.609/0.616). Under the mismatched-subject null, correlations remain substantial (p~0.46-0.47), indicating strong population-shared structure, yet matched pairs exceed this null (z~6.6-6.9, p<=10-4). Thus, in ADNI, individualized network attribution contains a modest but non-zero person-specific component atop a strong population-shared pattern. We recommend reporting scan-rescan reliability together with a permutation-null control, rather than within-session proxies alone, to avoid overstating individual specificity.
1 Introduction
Individualized network attribution is meaningful only if reproducible, yet its test–retest reliability over the multiyear intervals typical of aging cohorts has not been quantified. This study evaluates a frozen FC-based cognitive predictor using per-subject network-ablation signatures and multiple reliability checks.
1 Introduction
- Per-subject attribution estimates which functional networks drive a given subject’s prediction, and is meaningful only if reproducible.FC models mapping resting-state connectivity to cognitive and clinical phenotypes are increasingly used in AD neuroimaging.Source: §1 · p. 1
- Prior work reports short-interval reproducibility for subject-specific parcellations, but does not quantify attribution reliability across multiyear repeat scans typical of aging cohorts.FC test–retest reliability is frequently modest, with pooled mean edge-level ICC ≈0.29.Source: §1 · p. 2
- Per-subject attribution signatures can vary under resampling and noise even when predictive accuracy is similar, while FC attribution research has largely emphasized group-averaged importance.The paper identifies no prior study combining per-subject attribution of an FC-based cognitive predictor with quantitative test–retest evaluation on independent repeat scans.Source: §1 · p. 2
- The authors apply three reliability checks to a frozen FC-MLP trained on Schaefer-200 connectivity from 875 ADNI subjects to predict MMSE and ADAS-Cog13.The checks are within-session split-half, scan–rescan, and a cluster-aware subject-level permutation null.Source: §1 · p. 2
- The study quantifies split-half versus scan–rescan reliability of per-subject network attribution, estimates a mismatched-subject baseline with a permutation null, and offers reporting recommendations.The repeat-visit analysis includes 74 subjects and 205 session pairs.Source: §1 · p. 2
2 Methods
The methods use a frozen FC-MLP and network ablation to produce per-subject importance rankings for seven Yeo networks. Reliability is assessed within sessions, across repeat visits, and against a subject-level permutation null, with additional artifact checks and subject-level inference.
2 Methods
- For each target, the method zeros a Yeo network’s incident FC edges and corresponding mask entries, then uses the prediction change as network importance.The model maps each subject’s connectivity vector to MMSE and ADAS-Cog13 predictions.Source: §2 · p. 2
- Group-level importance is the held-out test-set explained-variance drop after network ablation, averaged over three independently trained runs and shown separately for MMSE and ADAS-Cog13.All seven networks are used in analyses; Somatomotor is omitted from the figure layout only.Source: Figure 1 · p. 3, §2 · p. 3
- The implementation reproduces the group-level ablation results for all 14 network–target combinations to six decimal places.The frozen model’s weights remain fixed throughout attribution and reliability analyses.Source: §2 · p. 3
- Reliability checks compare odd and even surviving volumes, each baseline with all repeat visits, and matched-subject reliability against 10,000 subject-level derangements.Repeat visits are kept together as a block in the permutation null; the figure describes these as progressively stricter checks.Source: §2 · pp. 3–4, Figure 2 · p. 4
- Artifact checks describe the association between network importance and ablated edge counts, and correlate split-half reliability with per-half scan duration.Edge-count correlations use seven networks and are treated descriptively because Spearman correlations are highly discrete with only seven items.Source: §2 · p. 4
- All Spearman correlations are computed within each of three runs and averaged; repeat-pair correlations are first averaged within subjects, and headline-mean uncertainty uses subject-level bootstrap intervals.The split-half versus scan–rescan gap is tested with a Wilcoxon signed-rank test across 74 subjects present in both analyses; the table caption specifies subject-level analysis and bootstrap CIs.Source: §2 · p. 4, Table 1 · p. 5
3 Experimental details
The experiments evaluate attribution reliability in ADNI on held-out subjects using a frozen predictor. Scan–rescan reliability is lower than split-half reliability, while matched subjects exceed a substantial mismatched-subject null.
3.1 Data description
- The 875-participant ADNI cohort spans four diagnostic groups and 53 acquisition sites; reliability analyses use the held-out 132-subject test split, with scan–rescan restricted to repeat-visit subjects.Diagnoses include 464 CN, 315 MCI, 94 Dementia, and 2 unknown participants.Source: §3.1 · p. 5
3.2 Implementation details
- A frozen 39,800-dimensional FC-MLP was used; ComBat was fit on training data only, while repeat-visit sessions used pre-ComBat features.The model input combines 19,900 FC edges with an edge-coverage mask.Source: §3.2 · pp. 5–6
3.3 Results
- ρ=0.609 (MMSE) and 0.616 (ADAS-Cog13) scan–rescan, below split-half 0.948/0.950; matched correlations exceed mismatched-subject nulls (ρ=0.463/0.472; z=6.89/6.63, p≤10^-4).The scan–rescan estimates use 205 session pairs from 74 subjects.Source: §3.3 · p. 6
- Split-half reliability increased with per-half scan duration (ρ=0.199/0.277), consistent with sampling-noise limits, although duration was confounded with acquisition protocol in ADNI.Source: §3.3 · p. 7
4 Discussion
Attribution reliability is substantially higher within sessions than across independent repeat visits, and the permutation null indicates that much of the apparent stability is population-shared. The authors recommend demonstrating individual specificity against an explicit population baseline.
4 Discussion
- Split-half reliability near ρ≈0.95 falls to ρ≈0.61–0.62 on independent repeat visits, while the permutation null indicates a modest person-specific excess atop shared structure.The authors report that robustness to other cohorts, preprocessing, parcellations, attribution operators, and models remains untested.Source: §4 · p. 7
- Between-network edges are removed in both corresponding network ablations, coupling attribution signatures; within-network-only ablation would remove this overlap.The weak predictor may also respond mainly to group-level FC structure, and the high shared baseline may leave little room to estimate individual excess.Source: §4 · p. 7
- The repeat-visit sample is modest (74 subjects; 205 session pairs), sessions were not ComBat-harmonized, and variable inter-visit intervals were not modeled.The median interval was 24.4 months (IQR 12.2–48.4).Source: §4 · p. 7
- The authors recommend reporting scan–rescan estimates when repeat visits exist and including a permutation-null or equivalent control.They recommend treating within-session proxies as expected upper bounds and call for validation in further aging/AD cohorts.Source: §4 · p. 7