PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking External Generalization of SPD Matrix Learning for Resting-State fMRI Connectome Prediction

Ce Ju, Antoine Collas, Florent Bouchard, Bertrand Thirion

arXiv:2608.30418v1eess.SPcs.LG

TL;DR

The paper asks whether strong rs-fMRI connectome prediction within one cohort generalizes to a new cohort. It benchmarks SPD-aware and baseline methods across six datasets under within-dataset, pooled, and leave-one-dataset-out evaluation, finding less favorable external-validation performance and narrower method differences.

  • Problem

    Within-cohort rs-fMRI prediction performance may not establish that a model has captured a generalizable connectome signal.

  • Method

    The study represents scans as regularized SPD correlation connectomes and compares baseline, Tangent-Space Ridge, SPDNet, and split-wise Riemannian harmonization across six datasets and validation protocols.

  • Results

    Under LODO, prediction error increases and differences among methods narrow, while age-range mismatch and cohort differences strongly shape performance.

  • Takeaways & Limitations

    External validation is necessary for judging how SPD matrix learning gains hold under realistic rs-fMRI dataset shift.

  • Takeaways & Limitations

    Residual motion effects could not be measured in the same way across all datasets.

Abstract

from arXiv · show

Resting-state functional magnetic resonance imaging (rs-fMRI) functional connectivity (FC) matrices are widely used for individual-level prediction, but strong performance within one cohort may not generalize to a new cohort. We ask whether within-dataset performance remains when the test data come from an entirely held-out rs-fMRI dataset. Each scan is represented as a regularized symmetric positive definite (SPD) correlation connectome, which allows methods to use the geometry of the SPD manifold. We introduce a reproducible age-prediction benchmark across six rs-fMRI datasets: COBRE, ADNIDOD, Cam-CAN, ABIDE, OASIS-3, and ADNI. The benchmark compares a vectorized correlation baseline, Tangent-Space Ridge, SPDNet, and split-wise Riemannian harmonization under within-dataset GroupKFold, pooled GroupKFold, and leave-one-dataset-out (LODO) evaluation. Within-dataset and pooled GroupKFold results are substantially more favorable than LODO results. When an entire dataset is held out, prediction error increases, differences among methods narrow, and performance is strongly affected by age-range mismatch and cohort heterogeneity. The benchmark provides common inputs, model settings, data splits, and analysis scripts so that future SPD matrix learning methods can be evaluated under the same external-validation protocol.

1 Introduction

The paper examines whether rs-fMRI connectome prediction generalizes beyond the cohort used for evaluation. It introduces a reproducible six-dataset benchmark comparing SPD-aware and baseline methods under increasingly demanding validation protocols.

  • Motivation: Within-dataset or pooled cross-validation can provide incomplete evidence of robustness when related cohorts contribute to training and testing.Scanner and site effects, preprocessing choices, age range, diagnosis mix, and implementation details influence the prediction problem.
  • Benchmark design: The study uses six heterogeneous datasets—COBRE, ADNIDOD, Cam-CAN, ABIDE, OASIS-3, and ADNI—for chronological-age regression.The datasets differ in age range, clinical composition, longitudinal structure, and acquisition setting.
  • Benchmark design: Each scan becomes a regularized SPD correlation matrix derived from regional time series, covariance estimation, correlation normalization, and numerical stabilization.This representation supports methods that use SPD matrix structure and Riemannian geometry directly.
  • Methods and evaluation: The comparison includes CorrVec, Tangent-Space Ridge, SPDNet, a Dummy regressor, and split-wise Riemannian harmonization under GroupKFold and LODO protocols.Harmonization is fitted within each split so data-dependent quantities are estimated from the training split only.
  • Motivation: The benchmark tests external generalization by holding out an entire rs-fMRI dataset, rather than evaluating only within the training distribution.Its central question is whether gains from SPD matrix learning remain when the cohort changes.
  • Expected benchmark insight: Under LODO, prediction error increases, method differences narrow, and age-range mismatch and cohort differences strongly shape performance.The benchmark therefore treats within-dataset performance as useful but incomplete evidence of robustness.

2 Methods

The benchmark applies a common rs-fMRI connectome pipeline across six heterogeneous datasets, representing scans as regularized SPD correlation connectomes and comparing SPD methods with reference baselines. It evaluates Tangent-Space Ridge and SPDNet, alongside CorrVec and Dummy, using standardized preprocessing and split-specific modeling procedures.

  • Datasets: Six rs-fMRI datasets spanning psychiatric, developmental, healthy lifespan, and aging cohorts form the benchmark.The datasets are COBRE, ADNIDOD, Cam-CAN, ABIDE, OASIS-3, and ADNI.
  • Datasets: 143 COBRE, 134 ADNIDOD, 652 Cam-CAN, 843 ABIDE, 1,035 OASIS-3, and 936 ADNI participants comprise the processed samples.ADNIDOD, OASIS-3, and ADNI also include 190, 1,792, and 1,997 scans, respectively.
  • Dataset heterogeneity: The cohorts differ in age range, sample size, diagnosis mix, sex balance, longitudinal structure, and site information.These differences combine demographic, clinical, longitudinal, and technical variation in cross-dataset comparisons.
  • Connectome construction: Each scan becomes a correlation-based SPD connectome using Oracle Approximating Shrinkage covariance and diagonal regularization with ϵ = 10^-5.Variance normalization preserves positive definiteness, with additional diagonal jitter used for numerical conditioning.
  • Prediction methods: The benchmark compares Tangent-Space Ridge and SPDNet against Dummy and CorrVec, including harmonized variants.Tangent-Space Ridge uses tangent coordinates and Ridge regression, while SPDNet processes full SPD connectomes through learned matrix mappings, eigenvalue rectification, logarithms, and regression.
  • Tangent-Space Ridge: Tangent-space preprocessing uses a training-set Fréchet mean, fixed evaluation transforms, symmetric-matrix vectorization, and training-split z-scoring.The vectorization uses unit diagonal weights and 2 weights on off-diagonal entries; Ridge strength α is selected by grouped inner cross-validation.

2.6 Harmonization Protocol

The benchmark uses split-wise Riemannian harmonization to address site-related distribution shifts while preserving age as a modeled biological covariate. It evaluates within-dataset, pooled, and leave-one-dataset-out generalization with grouped splits and complementary regression metrics.

  • Harmonization: Riemannian harmonization maps connectomes to tangent space, corrects batch effects with ComBat, and maps them back to the SPD manifold.Dataset or site identity is the batch effect, while chronological age is included as the biological covariate to preserve.
  • Split-wise protocol: Harmonization is fitted separately within each evaluation split so held-out data do not contribute to fitting the harmonization model.The implementation embeds harmonization within the outer training/test split to avoid leakage.
  • Protocol caveat: Pooled GroupKFold harmonization uses true test ages, so it represents harmonization with age as a covariate rather than prediction when test age is unavailable.In LODO, harmonization is restricted to pooled training datasets and does not use held-out-dataset ages.
  • Harmonization: Age is included because age distributions differ strongly across datasets and may be confounded with the batch variable.Without age in the covariate design, correction can attribute age-related functional-connectivity variation to site or dataset effects.
  • Evaluation design: The benchmark uses grouped procedures for within-dataset, pooled, and LODO evaluation, with K = 5 for all GroupKFold settings.Scans are grouped by subject, and LODO leaves one entire dataset out while retaining grouped train/validation splitting among the pooled training datasets.
  • Reproducibility: Fixed random seeds support reproducibility, but the reported ranking is not evidence that the same ranking holds across other seeds.This is an explicit scope boundary of the benchmark results.
  • Evaluation metrics: NegMAE, R2, and Spearman ρ distinguish prediction error, explained variance, and rank-order preservation under held-out-cohort evaluation.Spearman ρ is additionally reported for pooled LODO because held-out datasets may have different age distributions.

3 Results

Within-dataset and pooled GroupKFold evaluations were more favorable than strict LODO transfer, with performance varying substantially across datasets, metrics, and harmonization settings.

  • 3.1 Dataset Differences Define the External Generalization Problem: Dataset age ranges, cohort sizes, diagnosis mixes, sex balance, longitudinal structure, and site information differ substantially across the benchmark.ABIDE is younger, Cam-CAN spans a broad healthy lifespan, COBRE is smaller and psychiatric, and ADNIDOD, OASIS-3, and ADNI are older cohorts with repeated scans.
  • 3.2 Within-Dataset Accuracy Is Useful but Incomplete: Most informative models outperform Dummy within datasets, indicating that the shared connectome pipeline retains age-related signal.The main exception is the smaller and more variable ADNIDOD cohort, where R2 remains near or below zero for informative methods.
  • 3.2 Within-Dataset Accuracy Is Useful but Incomplete: CorrVec is usually competitive with Tangent-Space Ridge and SPDNet, while SPDNet often performs best in Cam-CAN, ABIDE, OASIS-3, and ADNI.Within-dataset performance is strongest in datasets where informative models reach positive R2, but model gaps are smaller in COBRE and ADNIDOD.
  • 3.3 Pooled Evaluation Gives a More Optimistic View Than LODO: Pooled GroupKFold favors SPDNet overall, whereas LODO narrows method differences and produces dataset-dependent winners.SPDNet has the best pooled negative MAE and R2, but Table 3 reports different best models across held-out datasets and metrics.
  • 3.3 Pooled Evaluation Gives a More Optimistic View Than LODO: Under LODO, prediction error increases, R2 becomes unstable and often negative, and variability across held-out datasets increases for all methods.Pooled GroupKFold can benefit from shared dataset identity, age structure, and cohort composition because every dataset appears in training and testing.
  • 3.4 Harmonization Benefits Depend on the Evaluation Protocol: Positive Spearman ρ can persist under LODO even when absolute error is large or R2 is poor, especially for Tangent-Space Ridge and SPDNet.This separates preservation of age rank ordering from calibration and absolute prediction accuracy.
  • 3.4 Harmonization Benefits Depend on the Evaluation Protocol: Harmonization improves CorrVec and Tangent-Space Ridge more consistently than SPDNet, with benefits depending on the evaluation protocol.Gains are clearer in pooled GroupKFold and smaller in LODO; LODO harmonization leaves the held-out dataset untouched during fitting.
  • 3.5 Larger SPDNet Configurations Do Not Consistently Improve Transfer: No SPDNet configuration is consistently best across pooled GroupKFold and LODO diagnostics.Smaller quarterdim and halfdim variants are competitive under pooled GroupKFold, while larger variants show no reliable advantage across error, explained variance, or rank preservation.

4 Discussion

The discussion identifies external validation as the central challenge: models retain age-related connectome signal within datasets, but transfer degrades under demographic, clinical, longitudinal, and technical shifts. It therefore favors multi-metric, held-out-cohort benchmarking over simple model rankings.

  • 4 Discussion: Shared connectome construction preserves age-related signal within datasets, but external validation is needed to assess robustness across cohorts.Within-dataset evidence can be partly driven by cohort structure and benchmark design.
  • 4 Discussion: CorrVec remains a strong Euclidean reference, Tangent-Space Ridge is competitive, and SPDNet provides only modest, inconsistent gains across settings.The model comparison does not support a simple hierarchy in which the deep SPD method always beats shallower alternatives.
  • 4 Discussion: The main performance contrast is between within-distribution evaluation and external validation, not between SPD matrix learning methods.Under LODO, method differences narrow, prediction error increases, and R2 often becomes negative.
  • 4 Discussion: Spearman ρ complements MAE and R2 by indicating whether age rank ordering persists when absolute accuracy and explained variance degrade.The discussion supports reporting multiple diagnostics rather than relying on a single error metric under dataset shift.
  • 4.2 Interpreting Harmonization and Model Performance: Harmonization benefits Tangent-Space Ridge more clearly than SPDNet, and the benchmark does not identify the mechanism behind this model dependence.Its measured benefit also depends on whether age is used during feature correction.
  • 4.1 Interpreting External Generalization: LODO combines site and acquisition shift with demographic, clinical, and longitudinal differences rather than isolating scanner effects.Held-out datasets expose different combinations of age-range extrapolation, age-related bias, and population mismatch.
  • 4.1 Interpreting External Generalization: ABIDE is difficult to hold out because its younger age distribution creates a mismatch with the predominantly adult and older training cohorts.Cam-CAN is relatively stable because its broad adult lifespan overlaps more strongly with the other datasets.
  • 4 Discussion: The SPDNet ablation shows that changing width or depth within the tested family does not reliably produce a configuration best across datasets and metrics.The ablation is limited to four prespecified configurations rather than a full architecture search.

5 Conclusions

The benchmark evaluates whether rs-fMRI connectome prediction generalizes across cohorts, finding that external performance is less favorable and strongly shaped by dataset shift. SPD geometry organizes the comparison, but does not remove the effects of age-range mismatch, cohort heterogeneity, and other limitations.

  • Benchmark scope: The benchmark tests rs-fMRI individual prediction across cohorts using shared connectome construction, subject-level splits, leakage control, and common model settings.It compares CorrVec, Tangent-Space Ridge, SPDNet, and Riemannian harmonization with chronological age as the common target.
  • External generalization: Within-dataset and pooled evaluations provide an important but incomplete view of external robustness, because their advantages become smaller under LODO.Informative models detect age-related connectome signal within datasets, but held-out-dataset evaluation is less favorable.
  • Dataset shift: Age-range mismatch and cohort differences strongly shape performance under external validation.Observed differences reflect age-range mismatch, cohort composition, longitudinal structure, diagnosis mix, and preprocessing differences, not scanner/site effects alone.
  • Model comparison: Tangent-Space Ridge is a strong transparent geometric baseline, while SPDNet is competitive within datasets but larger variants do not consistently improve transfer.Riemannian harmonization helps most for tangent-space features, whereas its benefit for SPDNet is smaller and more variable.
  • Conclusion: SPD geometry is useful for organizing the benchmark, but external generalization remains strongly shaped by dataset shift.The benchmark is intended as a transparent reference point for evaluating future SPD matrix learning methods under realistic rs-fMRI dataset shift.
  • Limitations: Age is a practical shared target rather than a complete biological or clinical outcome, limiting the scope of the conclusions.The study also notes that pooled GroupKFold may use dataset identity information unavailable in strict external validation.
  • Limitations: Harmonization uses true test age during test-fold correction, so it does not represent prediction when test age is unknown.LODO also mixes technical differences with age-range mismatch, especially for ABIDE.
  • Limitations: Residual motion may contribute to functional-connectivity features and vary with age or dataset membership, so neural aging effects are not isolated.Motion measures and exclusion logs were unavailable across all datasets; broader sensitivity analyses and nonlinear baselines would strengthen comparison.

Data and Code Availability

The benchmark is designed for reproducibility through documented splits, fixed seeds, shared derived inputs, and publicly available analysis resources. Raw data remain governed by source-provider access and redistribution policies.

  • Access constraints: ADNI, ADNI-DOD, and OASIS-3 data are not redistributed and must be obtained directly from their original providers.The benchmark does not redistribute raw neuroimaging data.
  • Derived data: Only redistribution-permitted derived functional-connectivity matrices from CamCAN, ABIDE, and COBRE are released through Zenodo.The release includes connectome inputs, derived metadata, and split definitions for those datasets.
  • Reproducibility: Splits are reconstructed from subject and dataset identifiers using documented protocols and fixed random seeds.This enables researchers to reproduce the benchmark and inspect its evaluation protocol.
  • Code: The analysis code, model configurations, split logic, and scripts are available through the project repository.These resources support comparison of new SPD matrix learning methods under identical experimental settings.

Funding

The study was supported by French national and institutional research funding programs.

  • Funding: The work was supported by grant ANR-22-PESN-0012 under the France 2030 program, managed by the Agence Nationale de la Recherche.
  • Funding: Additional support came from the DATAIA Convergence Institute under the Programme d’Investissement d’Avenir, operated by Inria.

Ethics Statement

The study reanalyzes deidentified data from existing public or controlled-access neuroimaging resources and collected no new participant data.

  • Data source: The benchmark uses deidentified data from public or controlled-access neuroimaging resources.
  • Study design: No new human participant data were collected for this benchmark.
  • Ethics: Ethical approval and informed-consent procedures were handled by the original studies and data providers.

Use of AI-Assisted Tools

OpenAI Codex supported English-language editing and submission-file organization, while the authors reviewed the output and retained responsibility for the scientific content and final manuscript.

  • OpenAI Codex was used for English-language editing.
  • The tool also organized the submission files.
  • The authors reviewed the resulting text and took full responsibility for the scientific content and final manuscript.

Supplementary Methods

The supplementary methods define dataset matching, quality control, connectome construction, covariance estimation, and model-specific transformations and evaluation safeguards. They also document missing motion metadata and a non-fully-nested tangent-space tuning step.

  • Data assembly: Scans were retained when phenotypic records, extracted time series, complete age information, and post-extraction quality control were available.Records were matched using subject and session identifiers, with predefined regular expressions used before matching.
  • Quality control: Quality control required at least 100 time points, covariance condition numbers from 10 to 10^6, and no regional time-series column with exactly zero ℓ2 norm.The lower condition-number cutoff is a heuristic excluding OAS covariance estimates with little eigenvalue variation, not a numerical-stability requirement.
  • Limitations: The processed benchmark tables include time-series and covariance checks but exclude scan-level motion measures and motion-based exclusion logs.
  • Model evaluation: Tangent-Space Ridge standardized training-derived features and selected Ridge α by grouped inner cross-validation without using outer test data.The tangent reference point came from the full outer training set and was not recomputed inside inner validation folds, so tuning was not fully nested with respect to that reference.
  • SPDNet: SPDNet used BiMap projections, ReEig eigenvalue rectification, and LogEig mapping to process SPD matrices while preserving positive definiteness.The BiMap output remains SPD when the learnable projection has full column rank, and ReEig applies an elementwise eigenvalue threshold.
Loading 2608.30418v1…