Source-linked AI summary
Deriving reproducible biomarkers from multi-site resting-state data: An Autism-based example
Alexandre Abraham, Michael Milham, Adriana Di Martino, R. Cameron Craddock, Dimitris Samaras, Bertrand Thirion, Gaël Varoquaux
TL;DR
The paper addresses the challenge of finding objective, reproducible autism biomarkers in heterogeneous multi-site R-fMRI data and pipeline choices. It evaluates connectome-based inter-site classification pipelines on ABIDE, achieving 66.8% accuracy while showing that performance continues to improve with sample size and depends strongly on functional-region definition.
Problem
Multi-site heterogeneity and differing functional-connectivity pipeline choices complicate the extraction and validation of objective psychiatric biomarkers.
Method
The study builds R-fMRI neurophenotypes by defining regions, extracting regional activity, estimating functional interactions, and training supervised classifiers to distinguish ASD from typical controls.
Results
66.8% accuracy was obtained for inter-site prediction on the whole ABIDE dataset, exceeding previously published findings and chance, with performance increasing as training-set size grew.
Takeaways & Limitations
Heterogeneous multi-site datasets can support imaging biomarkers robust to site differences, and data-driven functional atlases can improve classification relative to reference atlases.
Takeaways & Limitations
The representativeness of ABIDE has not been established, so identified features may be biased and require further replication.
Abstract
from arXiv · showhide
Resting-state functional Magnetic Resonance Imaging (R-fMRI) holds the promise to reveal functional biomarkers of neuropsychiatric disorders. However, extracting such biomarkers is challenging for complex multi-faceted neuropatholo-gies, such as autism spectrum disorders. Large multi-site datasets increase sample sizes to compensate for this complexity, at the cost of uncontrolled heterogeneity. This heterogeneity raises new challenges, akin to those face in realistic diagnostic applications. Here, we demonstrate the feasibility of inter-site classification of neuropsychiatric status, with an application to the Autism Brain Imaging Data Exchange (ABIDE) database, a large (N=871) multi-site autism dataset. For this purpose, we investigate pipelines that extract the most predictive biomarkers from the data. These R-fMRI pipelines build participant-specific connectomes from functionally-defined brain areas. Connectomes are then compared across participants to learn patterns of connectivity that differentiate typical controls from individuals with autism. We predict this neuropsychiatric status for participants from the same acquisition sites or different, unseen, ones. Good choices of methods for the various steps of the pipeline lead to 67% prediction accuracy on the full ABIDE data, which is significantly better than previously reported results. We perform extensive validation on multiple subsets of the data defined by different inclusion criteria. These enables detailed analysis of the factors contributing to successful connectome-based prediction. First, prediction accuracy improves as we include more subjects, up to the maximum amount of subjects available. Second, the definition of functional brain areas is of paramount importance for biomarker discovery: brain areas extracted from large R-fMRI datasets outperform reference atlases in the classification tasks.
1. Introduction
R-fMRI may provide objective functional neurophenotypes for psychiatric diagnosis, but their reproducibility and generalizability remain uncertain in heterogeneous multi-site data. The study addresses this challenge by evaluating inter-site autism classification pipelines on ABIDE.
- R-fMRI is promising for functional neurophenotypes because it is non-invasive and requires neither a constrained task nor active subject participation.
- Functional connectivity patterns from R-fMRI have been used to classify conditions including depression, ADHD, and autism.
- Reproducibility and generalizability remain debatable because acquisition, processing, and sampling differences across studies and sites are insufficiently quantified.
- Aggregating larger datasets can improve sample size and statistical power but introduces uncontrolled variation across scanners, protocols, recruitment, and sample composition.
- The study evaluates pipelines that learn inter-site psychiatric-status biomarkers from heterogeneous ABIDE data using whole acquisition sites as unseen test sets.
2. Material and methods
The study builds connectome-based classifiers through region definition, time-series extraction, connectivity estimation, and supervised learning. It evaluates preprocessing, atlas, connectivity, classifier, and validation choices on heterogeneous multi-site ABIDE data.
- A connectome is a functional-connectivity matrix linking a set of brain regions of interest.
- The pipeline estimates regions, extracts regional time series, estimates connectivity matrices, and classifies participants using connectivity features.
- Inter-site validation leaves an entire acquisition site out as the test set, measuring prediction on participants from an unseen site.
- 871 participants remained after quality assurance from the 1112-person ABIDE dataset aggregated across 17 independent sites.
- Regions are defined using reference or data-driven atlases, with all ROI definitions learned on the training set to avoid overfitting.
- Connectivity features use covariance-based estimation, correlations, partial correlations, or tangent embedding, followed by supervised ASD-versus-TC classification.
3. Results
Across ABIDE analyses, inter-site prediction reached 66.8% accuracy, improved with more training subjects, and depended strongly on pipeline choices. MSDL atlas extraction, tangent-space embedding, and l2-regularized classifiers performed best, while 40–100 regions gave the strongest results.
- 3.1. Overall prediction results: 66.8% was the highest whole-dataset inter-site accuracy, exceeding previously published ABIDE findings and the 53.7% chance level.
- 3.1. Overall prediction results: Prediction performance increased steadily with training-set size across subsamples, indicating that the largest tested dataset had not reached optimal classification performance.
- 3.1. Overall prediction results: Inter-site prediction showed higher variability than intra-site prediction, but this difference disappeared with a sufficiently large training set.
- 3.2. Effect of the choice of atlas: The best pipeline combined MSDL atlas extraction, tangent-space embedding, and l2-regularized classification.
- 3.3. Effect of the covariance matrix estimator: Tangent-space projection outperformed other connectivity estimators in all settings, although its matrices cannot be interpreted as correlation matrices.
- 3.6. Effect of the number of regions: Atlases with 40–100 regions performed best, whereas fewer than 20 regions performed poorly and more than 140 produced apparently unstable results.
4. Discussion
The study extracts neurophenotypes from aggregate resting-state fMRI through a four-step pipeline spanning region definition, time-series extraction, interaction estimation, and diagnostic classification.
- The pipeline defines brain regions from resting-state fMRI data.
- It extracts regional activity time series and estimates functional interactions between regions.
- The final step constructs a discriminant model for brain-based diagnostic classification.
Default Mode Network
Across heterogeneous ABIDE data, the study reports 67% autism-versus-control accuracy and finds that larger samples and pipeline choices, especially region definition, improve prediction.
- 67% accuracy classified autism versus control across new acquisition sites, exceeding prior work using the larger ABIDE sample.
- Increasing sample size improved inter-site prediction, which approached intra-site prediction as heterogeneity was better addressed.
- Training on 690 participants, 80% of 871 available subjects, had not reached optimal pipeline performance.
- Pipeline choices strongly affected accuracy; region selection was most important, with MSDL producing the best results overall.
- The most informative ASD-predictive features concentrated in default-mode, parieto-insular, and language networks, including decreased homotopic connectivity.
- Predictive biomarkers may help define diagnostic subgroups beyond providing neural correlates of autism.
Appendix A. Time series extraction
The pipeline extracts regional time series from overlapping brain maps using ordinary least squares and removes confound-related signal by orthogonalization.
- Ordinary least squares estimates participant-specific time series for each region from voxel signals and an atlas of overlapping maps.Y contains voxel signals across temporal scans, while V contains atlas maps.
- Confound removal orthogonalizes the signals by removing components projected onto an orthonormal basis of the confounds.The confounds are represented as a matrix C with c columns.
Appendix B. Covariance matrix estimation
Because empirical covariance estimates can be unreliable when the number of observations is smaller than the number of variables, the pipeline uses shrinkage and Ledoit-Wolf parameter selection.
- Shrinkage pulls extreme covariance values toward central values to avoid covariance matrices that cannot be inverted when n < p.The method uses an ℓ2-penalized maximum-likelihood covariance estimator.
- Ledoit-Wolf sets α by minimizing mean square error between the real and estimated covariance matrices.
Appendix C. Classifiers
Linear classifiers classify samples into two categories by finding a separating hyperplane in the input space.
- Linear classifiers seek a hyperplane that splits samples in the input space into two categories.
Support Vector Classification SVC
Support vector classification maximizes the margin between the hyperplane and the nearest training sample while using squared hinge loss and exploring sparse regularization.
- Support vector classification maximizes the distance between the hyperplane and the closest training sample.
- The classifier minimizes the classical squared hinge loss.
- The features are brain-region connection pairs, and the samples are subjects.The implementation explores ℓ1- and ℓ2-penalized support vector classifiers.
- The implementation uses scikit-learn with the LibLinear implementation.
Ridge classifier
The study uses Ridge regression for classification, assigning binary class values and predicting class membership from the regressor’s sign. Its ℓ2 penalty favors small coefficients and reduces distortion from colinear features.
- Ridge classification assigns binary values to classes and uses the sign of the regression prediction to classify each sample.
- The ℓ2 penalty rewards coefficients close to zero, producing a smaller model and preventing colinear features from skewing coefficients.
- The implementation uses scikit-learn.
- Movement is treated as a potential source of spurious connectome correlations because movement patterns may differ between healthy and ASD participants.
- The analysis removes movement effects and separately tests movement information to assess whether predictions rely on residual movement patterns.
Movement regression
During time-series extraction, the analysis regresses out estimated movement and other confounds, then compares diagnosis prediction with and without movement regression. No significant accuracy difference is observed.
- No significant difference appears between prediction with and without movement regression.This indicates that movement regression does not strongly affect connectome estimation in this analysis.
Movement-based prediction
The study evaluates movement-based diagnosis and multiple connectome-pipeline choices across intra-site and inter-site settings. Movement scores are at chance, while pipeline comparisons examine atlas design, feature handling, sample size, and region number.
- Prediction accuracy does not significantly differ with versus without movement regression.
- The movement analyses compare logistic regression, SVC, Ridge classifier, Gaussian Naive Bayes, and random forests, with and without feature extraction.
- 56 movement descriptors from movement estimates and their gradients are used as features in an SVC for diagnosis.
- Movement-based prediction scores are at chance level, with substantially larger score variance for inter-site prediction.
- Using 84 atlas regions outperforms using all Harvard Oxford and Yeo atlas regions, possibly because small regions induce spurious connectivity correlations.
- Overlapping atlases are evaluated against non-overlapping counterparts, with points above the identity line indicating better prediction by overlapping maps.
- Selecting 10% of features with ANOVA or using sparsity-inducing methods reduces accuracy relative to classification with all features.
- Pipeline comparisons report relative prediction impacts for each option, with circles and stars distinguishing the two cross-validation schemes.