Source-linked AI summary
Comparative Study of Anatomical and Learned Features in AI Models for Structural Brain MRI
Boyang Yu, Miquel Lopez Escoriza, Long Chen, Arjun V. Masurkar, Narges Razavian, Carlos Fernandez-Granda
TL;DR
The paper asks whether explicit anatomical features, supervised CNNs, or pretrained ViTs provide the strongest representations for structural brain MRI modeling. It compares these approaches across seven clinical tasks and introduces anatomy-segmentation pretraining; anatomical linear models remain competitive diagnostically, while ASP ViT performs best for biological age estimation.
Problem
Evidence is limited on how explicit anatomical features compare with supervised CNNs and pretrained ViTs across diverse structural MRI clinical tasks.
Method
The study evaluates anatomical linear models, CNNs, masked-autoencoder ViTs, and ASP ViTs across 18 public datasets and seven clinical tasks.
Results
Across clinical tasks, anatomical linear models remain highly competitive with deep neural networks, while ASP ViT achieves a mean absolute error of 2.54 years internally and 4.19 years on external age-estimation cohorts.
Takeaways & Limitations
CNNs and pretrained ViTs implicitly capture relevant anatomical information, while anatomy-guided pretraining improves biological age estimation.
Takeaways & Limitations
The study uses de-identified secondary data from publicly available repositories and collected no new human data.
Abstract
from arXiv · showhide
In this work, we comprehensively evaluate three popular feature-extraction paradigms in AI-based neuroimaging modeling: (1) computation of anatomical surfaces and volumes, (2) supervised learning with convolutional neural networks (CNNs), and (3) unsupervised pretraining of vision transformer (ViT) foundation models, followed by supervised finetuning. Our study is based on 18 publicly available datasets containing 3D structural T1-weighted MRI scans from approximately 80,000 participants across seven distinct clinical tasks. We observe that a linear model based on anatomical features matches the diagnostic performance of complex nonlinear features learned by sophisticated AI frameworks, including foundation models trained on thousands of scans. Conversely, CNNs and pretrained ViTs learn features that implicitly capture relevant anatomical information, bypassing the need for explicit feature extraction. Building upon these insights, we propose Anatomy Segmentation Pretraining (ASP), a novel method to incorporate anatomical information during foundation-model pretraining, which outperforms existing models in biological age estimation.
Introduction
The study compares anatomical features, CNNs, and pretrained ViTs for 3D structural T1-weighted MRI across seven clinical tasks and 18 public datasets. It finds anatomical features remain competitive while ASP adds anatomy-guided pretraining to ViTs.
- Prior approaches: Earlier neuroimaging models used handengineered features, whereas CNNs and ViTs learned increasingly complex representations from imaging data.CNN features can be difficult to interpret anatomically, while ViTs learn from large unlabeled datasets before downstream finetuning.
- Study scope: The study compares anatomical features, supervised CNNs, and pretrained ViTs for modeling 3D structural T1-weighted MRI.The datasets span seven clinical tasks and approximately 80,000 participants.
- Study scope: The comparison covers 18 publicly available datasets containing more than 96,000 MRI scans from approximately 80,000 participants.The study evaluates Alzheimer’s disease, epilepsy, Parkinson’s disease, tumor IDH mutations, brain age, and brain tumor and stroke lesion segmentation.
- Contribution: ViTs learn anatomical information during pretraining, and ASP incorporates brain-structure segmentation as an auxiliary pretraining task.ASP performs on par with other strategies across clinical tasks but achieves superior biological age estimation.
- Contribution: Explicit anatomical features remain competitive with representations learned by advanced deep-learning models across clinical cohorts ranging from dozens to thousands of training examples.The paper also introduces an anatomy-guided framework for neuroimaging foundation-model pretraining.
Results
Across the clinical tasks, pretrained ViTs, CNNs, and anatomical linear models show no significant overall performance differences. With a much larger age-estimation cohort, ASP ViT performs best, especially on external cohorts and older subjects.
- Clinical tasks: ASP ViT has the highest macro-AUC for Alzheimer’s diagnosis on the internal and six external cohorts, but significance versus CNNs and anatomical logistic regression occurs only for OASIS 2.The Parkinson’s task instead gives the highest AUC to the Pretrained ViT, with significance only versus BrainIAC.
- Cross-task comparison: There was no significant difference between pretrained ViTs, CNNs, and anatomical linear models across the different tasks.This overall comparison includes classification and segmentation settings.
- Age estimation: The age-estimation cohort contains 62,890 training subjects, with evaluation on a held-out UK Biobank cohort and external ADNI and NACC cohorts.This larger cohort was used to examine whether limited task-specific training data constrained deep-learning gains.
- Age estimation: ASP ViT achieves a mean absolute error of 2.54 years internally and 4.19 years on both NACC and ADNI test cohorts.The Pretrained ViT is close internally but more than 1 year worse externally; ASP ViT performs particularly well above age 80.
- Age estimation: Dementia patients show biological-age estimates consistently above their biological age across most age intervals in both NACC and ADNI.The same effect appears less strongly among mildly cognitively impaired subjects.
Methods
The methods combine masked-autoencoder pretraining, anatomy-segmentation pretraining, task-specific adaptation, anatomical linear models, CNNs, and standardized MRI preprocessing. ASP adds segmentation-based anatomical grounding to ViT representations.
- Anatomy Segmentation Pretraining: ASP first freezes the ViT while a lightweight decoder learns brain-region segmentation, then trains the encoder and decoder together.The procedure is designed to concentrate anatomical information in the ViT embedding.
- Anatomy Segmentation Pretraining: ASP uses 39 selected anatomical regions, consolidating symmetrical structures and related subunits to improve efficiency.Regions were selected using disease relevance and anatomical scale, excluding regions below 1% of total brain volume.
- ViT pretraining: MAE pretraining masks 75% of 3D MRI patches and trains a 3D ViT to reconstruct the original volumes.The ViT-Base encoder uses 16×16×16-voxel patches and is trained for 1000 epochs with MSE reconstruction loss.
- Downstream adaptation: Pretrained ViT embeddings feed binary-classification, linear-regression, or U-Net–style segmentation heads for downstream tasks.Linear probing and LoRA provide alternative adaptation strategies, with linear probing also monitoring anatomical and demographic representation quality.
- Baseline models: Anatomical baselines use 178 FreeSurfer-derived regional volume, cortical surface-area, and cortical-thickness features in linear models.CNN baselines use ResNet-18 for diagnostic classification and U-Net for segmentation.
- Preprocessing: All non-standardized scans undergo a uniform preprocessing sequence, while FreeSurfer recon-all provides processing and primary quality control.Images failing the initial recon-all phase are excluded.
Discussion
Anatomical linear models remain competitive with deep neural networks across structural MRI datasets, while anatomical segmentation accelerates ViT learning of relevant structure during pretraining.
- Comparative performance: Linear models using preselected anatomical features remain highly competitive with deep neural networks across structural MRI datasets.The anatomical features support a computationally efficient, interpretable, and robust baseline when extracted with modern segmentation tools.
- Dataset scale: Diagnostic performance may have reached a saturation point in public datasets containing only hundreds to a few thousand scans.The authors suggest that substantially larger cohorts may be needed for further progress.
- Representation learning: Vision transformers learn relevant anatomical structure during unsupervised pretraining, but slowly.The observed learning process motivates methods that provide stronger anatomical guidance during pretraining.
- Representation learning: Anatomical segmentation as an auxiliary pretraining task greatly accelerates ViT learning of relevant anatomical features.The approach is presented as a way to leverage clinically relevant tasks when medical pretraining datasets are smaller than general computer-vision datasets.
A.2 Pretraining Dataset
The pretraining data combine large, publicly available MRI cohorts spanning population aging, development, and healthy neuroscience studies, with varied acquisition settings.
- Pretraining cohorts: UK Biobank provides large-scale population health, demographic, lifestyle, and neuroimaging data for studying aging, disease, and cognition.The study uses T1-weighted MRI from participants in the imaging-genomics population with at least one brain scan.
- Pretraining cohorts: The data collection was based on information available in March 2024 and included some participants with a second scan within two years.This describes the temporal scope and repeat-imaging component of the collection.
- Pretraining cohorts: The Human Connectome Project Lifespan Project includes healthy adults across aging and development stages using a shared Siemens Prisma protocol.The cited cohorts cover ages 36–60 years and 5–21 years.
- Pretraining cohorts: CamCAN studies preserved cognitive abilities during healthy aging using interviews, brain scans, and cognitive experiments.Its T1-weighted MRI scans were collected with a standardized protocol.
- Pretraining cohorts: IXI provides MRI data from healthy participants scanned with varying scanners at three London hospitals.The cohort supports study of normal aging and brain structure without neurological-condition confounding.
A.3 Disease Specific Datasets
The disease-specific evaluation spans diverse Alzheimer’s, neurological, tumor, lesion-segmentation, and age-estimation datasets, including heterogeneous acquisition settings and multiple performance comparisons.
- Clinical datasets: NACC contributes longitudinal Alzheimer’s data collected with mixed scanner models, acquisition parameters, and imaging protocols.This variability is used to evaluate robustness across imaging conditions.
- Clinical datasets: The Alzheimer’s evaluation includes ADNI, NACC, OASIS, AIBL, and MIRIAD datasets with distinct cohort designs and imaging resources.OASIS contains cross-sectional and longitudinal components focused on aging and Alzheimer’s disease, while MIRIAD focuses on transition from mild cognitive impairment to Alzheimer’s disease.
- Clinical datasets: The study also evaluates Parkinson’s disease, epilepsy, tumor IDH mutations, brain tumors, and stroke lesions using PPMI, OpenNeuro, UCSF-PDGM, BraTS, and ATLAS datasets.EPISURG supplies an external epilepsy evaluation set, while UCSF-PDGM combines MRI with tumor genetic data.
- Model comparisons: ASP ViT achieved the highest macro-AUC in most Alzheimer’s cohorts, although statistical significance over CNN and anatomical logistic regression was reached only for OASIS 2.For Parkinson’s disease and epilepsy, Pretrained ViT achieved the highest AUC; for tumor IDH mutations, ASP ViT was highest without statistically significant differences.
- Model comparisons: Age estimation showed ASP ViT achieving MAE values of 2.54 years internally and 4.19 years in both NACC and ADNI external cohorts.The Pretrained ViT was 0.35 years worse internally and more than 1 year worse externally.
- Dataset coverage: BrainATLAS aggregates 18 neuroimaging datasets with diverse populations, imaging protocols, and dataset distributions.The appendix records exclusions, missing values, outliers, and FreeSurfer-derived BrainSegVol measurements.
A.5 Imaging Data Harmonization
The appendix describes FreeSurfer-based processing for structural MRI and the anatomical labels and features derived from it. The workflow produces grouped brain-region labels and 178 anatomical measurements for downstream modeling.
- Processing workflow: 6.11 hours is the average time required to process each 3D image with the 31-step workflow.The estimate comes from retrospective analysis of 1,500 successful image-processing logs.
- Processing workflow: FreeSurfer processing includes motion correction, intensity normalization, skull stripping, registration, volumetric labeling, white-matter segmentation, and cortical parcellation.The listed operations span preprocessing, volumetric reconstruction, and surface-based anatomical processing.
- Anatomical features: The study extracts 178 anatomical features covering regional volumes, cortical surface areas, and cortical thickness.These features are derived from FreeSurfer segmentation and surface-reconstruction statistics.
- Anatomical segmentation: FreeSurfer initially provides 112 anatomical regions, which are post-processed into 39 grouped regions for supervised training.Grouping merges symmetrical, related, or minor structures according to clinical relevance and volume thresholds.
- Anatomical segmentation: Appendix tables document the original region labels, scan coverage, brain-volume proportions, anatomical names, and final grouped-region mappings.The original and grouped segmentation statistics are reported separately before and after preprocessing.
B.1 Preliminary Analysis of Linear Model and Nonlinear Model Using Anatomical Features
This analysis tests whether nonlinear tabular models improve on logistic regression when using FreeSurfer-derived anatomical features for dementia detection. It compares models under cross-validation and varying training-set sizes.
- Model comparison: 178 FreeSurfer-derived features were used to compare linear logistic regression with nonlinear random forests for dementia detection.The features span regional volumes, cortical surface areas, and cortical thickness values.
- Evaluation design: Five-fold cross-validation selected hyperparameters for both models, while training with 10%, 30%, and 100% of the data tested sample-size dependence.The random-forest search varied estimator count, maximum depth, and split criterion.
- Evaluation design: Table B6 reports dementia-detection performance for anatomical-feature models, including external-cohort results with 95% confidence intervals.The table provides the requested model comparison and cross-dataset evaluation summary.
B.2 Preliminary Analysis of Label Encoding
The preliminary dementia analysis compares soft, rigid, and three-class label encodings. The soft-label formulation performed best on ADNI and was therefore used in the main experiments.
- Label encodings: The soft encoding assigns CN = 0, MCI = 0.5, and AD = 1, whereas rigid encoding treats MCI and AD as positive cases.A standard three-class encoding assigns CN = 0, MCI = 1, and AD = 2.
- Label-selection result: The soft-label formulation achieved the best performance on ADNI with a linear anatomical-feature model and was adopted for the main dementia experiments.The comparison is reported in Table B7.
B.3 Preliminary Analysis of Baseline Neural Network Model Choice
The preliminary neural-network analysis selected a ResNet-18 with 128^3 inputs as the supervised classification baseline. The appendix also describes downstream adaptation and the lightweight, region-conditioned decoder used for anatomy segmentation pretraining.
- CNN baseline selection: ResNet-18 with input size (128, 128, 128) achieved the best overall performance and was adopted as the CNN baseline.The preliminary comparison included ResNet-18 at two input sizes and ResNet-50 at 128^3.
- Downstream adaptation: The pretrained encoder uses linear probing for Parkinson’s disease and IDH mutation prediction, and LoRA for the remaining downstream tasks.Full fine-tuning was evaluated as an ablation against LoRA.
- Downstream adaptation: Healthy-aging evaluation uses healthy subjects from the pretraining cohort for finetuning and ADNI and NACC subjects for generalization testing.Dementia evaluation uses ADNI plus external cohorts, while Parkinson’s disease, IDH mutation, and epilepsy use task-specific datasets.
- ASP decoder: The ASP decoder receives a global image embedding and a selected region embedding, avoiding skip connections and dense logits for all 39 regions.This design keeps the decoder lightweight and encourages anatomical information to remain in the ViT embedding.
C.3 Ablation Study on ASP Training
The ablation study compares ASP training choices for age estimation and classification, showing that both optimization strategy and pretrained initialization affect performance. Full ASP performs best across age-estimation cohorts, while LoRA and full fine-tuning are comparable for Alzheimer’s disease and LoRA improves epilepsy detection.
- For Alzheimer’s disease detection, full fine-tuning and LoRA perform similarly, whereas LoRA improves epilepsy detection.
- Full ASP achieves the best age-estimation performance on both evaluated cohorts.
- Removing alternating masked-autoencoder updates reduces performance, with further degradation after also removing initial encoder freezing.
- Masked-autoencoder-only pretraining remains strong but is outperformed by ASP variants adding anatomical supervision.
- Training from scratch performs worst, indicating that ASP benefits from initialization with a pretrained masked autoencoder.