PaperlayerView exact revision on arXiv

Source-linked AI summary

Comparative Study of Anatomical and Learned Features in AI Models for Structural Brain MRI

Boyang Yu, Miquel Lopez Escoriza, Long Chen, Arjun V. Masurkar, Narges Razavian, Carlos Fernandez-Granda

arXiv:2609.06807v1cs.CVcs.AI

TL;DR

The relative performance of anatomical features and learned representations across clinical structural-MRI tasks is the question this study addresses. It compares anatomical features, CNNs, and pretrained ViTs across seven tasks and 18 datasets, finding that linear anatomical-feature models remain highly competitive with deep networks, while ASP achieves superior biological-age estimation.

  • Problem

    How anatomical features compare with CNN- and ViT-learned features across clinical structural-MRI tasks is the central question this study addresses.

    Source: Introduction · p. 3

  • Method

    The study compares anatomical features, CNNs, and pretrained ViTs across 18 datasets and seven clinical tasks, and introduces ASP using brain-region segmentation as an auxiliary pretraining task.

    Source: Introduction · p. 3, Anatomy Segmentation Pretraining of… · p. 7

  • Results

    Across public structural-MRI datasets, linear models using preselected anatomical features remain highly competitive with deep networks; ASP ViT achieves 2.54 years MAE in the internal age-test cohort.

    Source: Results · p. 6, Discussion · p. 10

  • Takeaways & Limitations

    Linear models using structured anatomical features provide a computationally efficient, interpretable, and robust baseline for structural-MRI diagnostic tasks.

    Source: Discussion · p. 10

    The study uses only de-identified secondary data from publicly available repositories and collects no new human data.

    Source: Ethics Approval and Consent to… · p. 10

Abstract

from arXiv · show

In this work, we comprehensively evaluate three popular feature-extraction paradigms in AI-based neuroimaging modeling: (1) computation of anatomical surfaces and volumes, (2) supervised learning with convolutional neural networks (CNNs), and (3) unsupervised pretraining of vision transformer (ViT) foundation models, followed by supervised finetuning. Our study is based on 18 publicly available datasets containing 3D structural T1-weighted MRI scans from approximately 80,000 participants across seven distinct clinical tasks. We observe that a linear model based on anatomical features matches the diagnostic performance of complex nonlinear features learned by sophisticated AI frameworks, including foundation models trained on thousands of scans. Conversely, CNNs and pretrained ViTs learn features that implicitly capture relevant anatomical information, bypassing the need for explicit feature extraction. Building upon these insights, we propose Anatomy Segmentation Pretraining (ASP), a novel method to incorporate anatomical information during foundation-model pretraining, which outperforms existing models in biological age estimation.

Introduction

The study compares anatomical features, CNNs, and pretrained ViTs for structural T1-weighted MRI modeling. It finds that anatomical features remain competitive with learned representations and introduces anatomy-guided pretraining that improves biological age estimation.

Introduction

  • Neuroimaging feature extraction has progressed from hand-engineered anatomical measures to CNN-learned features and, more recently, ViT foundation models pretrained on unlabeled data.Prior work adapted ViT pretraining to neuroimaging, including multimodal models.Source: Introduction · pp. 2–3
  • Across clinical tasks, anatomical features match the diagnostic performance of complex learned features, while CNNs and pretrained ViTs implicitly capture relevant anatomy.The comparison covers models trained on clinical cohorts ranging from dozens to thousands of examples.Source: Introduction · p. 3
  • The study evaluates seven clinical tasks using 18 public datasets with more than 96,000 MRI scans from approximately 80,000 participants.Splits were participant-level, and external cohorts came from independent data sources.Source: Figure 1 · p. 2, Introduction · p. 3
  • ASP adds brain-structure segmentation during ViT pretraining, performs on par with other strategies across clinical tasks, and achieves superior biological age estimation.Anatomical information becomes increasingly predictive of brain structures during pretraining; the age-estimation task has more training data available.Source: Figure 1 · p. 2, Introduction · p. 3

Results

Across the clinical and segmentation tasks, pretrained ViTs, CNNs, and anatomical-feature linear models showed no significant overall performance differences. For biological age estimation, ASP ViT outperformed the other models, with its advantage especially evident for older subjects and external cohorts.

Results

  • ASP ViT had the highest Alzheimer’s-diagnosis macro-AUC on the internal test and six external cohorts, while Pretrained ViT had the highest Parkinson’s-diagnosis AUC.For Alzheimer’s diagnosis, differences from CNN and anatomical logistic regression were significant only on OASIS-2; for Parkinson’s diagnosis, the difference was significant only against BrainIAC.Source: Results · p. 5
  • ASP ViT had the highest AUC for IDH-mutation detection, while Pretrained ViT had the highest epilepsy-diagnosis AUC; neither advantage was statistically significant.The CNN was not evaluated for epilepsy because training data were limited.Source: Results · p. 5
  • For tumor segmentation, U-Net had the highest Dice score, slightly above ASP ViT, while Pretrained ViT had the highest stroke-lesion Dice score, slightly above ASP ViT.With only 10% of tumor-segmentation training data, ASP and Pretrained ViT outperformed U-Net and UNETR; stroke-lesion differences were significant only against UNETR.Source: Results · pp. 5–6
  • Across the clinical and segmentation tasks, pretrained ViTs, CNNs, and anatomical-feature linear models showed no significant overall performance differences.Source: Results · p. 6
  • ASP ViT achieved 2.54-year MAE internally and 4.19-year MAE on both NACC and ADNI, outperforming the other models in biological age estimation.Pretrained ViT was 0.35 years worse internally and more than 1 year worse externally; ViT models, particularly ASP ViT, performed better for subjects over 80, and differences persisted by sex.Source: Results · p. 6
  • The best-performing age-estimation model predicted ages above biological age for dementia patients in most age intervals, with a less pronounced effect in MCI subjects.Source: Results · p. 6

Methods

The methods compare anatomical-feature models, CNNs, and pretrained 3D ViTs, and introduce anatomy segmentation pretraining (ASP) to add anatomical supervision during ViT pretraining. The study also describes downstream adaptation, MRI preprocessing, statistical evaluation, and ethics procedures.

Anatomy Segmentation Pretraining of Vision Transformer

  • ASP adds brain-region segmentation to MAE pretraining, first training a decoder with the encoder frozen, then jointly training them while alternating segmentation and MAE epochs; each component improved age estimation in ablations.The method uses 39 selected regions and a lightweight bottleneck decoder without skip connections to concentrate anatomical information in the ViT embedding.Source: Anatomy Segmentation Pretraining of… · pp. 7–8

Masked Autoencoder Pretraining of ViTs

  • 3D MAE pretraining used a 12-block ViT-Base with 16×16×16-voxel patches, masking 75% of patches and reconstructing volumes for 1000 epochs.Training used MSE loss, batch size 32 per GPU, and AdamW; convergence took about 10 days on two 80GB NVIDIA A100 GPUs.Source: Masked Autoencoder Pretraining of ViTs · p. 8

Downstream Application of Pretrained ViTs

  • Downstream ViT embeddings fed task-specific classification, regression, or segmentation heads; adaptation used linear probing, LoRA, or full finetuning for ablations.LoRA froze the original encoder and trained low-rank attention matrices with task heads; inference used center crops for classification and sliding windows for full-volume segmentation.Source: Downstream Application of Pretrained… · pp. 8–9

Linear Models with Anatomical Features

  • Anatomical models used 178 FreeSurfer features—regional volumes, cortical surface areas, and cortical thickness—for linear prediction across diagnostic tasks and age estimation.For Alzheimer’s diagnosis, healthy, mild cognitive impairment, and Alzheimer’s labels were encoded as 0, 0.5, and 1, respectively.Source: Linear Models with Anatomical Features · p. 9

Convolutional Neural Networks

  • CNN baselines used ResNet-18 for diagnostic classification and U-Net for segmentation, with classification trained using SGD and early stopping based on validation loss.Classification ran for up to 60 epochs with batch size 16 and an initial learning rate of 0.01.Source: Convolutional Neural Networks · p. 9

Data Preprocessing

  • MRI preprocessing converted DICOM to NIFTI, applied FreeSurfer recon-all except for pre-standardized segmentation cohorts, excluded scans failing initial processing, and coregistered images and masks to MNI152.Final scans had 197 × 233 × 189 voxels at one-cubic-millimeter isotropic resolution.Source: Data Preprocessing · pp. 9–10

Statistical analysis

  • Each experiment reports an overall metric and 95% confidence interval estimated from 1000 bootstrap resamples of the held-out test set.The interval uses the 2.5th and 97.5th percentiles of the bootstrap metric distribution.Source: Statistical analysis · p. 10

Ethics Approval and Consent to Participate

  • The study used de-identified secondary data from publicly available repositories collected under the original institutions’ ethics approvals and informed consent; no new human data were collected.The repositories included UKBB, HCP, CamCAN, IXI, NACC, ADNI, OASIS, AIBL, MIRIAD, PPMI, OpenNeuro, EPISURG, UCSF-PDGM, BraTS, and ATLAS.Source: Ethics Approval and Consent to… · p. 10

Discussion

Linear models using preselected anatomical features remain highly competitive with deep neural networks across public structural MRI datasets, while anatomical segmentation can accelerate anatomical learning during ViT pretraining. The discussion also argues that larger cohorts may enable further progress as diagnostic performance on existing datasets appears to be saturating.

Discussion

  • Linear models trained on preselected anatomical features remain highly competitive with deep neural networks across public structural MRI datasets.FastSurfer offers an efficient alternative for extracting anatomical features, supporting computationally efficient, interpretable, and robust linear-model baselines.Source: Discussion · p. 10
  • Diagnostic performance may be saturating on public structural MRI datasets with hundreds to a few thousand scans, while age-estimation experiments on 60,000 subjects suggest scaling datasets could enable further innovation.The authors argue that further progress will likely require curating substantially larger cohorts.Source: Discussion · p. 10
  • Vision transformers learn relevant anatomical structure slowly during unsupervised pretraining, but anatomical segmentation as an auxiliary task can greatly accelerate that learning.Source: Discussion · pp. 10–11
  • Leveraging clinically relevant tasks during pretraining could mitigate the small size of medical datasets relative to general computer-vision databases with millions of examples.Source: Discussion · p. 11

A.2 Pretraining Dataset

The pretraining dataset draws on UKBB, HCP, CamCAN, and IXI cohorts, including population-based and healthy-participant samples.

A.2 Pretraining Dataset

  • UK Biobank (UKBB)[18] contributed imaging-genomics participants with at least one T1 MRI, acquired on 3T Siemens Skyra with MPRAGE; 7.79% had a second scan within two years, using data available March 2024.Source: §A.2 · pp. 19–20
  • HCP[47, 48] contributed cross-sectional visit 1 structural scans from HCP-Aging adults aged 36–60 and HCP-Development participants aged 5–21, all imaged with the same Siemens Prisma protocol.Source: §A.2 · p. 20
  • CamCAN[49] studies healthy aging and cognitive abilities using interviews, brain scans, and cognitive experiments, with T1 MRI collected under a standardized protocol.Source: §A.2 · p. 20
  • IXI[50] provides healthy participants scanned at three London hospitals using varying scanners, supporting study of normal aging and brain structure without neurological-condition confounds.Source: §A.2 · p. 20

A.3 Disease Specific Datasets · A.4 Study Population in Each Dataset

The sections describe disease-specific MRI cohorts spanning Alzheimer’s disease, Parkinson’s disease, epilepsy, tumors, and stroke, alongside population and acquisition summaries for BrainATLAS’s 18 studies. Across evaluated tasks, anatomical features, CNNs, and pretrained ViTs showed no significant overall performance differences, while ASP led biological-age estimation.

A.3 Disease Specific Datasets

  • Alzheimer’s cohorts included longitudinal NACC visits through June 2023, available ADNI T1 scans through October 2023, and separately analyzed OASIS1–4 cohorts.NACC includes mixed scanner and acquisition protocols; OASIS1–4 contributed 436/373/2,275/181 MRI images from 416/150/1,044/180 unique subjects. AIBL T1 scans were accessed through LONI, while MIRIAD contributed 705 images from 69 subjects acquired on the same 1.5 T scanner by one radiographer.Source: §A.3 · pp. 20–21
  • Other disease-specific cohorts covered Parkinson’s disease in PPMI and epilepsy in OpenNeuro and EPISURG.OpenNeuro included 82 people with focal cortical dysplasia type II and 81 healthy controls; EPISURG contributed 268 available preoperative scans from 430 surgical epilepsy patients as external validation. PPMI data available in November 2023 included baseline-only control imaging and potentially multiple images for prodromal and Parkinson’s disease participants.Source: §A.3 · p. 21
  • Tumor and stroke cohorts included UCSF-PDGM and BraTS glioma scans and ATLAS lesion images with manual segmentation masks.UCSF-PDGM provided 501 grade 2–4 diffuse glioma patients with standardized 3-T preoperative MRI and tumor genetic data; BraTS 2023 Adult Glioma included paired contrast and non-contrast T1 scans with neuro-radiologist annotations, while ATLAS contributed 655 images from 33 subjects.Source: §A.3 · pp. 21–22
  • Across evaluated tasks, no significant overall performance differences were observed between pretrained ViTs, CNNs, and anatomical logistic regression.ASP ViT achieved the highest macro-AUC in most Alzheimer’s cohorts, with significance over CNN and anatomical logistic regression only for OASIS 2; pretrained ViT led PPMI and OpenNeuro AUC, and U-Net and pretrained ViT led BraTS and ATLAS segmentation, respectively, with significance only relative to UNETR in both segmentation tasks.Source: Figure 2 · p. 23
  • ASP ViT achieved MAE 2.54 years internally and 4.19 years in both NACC and ADNI external cohorts.Pretrained ViT’s MAE was 0.35 years higher internally and more than 1 year higher externally; age-stratified errors were more pronounced above age 80, and NACC and ADNI dementia estimates exceeded biological age across most age intervals.Source: Figure 3 · p. 24
  • The full ASP training variant achieved the lowest age-estimation error on external NACC among the reported training variants.Removing alternating MAE updates or encoder freezing, using MAE alone, or training from scratch reduced performance.Source: §A.3 · p. 25

A.4 Study Population in Each Dataset

  • Table A1 summarizes demographic and imaging-acquisition characteristics across BrainATLAS’s 18 studies.The table excludes missing values and outliers; its brain-volume measure is FreeSurfer BrainSegVol, and NA denotes unavailable data for the analyzed cohort.Source: §A.4 · p. 26

A.5 Imaging Data Harmonization … Appendix B Preliminary Analyses

The appendix describes FreeSurfer processing and anatomical-feature extraction, including a reduction from 112 raw segmentation regions to 39 grouped regions for supervised training. It also specifies that preliminary model selection used only a 10% ADNI training validation subset.

A.5 Imaging Data Harmonization

  • 6.11 hours was the average FreeSurfer workflow runtime per 3D image, estimated retrospectively from 1,500 successful processing logs across 31 workflow steps.The workflow covers operations from motion correction and conforming through cortical parcellation mapping.Source: §A.5 · p. 27

A.6 Brain Anatomical Features

  • 178 anatomical brain features derived from FreeSurfer segmentation and surface-reconstruction statistics cover volume, surface area, and thickness.The features collectively capture structural information about the brain.Source: §A.6 · p. 29, Table A3 · p. 34

A.7 Anatomical Brain Segmentation Workflow

  • All pretraining MRIs were processed with FreeSurfer, which automatically segments brain regions using volumetric and surface-based techniques.The segmentation workflow generates region-level labels from raw MRI volumes.Source: §A.7 · p. 33
  • The raw FreeSurfer segmentation contains 112 anatomical regions, providing the label space used as the basis for supervised learning.The label space was subsequently simplified to balance granularity, practicality, and preservation of disease-relevant neuroanatomical information.Source: §A.7 · p. 33
  • 39 grouped regions were used for supervised training after merging symmetrical, related, or minor structures according to clinical relevance and volume thresholds.The grouped labels combine original FreeSurfer region IDs.Source: §A.7 · p. 33, Table A5 · p. 37

Appendix B Preliminary Analyses

  • Preliminary model selection used only a validation subset comprising 10% of the ADNI training split; external-cohort results were reported for completeness, not selection.The corresponding tables underline the validation results used for model selection.Source: Appendix B · p. 37

B.1 Preliminary Analysis of Linear Model and Nonlinear Model Using Anatomical Features

In dementia detection using FreeSurfer anatomical features, performance differences between logistic regression and random forest were generally small, supporting logistic regression as the main anatomical baseline.

B.1 Preliminary Analysis of Linear Model and Nonlinear Model Using Anatomical Features

  • The analysis used 178 FreeSurfer-derived features covering regional brain volumes, cortical surface areas, and cortical thickness.The features were extracted from the recon-all pipeline.Source: §B.1 · p. 37
  • Models were tuned with 5-fold cross-validation, comparing logistic regression with a random forest across specified hyperparameter settings.Logistic regression used no regularization or L2 regularization with coefficient 1; the random forest varied n_estimators, max_depth, and criterion.Source: §B.1 · p. 37
  • Performance differences were generally small, supporting logistic regression as the main anatomical baseline for dementia detection with anatomical features.The comparison was repeated with 10%, 30%, and 100% of the training data; Table B6 reports external-cohort performance with 95% confidence intervals by dataset.Source: §B.1 · p. 37, Table B6 · p. 38

B.2 Preliminary Analysis of Label Encoding

The study compared soft, rigid, and three-class label encodings for dementia progression, and adopted soft labels after they performed best in preliminary ADNI experiments.

B.2 Preliminary Analysis of Label Encoding

  • The analysis compared soft labels (CN = 0, MCI = 0.5, AD = 1) trained with log loss, rigid labels (CN = 0, MCI/AD = 1), and three-class labels (CN = 0, MCI = 1, AD = 2).The encodings were evaluated because dementia progression is not strictly discrete.Source: §B.2 · p. 38
  • Soft-label encoding performed best on ADNI with a linear model using anatomical features and was adopted for the main dementia experiments.Table B7 compares label encoding methods using this model and feature set.Source: Table B7 · p. 38, §B.2 · p. 38

B.3 Preliminary Analysis of Baseline Neural Network Model Choice … C.2 Additional details on ASP architecture

The appendix selects ResNet-18 at 128³ as the CNN baseline and details the pretraining, adaptation, evaluation, and lightweight region-conditioned ASP decoder setup.

B.3 Preliminary Analysis of Baseline Neural Network Model Choice

  • ResNet-18 with (128, 128, 128) input achieved the best overall performance among the tested ResNets and was adopted as the CNN baseline.Source: Table B8 · p. 39, §B.3 · p. 39

C.1 Additional details on MAE pretraining and downstream adaptation

  • The pretraining cohort comprised UKBB, HCP, CamCAN, and IXI; a participant-level UKBB holdout of 6,950 scans monitored linear probes and served as the internal held-out age-estimation cohort.Source: §C.1 · p. 39
  • Downstream adaptation used linear probing for Parkinson’s disease and IDH prediction, LoRA for other tasks, and full fine-tuning as an ablation; regression used MSE and classification used binary cross-entropy.Source: §C.1 · p. 39
  • Healthy-aging models trained on healthy pretraining-cohort subjects and tested on ADNI and NACC; dementia models trained on ADNI and tested on held-out ADNI plus NACC, AIBL, OASIS1-4, and MIRIAD.Source: §C.1 · p. 39
  • Parkinson’s disease, IDH mutation, and epilepsy used task-specific train/test splits from PPMI, UCSF-PDGM, and OpenNeuro, respectively.Source: §C.1 · pp. 39–40

C.2 Additional details on ASP architecture

  • ASP avoids skip connections and uses a small bottleneck decoder driven by the global image embedding, encouraging the ViT embedding to carry anatomical information rather than relying on a large segmentation head.Source: §C.2 · p. 40
  • At each training step, a learned region embedding conditions the decoder to predict one of 39 binary masks, avoiding memory-intensive dense logits for all regions in full 3D volumes.Source: §C.2 · p. 40
  • The decoder combines a 768-dimensional image embedding with the selected region embedding, then uses six 3D transposed-convolution blocks to upsample from 13 to 128³; it has around 15 million parameters.Source: §C.2 · p. 40

C.3 Ablation Study on ASP Training

The full ASP model performs best on both age-estimation cohorts; its ablations indicate benefits from anatomical supervision and pretrained masked autoencoding. For classification, full fine-tuning and LoRA perform similarly on Alzheimer’s disease, while LoRA improves epilepsy detection.

C.3 Ablation Study on ASP Training

  • For Alzheimer’s disease, full fine-tuning and LoRA perform similarly, while LoRA improves epilepsy detection.These comparisons evaluate ASP with full fine-tuning versus LoRA on the two classification tasks.Source: §C.3 · p. 41, Table C9 · p. 41
  • Age-estimation ablations include removing MAE alternation, encoder freezing, or anatomical supervision, as well as training from scratch; table errors are reported as absolute error in years.This label distinguishes mean absolute error from masked autoencoding; the variants correspond to Figure 4(e).Source: §C.3 · p. 41, Table C10 · p. 42
  • Removing alternating masked-autoencoder updates reduces performance, and also removing initial encoder freezing reduces it further.The masked-autoencoder-only model remains a strong baseline, but ASP variants with anatomical supervision outperform it; training from scratch performs worst, indicating a benefit from starting with a pretrained masked autoencoder.Source: §C.3 · p. 41, Table C10 · p. 42
  • The full ASP model achieves the best age-estimation performance on both UK Biobank and NACC.UK Biobank is the internal test cohort and NACC the external test cohort.Source: §C.3 · p. 41, Table C10 · p. 42

Related papers

Loading 2609.06807v1…