PaperlayerView exact revision on arXiv

Source-linked AI summary

How Robust Are CNNs and Vision Transformers for Alzheimer's MRI Classification Across Datasets?

Hafija Aktar, Molly Croes, Frank S. Menniti, Abdeltawab Hendawi

arXiv:2610.33416v1cs.CV

TL;DR

Single-dataset Alzheimer’s MRI results do not establish whether architecture performance patterns hold across different data conditions, and adjacent early stages can be difficult to distinguish. The paper benchmarks five architectures across three datasets using standardized preprocessing and evaluation. ViT-Large is strongest across datasets, while architecture gaps vary with dataset condition.

  • Problem

    Single-dataset evaluations leave uncertain whether model rankings are consistent across data conditions, while subtle early-stage changes make adjacent disease categories difficult to distinguish.

    Source: §I · p. 1

  • Method

    The study benchmarks five CNN and vision-transformer architectures across three MRI datasets representing class imbalance, train–test mismatch, and large-scale augmentation, using standardized preprocessing and evaluation.

    Source: §I · pp. 1–2

  • Results

    ViT-Large achieves the strongest overall performance across the three datasets; architecture differences are more pronounced under distribution mismatch and less apparent with augmented data.

    Source: F. Summary of Experimental Results · pp. 10–11, §VII · p. 11

  • Takeaways & Limitations

    Evaluating multiple datasets provides a more reliable view of performance consistency than relying on a single dataset.

    Source: §VII · p. 11

    The findings may not fully generalize to independent clinical cohorts, multi-site hospital data, or scans acquired under different imaging protocols.

    Source: §VI · p. 11

Abstract

from arXiv · show

Accurate assessment of Alzheimer's disease stages from MRI is important for understanding disease progression and supporting the development of computer-aided analysis tools. Deep learning models have shown promising results for Alzheimer's disease classification from MRI. However, when evaluated on single datasets, it is difficult to determine whether the observed performance reflects consistent architectural behavior or dataset-specific effects. In this work, we present a multi-dataset benchmarking framework with standardized preprocessing and evaluation. We compare five architectures: ResNet-50, ReXNet-150, Swin Transformer, ViT-Base, and ViT-Large. The models are evaluated across three MRI datasets representing severe class imbalance, train-test distribution mismatch, and large-scale augmentation. The results show broadly consistent performance patterns across datasets, with ViT-Large achieving the best performance on all three datasets and reaching up to 99.84% accuracy and 99.86% macro F1. Performance differences between architectures are most pronounced under distribution mismatch and considerably smaller in the augmented-data setting. Across datasets, lower class-level performance is observed primarily among adjacent early-stage categories, particularly Very Mild and Mild impairment. These findings highlight the importance of evaluating deep learning architectures across multiple data conditions rather than relying on single-dataset performance.

I. INTRODUCTION

Single-dataset results leave uncertain whether model rankings reflect consistent architecture behavior or dataset-specific effects. The study benchmarks five architectures across three data conditions and finds ViT-Large strongest overall, with performance gaps varying by condition and weaker results concentrated in adjacent early stages.

I. INTRODUCTION

  • Because prior Alzheimer’s MRI results often come from single datasets, it remains unclear whether architecture rankings are consistent across diverse data conditions.The paper also notes that early-stage structural changes can be subtle, making adjacent categories particularly difficult to distinguish.Source: §I · p. 1
  • The benchmark compares five CNN and vision-transformer architectures across severe class imbalance, train–test distribution mismatch, and large-scale augmentation.It uses standardized preprocessing and evaluation, with dataset- and architecture-specific optimization parameters.Source: §I · pp. 1–2
  • The analysis examines changes in model ranking, sensitivity to distribution mismatch, and class-level performance across disease stages.Source: §I · pp. 1–2
  • Across all three datasets, ViT-Large achieves the strongest overall performance; architecture differences widen under distribution mismatch, while Very Mild and Mild impairment show lower class-level performance.The introduction also notes that Swin Transformer consistently ranks among the top models.Source: §I · p. 2

II. RELATED WORK

Prior Alzheimer’s MRI studies span CNNs, transformers, and hybrid models, but differing datasets and evaluation protocols limit direct comparisons. This paper applies a consistent benchmark across three dataset conditions, with dataset-specific partitioning where needed.

II. RELATED WORK

  • Prior work spans CNN, transformer, and hybrid approaches, but single-dataset evaluations and inconsistent protocols make architecture effects difficult to separate from dataset effects.The related-work review also discusses cross-cohort performance, dataset shift, synthetic balancing, augmentation, and possible leakage; Table I summarizes representative studies.Source: §II · p. 2, Table I · p. 3

III. DATASETS

  • The three datasets provide complementary tests of severe class imbalance, train–test distribution mismatch, and large-scale augmentation.The study uses them to examine whether CNN and vision-transformer behavior remains stable as data properties change.Source: §III · p. 3
  • Datasets I and III are repartitioned into stratified training, validation, and test sets, while Dataset II retains its original test set to preserve the imbalanced evaluation condition.Dataset II’s provided training set is split into training and validation subsets.Source: §III · p. 3

A. Dataset Descriptions

The benchmark selects datasets to represent different data conditions. Dataset I supplies a real-image setting with severe class imbalance.

A. Dataset Descriptions

  • The datasets are selected to represent real-world class imbalance, synthetic balancing with distribution mismatch, and large-scale augmentation.The study uses these conditions to compare five architectures across different dataset characteristics.Source: A. Dataset Descriptions · pp. 3–4
  • Fig. 1 presents representative axial MRI slices for the study’s four diagnostic categories.Source: Figure 1 · p. 4

1) Dataset I: Alzheimer MRI Disease Classification Dataset:

  • Dataset I contains 6,400 images across four diagnostic categories, with 64 Moderate Demented samples in total.After stratified partitioning, it has 4,572 training, 914 validation, and 914 test images; only nine Moderate Demented samples are in each of the validation and test sets.Source: 1) Dataset I: Alzheimer MRI Disease… · p. 4

2) Dataset II: Best Alzheimer MRI Dataset:

Dataset II pairs balanced, partly synthetic training data with a highly imbalanced test set, while Dataset III uses augmented images derived from a small number of originals. The datasets also differ in their class-distribution patterns.

2) Dataset II: Best Alzheimer MRI Dataset:

  • Dataset II combines real and WGAN-GP-generated training images in a roughly balanced set, but preserves a highly imbalanced test set to create train–test distribution mismatch.The test set contains 640 No Impairment samples and 12 Moderate Impairment samples.Source: 2) Dataset II: Best Alzheimer MRI… · p. 4

3) Dataset III: Augmented Alzheimer MRI Dataset:

  • Dataset III uses augmented images generated from a small number of originals, so redundancy across splits may inflate performance and its results are not evidence of clinical generalization.The dataset has 33,984 images after stratified partitioning.Source: 3) Dataset III: Augmented Alzheimer… · p. 4

B. Class Distribution

  • Dataset I has only 46 Moderate Demented training samples versus 2,286 Non Demented samples, while Dataset II’s balanced training data contrasts with its highly imbalanced test set.Dataset III has moderate imbalance, with similar class distributions across train, validation, and test splits.Source: B. Class Distribution · p. 5
  • Across the three datasets, the distribution conditions are severe class imbalance, train–test mismatch, and large-scale augmentation with more moderate class imbalance.Source: B. Class Distribution · p. 5

C. MRI Data Preparation

MRI inputs are standardized before models are fine-tuned and evaluated, with augmentation applied only to training images. The setup supports architectural comparisons under consistent preprocessing and evaluation while allowing dataset-specific training adaptations.

C. MRI Data Preparation

  • 224×224 resizing, three-channel replication, [0, 1] scaling, and ImageNet normalization standardize MRI inputs for pretrained models.The preprocessing is intended to reduce input-related variability when comparing architectures and training configurations.Source: C. MRI Data Preparation · p. 5
  • Training-only augmentation combines spatial and intensity perturbations, while validation and test images receive deterministic processing.The pipeline includes random cropping, flips, rotations, color jitter, grayscale conversion, and RandAugment; on replicated grayscale images, color operations act as intensity perturbations.Source: C. MRI Data Preparation · p. 5

IV. METHODOLOGY

  • Standardized preprocessing and evaluation support comparisons across CNN and transformer models, while training strategies are adapted to each dataset.The methodology evaluates representative architectures from both families under consistent preprocessing and evaluation settings.Source: §IV · p. 5

A. Benchmark Architectures

The benchmark compares five CNN and transformer architectures spanning residual and efficient convolutions, hierarchical attention, and patch-based global attention. The models differ in architectural design and capacity for Alzheimer’s MRI classification.

A. Benchmark Architectures

  • The five-model benchmark spans CNNs and transformers with different feature-extraction designs and model capacities.The set comprises ResNet-50, ReXNet-150, Swin Transformer, ViT-Base, and ViT-Large.Source: A. Benchmark Architectures · p. 6

1) ResNet-50:

  • ResNet-50 serves as the main CNN reference, using residual shortcuts to support optimization of a deep convolutional model.Its bottleneck blocks use 1 × 1, 3 × 3, and 1 × 1 convolutions to reduce, process, and restore feature dimensions.Source: 1) ResNet-50 · p. 6

2) ReXNet-150:

  • ReXNet-150 is an efficient CNN baseline that progressively expands channels to reduce representational bottlenecks.The architecture is used to assess whether channel allocation and parameter-efficient convolution benefit MRI-based classification.Source: 2) ReXNet-150 · p. 7

3) Swin Transformer:

  • Swin Transformer uses local-window attention and shifted windows to exchange information across windows while retaining an efficient hierarchical design.It progressively reduces spatial resolution and increases feature dimensionality to produce multi-scale representations.Source: 3) Swin Transformer · p. 7

4) Vision Transformer (ViT-Base):

  • ViT-Base splits images into 16 × 16 patches and processes the resulting sequence with transformer encoders for global patch-level attention.It serves as the baseline transformer model for assessing global self-attention in MRI classification.Source: 4) Vision Transformer (ViT-Base) · p. 7

5) Vision Transformer (ViT-Large):

ViT-Large increases the capacity of ViT-Base while retaining its patch-based design. Fine-tuning uses classifier heads and training strategies adapted to dataset characteristics.

5) Vision Transformer (ViT-Large):

  • ViT-Large has 24 encoder layers, 16 attention heads, and approximately 307M parameters, versus ViT-Base’s 12 layers, 12 heads, and approximately 86M parameters.The comparison is used to assess whether increased transformer capacity improves performance across dataset conditions.Source: 5) Vision Transformer (ViT-Large) · p. 7

B. Model Fine-Tuning Strategy

  • All architectures start from ImageNet-pretrained weights and are adapted to predict four Alzheimer’s MRI diagnostic categories.CNNs use a replacement linear classifier; transformer heads vary by dataset, with custom heads on Datasets II and III.Source: B. Model Fine-Tuning Strategy · pp. 7–8
  • Dataset I uses a two-epoch head-only warmup before end-to-end fine-tuning, whereas Datasets II and III fine-tune all parameters from the start.Transformer models use split backbone/head learning rates on Datasets I and III, but a single rate on Dataset II.Source: B. Model Fine-Tuning Strategy · p. 8

C. Loss Function and Optimization

Optimization and training settings are adapted to dataset size and class distribution, with common optimizer and stopping procedures across model comparisons. Performance is evaluated using accuracy and macro F1.

C. Loss Function and Optimization

  • Dataset I uses tempered inverse-frequency sampling for severe imbalance, Dataset II uses inverse-frequency loss weights, and Dataset III uses no explicit imbalance correction.Dataset I’s sampling weights use inverse class frequency raised to α = 0.5.Source: C. Loss Function and Optimization · p. 8
  • All configurations use AdamW, while learning-rate schedules and computational stabilization techniques vary by dataset.Dataset I uses cosine annealing; Datasets II and III use ReduceLROnPlateau, and mixed precision, gradient accumulation, and clipping are applied on Datasets I and III.Source: C. Loss Function and Optimization · p. 8

D. Training Configuration

  • Training runs for up to 20 epochs with early stopping, using Balanced Accuracy for Dataset I validation and Macro F1 for Datasets II and III.The model with the best validation score is selected for final testing.Source: D. Training Configuration · p. 8

V. EXPERIMENTAL RESULTS

  • Accuracy and macro-averaged F1 are used to evaluate models, with macro F1 giving equal importance to each class.This makes macro F1 useful for assessing performance when class distributions are imbalanced.Source: §V · p. 8

A. Model Performance Across Datasets

Model rankings are broadly consistent across datasets, but Dataset II’s train-test distribution mismatch produces the largest performance differences. Very Mild and Mild impairment remain difficult categories, including in the severely imbalanced Dataset I setting.

A. Model Performance Across Datasets

B. Results on Dataset I

  • On Dataset I, ViT-Large achieves 98.47% accuracy and 98.85% macro F1, followed by Swin Transformer at 98.36% accuracy and 98.75% macro F1.ReXNet-150 outperforms ResNet-50, while ViT-Base has lower accuracy but a macro F1 of 98.15%.Source: B. Results on Dataset I · p. 9
  • Dataset I’s Moderate Demented class has an F1-score of 1.00 for every model, but only nine test samples; Very Mild and Mild Demented scores are generally lower.The severe class imbalance makes macro F1 and per-class results useful complements to accuracy.Source: B. Results on Dataset I · p. 9

C. Results on Dataset II

  • On Dataset II, ViT-Large leads with 97.81% accuracy and 98.30% macro F1, followed by Swin Transformer at 97.19% accuracy and 97.60% macro F1.The test set is highly imbalanced despite balanced training data.Source: C. Results on Dataset II · p. 9
  • On Dataset II, ViT-Base has the largest degradation, with 91.30% accuracy and 86.00% macro F1; early-stage Very Mild and Mild categories also have lower F1-scores for several models.Moderate Impairment class-level results are based on only 12 test samples and warrant caution.Source: C. Results on Dataset II · p. 9

D. Results on Dataset III

ViT-Large leads on the augmented Dataset III, where architectural performance differences are smaller than under Dataset II’s distribution mismatch. Across settings, early-stage categories remain difficult, and the authors caution that augmented-data results do not establish clinical generalization.

D. Results on Dataset III

  • On Dataset III, ViT-Large achieves 99.84% accuracy and 99.86% macro F1, the best individual-model results; the larger, less imbalanced dataset has smaller performance differences across models.Swin Transformer follows at 99.57% accuracy and 99.60% macro F1.Source: D. Results on Dataset III · pp. 9–10, Table VIII · p. 10

F. Summary of Experimental Results

  • Architecture gaps are substantially larger under Dataset II’s distribution mismatch and relatively small on Datasets I and III; dataset conditions influence how clearly models separate.Across the three settings, model performance patterns remain broadly consistent, with ViT-Large and Swin Transformer among the strongest.Source: E. Cross-Dataset Study · p. 10, F. Summary of Experimental Results · p. 10
  • Lower class-level performance generally occurs in adjacent early-stage categories, particularly Very Mild and Mild impairment.Dataset III’s near-ceiling results may be influenced by redundant or visually similar samples.Source: F. Summary of Experimental Results · p. 10, Table IX · p. 10
  • Independent clinical validation is still needed to assess generalizability beyond the evaluated benchmark conditions.The authors characterize Dataset III results as augmented-data performance rather than evidence of clinical generalization.Source: F. Summary of Experimental Results · p. 11

E. Cross-Dataset Study

  • From Dataset I to Dataset II, ViT-Base’s accuracy drops 5.86 points, versus 0.66 points for ViT-Large.Figure 3 orders models by degradation under the evaluated train-test distribution mismatch.Source: E. Cross-Dataset Study · p. 10, Figure 3 · p. 11

VI. LIMITATIONS

The authors caution that Dataset III’s near-ceiling results come from augmented MRI images derived from limited originals, and that all datasets use similar labels and preprocessed structural MRI. The conclusions call for independent clinical cohorts and broader data, while noting that dataset- and architecture-specific tuning prevents treating observed gaps as isolated architecture effects.

VI. LIMITATIONS

  • Dataset III’s near-ceiling performance may be inflated by visually similar or source-related images appearing across training, validation, and test splits, so it is not evidence of clinical generalization.The authors say to interpret these results as augmented-data performance.Source: §VI · p. 11
  • Similar four-class labels and preprocessed structural MRI limit generalization to independent clinical cohorts, multisite hospital data, or scans acquired under different protocols.The study also excludes longitudinal information, clinical variables, cognitive scores, and demographic factors.Source: §VI · p. 11
  • Because optimization settings vary by dataset and architecture, performance differences compare selected fine-tuning configurations rather than isolated causal effects of architecture or model capacity.The conclusions call for future work using independent cohorts, subject-level splits, multiple scanners and sites, and longitudinal or multimodal information.Source: §VI · p. 11, §VII · p. 11
Loading 2610.33416v1…