PaperlayerView exact revision on arXiv

Source-linked AI summary

Anatomical Grounding and Leakage-Aware Multimodal Contrastive Learning for Alzheimer's Disease Classification from Structural MRI

Paul-Gabriel Nicolae, Irina Georgiana Mocanu

arXiv:2609.15888v1cs.CVcs.AI

TL;DR

MRI classifiers can attend to irrelevant anatomy, while clinical inputs may encode the diagnostic labeling rule. The paper uses FastSurfer references and leakage-aware image-tabular contrastive learning to study both issues; volume alignment yields an image-only MCI-versus-CN representation 21 points better than cognitive-score alignment, and an MTL crop improves image-only accuracy. These findings are bounded by a small test set, single runs, and differing evaluation protocols.

  • Problem

    MRI classifiers may attend to anatomically irrelevant regions, and multimodal models may use clinical variables that helped assign the diagnostic label.

    Source: §1 · p. 2

  • Method

    The study uses FastSurfer segmentations to audit and guide a slice-based MRI classifier and adapts CLIP-style contrastive learning to evaluate ADNIMERGE groups by label overlap.

    Source: §1 · p. 2

  • Results

    21 points better on MCI vs. CN when the image encoder is aligned to volumes rather than cognitive scores, with the image-only head using no tabular input at inference.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FastSurfer-derived anatomy can serve both as an audit reference and as a way to constrain classifier input, while the contrastive target influences the image representation.

    Source: §11 · p. 10

    The test set is small, each configuration is a single run, and the MTL crop requires a FastSurfer segmentation at inference.

    Source: §10 · p. 10

Abstract

from arXiv · show

Deep networks trained on structural MRI for Alzheimer's disease (AD) staging often reach reasonable accuracy while attending to anatomically irrelevant regions, and multimodal models that add clinical tables frequently rely on variables that were used to assign the diagnostic label in the first place. We study both issues with a deliberately lightweight slice-based encoder (ResNet18 with a one-layer Transformer over slices) on 1,075 baseline T1-weighted scans from ADNI-1. First, we use FastSurfer segmentations as an anatomical reference: YOLOv8 models trained on segmentation-derived labels localize Alzheimer-relevant structures with mAP_50 above 0.96, and a Grad-CAM comparison shows that the image-only classifier frequently attends to the skull, orbits and background. Second, we adapt a CLIP-style image - tabular contrastive framework and organize ADNIMERGE variables along a label-leakage spectrum. Fusion with cognitive scores yields 87.3% three-way accuracy, which we treat as a leakage-driven upper bound rather than an imaging result; fusion with regional volumes yields 73.0%. We observe that the choice of contrastive target changes what the image encoder learns: on MCI vs. CN, the image-only head reaches 52.4% when the encoder is aligned to cognitive scores and 73.8\% when aligned to volumes, although no tabular input is used at inference. Third, restricting the input to a per-subject crop of the medial temporal lobe raises image-only three-way accuracy from 58.7% to 65.1%. All results come from single runs on a small balanced test set, and we report confidence intervals and the protocol differences that prevent direct comparison with published numbers.

1 Introduction

The paper addresses anatomically irrelevant attention in MRI classifiers and label leakage from clinical variables used to assign diagnoses. It combines anatomical references with a leakage-aware multimodal protocol to examine these problems and the image representations they produce.

1 Introduction

  • Clinical tables can include MMSE and CDR scores used in ADNI diagnosis, so high accuracy with these inputs can reflect recovery of the labeling rule rather than image contribution.The paper evaluates ADNIMERGE modality groups separately according to their overlap with the label.Source: §1 · p. 2
  • The study examines both problems in one lightweight pipeline: a 2D ResNet18 slice encoder aggregated by a single Transformer layer, trained on ADNI-1 baseline scans.Source: §1 · p. 2
  • The authors use FastSurfer segmentations to train YOLOv8 structure detectors and compare their outputs with classifier Grad-CAM maps.This uses segmentation-derived slice ranges and detection/segmentation labels for Alzheimer-relevant structures.Source: §1 · p. 2
  • Aligning the image encoder to volumes instead of cognitive scores produces a representation 21 points better on MCI vs. CN, with architecture, split, and training held fixed.The image-only head uses no tabular input at inference.Source: §1 · p. 2
  • A per-subject medial temporal lobe crop derived from the segmentations improves image-only accuracy without changing the encoder.The crop is presented as an anatomically guided input strategy.Source: §1 · p. 2
  • The test set contains 63 scans, and single-run results and protocol differences constrain comparisons with published numbers.The authors state that these differences prevent treating their numbers as evidence of outperforming prior work.Source: §1 · p. 2

2 Related Work

Prior MRI work reports stronger AD-versus-CN than MCI-versus-CN performance, while leakage and multimodal contrastive fusion are established concerns. This paper combines anatomical priors with segmentation-based auditing and input constraints.

2 Related Work

  • Prior ADNI-1 studies report AD vs. CN accuracies around 90% with 3D networks, while MCI vs. CN performance remains considerably lower.Wen et al. documented inflation from slice-level rather than subject-level splitting and argued for reproducible subject-level evaluation.Source: §2 · p. 2
  • Huang [3] reports 83.8% three-way accuracy and 95.5% on AD vs. CN for a fused image-tabular model.The model contrasts MR images with groups of tabular variables and adds a tabular attention module.Source: §2 · p. 2
  • ADNIMERGE modality groups are ordered by overlap with the diagnostic label: cognitive scores are diagnostic criteria, whereas volumes correlate with atrophy but are not criteria.Cognitive scores are part of the ADNI diagnostic criteria [5].Source: Table 1 · p. 3
  • The paper combines FastSurfer segmentation with Grad-CAM to audit MRI classifiers and constrain their input anatomically.FastSurfer provides subcortical segmentation and cortical parcellation; Grad-CAM is used to inspect regions driving classifiers.Source: §2 · p. 3

3 Data

The study uses ADNI-1 baseline T1-weighted scans, subject-level splits, and matched ADNIMERGE clinical variables. It focuses on cognitive scores and regional volumes as contrasting points on a label-leakage spectrum.

3 Data

  • The dataset contains 1,075 baseline scans from 982 subjects, with approximately 289 CN, 507 MCI, and 279 AD scans.FastSurfer processed the scans into 1 mm isotropic volumes and DKT cortical parcellations with subcortical labels.Source: §3 · p. 3
  • Subject-level splitting keeps each subject in one split; the balanced test set has 21 scans per class, or 63 scans total.The remaining 1,012 scans are divided into approximately 862 training and 150 validation scans.Source: §3 · p. 3
  • Each scan is matched to an ADNIMERGE row by subject, preferred baseline visit codes, and examination-date proximity; all 1,075 scans were matched.Features are standardized using training-set statistics, and missing cog and vol values are imputed with training-set feature means.Source: §3 · p. 3
  • The experiments use cognitive and volume groups because they sit at opposite ends of the leakage spectrum; the bio group covers only about 14% of ADNIMERGE rows.The authors expect heavy imputation of bio features to distort contrastive alignment.Source: §3 · p. 3

4 Image Encoder

The image encoder processes contiguous 2D slices with a shared ResNet18 and aggregates slice features through a single Transformer layer. The slice-based design supports anatomical cropping and comparison of Grad-CAM maps with detections.

4 Image Encoder

  • Each slice passes through an ImageNet-pretrained ResNet18, and a single Transformer layer aggregates projected slice features into a 128-dimensional subject embedding.Only the final ResNet residual stage is fine-tuned; mean pooling produces the 512-dimensional subject descriptor before projection.Source: §4 · p. 4
  • The raw configuration uses 90 axial slices, indices 62–152, selected to cover AD-relevant structures while excluding the orbits and neck.The selection uses slice statistics described in Section 5.1.Source: §4 · p. 4
  • The authors chose the 2D design despite a preliminary earlier-split comparison in which a 3D encoder was about 20 points ahead.Slicewise processing makes per-slice Grad-CAM and detection comparison, anatomical cropping, and fusion easier while keeping parameter count low.Source: §4 · p. 4

5 Anatomical Reference from FastSurfer Segmentations

FastSurfer segmentations provide anatomical references for selecting slice ranges and training YOLOv8 structure-localization models. Comparing Grad-CAM with detected structures shows that the image classifier attends both to relevant tissue and to regions outside the brain.

5.1 Structures and slice statistics

  • Slice ranges covering 90% of structure occurrences across 50 subjects guided encoder-window selection and decisions about which structures to annotate on each slice.The selected structures included bilateral medial temporal, temporal, parietal, frontal, ventricular, and cerebellar regions.Source: §5.1 · pp. 4–5

5.2 Detection and segmentation with YOLOv8

  • All YOLOv8 variants localized structures reliably, with mAP50 ≥0.95; stricter mAP50–95 increased with model size and had not saturated within the training budget.Models were trained for detection and instance segmentation using segmentation-derived labels on slices along the encoder’s input axis.Source: §5.2 · p. 5, Table 2 · p. 5

5.3 What does the image classifier look at?

  • Grad-CAM often overlapped detected diagnostically meaningful tissue, but on many other slices attention fell on the skull, neck, orbits, or outside the head.The analysis was qualitative; quantifying the fraction of Grad-CAM mass inside brain or MTL masks was left for future work.Source: §5.3 · p. 5

6 Leakage-Aware Multimodal Contrastive Learning

The multimodal framework contrasts image and tabular embeddings, then trains image-only and fusion heads under a two-phase schedule. It treats cognitive-score fusion as label-leakage-prone and regional-volume fusion as the imaging-relevant comparison.

6.1 Architecture

  • A TabTransformer-style branch encodes modality features, while a symmetric CLIP loss aligns image and tabular embeddings and two heads measure image-only and fused prediction.Each of four features is tokenized into a 32-dimensional embedding; the six-layer tabular Transformer uses eight heads, and the fusion head receives concatenated 128-dimensional embeddings.Source: §6.1 · pp. 5–6

6.2 Training

  • Training uses 8–10 epochs of contrastive-only pretraining followed by 20–25 epochs of joint fine-tuning with CE + 0.3 LCLIP.Joint optimization from the start left the CLIP loss near its chance value log N because cross-entropy gradients dominated.Source: §6.2 · p. 6

6.3 Why the leakage spectrum matters

  • Cognitive-score fusion is treated as a leakage-driven upper bound and implementation check, whereas volume fusion is the imaging-relevant result because cognitive scores help define ADNI labels.The supplied Table 3 and Table 4 captions describe three-way and binary test accuracy, respectively, and identify reference rows as using different protocols.Source: §6.3 · p. 6, Table 3 · p. 7, Table 4 · p. 7

7 Results

On three-way classification, cognitive-score fusion outperformed the image-only model, while volume fusion also improved on it. Binary results show a substantial difference between image heads aligned to cognitive scores versus regional volumes on MCI versus CN.

7 Results

  • Results were single runs on balanced subject-level test sets of 63 scans for three-way classification and 42 scans per binary task, with 95% Wilson intervals.On 63 scans, the interval spans roughly ±12 percentage points.Source: §7 · p. 7

7.1 Three-way classification

  • 87.3% three-way accuracy with cognitive-score fusion compared with 58.7% image-only accuracy; volume fusion reached 73.0%.Cognitive-score fusion had balanced per-class performance: 19/21 MCI, 17/21 CN, and 19/21 AD; it is not evidence of better image understanding.Source: §7.1 · p. 7

7.2 Binary tasks

  • On MCI vs. CN, both fusion variants improved over the image head, and the vol-aligned head substantially outperformed the cog-aligned head; the test set contained 42 scans.On AD vs. CN, the vol-aligned image head reached 85.7%, and volume fusion did not improve over that image head.Source: §7.2 · p. 7, Figure 3 · p. 8

8 The Contrastive Target Shapes the Image Encoder

With architecture and split fixed, volume alignment produced a stronger image-only MCI-versus-CN head than cognitive-score alignment, despite the tabular branch being absent at evaluation. The fused three-way ordering reversed, while the authors regard the image-only gap as suggestive given the single-run evidence.

8 The Contrastive Target Shapes the Image Encoder

  • 73.8% (31/42) vs. 52.4% (22/42) on MCI vs. CN: the vol-aligned image-only head exceeded the cog-aligned head by 21 points.The comparison holds architecture, training data, split, schedule, hyperparameters, and test set fixed; neither head receives tabular input at inference.Source: §8 · p. 8
  • 87.3% for cog vs. 73.0% for vol on three-way fusion reversed the image-only ordering; the cog configuration had the weaker visual representation.The authors argue that cognitive scores nearly determine the label, allowing the fusion head to rely on the tabular branch.Source: §8 · p. 8
  • Reporting the image-only head alongside fusion helps distinguish imaging contribution from gains attributable to access to the labelling rule.The authors also frame the tabular target as a training design choice that can shape the encoder, rather than only as extra inference input.Source: §8 · p. 8
  • The gap corresponds to nine test scans; a single-run two-proportion test gave p ≈0.04, so the authors call the effect suggestive and its size non-literal.They say confirmation requires paired per-scan predictions, several seeds, and ideally a target with intermediate leakage.Source: §8 · pp. 8–9

9 Anatomically Guided Input: MTL Crop

A per-subject medial temporal lobe crop, centered from FastSurfer labels and passed through the unchanged encoder, improved image-only three-way accuracy over raw slices. The authors report a larger benefit for MCI vs. CN than for AD vs. CN, while noting the small test-set difference.

9 Anatomically Guided Input: MTL Crop

  • The crop centers a fixed 96 × 48 × 64-voxel box on the centroid of eight FastSurfer MTL labels, then supplies 64 resized coronal slices to the unchanged encoder.The labels cover bilateral hippocampus, amygdala, entorhinal cortex, and parahippocampal cortex; the crop position follows each subject’s anatomy.Source: §9 · p. 9
  • 65.1% (41/63) vs. 58.7% (37/63) image-only three-way accuracy: the coronal MTL crop improved over raw slices.The 95% CI for the crop result was [52.8, 75.7], and the improvement corresponds to four scans.Source: §9 · p. 9
  • The MCI vs. CN task benefited more than AD vs. CN, consistent with the MTL carrying an early subtle signal while late-stage atrophy is visible in full slices.Axial and sagittal crop results are omitted, and combining the crop with volume alignment helped further but sub-additively.Source: §9 · p. 9

10 Discussion and Limitations

The discussion interprets cognitive-score fusion as entangled with the diagnostic labeling rule and reports that contrastive target choice may also affect image-only representations. It cautions that small, single-run evaluation and protocol differences constrain comparisons and calls for broader follow-up.

10 Discussion and Limitations

  • Protocol differences preclude ranking the numerically close cognitive-fusion results against Huang [3]; image-only baselines remain below 3D encoders, while the MTL crop narrows but does not close the gap.Differences include subject-level versus slice-level evaluation, cohort selection, fused modality groups, and single runs versus five splits.Source: §10 · p. 10
  • A leaky contrastive target may degrade the image representation; the cognitive-aligned image-only head is near chance on MCI vs. CN.The authors recommend reporting fusion results by modality group and alongside the image-only head.Source: §10 · p. 10
  • The test set is small and each configuration is a single run, so differences of a few points are within noise.The MTL crop requires FastSurfer segmentation at inference, with segmentation errors propagating to crop position; the data are from ADNI-1 at 1.5 T, Grad-CAM findings are qualitative, and a 12-hour job limit prevented longitudinal multi-phase training.Source: §10 · p. 10
  • Future work proposes repeated-seed experiments on larger longitudinal ADNI-1/2/3 data, testing the bio group, quantifying Grad-CAM overlap, and reducing dependence on external segmentation.Source: §10 · p. 10

11 Conclusion

The conclusion presents FastSurfer-derived anatomy as a means to audit and constrain an Alzheimer’s classifier, while emphasizing that cognitive-score fusion reflects label leakage and contrastive targets shape image-only representations. It also reports improved image-only accuracy from medial temporal lobe cropping, with confirmation needed in larger repeated experiments.

11 Conclusion

  • FastSurfer-derived anatomical references can both audit and constrain an Alzheimer’s classifier.Source: §11 · p. 10
  • Cognitive-score fusion gains largely reflect label leakage, and volume alignment yields a markedly better image-only MCI vs. CN representation than cognitive-score alignment.Source: §11 · p. 10
  • Restricting input to a medial temporal lobe crop improves image-only accuracy, but the findings need confirmation with repeated, larger-scale experiments.Source: §11 · p. 10

Related papers

Loading 2609.15888v1…