PaperlayerView exact revision on arXiv

Source-linked AI summary

Global-Local Transformer for Brain Age Estimation

Sheng He, P. Ellen Grant, Yangming Ou

arXiv:2109.01663v1cs.CVeess.IV

TL;DR

Whole-image brain age estimation can miss fine-grained details and makes decisions difficult to understand. The global-local transformer combines global context with local patch information through attention, achieving the best reported performance among evaluated models while highlighting informative brain regions.

  • Problem

    Whole-image CNNs capture global information but can miss fine-grained details, while their decisions are difficult to understand.

  • Method

    A two-pathway network extracts global context and local fine-grained information, then fuses them with an attention mechanism.

  • Results

    The proposed method provides the best performance for brain age estimation across MAE, correlation, and cumulative scores against the evaluated baseline models.

  • Takeaways & Limitations

    The method highlights subject-level and group-level salient brain regions and supports comparisons of brain aging across regions.

  • Takeaways & Limitations

    The study evaluates healthy-cohort MRIs and does not include patients’ MRIs.

Abstract

from arXiv · show

Deep learning can provide rapid brain age estimation based on brain magnetic resonance imaging (MRI). However, most studies use one neural network to extract the global information from the whole input image, ignoring the local fine-grained details. In this paper, we propose a global-local transformer, which consists of a global-pathway to extract the global-context information from the whole input image and a local-pathway to extract the local fine-grained details from local patches. The fine-grained information from the local patches are fused with the global-context information by the attention mechanism, inspired by the transformer, to estimate the brain age. We evaluate the proposed method on 8 public datasets with 8,379 healthy brain MRIs with the age range of 0-97 years. 6 datasets are used for cross-validation and 2 datasets are used for evaluating the generality. Comparing with other state-of-the-art methods, the proposed global-local transformer reduces the mean absolute error of the estimated ages to 2.70 years and increases the correlation coefficient of the estimated age and the chronological age to 0.9853. In addition, our proposed method provides regional information of which local patches are most informative for brain age estimation. Our source code is available on: \url{https://github.com/shengfly/global-local-transformer}.

I. INTRODUCTION

Brain age estimation is useful for assessing brain health, but whole-image models can miss fine-grained age information and patch-only models have limited receptive fields. The proposed global-local transformer combines global context with local details through attention.

  • Motivation: Brain age gap is the difference between predicted and chronological age and is an informative biomarker of brain health.Positive brain age gap is associated with cognitive decline and several neurodegenerative and psychiatric conditions.
  • Motivation: Whole-image CNNs capture global information but can miss fine-grained details and are difficult to interpret.Deep networks may be dominated by salient whole-image regions, while subtle age-related regions are ignored.
  • Proposed approach: The proposed two-pathway network extracts global context from the whole MRI and fine-grained information from local patches.The pathways interact in global-local transformer blocks to fuse both feature types for brain age estimation.
  • Motivation: Patch-based methods capture local evidence but are constrained by the local patch’s receptive field.Local pathways force learning within small regions, which can limit performance.
  • Proposed approach: Attention fuses weighted global-context information with local features without requiring spatial feature alignment.The weights are computed from similarities between global and local deep features.

C. Our method can be interpreted.

The paper positions global-local transformer as an interpretable 2D-slice method that combines patch-level evidence with global context. It reports state-of-the-art brain age estimation performance and regional evidence visualization.

  • Interpretation: Direct interpretation evaluates prediction accuracy on local patches to identify the most informative image regions.The approach supports patch-level evidence that can be visualized for each subject.
  • Interpretation: The method retains patch-level interpretability while incorporating global-context information through attention.Its local-patch performance is reported as higher than BagNet because corresponding global context is also learned.
  • Method and comparison: The study compares convolutional neural network approaches for brain age estimation on healthy brain MRIs.Reported prior results include MAE values of 4.16 years, 3.631 years, 2.428 years, and 3.00 years for cited methods.
  • Method and comparison: The proposed method uses a two-pathway network and applies it to 2D slices extracted from 3D brain MRI volumes.The 2D-slice approach is described as computationally efficient and achieves an MAE of 2.70 years over ages 0-97.
  • Transformer distinction: Unlike self-attention, the global-local transformer derives query features from the local pathway and key-value features from the global pathway.This design applies attention between global and local information to compute contextual features for local patches.

III. METHOD

The method extracts deep features with a CNN backbone from whole images and local patches, then uses asymmetric global-local attention to transfer global context into local features. The fused representations support brain age estimation.

  • A. Backbone for deep feature extraction: The CNN backbone contains eight convolutional blocks, each combining convolution, batch normalization, and ReLU operations.Max-pooling reduces spatial resolution after blocks, and the backbone converts a brain image into a deep feature.
  • B. Global-local attention mechanism: The global pathway processes the whole image, whereas the local pathway processes smaller patches with shared channel dimension d.The resulting global feature contains whole-image context and the local feature contains fine-grained patch information.
  • B. Global-local attention mechanism: The local feature is projected into a query, while global features are projected into key and value representations.These projections enable attention-based fusion between local details and global context.
  • B. Global-local attention mechanism: At each local-feature location, attention compares the query with all global-feature positions and normalizes the similarities with softmax.The normalized weights determine the contribution of global values to the contextual local feature.
  • B. Global-local attention mechanism: Global-local attention is asymmetric because the query comes from the local pathway while key and value come from the global pathway.The feature count satisfies N1 ≪ N2, and the stated attention complexity is O(N1N2).
  • B. Global-local attention mechanism: The attention output matches the local-query size and is computed as a weighted sum of global values without spatial alignment.Multi-head attention splits features into h = 8 channel-wise parts before concatenation and projection.

C. Global-local Transformer

The global-local transformer repeatedly integrates global-context and local fine-grained information, then maps branch features to brain age.

  • C. Global-local Transformer: The model concatenates global-local attention output with local features to preserve distinct global-context information.The fused features pass through convolutional transformations, batch normalization, and ReLU activation.
  • C. Global-local Transformer: The global-context representation is an attention-weighted sum of global features, with weights determined by global-local feature similarity.
  • C. Global-local Transformer: The global-local transformer structure is repeated N times to iteratively integrate global-context and local detailed information.
  • C. Global-local Transformer: Each branch ends with average pooling and a fully connected layer that maps a 512-dimensional feature vector to brain age.

IV. EXPERIMENTS

The experiments evaluate the proposed method on a large healthy cohort and compare it with baseline models and state-of-the-art neural-network architectures.

  • IV. EXPERIMENTS: The proposed method is evaluated on a large healthy cohort.
  • IV. EXPERIMENTS: The experiments compare the proposed method with baseline models.
  • IV. EXPERIMENTS: The experiments also compare it with state-of-the-art neural-network architectures.

A. Dataset

The study uses healthy T1-weighted MRI data from eight public datasets, processed into standardized 2D slices and evaluated through cross-validation and generality testing.

  • A. Dataset: 8,379 healthy T1-weighted MRI scans from eight public datasets cover ages 0-97 years.
  • A. Dataset: Six datasets are randomly partitioned into five parts for 5-fold cross-validation, while CMI and CoRR assess model generality across scanners.
  • A. Dataset: Preprocessing applies N4 bias correction, field-of-view normalization, MASS skull stripping, affine registration, and cropping to 130×170×120 voxels.
  • A. Dataset: The method extracts central axial, coronal, and sagittal 2D slices because 2D networks require fewer parameters than 3D networks.
  • A. Dataset: Training uses mean absolute error as the loss, with the final loss summing global-pathway and local-pathway losses.

C. Performance evaluation of age estimation

Performance is assessed with MAE, Pearson correlation, and cumulative score, using comparisons against conventional, transformer-based, and brain-age-specific networks.

  • C. Performance evaluation of age estimation: Three metrics evaluate age estimation: mean absolute error, correlation coefficient, and cumulative score.The correlation coefficient is Pearson correlation between predicted and chronological ages.
  • C. Performance evaluation of age estimation: A higher cumulative score indicates better performance, based on the proportion of samples whose absolute error is within threshold α.
  • C. Performance evaluation of age estimation: Baselines include whole-image ResNet18, local-patch BagNet-ResNet18, and transformer variants using global or local patch features.
  • C. Performance evaluation of age estimation: Additional comparisons use ResNet, wide residual, DenseNet, SFCN, DeepBrainNet, and FiA-Net architectures.
  • C. Performance evaluation of age estimation: The final estimated age averages predictions from all local patches obtained with a sliding window whose step equals half the patch size.

H. Interpretation with multiple patch sizes

The method estimates brain age from local patches of multiple sizes, summarizing their predictions with a mean and standard deviation. Parameter evaluation selects settings balancing performance with computational cost.

  • Multiple patch-size interpretation: Multiple patch sizes are cropped from each input image and processed by the same local pathway for brain-age estimation.Training samples 30 patches across sizes and locations; testing samples 3,000 patches per subject.
  • Multiple patch-size interpretation: The distribution of patch-based age estimates is summarized by the mean m and standard deviation σ.The standard deviation measures variation among estimates from different local patches.
  • Parameter selection: Patch sizes above 48 produce no significant performance differences, so the experiments use a local-pathway patch size of 64.This setting is selected for the subsequent evaluations.
  • Parameter selection: Performance is not significantly different when using 5 to 20 slices, while approximately N=6-10 global-local transformer blocks yields lower MAE.The study uses 5 slices and N=6 blocks as a tradeoff between performance, time, and memory.
  • Parameter selection: The eight-layer VGG backbone provides the best results among the evaluated backbones, and combining global and local pathways improves performance.The local pathway captures detailed patch information while the global pathway supplies global context.

B. Comparison with different baseline models

The proposed Global-Local Transformer is compared with whole-image and local-patch baselines across three imaging planes and their fusion. It provides the strongest reported performance across age groups and datasets, while axial slices are the most informative individual plane.

  • Evaluation setup: The evaluation compares baseline models on axial, coronal, and sagittal slices, plus predictions fused by averaging the three plane-specific ages.The fused prediction is y = (y_axial + y_coronal + y_sagittal)/3.
  • Baseline comparison: Local-patch-only BagNet methods perform worse than whole-image ResNet18 and VGG models, showing that local patches alone have limited performance.Local-Transformer improves over some local-patch baselines but remains below ResNet18 with the VGG backbone.
  • Baseline comparison: The proposed Global-Local Transformer gives the best performance, supporting fusion of global-context and local detailed information.Its attention-based design combines information from both pathways.
  • Plane comparison: Axial slices are more informative than coronal and sagittal slices, while fusing all three planes improves performance for the evaluated whole-image and transformer models.Subsequent results therefore report the three-plane fusion.
  • Age-range comparison: The proposed method provides the best performance across the four evaluated age groups, although ages 30-60 years have higher MAE for all models.This age range is reported as more challenging than the other groups.
  • Dataset comparison: The proposed method achieves the lowest MAE on all six cross-validation datasets and on the CMI and CoRR generality datasets.The authors report generalizability across datasets from different sites and scanners.

C. Comparison with state-of-the-art neural networks and models of brain age estimation

The paper compares its model with general-purpose and brain-age-specific neural networks, then examines regional evidence and uncertainty in patch-based predictions. The proposed method outperforms the compared state-of-the-art models and yields interpretable regional patterns.

  • State-of-the-art comparison: The comparison covers eight state-of-the-art deep networks and two recently published brain-age estimation models using MAE, correlation r, and CS(α=5 years).All models use five-fold validation and fused predictions from three planes.
  • State-of-the-art comparison: The proposed method outperforms the compared general-purpose networks and the brain-age-specific SFCN, DeepBrainNet, and FiA-Net models.The comparison includes both 2D and 3D SFCN variants.
  • State-of-the-art comparison: Efficient networks have MAE greater than 3.5 years, correlation r below 0.98, and CS(α) below 80% in the reported comparison.DenseNet performs better than the other listed neural networks, while SFCN 3D performs better than SFCN 2D.
  • Interpretation: Subject-level heatmaps identify patches with the lowest MAE, while group-level heatmaps show the probability of the best-prediction regions across age ranges.The visualizations provide regional information about local patch informativeness.
  • Interpretation: The most informative patches generally cover the same region within a subject, while salient regions vary slightly across subjects of different ages.This indicates age-related variation in the regions highlighted by the patch-based interpretation.
  • Uncertainty: Patch-estimate uncertainty is lowest around ages 20 and 65, highest around age 40, and may reflect fewer training samples near age 40.Here σ measures differences among brain-age estimates from different brain regions.
  • Regional analysis: Predicted brain-age error has no significant correlation with normalized regional brain volumes in the cross-validation dataset, with r < 0.1.The analysis uses n=5,441 subjects and examines auto-segmented regions based on the SRI atlas.

VI. DISCUSSION AND CONCLUSION

The global-local transformer fuses global-context and local detailed information with attention for brain age estimation, while also providing interpretable regional evidence. Across six public datasets, it achieved the best reported performance against baseline and state-of-the-art models, but the study identifies important data and cohort limitations.

  • Method: The proposed global-local transformer optimally fuses global-context and local detailed information with an attention mechanism.
  • Results: Across six public datasets containing 5,441 healthy subjects aged 0–97 years, the method achieved the best performance against six baselines across MAE, correlation, and cumulative scores.
  • Results: Comparisons with eight state-of-the-art neural networks and two brain-age-specific networks showed that fusing global-context and local detailed information improved brain age estimation on 2D slices.
  • Interpretation: The method provides subject-level and group-level salient regions, enabling visualization of estimated evidence and regional differences in brain aging.
  • Limitations: The study excludes patient MRIs, uses a dataset that is not the largest, and has an age imbalance with the largest MAE in the 30–60-year group reaching 5.12 years.
  • Future work: The conclusion identifies larger and more balanced datasets, gender-information fusion, and application to patient MRIs as future directions.

APPENDIX

The appendix applies the model to brain MRIs containing tumors and examines whether its informative regions and errors distinguish tumor from non-tumor tissue. The reported visualizations indicate that age predictions primarily rely on non-tumor regions, while errors are higher in tumor regions.

  • Motivation: The appendix evaluates chronological-age estimation on pathology-bearing MRIs, including brain-tumor images, where biological age is unavailable and radiologist assessment is subjective and time-consuming.
  • Data: The BraTS experiment includes 382 subjects aged 17.4–86.6 years with four concatenated MRI modalities used as multichannel input.
  • Evaluation: Five-fold cross-validation on BraTS compares the proposed method with baseline and state-of-the-art networks using fusion results from three imaging planes.
  • Method: The attention mechanism can identify tumor regions by comparing healthy and tumor regions, reducing the effect attributed to tumors.
  • Interpretation: The most informative regions do not overlap tumor regions, indicating that predicted age mainly comes from non-tumor brain regions.
  • Results: 7.09±5.91 years was the MAE for non-tumor regions versus 8.45±7.72 years for tumor regions, with p < 0.0001 in a two-sided t-test.
Loading 2109.01663v1…