Source-linked AI summary
Machine Learning with Multi-Site Imaging Data: An Empirical Study on the Impact of Scanner Effects
Ben Glocker, Robert Robinson, Daniel C. Castro, Qi Dou, Ender Konukoglu
TL;DR
Pooling multi-site neuroimaging data can be problematic, and this study examines scanner effects in machine learning using matched MRI data from two studies. Despite preprocessing, classifiers distinguish data origin with high accuracy, while sex-classification performance depends on data arrangement and cross-site testing. The authors conclude that multi-site harmonization remains an open challenge.
Problem
Pooling data across sites is important for large-population analyses, but differences in acquisition, demographics, disease characteristics, and other factors make naive combination problematic.
Source: §1 · p. 1Method
The authors construct a matched Cam-CAN and UK Biobank T1-weighted MRI cohort and use two classification experiments to examine scanner effects after preprocessing.
Source: §2 · p. 1, §2.1 · p. 2, §3 · p. 3Results
After preprocessing, classifiers predict data origin with high accuracy, and sex-classification results show optimistic estimates with imbalance and poorer generalization across sites.
Source: §3 · p. 3Takeaways & Limitations
Scanner effects can be subtle yet significantly affect machine learning, and multi-site data harmonization remains an open challenge.
Source: §1 · p. 1, §3 · p. 3The cohort is a somewhat best-case setting: its acquisition protocols are remarkably similar and participants should be normal, while such age- and sex-matching is rarely possible in practice.
Source: §2 · pp. 1–2
Abstract
from arXiv · showhide
This is an empirical study to investigate the impact of scanner effects when using machine learning on multi-site neuroimaging data. We utilize structural T1-weighted brain MRI obtained from two different studies, Cam-CAN and UK Biobank. For the purpose of our investigation, we construct a dataset consisting of brain scans from 592 age- and sex-matched individuals, 296 subjects from each original study. Our results demonstrate that even after careful pre-processing with state-of-the-art neuroimaging pipelines a classifier can easily distinguish between the origin of the data with very high accuracy. Our analysis on the example application of sex classification suggests that current approaches to harmonize data are unable to remove scanner-specific bias leading to overly optimistic performance estimates and poor generalization. We conclude that multi-site data harmonization remains an open challenge and particular care needs to be taken when using such data with advanced machine learning methods for predictive modelling.
1 Motivation
Pooling data from multiple sites and studies supports large-population analyses with sufficient statistical power, but naive combination can be problematic. This study illustrates the issue for machine learning and reiterates that multi-center harmonization remains an open challenge, not a solution it addresses.
1 Motivation
- Pooling data across sites and studies supports large-population analyses with sufficient statistical power.The authors cite Smith and Nichols (2018).Source: §1 · p. 1
- Naively combining datasets can be problematic because acquisition, demographics, disease characteristics, and other factors differ.Source: §1 · p. 1
- The study illustrates this issue for machine learning and reiterates that multi-center data harmonization remains an open research challenge.The authors explicitly state that they are not proposing a solution.Source: §1 · p. 1
2 Data
The study constructs a matched Cam-CAN and UK Biobank MRI cohort and applies a common preprocessing pipeline. The data are an unusually favorable case for multi-site analysis, while visual examples do not make site differences obvious.
2 Data
- The age- and sex-matched T1-weighted brain MRI dataset contains 592 individuals, with 296 drawn from each of the Cam-CAN and UK Biobank imaging studies.The authors describe this as simulating a somewhat ‘best case scenario’ for multi-site data, with matching intended to remove population bias.Source: §2 · p. 1
- Example histograms, axial slices, and SPM12 graymatter maps show site-specific differences that are non-obvious by visual inspection.Source: Figure 1 · p. 2
- The studies’ acquisition protocols are remarkably similar and participants should be normal, while such matching is rarely feasible in practice.The authors expect pooled analyses to suffer from larger site-specific biases than this dataset.Source: §2 · pp. 1–2
2.1 Pre-Processing Pipeline
- Preprocessing reorients scans, skull-strips them, corrects bias fields, and registers them to MNI space before intensity normalization and tissue-map generation.The pipeline uses whitening within brain regions; SPM12 and FSL-FAST produce tissue probability maps through different preprocessing routes.Source: §2.1 · p. 2
3 Experiments, Results & Conclusion
The experiments test whether preprocessing removes site information and how sex-classification performance changes across data arrangements. Site remains classifiable after preprocessing, and sex-classification results vary with matching, imbalance, and cross-site testing.
3 Experiments, Results & Conclusion
- Classifiers predict data origin with high accuracy after preprocessing, with scanner effects also present in tissue maps and amplified by higher spatial normalization.The authors note that amplification may be related to interpolation.Source: §3 · p. 3
- Age- and sex-matched multi-site data gives realistic sex-classification accuracy, while sex imbalance yields overly optimistic accuracy and cross-site testing shows poorer generalization.The performance drop is more severe when affine registration removes discriminative features such as brain size.Source: §3 · p. 3
- Scanner effects can be subtle yet significantly affect machine learning.Source: §3 · p. 3