Source-linked AI summary
Detect and Correct Bias in Multi-Site Neuroimaging Datasets
Christian Wachinger, Anna Rieckmann, Sebastian Pölsterl
TL;DR
Pooling neuroimaging studies can increase sample sizes, but selection, measurement, and confounding biases may affect analyses. The paper tests dataset-specific information in MRI features and evaluates causal-inference and harmonization methods; it reports dataset identification above 70% and reductions in age-prediction error and confounding bias.
Problem
Pooling scans from independent neuroimaging studies can introduce selection, measurement, and confounding bias, while larger datasets are sought for machine learning and association studies.
Source: §1 · pp. 1–2Method
The paper combines 17 studies, tests dataset prediction from scan features, models confounders as latent variables, and evaluates harmonization methods.
Source: §1 · p. 2Results
Dataset identification exceeded 70% accuracy, while harmonization significantly reduced age-prediction errors and reduced confounding bias in evaluated models.
Source: §6.2 · p. 10, §6.3 · p. 10, §8 · p. 13Takeaways & Limitations
The authors conclude that novel methods for handling bias in datasets and image-derived features are needed, with causal-inference methods potentially promising for further insights.
Source: §8 · p. 13Causal inference relies on untestable assumptions, and the true causal effects and their sizes are unknown, making quantitative evaluation impossible.
Source: §7.3 · p. 13
Abstract
from arXiv · showhide
The desire to train complex machine learning algorithms and to increase the statistical power in association studies drives neuroimaging research to use ever-larger datasets. The most obvious way to increase sample size is by pooling scans from independent studies. However, simple pooling is often ill-advised as selection, measurement, and confounding biases may creep in and yield spurious correlations. In this work, we combine 35,320 magnetic resonance images of the brain from 17 studies to examine bias in neuroimaging. In the first experiment, Name That Dataset, we provide empirical evidence for the presence of bias by showing that scans can be correctly assigned to their respective dataset with 71.5% accuracy. Given such evidence, we take a closer look at confounding bias, which is often viewed as the main shortcoming in observational studies. In practice, we neither know all potential confounders nor do we have data on them. Hence, we model confounders as unknown, latent variables. Kolmogorov complexity is then used to decide whether the confounded or the causal model provides the simplest factorization of the graphical model. Finally, we present methods for dataset harmonization and study their ability to remove bias in imaging features. In particular, we propose an extension of the recently introduced ComBat algorithm to control for global variation across image features, inspired by adjusting for population stratification in genetics. Our results demonstrate that harmonization can reduce dataset-specific information in image features. Further, confounding bias can be reduced and even turned into a causal relationship. However, harmonziation also requires caution as it can easily remove relevant subject-specific information. Code is available at https://github.com/ai-med/Dataset-Bias.
1. Introduction
Pooling neuroimaging studies can introduce selection, measurement, and confounding bias. The paper tests dataset-specific information in scans and presents approaches to identify and reduce bias.
1. Introduction
- Scans could be assigned to their source datasets with substantially higher accuracy than basic demographics would predict, revealing dataset-specific information in image-derived measures.The introduction frames this as evidence that image features encode information beyond subject-specific neuroanatomy.Source: §1 · p. 1
- Neuroimaging bias can arise through non-random subject enrollment, scanner and acquisition differences, and variation in image-analysis procedures.These include selection bias and measurement bias; the introduction also describes confounding as a common cause of an observed association.Source: §1 · pp. 1–2
- Pooling studies designed around different research questions can introduce bias into machine-learning models trained on the combined data.The text notes that study inclusion criteria are tailored to their research questions.Source: §1 · p. 2
- The paper combines 17 studies to test dataset bias, model confounders as unobserved variables, and evaluate harmonization methods, including an extension addressing global feature variation.It also evaluates dataset prediction, brain-age prediction, and causal models on harmonized data.Source: §1 · pp. 2–3
2. Neuroimaging Data
The study assembled T1 brain MRI data from 17 public datasets and processed scans consistently before analysis. It retained 35,320 subjects after exclusions and downsampled the two largest datasets for dataset prediction.
2. Neuroimaging Data
- 35,320 subjects remained from 17 public T1 MRI datasets after processing-error and metadata exclusions.All datasets were processed with FreeSurfer 5.3, and one scan per subject was retained for longitudinal or test-retest datasets.Source: §2 · p. 3
- ABCD and UK Biobank were the largest datasets, so they were downsampled to sizes closer to the next-largest datasets for dataset prediction.The subset retained representation from all imaging sites.Source: §2 · p. 3
3. Name That Dataset
The Name That Dataset experiment tests whether MRI-derived measures identify a scan’s source dataset. Image features classified datasets far better than chance and better than age and sex alone, indicating substantial inter-dataset information.
3. Name That Dataset
- The experiment used a random-forest classifier on volume and thickness measures, with stratified training and testing splits across 17 datasets.It compared these features with age and sex and varied the training share from 0.1% to 70%.Source: §3 · p. 4
- The highest dataset-classification accuracy reported was 71.5% for the combination of volume and thickness measures, which performed better than either measure alone; age and sex reached 37.9%.Source: §3 · p. 4
- Image-feature accuracy exceeded age-and-sex accuracy, indicating that basic demographics did not explain all dataset-specific information.The authors point to measurement and confounding bias as possible additional sources.Source: §3 · p. 4
- Confusion was greater among datasets with similar populations, while the single-site HCP showed almost no confusion with other datasets.The multi-site CORR dataset and UK Biobank showed confusion with other datasets.Source: §3 · p. 4
- The observed dataset differences are consistent with prior reports of heterogeneity and cross-dataset prediction changes, including after scanner standardization efforts.The cited examples include dataset heterogeneity explaining over 10% of variance per structure in an ENIGMA study.Source: §3 · p. 5
4. Telling Causal from Confounded with Causal Inference
The paper distinguishes statistical dependence from causal structure and addresses confounding when relevant variables are unobserved. It compares causal and latent-confounder models using complexity-based criteria.
4.1. Confounding Bias in Causal Inference
- A feature–dataset association measures statistical dependence but cannot by itself distinguish a causal relationship from confounding.An open backdoor path from a shared cause can make the source of an association unclear.Source: §4 · p. 5, §4.1 · p. 5
4.2. Assumptions for Inferring Causality with Unknown Confounders
- The approach models an unobserved confounder because causal sufficiency—the knowledge of all potential confounders—is often unavailable in neuroimaging datasets.It uses probabilistic PCA to represent the hidden variable and compare causal with confounded models.Source: §4.1 · p. 5, §4.2 · pp. 5–6
4.3. Causal and Confounded Models
- Causal inference requires assumptions including faithfulness, and the method compares the causal and confounded factorizations of the joint distribution.The model parameters were estimated using MCMC and bridge sampling in the experiments.Source: §4.2 · p. 5, §4.3 · p. 6
4.2. Assumptions for Inferring Causality with Unknown Confounders
- The algorithmic Markov condition favors the causal direction when its factorization has lower Kolmogorov complexity, which the method approximates using minimum description length.The approach extends the comparison to a model with a latent confounder.Source: §4.2 · pp. 5–6
4.3. Causal and Confounded Models
- The causal model factors P(X, Y) as P(Y|X)P(X|Z)P(Z), while the confounded model uses P(Y|Z)P(X|Z)P(Z), with the confounder estimated through probabilistic PCA.The sign of Δ indicates which model better describes the data; a positive value favors the causal model, and a negative value favors the confounded model.Source: §4.3 · p. 6
5. Data Harmonization
The paper focuses on feature-stage harmonization to remove unwanted scanner- and site-associated variation while retaining subject biology, and extends harmonization to address unknown confounders.
5. Data Harmonization
- Feature-stage harmonization adjusts image-derived measures after acquisition, with imaging site or dataset as possible targets.The paper focuses on feature-stage adjustments because acquisition cannot be changed and FreeSurfer already provides an image-processing normalization pipeline.Source: §5 · pp. 6–7
5.1. Harmonization only based on imaging sites
- Site-wise Z-score normalization uses each site's feature mean and standard deviation, but can erase age-related differences; site-regression residuals provide another adjustment.The paper notes that normalization could give young-only and elderly-only sites the same mean, which is inadequate for structures such as the ventricles.Source: §5.1 · p. 7
5.2. ComBat Harmonization
- ComBat models additive and multiplicative site effects with empirical-Bayes parameter sharing, but assumes feature parameters are homogeneous; age and sex can be retained.The model includes covariates whose influence should remain after harmonization.Source: §5.2 · p. 7
5.3. Harmonization for non-biological variability with ComBat
- The ComBat extension adds variables r for non-biological variation to remove, including manufacturer, field strength, and principal components capturing global image-feature variation.The approach is motivated by using principal components to capture generic variation across features, and its causal-effect identification depends on the neurological process affecting only a few brain structures.Source: §5.3 · pp. 7–8, Table 2 · p. 8
6. Results
Harmonization reduced confounding in both causal-inference experiments, with PC inclusion producing positive causal-model evidence for NKI and OASIS. It also lowered age-prediction error across held-out datasets, while source-dataset information and biological signals remained a trade-off in dataset harmonization.
6.1. Name That Dataset with Harmonization
- Extending ComBat to remove additional variables—manufacturer, magnetic field strength, and principal component—lowered classification accuracy to 45% while strengthening age correlations to 0.62 for lateral ventricles and -0.38 for hippocampus.Source: §6.1 · p. 8, Figure 4 · p. 9
- Using dataset rather than imaging site as the grouping variable produced about 50% ComBat classification accuracy, versus better results from site-level harmonization.The authors interpret this as evidence of heterogeneity within the multi-site datasets and a need to model site-specific variation.Source: §6.1 · p. 9, Figure 5 · p. 10
6.2. Results on Brain Age Prediction
- Age-prediction errors were significantly lower after ComBat and ComBat with PC than with raw measurements; the two harmonization methods did not differ significantly.The raw-versus-ComBat and raw-versus-ComBatPC Wilcoxon tests gave p = 0.0064 and p = 0.0197; ComBat versus ComBatPC gave p = 0.5171.Source: Figure 6 · p. 10, §6.2 · p. 10
6.3. Results on Causality with Harmonization
- Adding ADAS to the variables kept during harmonization further reduced confounding bias, with non-overlapping Δ intervals.In the ADNI experiment, including PC also raised median Δ relative to site-only or site-plus-age-and-sex harmonization.Source: §6.3 · p. 10
- After adding PC, the causal-model complexity drop yielded positive Δ values for NKI and OASIS, indicating a causal relationship.The confounded-model complexity changed little; the causal-model change produced the positive Δ.Source: §6.3 · p. 10
7. Discussion
Across 17 datasets, scans were classified by dataset at over 70% accuracy, while age-and-sex classification stayed below 38%. Harmonization can reduce dataset information and confounding, but may also remove relevant biological information; causal conclusions remain constrained by assumptions.
7.1. Name That Dataset
- Over 70% accuracy for dataset prediction contrasted with below 38% using age and sex, indicating substantial dataset-specific information in image features.The discussion links this pattern to scanner-specific information reported in prior work.Source: §7.1 · pp. 10–11
7.2. Dataset Harmonization
- Z-score normalization and site-based harmonization sharply reduced dataset prediction, but also removed biologically relevant age information from hippocampus and ventricle volumes.Adding age and sex as variables to preserve improved age correlations, while dataset prediction increased compared with site-only harmonization.Source: §7.2 · p. 12
- The proposed harmonization extension adds variables whose effects should be removed, including principal components across image features to capture generic variation.Removing manufacturer and field-strength effects had little additional impact when imaging site was the grouping variable, but mattered more when dataset was the grouping variable.Source: §7.2 · p. 12
- In leave-one-dataset-out experiments, harmonization significantly reduced brain-age prediction error.The discussion notes that brain-age prediction has been proposed as a diagnostic biomarker for several diseases.Source: §7.2 · p. 12
7.3. Causal Inference for Identifying Confounding
- Most investigated relationships were judged confounded rather than causal, although harmonization sometimes reduced confounding or yielded causal relationships.Principal components in ComBat were preferred for both causal experiments; for ADAS, retaining the target reduced confounding but the relationship remained non-causal.Source: §7.3 · p. 12
- Causal inference relies on domain assumptions that may not transfer across applications, and observational data provide no known true effects for quantitative validation.The illustrated models assume linear relationships between continuous-valued variables; results therefore require cautious interpretation alongside prior knowledge.Source: §7.3 · pp. 12–13
8. Conclusion
Using data from more than 35,000 individuals, the paper demonstrates substantial bias when scans from distinct studies are pooled. It introduces an approach for unknown-confounder causal inference and a ComBat extension in which principal components capture generic variation.
8. Conclusion
- Pooling scans from distinct studies can introduce substantial bias that is passed to machine-learning models trained on the pooled data.The study examined data from more than 35,000 individuals.Source: §8 · p. 13
- The proposed causal-inference approach models confounders as unknown variables, addressing the difficulty of assuming causal sufficiency in complex neuroimaging studies.Source: §8 · p. 13
- The proposed ComBat extension adds variables to remove, with principal components capturing generic variation and producing the best harmonization results.Source: §8 · p. 13
- The authors argue that growing medical-image datasets require methods for handling bias in datasets and image-derived features.They identify causal-inference methods as potentially promising for gaining insights.Source: §8 · p. 13
Appendix A. Additional Results
Appendix A reports normalized ∆ from the causal inference model across all datasets and ten repetitions.
Appendix A. Additional Results
- Table A.4 reports median, minimum, and maximum normalized ∆ across 10 repetitions for the causal inference model across all datasets.Source: Table A.4 · p. 16