Source-linked AI summary
Reading Race: AI Recognises Patient's Racial Identity In Medical Images
Imon Banerjee, Ananth Reddy Bhimireddy, John L. Burns, Leo Anthony Celi, Li-Ching Chen, Ramon Correa, Natalie Dullerud, Marzyeh Ghassemi, Shih-Cheng Huang, Po-Chih Kuo, Matthew P Lungren, Lyle Palmer, Brandon J Price, Saptarshi Purkayastha, Ayis Pyrros, Luke Oakden-Rayner, Chima Okechukwu, Laleh Seyyed-Kalantari, Hari Trivedi, Ryan Wang, Zachary Zaiman, Haoran Zhang, Judy W Gichoya
TL;DR
The paper asks how medical-imaging AI can produce racial disparities when human experts cannot identify race from images. Using standard deep-learning models across datasets, modalities, clinical tasks, and validation settings, it finds that self-reported race is highly predictable from medical-image pixels alone, creating a risk that race-based errors could evade human oversight.
Problem
The study addresses how AI systems can produce racial disparities despite no known race-related visual correlation apparent to human medical-image experts.
Method
The authors train and evaluate deep-learning models for race detection across large public and private datasets, imaging modalities, body locations, clinical tasks, and external validation settings.
Results
Self-reported race is trivially predictable from medical-image pixels alone, with high performance across modalities, datasets, tasks, and training objectives.
Takeaways & Limitations
Race-related information may be present in many medical-image models, while human oversight is limited because experts cannot similarly identify racial identity from the images.
Takeaways & Limitations
The study operationalizes racial identity using patients' self-reported race, a social, political, and legal construct that is distinct from genetic ancestry.
Abstract
from arXiv · showhide
Background: In medical imaging, prior studies have demonstrated disparate AI performance by race, yet there is no known correlation for race on medical imaging that would be obvious to the human expert interpreting the images. Methods: Using private and public datasets we evaluate: A) performance quantification of deep learning models to detect race from medical images, including the ability of these models to generalize to external environments and across multiple imaging modalities, B) assessment of possible confounding anatomic and phenotype population features, such as disease distribution and body habitus as predictors of race, and C) investigation into the underlying mechanism by which AI models can recognize race. Findings: Standard deep learning models can be trained to predict race from medical images with high performance across multiple imaging modalities. Our findings hold under external validation conditions, as well as when models are optimized to perform clinically motivated tasks. We demonstrate this detection is not due to trivial proxies or imaging-related surrogate covariates for race, such as underlying disease distribution. Finally, we show that performance persists over all anatomical regions and frequency spectrum of the images suggesting that mitigation efforts will be challenging and demand further study. Interpretation: We emphasize that model ability to predict self-reported race is itself not the issue of importance. However, our findings that AI can trivially predict self-reported race -- even from corrupted, cropped, and noised medical images -- in a setting where clinical experts cannot, creates an enormous risk for all model deployments in medical imaging: if an AI model secretly used its knowledge of self-reported race to misclassify all Black patients, radiologists would not be able to tell using the same data the model has access to.
Author Affiliations and addresses
The authors are affiliated with institutions in the United States and Taiwan, with support from multiple governmental and research funding agencies.
- Authors represent Emory University, Indiana University Purdue University, MIT, Beth Israel Deaconess Medical Center, and National Tsing Hua University.
- Several authors received support from the National Institutes of Health, including NIBIB, NLM, and related awards.
- Additional funding came from the National Science Foundation and Taiwan’s Ministry of Science and Technology.
Introduction
The paper examines how AI can predict self-reported race from medical images despite the absence of an obvious visual correlation for human experts. Across modalities, datasets, tasks, and perturbations, the authors investigate performance, confounding factors, and possible mechanisms.
- Definitions: The study defines racial identity as a social, political, and legal construct measured through patients’ self-reported race, rather than genetic ancestry.
- Motivation: Prior work established racial disparities in AI performance, but the mechanism of race-related differences in medical imaging remained unexplored.
- Implications: The paper argues that human radiologist oversight may not reveal race-based model behavior because experts cannot identify racial identity from the same medical images.
- Study design: The authors train deep learning models to detect race across chest x-rays, other imaging modalities, multiple datasets, external environments, and clinically motivated tasks.
- Mechanism analysis: Experiments assess whether body habitus, tissue density, disease distribution, anatomical regions, or image-frequency features act as proxies for race.
- Results: MXR / CXP models achieved AUC values of 0.96/0.94 in the reported main-text comparison.
- Conclusion: Race detection remained highly learnable across modalities, datasets, tasks, and training objectives, while learned features appeared to involve all image regions and the frequency spectrum.
B. RACE DETECTION IS NOT DUE TO OBVIOUS ANATOMIC AND PHENOTYPE CONFOUNDER VARIABLES
Across body habitus, tissue density, diagnostic labels, bone density, age, and sex, obvious phenotype and anatomic variables explain substantially less race-prediction performance than full medical images.
- B1. Race detection using body habitus: BMI-only models performed far below chest-x-ray image models for race prediction.The GLM AUC was 0.55 with BMI data and 0.52 with the binary BMI-data label.
- B1. Race detection using body habitus: Race-prediction performance remained similar across body-habitus strata and patient race groups.AUCs were [0.89, 0.98] in MXR and [0.92, 0.99] across race groups in EMX.
- B2. Tissue density analysis on mammograms: Tissue density and age together achieved AUC = 0.61, below the image-model AUC = 0.91 on mammograms.Tissue density alone achieved AUC = 0.54, suggesting these variables do not account for most image-model performance.
- B3. Race detection using diagnostic labels: Diagnostic labels alone produced much lower AUCs than chest-x-ray images, while “no finding” labels retained similar performance to the full dataset.Diagnostic-label AUCs were 0.54–0.61 on MXR and 0.52–0.57 on CXP; “no finding” AUCs were 0.93–0.94 versus 0.91–0.95 overall.
- B4. Race detection using bone density: Bone-density removal preserved race prediction, including Black AUCs of 0.96 on MXR and 0.94 on CXP.This indicates race information is not localized within the brightest image pixels, such as bone.
- B5. Race detection using age and sex: Race-recognition performance did not differ significantly across age groups or between male and female patients.The age-stratified results were from MXR, with no significant age-group difference or sex-based AUC difference.
C. RACE INFORMATION PERSISTS IN ALL SPECTRAL RANGES AND IN THE PRESENCE OF HIGHLY DEGRADED IMAGES
Race-prediction performance persists across frequency ranges, severe image degradation, resolutions, anatomical segments, and image patches, indicating that race information is broadly distributed rather than confined to an obvious visual region.
- C1. Race information is present across the frequency domain: High performance persisted under low-pass and high-pass filtering, including high-pass images lacking visually discernible anatomy.Low-pass performance degraded around diameter 10, whereas high-pass performance remained high up to diameter 100.
- C2. Race information persists in degraded image resolution and low image quality: Race information remained above chance at resolutions as small as 4x4, with AUC > 0.95 at 160x160 resolution or larger.Noisy images achieved AUC 0.74–0.80 and blurred images achieved AUC 0.64–0.72.
- C3. Race information is not localised to a specific anatomical region or body segment: No single anatomical segment clearly contributed most to race prediction across the tested experiments.Non-lung segmentations performed better than lung segmentations, but both were below predictions from the original images.
- C3. Race information is not localised to a specific anatomical region or body segment: CT slice-wise performance was similar across the top, middle, and lower chest, while heatmap occlusion preserved AUC > 0.86.These findings support distributed race information across lung and non-lung regions rather than a single localized segment.
- C4. There is no clear contribution to race prediction from specific patches: Removing any particular patch did not substantially reduce race prediction, and most individual ninth-image patches nearly matched full-image performance.Some lower-row patches contained little race information, but race information was not localized to one part of the 3x3 grid.
Discussion
AI models can predict self-reported race from medical images across clinical environments and modalities, despite no identifiable human-perceptible anatomical or imaging proxy. Race-related information persists after substantial image perturbations, making selective debiasing difficult.
- Race-recognition performance generalised across clinical environments and medical imaging modalities, suggesting models were not relying on local hospital processes.The datasets included settings where patients were imaged in the same locations and with the same processes independent of race.
- Disease distribution and body habitus were not strongly predictive of racial group, so neither feature alone explained the models’ recognition performance.The authors also considered scanner, equipment, and acquisition factors, but could not identify specific image-based covariates explaining the results.
- Features relevant to racial-identity recognition appeared throughout the image frequency spectrum, with high-pass models retaining performance after recognizable structures disappeared to human observers.Low-pass models maintained high performance until obvious blurring, while high-pass models remained effective beyond recognizable anatomy.
- Reasonable resolution reduction, noise injection, frequency filtering, and patch masking did not eliminate race detection.The ability to detect race was reduced only by extreme image degradation, often beyond the point of human diagnosis.
- Models trained for pathology detection and patient re-identification also learned features predictive of self-reported race, creating a risk of unintended racial cues in clinical decisions.The authors state that this risk may be underappreciated and that regulatory processes for identifying or mitigating unexpected racial recognition remain weak.
- The analysis used self-reported race as ground truth and focused on White, Black, and Asian patients because other categories lacked adequate or consistent labels.The authors frame self-reported race as a strong proxy for racial identity while noting race is a social rather than biological construct.
Conclusion
The study concludes that medical AI can readily learn racial identity from x-ray and CT images, and that this capability is difficult to isolate or mitigate. It therefore calls for caution and explicit auditing of model performance across demographic groups.
- Medical AI systems can easily learn racial identity from medical images, and this capability is extremely difficult to isolate or mitigate.
- In x-ray and CT imaging, racial identity is readily learnable from image data alone and generalises to new settings.
- Future medical-imaging AI work should emphasize explicit performance audits by racial identity, sex, and age, with self-reported race included in datasets where possible.
Supplemental material
The supplemental material documents common model architectures, training parameters, standardized evaluation procedures, and open-source code for the experiments.
- The study used ResNet34, DenseNet121, and EfficientNetB0 architectures for chest-x-ray race classification, with architecture choice based on dataset size and task complexity.
- Images were resized to 224–320 pixels and trained with augmentation, Adam optimization, categorical cross-entropy, and a starting learning rate of 1e-3.
- Detection performance was characterized using one-vs-all ROC-AUC, with sensitivity and specificity reported at the default 0.5 operating point.
- All experiment code was released under an open-source license through the AI-Vengers GitHub repository.
Chest CT image preprocessing
The supplemental analyses describe image preprocessing and experiments testing whether race information reflects disease, anatomy, frequency content, localization, or image degradation. Across these tests, race information remained detectable in altered or restricted images.
- Chest CT image preprocessing: Chest CT images were standardized using a rescale intercept of 1024 and pixel-value normalization by division by 3000.Images with abnormal air-patch pixel values outside [-30, 30] were dropped.
- Frequency-domain imaging features: Frequency filtering used Fourier-domain low-pass, high-pass, bandpass, and notch filters before inverse transformation and model training.
- Frequency-domain imaging features: Race information remained present in bandpass- and notch-filtered images, including examples barely perceptible to human observers.
- Image degradation: Noisy and blurred images still contained race information, with AUCs greater than 0.59 despite reduced performance.
- Anatomical localisation: Masking salient regions reduced race-detection AUCs but left performance above random chance, while lung-only segmentation produced the lowest AUCs.The original, non-segmented images had the highest AUCs, suggesting information was distributed across image portions.
- Anatomical localisation: Saliency-map review by radiologists did not reveal an identifiable pattern explaining race prediction across breast-density classes.