Source-linked AI summary
Can we trust deep learning models diagnosis? The impact of domain shift in chest radiograph classification
Eduardo H. P. Pooch, Pedro L. Ballester, Rodrigo C. Barros
TL;DR
This study examines whether chest-radiograph models generalize across datasets despite domain shift from differing acquisition settings and populations. Training identical convolutional networks on four datasets and testing them across datasets shows substantial out-of-domain performance drops, while CheXpert- and MIMIC-CXR-trained models generalize better.
Problem
Deep learning models may not generalize reliably beyond their source domain because medical-imaging datasets differ in acquisition protocols, machines, and label distributions.
Method
The study trains the same multi-label convolutional neural network separately on four chest-radiograph datasets and evaluates each model on its own and the other datasets.
Results
Models show substantial performance drops on unseen datasets; on CheXpert, mean AUC decreases by 0.04 for MIMIC-CXR-trained models and 0.12 for ChestX-ray14-trained models.
Takeaways & Limitations
CheXpert- and MIMIC-CXR-trained models generalize better to other datasets, supporting external and case-by-case validation before clinical use.
Takeaways & Limitations
The study argues that validation should account for center-specific data and changing label distributions because available datasets may not capture the heterogeneity required for generalizable models.
Abstract
from arXiv · showhide
While deep learning models become more widespread, their ability to handle unseen data and generalize for any scenario is yet to be challenged. In medical imaging, there is a high heterogeneity of distributions among images based on the equipment that generates them and their parametrization. This heterogeneity triggers a common issue in machine learning called domain shift, which represents the difference between the training data distribution and the distribution of where a model is employed. A high domain shift tends to implicate in a poor generalization performance from the models. In this work, we evaluate the extent of domain shift on four of the largest datasets of chest radiographs. We show how training and testing with different datasets (e.g., training in ChestX-ray14 and testing in CheXpert) drastically affects model performance, posing a big question over the reliability of deep learning models trained on public datasets. We also show that models trained on CheXpert and MIMIC-CXR generalize better to other datasets.
1 Introduction
Chest radiograph deep learning models face a domain-shift problem because acquisition settings and populations alter data distributions between training and deployment. The paper evaluates cross-dataset generalization to assess whether models remain reliable outside their source domain.
- Radiography is widely used for chest diagnosis, but observer error accounts for 90% of lung cancer misdiagnoses in radiographs.
- Deep learning models have achieved near radiologist-level accuracy for identifying radiological findings in chest radiographs.
- Domain shift occurs when training and target data distributions differ because of acquisition settings, machines, protocols, or population label distributions.
- As source and target distributions diverge, trained models tend to perform poorly on target data.
- The study trains a multi-label convolutional neural network separately on four large chest-radiograph datasets and tests each model on the other three.
2 Related work
Prior work shows that domain shift can impair medical-image models across institutions and modalities. Related approaches attempt to reduce this problem through domain adaptation, semi-supervised learning, or synthetic image generation.
- Cross-institution training and testing affected brain-tumor segmentation, and using multiple institutions for training did not necessarily remove the limitation.
- Unsupervised and semi-supervised domain-adaptation methods use unlabeled target data or adversarial objectives to mitigate domain shift.
- A semi-supervised GAN-based method improved cardiac-abnormality detection when testing on a different domain.
- Other approaches learn domain-specific and shared representations for cross-modality segmentation or generate translated and synthetic images for augmentation.
3 Materials and methods
The study compares four chest-radiograph datasets and evaluates domain shift by training identical multi-label models on each dataset, then testing them across datasets. Dataset distributions, label schemes, image formats, and split procedures are explicitly examined.
- 3.1 Datasets: The four datasets differ in size, institutions, patient populations, image views, and label vocabularies, with CheXpert and MIMIC-CXR sharing 13 findings.
- 3.1 Datasets: Pixel-intensity distributions differ across centers, while CheXpert and MIMIC-CXR show similar distributions; PadChest also contains a common top-left artifact.
- 3.2 Experiment design: The experiments reproduce CheXNet with an ImageNet-pretrained DenseNet121, training one model per dataset and evaluating each on its own and the other three test sets.
- 3.2 Experiment design: Because label sets differ, results are reported on eight common findings after merging compatible labels across datasets.
- 3.2 Experiment design: AUC evaluates individual findings and their average, incorporating true- and false-positive rates; higher AUC indicates better performance.
4 Results
Across four chest-radiograph datasets, models performed best on their own test distributions and often lost performance on other datasets, with CheXpert and MIMIC-CXR showing greater cross-dataset stability. These results also highlight label-reliability and dataset-representativeness concerns.
- Cross-dataset performance: Models achieved their best average result when training and testing used the same dataset, indicating substantial cross-dataset performance variation.The experiments used one model architecture and shared hyperparameters across datasets.
- Cross-dataset performance: 0.12 mean AUC reduction occurred when a ChestX-ray14-trained model was evaluated on CheXpert, compared with 0.04 for a MIMIC-CXR-trained model.On MIMIC-CXR, the ChestX-ray14-trained model also showed a 0.08 reduction, while the MIMIC-CXR-trained model’s decrease was 0.04.
- Cross-dataset performance: CheXpert and MIMIC-CXR models showed smaller AUC variations across test sets, whereas ChestX-ray14 and PadChest models dropped more outside their source datasets.Figure 4 represents each trained model’s performance across all four test sets as lines.
- Cross-dataset performance: The best AUC occurred on the matching training dataset for 7 of 8 findings in ChestX-ray14 and all 8 findings in both CheXpert and MIMIC-CXR.Across all four test sets, the best average AUC came from the corresponding training dataset.
- Dataset limitations: ChestX-ray14 label accuracy was estimated to be 10–30% lower than originally reported, raising concerns about label reliability.The paper notes that the ChestX-ray14 labeler may not accurately reflect image content.
5 Discussion and Conclusion
The study concludes that models may fail to generalize across chest-radiograph datasets with differing distributions, despite strong source-domain performance. It therefore emphasizes representative training data and external, center-specific validation.
- Conclusion: Models failed to generalize reliably to unseen datasets when those datasets followed somewhat different distributions.The paper reports a large performance drop outside the source dataset, including for a model with reported radiologist-level performance.
- Dataset representativeness: CheXpert and MIMIC-CXR appeared most representative of the other analyzed datasets because their models showed smaller performance drops relative to baseline.ChestX-ray14 appeared least representative, while PadChest’s drop may relate to fewer examples per finding.
- Validation: The authors recommend greater emphasis on external validation, ideally validating models on new data from each center where they will be used.They also describe fine-tuning on small datasets from specific machines after pre-training on larger datasets as one alternative.