Source-linked AI summary
On the limits of cross-domain generalization in automated X-ray prediction
Joseph Paul Cohen, Mohammad Hashir, Rupert Brooks, Hadrien Bertrand
TL;DR
The paper asks which chest X-ray prediction tasks generalize across datasets despite differing institutions and labeling practices. It evaluates cross-domain performance, prediction agreement, and model representations, finding task-dependent generalization and evidence that concept or label shift is more important than covariate shift alone.
Problem
The study addresses limited evidence about which chest X-ray prediction tasks generalize reliably across datasets with differing labels and annotators.
Method
The authors analyze models trained on multiple public datasets using cross-domain performance, prediction agreement, and representation comparisons.
Results
Generalization varies by task, with strong performance for Cardiomegaly, Edema, and Effusion and highly variable results elsewhere; agreement and performance can also disagree.
Takeaways & Limitations
The findings support focusing on concept shift and treating predictions as defined by their training data rather than assuming a universal medical concept.
Takeaways & Limitations
The analysis uses only dataset-associated labels and AP/PA views, excluding patient outcomes and lateral views that may contain needed findings.
Abstract
from arXiv · showhide
This large scale study focuses on quantifying what X-rays diagnostic prediction tasks generalize well across multiple different datasets. We present evidence that the issue of generalization is not due to a shift in the images but instead a shift in the labels. We study the cross-domain performance, agreement between models, and model representations. We find interesting discrepancies between performance and agreement where models which both achieve good performance disagree in their predictions as well as models which agree yet achieve poor performance. We also test for concept similarity by regularizing a network to group tasks across multiple datasets together and observe variation across the tasks. All code is made available online and data is publicly available: https://github.com/mlmed/torchxrayvision
1. Introduction
The paper investigates why chest X-ray prediction models generalize inconsistently across institutions and labeling systems. It focuses on whether differences arise from image distributions or label concepts, motivated by documented annotation and interobserver variability.
- Motivation: Cross-dataset generalization is studied because prior work established the problem but not its precise cause, with possible sources including artifacts, sampling, superficial image statistics, and label differences.The paper frames the investigation around models trained and tested across institutions with different clinicians or labeling tools.
- Motivation: Annotation disagreement can reflect errors, differing interpretations of radiology reports, clinical-practice bias, and interobserver variability.These factors may arise from differences in medical culture, language, textbooks, politics, or conceptual understanding.
- Motivation: Relabeling studies found imperfect agreement, including 10% and 6% rater disagreement for Airspace opacity and Nodule/mass, respectively, and an F1 score as low as 10% for NIH Pneumonia labels.Kaggle and Google relabelings agreed better for overlapping Opacity labels, but their F1 score was still only 73%.
- Motivation: MIMIC-CXR’s automatic labeling methods differed by up to 0.6% for Fracture or 2.6% when uncertain and missing labels were included, while radiologist agreement reached a lowest F1 of 0.462 for Enlarged Cardiomediastinum.The latter discrepancy may reflect uncertainty about the clinically relevant cardio-thoracic ratio.
- Approach: The study analyzes over 200k unique chest X-rays from four large public datasets and compares performance, prediction agreement, and learned representations across domains.The paper also makes its code and data publicly available.
2. Data
The study assembles multiple public chest X-ray datasets and harmonizes their labels for cross-domain analysis. Eighteen common labels were identified manually, and a canonical loading framework was released.
- Data: The dataset collection includes NIH, PadChest, CheXpert, MIMIC-CXR, OpenI, Google, and Kaggle chest X-ray datasets.The paper provides full dataset details in its appendix.
- Data: Eighteen common labels were identified by manually reviewing the label descriptions supplied by each dataset.The exact mapping is documented in the released code.
- Data: The authors release a framework for loading the datasets in a canonical way.
3. Models
The models use DenseNet-based training with label-imbalance weighting, calibration, and augmentation. Calibration is performed separately on each evaluated test domain to remove miscalibration as an explanation for generalization error.
- Models: An ensemble of three DenseNet models is trained per dataset, with losses computed only for available labels in newer datasets and results averaged to reduce noise.DenseNet was selected based on prior evidence that it performs well for X-ray prediction.
- Models: Task weights are balanced using positive-label frequencies so overrepresented tasks receive less dominance in the loss.The weighting formula raises the minimum task weight and normalizes task weights between 0 and 1.
- Models: A piecewise linear transformation calibrates outputs so each disease’s optimal operating point maps to 50%.The operating point maximizes true positive rate minus false positive rate, removing miscalibration as a considered source of generalization error.
- Models: Each model is calibrated separately on the test dataset being evaluated because the optimal decision boundary differs across datasets.The calibrated outputs of the three ensemble members are then averaged.
- Models: Training augments images through rotations up to 45 degrees, translations up to 15%, and scaling up or down by up to 10%.
4. Performance
Cross-domain performance varies substantially by task: Cardiomegaly, Edema, and Effusion generalize best, while Infiltration, Pneumonia, and Fracture show pronounced domain-specific failures. Training on all datasets performs well partly because it includes same-domain training data.
- Performance: Cardiomegaly, Edema, and Effusion show the best generalization, while Atelectasis, Consolidation, Emphysema, Hernia, and Lung Opacity generalize reasonably.
- Performance: Infiltration generalizes worst, with performance inverted between the PadChest and NIH datasets.
- Performance: NIH Pneumonia performance is poor for the NIH model and for other models evaluated on NIH, despite better performance on other datasets.
- Performance: For Fracture, models perform much better on their own test sets than on the hand-labeled NIH_Google dataset, while Pneumothorax also performs better in-domain but remains strong on NIH_Google.
- Performance: All-dataset training outperforms almost all other models except for Pneumonia on NIH, but this advantage partly reflects inclusion of same-domain training data rather than better generalization.A leave-the-test-domain-out evaluation is used to check this interpretation.
5. Agreement
The study measures agreement among models trained on different datasets using Cohen’s Kappa, revealing that cross-dataset prediction agreement varies substantially by task. Agreement and performance can diverge: models may disagree while achieving high AUC, or agree while achieving low AUC.
- Agreement metric: Kappa ranges from 0 for chance agreement to 1 for total agreement, with 40% considered moderate and 70% excellent.The metric is used to quantify agreement between networks trained on different datasets.
- Cross-dataset agreement: Agreement is poor for labels that are not common across datasets.The study calculates pairwise Cohen’s Kappa between models trained on different datasets for the same task.
- Agreement and performance: Models can disagree yet achieve high AUC, while other tasks show strong agreement but low AUC.These discrepancies indicate that predictive performance and inter-model agreement capture different aspects of cross-domain behavior.
6. Representation
The representation analysis compares dataset-task weight vectors and tests whether L2 regularization makes vectors for the same task more similar. Some tasks remain separated despite alignment pressure, indicating dataset-dependent task representations.
- Representation setup: Each dataset-task output has an independent weight vector applied to 1024-dimensional image features, making vector distance a proxy for prediction similarity.The logistic output layer means similar vectors produce similar predictions.
- Representation structure: PCA shows that task vectors are generally very different, although Cardiomegaly and Effusion exhibit some similarity.The plotted vectors represent dataset-task combinations, and PCA preserves distances because it is linear.
- Representation alignment: Some tasks, including Mass, Nodule, Fibrosis, Lung Lesion, and Pleural Thickening, remain separated despite L2 regularization encouraging same-task weight vectors to align.The regularizer jointly optimizes prediction and vector alignment.
- Representation divergence: Greater variation between task vectors provides more evidence that identical feature vectors require different predictions across datasets.The study interprets these differences as dataset-driven divergence in task representations and examines them through averaged, normalized vector distances.
7. Discussion
The discussion argues that cross-domain failures are more consistent with concept shift than image covariate shift, while prediction agreement and performance can diverge. It cautions that model outputs inherit training-data biases and that local fine-tuning may adapt to local biases rather than clinical reality.
- Discussion: The authors argue that concept shift deserves more attention than covariate shift because task performance varies substantially across datasets and agreement does not reliably track performance.They frame the discrepancy as raising questions for further study rather than as a resolved mechanism.
- Discussion: Automatic relabeling alone may not solve the problem because disagreement can reflect differing schools of thought, radiologist variation, and subjective clinical relevance.The limitation concerns label concepts and interpretation, not only labeling machinery.
- Discussion: Models trained on particular datasets carry those datasets’ biases, so predictions should be interpreted with their training context and origin.The discussion gives “NIH Pneumonia” as an example of a context-defined task and suggests showing multiple model outputs with provenance.
- Discussion: Local fine-tuning is not presented as a solution because it may adapt models to local data biases that do not match reality in the images.The authors also note that strong performance from a model trained on all datasets does not establish true generalization.
8. Limitations
The study uses only dataset-associated labels and AP/PA views, excludes patient outcomes, and omits lateral views that may contain necessary finding features.
- The analysis excludes patient outcomes, so it does not establish disagreement risk for specific tasks.
- Only AP/PA views are used, while lateral views are omitted despite potentially containing needed features.
Appendix A. Data details
The appendix documents the datasets, label counts, label-agreement analyses, and visual diagnostics used to characterize cross-dataset X-ray prediction.
- The study includes NIH, PadChest, CheXpert, MIMIC-CXR, OpenI, Google, and Kaggle datasets with differing label sources and annotation procedures.
- MIMIC-CXR provides labels from both the CheXpert and NIH NegBio automatic labellers, denoted MIMIC_CH and MIMIC_NB.
- Table A.1 reports dataset sample counts as positive/negative labels, noting that some datasets omit labels while others encode missing labels as negative.
- Figure A.1 compares labels for matched NIH images that were relabelled in the Kaggle and Google datasets using confusion matrices.
- Appendix figures examine model disagreement for Infiltration, Hernia, and Fracture, including calibrated probabilities and dataset- or labeller-specific predictions.