Source-linked AI summary
Exploring large scale public medical image datasets
Luke Oakden-Rayner
TL;DR
Large public medical image datasets are essential for radiology AI, but their labels and documentation may not reliably reflect image content. This paper visually examines two such datasets and finds substantial labeling and documentation problems, motivating expert review and clearer dataset reporting.
Problem
Large, well-characterised datasets are essential for radiology AI, but privacy, cost, restricted access, and proprietary incentives limit their availability.
Method
The paper explores two public datasets through expert visual review and examines label accuracy, label definitions, and clinically relevant subsets.
Results
The datasets had variable quality: ChestXray14 labels often had low positive predictive value and suffered disambiguation and stratification failures, while MURA labels under-reported degenerative joint disease.
Takeaways & Limitations
Public dataset releases should include expert visual label review, thorough documentation of development and labeling rules, clinically relevant subset analyses, and separate visually accurate test sets.
Takeaways & Limitations
The main limitation identified is inadequate dataset documentation, compounded for CXR14 by reports that can incompletely describe image findings.
Abstract
from arXiv · showhide
Rationale and Objectives: Medical artificial intelligence systems are dependent on well characterised large scale datasets. Recently released public datasets have been of great interest to the field, but pose specific challenges due to the disconnect they cause between data generation and data usage, potentially limiting the utility of these datasets. Materials and Methods: We visually explore two large public datasets, to determine how accurate the provided labels are and whether other subtle problems exist. The ChestXray14 dataset contains 112,120 frontal chest films, and the MURA dataset contains 40,561 upper limb radiographs. A subset of around 700 images from both datasets was reviewed by a board-certified radiologist, and the quality of the original labels was determined. Results: The ChestXray14 labels did not accurately reflect the visual content of the images, with positive predictive values mostly between 10% and 30% lower than the values presented in the original documentation. There were other significant problems, with examples of hidden stratification and label disambiguation failure. The MURA labels were more accurate, but the original normal/abnormal labels were inaccurate for the subset of cases with degenerative joint disease, with a sensitivity of 60% and a specificity of 82%. Conclusion: Visual inspection of images is a necessary component of understanding large image datasets. We recommend that teams producing public datasets should perform this important quality control procedure and include a thorough description of their findings, along with an explanation of the data generating procedures and labelling rules, in the documentation for their datasets.
1 Introduction
Large public medical image datasets expand access for AI research but create risks when dataset development and usage are disconnected. Understanding label accuracy, hidden visual subgroups, and label meanings requires medical expertise.
- Large, well-characterised datasets are essential for modern medical AI but remain difficult to produce and access because of cost, privacy, and proprietary interests.
- Public releases such as ChestXray14 and MURA contain over 112,000 chest radiographs and over 40,000 upper limb radiographs, respectively.
- These datasets commonly provide labels intended to support training of clinically useful AI systems.
- Disconnects between data gathering, cleaning, labelling, and later use can obscure design decisions and dataset nuances from end-users.
- Users may overestimate label accuracy, miss hidden stratification, or misunderstand clinical label meanings, especially when relying on test data generated through the same procedures.
- The study uses expert medical review to assess two large public datasets and demonstrate why direct evaluation of labels and label schemas is necessary.
2.1 Datasets
ChestXray14 and MURA are large single-centre radiograph datasets with different image types, label-generation procedures, and repetition patterns. Their labels and source data require careful interpretation.
- 2.1.1 CXR14: 112,120 frontal chest films from 30,805 patients comprise ChestXray14, drawn from the NIH Clinical Center across ICU and non-ICU settings.
- 2.1.1 CXR14: ChestXray14 labels were generated by natural language processing that matched pathology keywords and identified report negations.
- 2.1.1 CXR14: The dataset expanded from 8 to 14 pathology classes, while the normal class accounts for around 75% of images, or roughly 84,000 studies.
- 2.1.1 CXR14: 13,302 ChestXray14 patients had multiple studies, together accounting for 84% of the data.
- 2.1.2 MURA: 40,561 MURA images from 14,863 studies were obtained at Stanford Hospital across seven upper-limb study types.
- 2.1.2 MURA: MURA labels were assigned during clinical interpretation by board-certified radiologists as either normal or abnormal, without pathology-specific labels.
- 2.1.2 MURA: In a review of 100 abnormal MURA cases, fractures, implanted hardware, degenerative joint disease, and other abnormalities were identified.
2.2 Visual inspection
The study visually inspected randomly selected images and relabelled them with dataset-specific procedures. CXR14 required enriched class sampling because its labels were rare and clinically ambiguous, whereas MURA allowed prevalence-preserving review.
- A board-certified radiologist visually inspected randomly selected cases to understand image-label relationships and identify problems in label schemas.
- 2.2.1 CXR14: CXR14 could not be relabelled at original prevalence because many classes were rare and clinically ambiguous, including pneumonia, consolidation, and infiltration.
- 2.2.1 CXR14: 700 CXR14 cases were reviewed as 50 randomly selected cases from each of 14 classes, assessing only whether each assigned label was visually present.
- 2.2.1 CXR14: CXR14 relabelling used permissive rules that counted a finding when it plausibly matched the label and was visually apparent.
- 2.2.2 MURA: 714 randomly selected MURA cases were relabelled at the dataset’s original prevalence because abnormal and normal cases occurred at approximately a 45:55 ratio.
- 2.2.2 MURA: MURA pathology-specific labels covered fractures, implanted hardware, degenerative joint disease, bone tumours, and miscellaneous pathologies without access to original normal/abnormal labels.
- 2.2.2 MURA: MURA pathology labels followed a less permissive rule intended to reflect the radiologist’s best clinical accuracy.
2.3 Analysis
Label quality was assessed against expert visual review using dataset-appropriate metrics and subgroup analyses. CXR14 used PPV because comprehensive relabelling was not feasible, while MURA permitted sensitivity and specificity estimates.
- 2.3.1 CXR14: CXR14 label quality was measured with positive predictive value using expert visual labels as the gold standard.
- 2.3.1 CXR14: CXR14 negative predictive value was not determined because the dataset could not be efficiently relabelled wholesale.
- 2.3.2 MURA: MURA labels were evaluated against expert visual ground truth using sensitivity and specificity because comprehensive relabelling was possible.
- Both datasets underwent subgroup analysis, including body region and visually identified pathology groups in MURA and exploratory visual subgroups in CXR14.
3 Results
Visual review identified systematic label problems in CXR14 and more specific performance weaknesses in MURA, including hidden stratification, label disambiguation failure, and poor detection of degenerative joint disease.
- 3.1 CXR14: CXR14 visual-assessment PPV was below Wang et al.’s estimated PPV in all 15 class groups.Fifty cases from each class group were assessed using permissive labelling rules.
- 3.1 CXR14: 90% overall PPV for pneumothorax fell to 60% among cases without chest drains.Chest drains were present in 80% of positive pneumothorax cases, often without other visible pneumothorax features.
- 3.1 CXR14: 86% of emphysema cases had subcutaneous rather than pulmonary emphysema, producing very low PPV for the labels.The finding was attributed to failure to disambiguate keywords during the original labelling process.
- 3.2 MURA: MURA specificity values were inflated by the low prevalence of per-class conditions relative to normal studies.Sensitivity and specificity were reported overall and by region, with subgroup performance summarized in Table 4.
- 3.2 MURA: MURA labels showed poor sensitivity for degenerative joint disease at 60%, compared with 92% for fractures and 85% for hardware.Lower pathology-identification sensitivity also occurred in the wrist and forearm, regions commonly affected by joint disease.
- 3.2 MURA: MURA visual review found unexpected shoulder false positives and several missed diagnoses, including subtle fractures and two lytic bone lesions.Most shoulder false positives showed no identifiable pathology.
4 Discussion
The two datasets had important weaknesses linked to their labelling methods and documentation, although MURA labels were generally more accurate than CXR14 labels. The authors argue that expert visual review and clearer documentation are needed to understand dataset usefulness and prevent models from learning flawed label proxies.
- ChestXray14: CXR14 labels had low positive predictive value and suffered from report-based labelling, label disambiguation failure, and clinically important hidden stratification.Natural-language processing extracted labels from incomplete clinical reports; many pneumothorax cases showed chest tubes rather than visible pleural gas, while “emphysema” often indicated subcutaneous emphysema.
- MURA: MURA labels were more accurate overall, but degenerative joint disease was under-reported because “normal” and “abnormal” were not operationally defined.The paper notes that some radiologists may have interpreted “normal” as normal for age, while patient ages were unavailable to test this explanation.
- Documentation: The main limitation of these datasets was inadequate documentation of label-generation procedures, label meanings, and known weaknesses.The supporting documents did not adequately discuss the issues identified by visual review and could create misleading impressions of label accuracy or coverage.
- Quality control: Centralised expert inspection of a small subset is presented as a feasible quality-control step for dataset builders.The authors argue that dataset creators are best positioned to perform this review because they understand the data-generating process and existing labelling decisions.
- Evaluation: A visually accurate test set can reduce the risk that models silently reproduce flawed labels and appear successful against an equally flawed test set.MURA included a 207-case test set labelled by three radiologists, whereas CXR14 lacked a publicly provided manually labelled test set; the test-set labelling rules were not explicitly stated in either case.
- Limitations: Interpretation of the review was limited by imperfect expert labels, inter-observer variability, and reduced image quality that obscured some findings.The authors specifically describe difficulty identifying retrocardiac pathology in CXR14 because dense regions appeared purely white.
5 Conclusion
Public-dataset label quality and clinical relevance can be significantly compromised when dataset development is disconnected from data usage. Expert visual review and thorough documentation of dataset development, strengths, and weaknesses can mitigate these problems.
- Dataset development–usage disconnect can significantly impact the accuracy, meaning, and clinical relevance of public-dataset labels.
- Expert visual review should assess label accuracy and identify clinically relevant subsets within each label class.
- Dataset releases should explain the development process, strengths, weaknesses, and labelling findings in detail.