Source-linked AI summary
CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Matthew P. Lungren, Andrew Y. Ng
TL;DR
Chest radiograph interpretation lacks large datasets with strong reference standards and expert comparisons. CheXpert provides a large uncertainty-labeled dataset and evaluates models against radiologists, finding its best model outperformed at least two of three radiologists on four of five pathologies.
Problem
Chest radiograph interpretation lacks datasets that are large, have strong reference standards, and provide expert human performance metrics for comparison.
Method
CheXpert automatically extracts uncertain labels for 14 observations from radiology reports and evaluates uncertainty-handling approaches in convolutional neural networks.
Results
The best model outperformed at least 2 of 3 radiologists on 4 of 5 clinically relevant pathologies, with different uncertainty approaches useful for different observations.
Takeaways & Limitations
CheXpert provides a publicly available benchmark with uncertainty labels and radiologist-labeled reference-standard evaluation sets for developing and validating chest radiograph interpretation models.
Takeaways & Limitations
The model–radiologist comparison lacked patient history and prior examinations, and no statistical test assessed whether performance differences were significant.
Abstract
from arXiv · showhide
Large, labeled datasets have driven deep learning methods to achieve expert-level performance on a variety of medical imaging tasks. We present CheXpert, a large dataset that contains 224,316 chest radiographs of 65,240 patients. We design a labeler to automatically detect the presence of 14 observations in radiology reports, capturing uncertainties inherent in radiograph interpretation. We investigate different approaches to using the uncertainty labels for training convolutional neural networks that output the probability of these observations given the available frontal and lateral radiographs. On a validation set of 200 chest radiographic studies which were manually annotated by 3 board-certified radiologists, we find that different uncertainty approaches are useful for different pathologies. We then evaluate our best model on a test set composed of 500 chest radiographic studies annotated by a consensus of 5 board-certified radiologists, and compare the performance of our model to that of 3 additional radiologists in the detection of 5 selected pathologies. On Cardiomegaly, Edema, and Pleural Effusion, the model ROC and PR curves lie above all 3 radiologist operating points. We release the dataset to the public as a standard benchmark to evaluate performance of chest radiograph interpretation models. The dataset is freely available at https://stanfordmlgroup.github.io/competitions/chexpert .
Introduction
CheXpert introduces a large chest-radiograph dataset with uncertainty-aware report labels for predicting 14 observations from multi-view images. The study evaluates uncertainty-training approaches against radiologist annotations and compares the final model with radiologists.
- Motivation: Chest radiography is globally common and supports screening, diagnosis, and management, motivating automated interpretation for workflow prioritization, decision support, and large-scale screening.
- Dataset: 224,316 chest radiographs from 65,240 patients are labeled for 14 common observations using a report labeler that captures uncertainty.
- Task: The CheXpert task predicts probabilities for 14 observations from multi-view radiographs while investigating approaches for incorporating uncertainty labels during training.
- Validation: Different uncertainty approaches are useful for different observations, based on evaluation across 5 clinically significant and prevalent observations in 200 studies annotated by 3 radiologists.
- Radiologist comparison: On 4 out of 5 pathologies, the final model’s ROC and PR curves lie above at least 2 of 3 radiologist operating points on 500 studies with 5-radiologist consensus ground truth.The dataset is made publicly available to encourage further model development.
Dataset
CheXpert is a large public chest-radiograph dataset containing 224,316 radiographs from 65,240 patients, labeled for 14 observations as positive, negative, or uncertain. Labels are generated from radiology reports using a three-stage automated rule-based labeler that extracts, classifies, and aggregates mentions.
- Dataset: 224,316 chest radiographs from 65,240 patients comprise the CheXpert dataset, with labels for 14 observations.The observations include 12 pathologies, Support Devices, and No Finding.
- Dataset: The dataset was retrospectively collected at Stanford Hospital between October 2002 and July 2017 from inpatient and outpatient centers with associated radiology reports.A sample of 1000 reports was manually reviewed to assess observation extraction feasibility.
- Label generation: An automated rule-based labeler processes free-text radiology reports through mention extraction, mention classification, and mention aggregation.Mentions are extracted from the Impression section using manually curated phrases developed by multiple board-certified radiologists.
- Label generation: Mention classification assigns negative, uncertain, or positive labels, with uncertainty capturing diagnostic uncertainty and ambiguity in the report.Classification rules use universal dependency parses of report sentences.
- Label generation: Final observation labels are positive when any mention is positive, uncertain when no mention is positive but at least one is uncertain, and negative when a negative mention is present.This aggregation produces labels for the 14 observations.
Labeler Results
The CheXpert labeler was evaluated against board-certified radiologist annotations and the NIH labeler across mention extraction, negation detection, and uncertainty detection. It achieved higher F1 scores across all observations and tasks, with especially large gains in uncertainty detection, supported by expanded rules and separate pre- versus post-negation uncertainty classification.
- Evaluation set: 1000 radiology reports from 1000 distinct patients formed the evaluation set, with two board-certified radiologists labeling observations as present, absent, uncertain, or not mentioned.The patients did not overlap with those used to develop the labeler.
- Evaluation tasks: The labeler was compared with the NIH labeler on mention extraction, negation detection, and uncertainty detection.Mention extraction treated any assigned label as positive and blank as negative.
- Results: Across all observations and tasks, the CheXpert labeling algorithm achieved a higher F1 score than the NIH labeler.It significantly outperformed the NIH labeler on negation detection for Atelectasis and Cardiomegaly, with notably better performance on Consolidation and Pneumonia.
- Results: On uncertainty detection, the CheXpert labeler showed large gains over the NIH labeler, particularly for Cardiomegaly, Pneumonia, and Pneumothorax.These gains were reported across the evaluated observations.
- Method differences: The method used expanded rules for negation and uncertainty variation and split uncertainty classification into pre-negation and post-negation.The authors also avoided automatic mention extractors such as MetaMap and DNorm because they produced weak extractions on the report collection.
Model
The models predict probabilities for 14 observations from single-view chest radiographs, taking the maximum probability across views when multiple views are available. The study compares approaches for training with negative, positive, and uncertain labels, including ignoring, replacing, or self-training on uncertainty labels.
- Model inputs and outputs: Models output probabilities for 14 observations from a single-view chest radiograph, using the maximum observation probability across views when multiple views are available.
- Uncertainty-label strategies: Training labels are 0, 1, or u, and the study investigates different strategies for incorporating uncertainty labels during training.
- Uncertainty-label strategies: U-Ignore masks uncertain observations from the binary cross-entropy loss, providing a baseline for approaches that explicitly incorporate uncertainty.Ignoring uncertainty is analogous to listwise deletion and can produce biased models when cases are not missing completely at random.
- Uncertainty-label strategies: U-Zeroes maps every uncertain label to 0, whereas U-Ones maps every uncertain label to 1.These approaches can distort classifier decision-making and degrade performance if uncertainty conveys semantically useful information.
- Uncertainty-label strategies: U-SelfTrained first trains U-Ignore to convergence, then relabels uncertain observations with the model’s probability predictions while leaving 0 and 1 labels unchanged.The relabeled examples use the mean binary cross-entropy loss across observations.
3-Class Classification
The U-MultiClass model treats uncertainty as a third class for each of 14 observations, outputting probabilities for negative, positive, and uncertain labels. All uncertainty approaches use the same training process, with DenseNet121 selected as the best-performing architecture.
- 3-Class Classification: U-MultiClass treats the uncertainty label as its own class for each of the 14 observations.The model outputs probabilities for the three classes {p0, p1, pu}.
- 3-Class Classification: DenseNet121 produced the best results among the evaluated convolutional neural network architectures and was used for all experiments.The compared architectures were ResNet152, DenseNet121, Inception-v4, and SE-ResNeXt101.
Validation Results
Validation on 200 studies annotated by three board-certified radiologists showed that uncertainty-labeling strategies affected performance differently across the five competition tasks. U-Ones was strongest for Atelectasis, while U-MultiClass improved Cardiomegaly detection over ignoring uncertainty.
- Validation setup: The validation set comprised 200 studies from 200 patients, each independently annotated by three board-certified radiologists and binarized for evaluation.Annotations distinguished present, uncertain likely, uncertain unlikely, and absent observations; present and uncertain likely cases were treated as positive.
- Evaluation: Performance was evaluated using AUC on five competition tasks: Atelectasis, Cardiomegaly, Consolidation, Edema, and Pleural Effusion.The analysis reported 95% two-sided confidence intervals and used Benjamini-Hochberg correction for multiple hypothesis testing.
- Uncertainty approaches: U-Ones significantly outperformed U-Zeros on Atelectasis, achieving AUC=0.858 versus AUC=0.811 (p = 0.03).The authors attribute U-Ones’ performance to uncertain phrases such as “possible atelectasis” describing the most likely image finding.
- Interpretation: The study concluded that ignoring uncertainty labels was not effective, whereas the best uncertainty treatment depended on the pathology.The reported differences indicate pathology-specific usefulness of uncertainty handling rather than one universally superior approach.
Test Results
On a 500-study test set evaluated against a majority-vote ground truth from five radiologists, the model achieved strong discrimination across observations and exceeded all three individual radiologists on Cardiomegaly, Edema, and Pleural Effusion. The comparison lacked patient history and statistical significance testing.
- Test set: 500 studies from 500 patients were evaluated using the majority vote of 5 radiologist annotations as ground truth.Eight board-certified radiologists independently annotated the studies; three also annotated the validation set.
- Model performance: 0.97 AUC was achieved on Pleural Effusion, while Atelectasis had the lowest AUC at 0.85; all other observations reached at least 0.9 AUC.The model’s best AUPRC was 0.91 on Pleural Effusion, and its worst was 0.44 on Consolidation.
- Radiologist comparison: On Cardiomegaly, Edema, and Pleural Effusion, model performance exceeded all 3 individual radiologists but not their majority vote.Radiologist operating points were compared with model ROC and Precision-Recall curves.
- Limitations: Neither radiologists nor the model had access to patient history or previous examinations, and no statistical test assessed performance differences.Both limitations may affect interpretation of the model–radiologist comparison.
Existing Chest Radiograph Datasets
Chest radiograph interpretation research has been constrained by the scarcity of publicly available datasets with strong radiologist-annotated ground truth and expert comparison scores. OpenI and ChestXray14 enabled progress, but existing public datasets lacked test sets with strong ground truth or radiologist performances.
- A major obstacle is the lack of datasets with strong radiologist-annotated ground truth and expert scores for model comparison.
- OpenI provides 7,470 frontal-view radiographs and radiology reports labeled with key findings by human annotators.The dataset is hosted by the Indiana Network for Patient Care.
- Publicly available chest radiographic imaging datasets had no test sets with strong ground truth or radiologist performances.
- ChestXray14 became the most commonly used benchmark and enabled substantial progress in automated chest radiograph interpretation models.
Conclusion
CheXpert is a large chest-radiograph dataset with uncertainty labels and radiologist-labeled reference-standard evaluation sets. The study evaluates uncertainty-handling approaches and finds that its best model outperforms at least two of three radiologists on four clinically relevant pathologies.
- CheXpert provides a large chest-radiograph dataset with uncertainty labels and radiologist-labeled reference-standard evaluation sets.
- The study investigates different approaches to handling uncertainty and validates them on the evaluation sets.
- At least 2 of the 3 radiologists were outperformed by the best model in detecting 4 clinically relevant pathologies on a test set with strong ground truth.