Source-linked AI summary
CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, Matthew P. Lungren, Andrew Y. Ng
TL;DR
Interpreting pneumonia on chest X-rays is challenging and depends on expert radiologists. CheXNet uses a deep neural network to detect pneumonia and all 14 ChestX-ray14 diseases, exceeding average radiologist performance and prior state-of-the-art results.
Problem
Pneumonia detection in chest X-rays is challenging and depends on expert radiologists, whose diagnoses can vary.
Method
CheXNet is a 121-layer convolutional neural network trained on ChestX-ray14 to detect pneumonia, then extended to recognize all 14 diseases.
Results
CheXNet achieves an F1 score of 0.435 versus 0.387 for radiologists and state-of-the-art results on all 14 pathology classes.
Takeaways & Limitations
Expert-level automated chest X-ray disease detection could improve healthcare delivery and access to medical imaging expertise where radiologists are limited.
Takeaways & Limitations
The comparison used only frontal radiographs and excluded patient history, although lateral views and clinical context can affect diagnosis.
Abstract
from arXiv · showhide
We develop an algorithm that can detect pneumonia from chest X-rays at a level exceeding practicing radiologists. Our algorithm, CheXNet, is a 121-layer convolutional neural network trained on ChestX-ray14, currently the largest publicly available chest X-ray dataset, containing over 100,000 frontal-view X-ray images with 14 diseases. Four practicing academic radiologists annotate a test set, on which we compare the performance of CheXNet to that of radiologists. We find that CheXNet exceeds average radiologist performance on the F1 metric. We extend CheXNet to detect all 14 diseases in ChestX-ray14 and achieve state of the art results on all 14 diseases.
1. Introduction
CheXNet is a 121-layer convolutional neural network trained on ChestX-ray14 to detect pneumonia from chest X-rays and localize indicative image regions. It exceeds average radiologist performance on pneumonia detection and achieves state-of-the-art results across all 14 diseases in ChestX-ray14.
- Model and data: 121-layer CheXNet takes a chest X-ray image as input and outputs the probability of a pathology.The model also localizes areas most indicative of the detected pathology.
- Model and data: 112,120 frontal-view chest X-ray images in ChestX-ray14 are labeled with up to 14 thoracic diseases, including pneumonia, to train CheXNet.CheXNet outputs pneumonia probability and a heatmap localizing the most indicative image areas.
- Pneumonia detection: 0.435 F1 is achieved by CheXNet, versus 0.387 for the radiologist average, with the difference statistically significant.The reported 95% confidence intervals are 0.387–0.481 for CheXNet and 0.330–0.442 for the radiologist average.
- Pneumonia detection: Four practicing academic radiologists provide annotations for estimating radiologist performance on pneumonia detection.Pneumonia can be difficult to detect because its appearance is often vague, overlaps with other diagnoses, and mimics benign abnormalities.
- All-disease detection: CheXNet modifications outperform the best published results on all 14 diseases in ChestX-ray14.The model is extended from pneumonia detection to automated detection of every disease in the dataset.
2. CheXNet
CheXNet frames pneumonia detection as binary classification of frontal-view chest X-rays and uses a 121-layer DenseNet adapted to produce a single sigmoid output. It is trained end-to-end with weighted binary cross-entropy, ImageNet initialization, Adam, and validation-loss-based learning-rate decay.
- Pneumonia detection maps a frontal-view chest X-ray X to a binary label y ∈ {0, 1} indicating absence or presence of pneumonia.
- The model optimizes a weighted binary cross-entropy loss using class-count-based weights for positive and negative pneumonia cases.The positive weight is w+ = |N|/(|P|+|N|), while the negative weight is w− = |P|/(|P|+|N|).
- CheXNet is a 121-layer DenseNet trained on ChestX-ray14, with its final fully connected layer replaced by a single sigmoid-output unit.DenseNets improve information and gradient flow, making very deep networks tractable to optimize.
- Training uses ImageNet-pretrained weights, Adam, minibatches of 16, and an initial learning rate of 0.001 decayed tenfold when validation loss plateaus.The selected model has the lowest validation loss; Adam uses β1 = 0.9 and β2 = 0.999.
3. Data
The study uses ChestX-ray14, a large frontal-view chest X-ray dataset with 14 automatically extracted thoracic pathology labels, defining pneumonia-positive and other images as negative. Images are standardized and augmented before training, and performance is evaluated on an independently annotated test set from four practicing radiologists.
- Dataset: 112,120 frontal-view X-ray images from 30,805 unique patients comprise the ChestX-ray14 dataset.Each image can have up to 14 thoracic pathology labels automatically extracted from radiology reports.
- Dataset: Pneumonia-labeled images are positive examples, while all other images are negative examples for pneumonia detection.
- Preprocessing: 224×224 downscaling, ImageNet-based normalization, and random horizontal flipping preprocess and augment the training images.
- Benchmarking: >0.05 AUROC margins over previous state of the art results occur for Mass, Nodule, Pneumonia, and Emphysema.CheXNet outperforms the best published results on all 14 pathologies in ChestX-ray14.
- Test set: 420 frontal chest X-rays form a test set independently annotated for all 14 pathologies by four practicing Stanford radiologists.The radiologists had 4, 7, 25, and 28 years of experience, including one fellowship-trained thoracic radiologist, and lacked patient information and disease-prevalence knowledge.
4. CheXNet vs. Radiologist Performance
On the pneumonia test set, CheXNet achieved a higher F1 score than the radiologist average, with a positive bootstrap difference. The comparison was conservative because evaluation used only frontal radiographs and excluded patient history, both of which can affect diagnosis.
- Evaluation method: F1 scores were computed for each radiologist and CheXNet against the other four radiologists’ labels as ground truth.The mean of the four resulting F1 scores was reported for each radiologist and CheXNet, with 95% bootstrap CIs from the 2.5th and 97.5th percentiles.
- Performance comparison: CheXNet achieved an F1 score of 0.435, compared with 0.387 for the radiologist average.The reported 95% CIs were 0.387–0.481 for CheXNet and 0.330–0.442 for the radiologist average.
- Limitations: The setup was conservative because only frontal radiographs were available, although up to 15% of accurate diagnoses require the lateral view.Neither the model nor the radiologists could use patient history, which has been shown to decrease radiologist diagnostic performance in interpreting cough.
5. ChexNet vs. Previous State of the Art on the ChestX-ray14 Dataset
CheXNet is extended to predict 14 thoracic pathologies using multilabel outputs and unweighted binary cross-entropy training. On a patient-disjoint ChestX-ray14 test split, it achieves state-of-the-art per-class AUROC on all 14 classes, with gains exceeding 0.05 for four pathologies.
- Multilabel extension: 14 pathology classes are predicted simultaneously using a 14-dimensional sigmoid output vector of presence probabilities.The classes include Atelectasis, Cardiomegaly, Consolidation, Edema, Effusion, Emphysema, Fibrosis, Hernia, Infiltration, Mass, Nodule, Pleural Thickening, Pneumonia, and Pneumothorax.
- Multilabel extension: CheXNet optimizes the sum of unweighted binary cross-entropy losses across pathology classes.The loss uses predicted probabilities for both pathology presence and absence.
- Evaluation: The dataset is divided into patient-disjoint training (70%), validation (10%), and test (20%) sets, with per-class AUROC compared against prior state of the art.Yao et al. (2017) provides the comparison for 13 classes, while Wang et al. (2017) provides it for the remaining class.
- Results: CheXNet achieves state-of-the-art results on all 14 pathology classes by per-class AUROC on the test set.The comparison shows especially large improvements on Mass, Nodule, Pneumonia, and Emphysema.
- Results: > 0.05 increase in AUROC is achieved over previous state of the art for Mass, Nodule, Pneumonia, and Emphysema.These four classes are the ones identified as showing considerable improvement.
6. Model Interpretation
The network is interpreted with class activation mappings (CAMs) that visualize image regions most indicative of a pathology. CAMs are generated from final-convolutional-layer feature maps and overlaid on images to identify important predictive features.
- Class activation mappings: CAMs visualize image areas most indicative of a disease using the network’s predictions.They are produced by extracting feature maps from the fully trained network’s final convolutional layer.
- Class activation mappings: The model’s important predictive features are identified by upscaling the pathology map Mc to image dimensions and overlaying it on the image.The map is constructed from final-layer weights associated with feature maps leading to pathology c.
- Class activation mappings: Figure 2 presents CAM examples for pneumonia detection and 14-class pathology classification.
7. Related Work
Deep learning and large datasets have enabled algorithms to surpass medical professionals across several medical imaging tasks. Chest-radiograph automation has expanded from tuberculosis and lung-nodule analysis to studies using OpenI and the substantially larger ChestX-ray-14 dataset.
- Related Work: Deep-learning algorithms have surpassed medical professionals in diabetic retinopathy detection, skin cancer classification, arrhythmia detection, and hemorrhage identification.The cited work spans multiple medical imaging applications enabled by advances in deep learning and large datasets.
- Related Work: Automated chest-radiograph diagnosis has been studied for pulmonary tuberculosis classification and lung nodule detection.These efforts represent applications of algorithms to chest radiographs.
- Related Work: Islam et al. evaluated convolutional architectures across abnormalities using the publicly available OpenI dataset, while Wang et al. released ChestX-ray-14, an order of magnitude larger than previous comparable datasets.The related work includes both architecture comparisons and expansion of publicly available chest-X-ray data.
8. Conclusion
The paper presents an algorithm that detects pneumonia from frontal-view chest X-rays at a level exceeding practicing radiologists and extends it to outperform prior state of the art across multiple diseases. The authors suggest expert-level automation could improve healthcare delivery and access to imaging expertise.
- Motivation: Pneumonia causes substantial morbidity and mortality, making early diagnosis and treatment critical to preventing complications including death.Chest X-rays are the most common imaging examination tool, with approximately 2 billion procedures performed annually.
- Contribution: The algorithm detects pneumonia from frontal-view chest X-ray images at a level exceeding practicing radiologists.The conclusion describes this as expert-level performance in pneumonia detection.
- Contribution: A simple extension detects multiple diseases and outperforms previous state of the art on ChestX-ray14.ChestX-ray14 is identified as the largest publicly available chest X-ray dataset.
- Implications: Expert-level automation could improve healthcare delivery and increase access to medical imaging expertise in underserved parts of the world.The authors frame this as a potential benefit of the technology.