Source-linked AI summary
ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, Ronald M. Summers
TL;DR
Hospital-scale chest X-ray data with loosely labeled reports remain difficult to use for high-precision automated diagnosis. This paper introduces ChestX-ray8 and a unified weakly supervised framework that detects and spatially localizes eight thoracic diseases, with promising initial quantitative results.
Problem
It remains unclear how deep-learning methods scale from small datasets to tens of thousands of patients for chest X-ray analysis.
Method
The paper constructs ChestX-ray8 and applies unified weakly supervised multi-label classification and pathology localization to frontal chest X-rays.
Results
The framework detects and spatially localizes common thoracic diseases, with promising initial quantitative results on the proposed dataset.
Takeaways & Limitations
ChestX-ray8 establishes a hospital-scale public dataset and benchmark for eight-pathology classification and weakly supervised localization.
Takeaways & Limitations
Only a small number of pathology images have hand-labeled bounding boxes for evaluating localization and potential low-shot learning.
Abstract
from arXiv · showhide
The chest X-ray is one of the most commonly accessible radiological examinations for screening and diagnosis of many lung diseases. A tremendous number of X-ray imaging studies accompanied by radiological reports are accumulated and stored in many modern hospitals' Picture Archiving and Communication Systems (PACS). On the other side, it is still an open question how this type of hospital-size knowledge database containing invaluable imaging informatics (i.e., loosely labeled) can be used to facilitate the data-hungry deep learning paradigms in building truly large-scale high precision computer-aided diagnosis (CAD) systems. In this paper, we present a new chest X-ray database, namely "ChestX-ray8", which comprises 108,948 frontal-view X-ray images of 32,717 unique patients with the text-mined eight disease image labels (where each image can have multi-labels), from the associated radiological reports using natural language processing. Importantly, we demonstrate that these commonly occurring thoracic diseases can be detected and even spatially-located via a unified weakly-supervised multi-label image classification and disease localization framework, which is validated using our proposed dataset. Although the initial quantitative results are promising as reported, deep convolutional neural network based "reading chest X-rays" (i.e., recognizing and locating the common disease patterns trained with only image-level labels) remains a strenuous task for fully-automated high precision CAD systems. Data download link: https://nihcc.app.box.com/v/ChestXray-NIHCC
1 Introduction
The paper introduces ChestX-ray8 to address the difficulty of large-scale chest X-ray diagnosis when disease labels must be mined from radiological reports rather than crowd-sourced. It demonstrates weakly supervised multi-label disease detection and spatial localization using 108,948 images from 32,717 patients.
- Motivation: Fully automated high-precision chest X-ray CAD remains strenuous despite deep learning’s progress in medical image detection and segmentation.The introduction situates the work among prior applications including lymph node, interstitial lung disease, cerebral microbleed, pulmonary nodule, pancreas, and cell-image analysis.
- Motivation: Large-scale medical image diagnosis is difficult because generic image-level anatomy and pathology labels cannot be obtained through crowd-sourcing by non-medical annotators.The paper therefore mines per-image, potentially multiple, thoracic pathology labels from attached radiological reports using NLP.
- Dataset: ChestX-ray8 contains 108,948 frontal-view X-ray images from 32,717 unique patients collected from 1992 to 2015.The images carry text-mined labels for eight common diseases extracted from radiological reports via NLP.
- Method: The proposed framework performs weakly supervised multi-label classification and spatial localization of commonly occurring thoracic diseases.This uses image-level labels mined from reports rather than detailed spatial annotations.
- Novelty: ChestX-ray8 is at least one order of magnitude larger than the prior largest public chest X-ray dataset, OpenI.OpenI contains 3,955 radiology reports and 7,470 associated chest X-rays, while the paper focuses on quantitative disease detection and localization.
2 Construction of Hospital-scale Chest X-ray Database
ChestX-ray8 is constructed by mining PACS radiology reports for eight thoracic disease labels using multi-stage clinical concept extraction and syntactic negation/uncertainty handling. The labeling method achieves strong agreement with human-annotated OpenI reports, while a smaller subset receives radiologist-drawn bounding boxes for localization evaluation.
- Database construction: Eight common thoracic pathologies are selected from radiologists’ feedback as target labels for PACS mining.The targets are Atelectasis, Cardiomegaly, Effusion, Infiltration, Mass, Nodule, Pneumonia and Pneumathorax.
- Report processing: Reports are processed in two passes, prioritizing Findings and Impression sections before considering the full report when those sections are absent.The first pass detects disease concepts, and the second pass codes the resulting concepts for labeling.
- Clinical concept labeling: DNorm and MetaMap outputs are merged to maximize recall, while dependency-based rules remove negated and uncertain disease mentions.MetaMap contributes disease or syndrome and finding concepts, and syntactic dependency labels and directions define the negation/uncertainty rules.
- Label validation: 0.90 precision, 0.91 recall, and 0.90 F1-score are achieved on human-annotated OpenI reports, outperforming MetaMap with NegEx enabled.For pneumothorax, precision is 0.90 versus 0.32 for MetaMap, and the method obtains higher precision across all diseases.
- Localization annotations: 1,600 pathology instances across 983 images are selected for radiologist-drawn bounding boxes to evaluate localization.A board-certified radiologist selects 200 instances for each pathology and labels each corresponding disease instance with a B-Box stored as XML.
3 Common Thoracic Disease Detection and Localization
The section presents a unified weakly supervised multi-label DCNN framework that detects multiple thoracic pathologies and localizes them with heatmap-derived bounding boxes. It defines an eight-dimensional label setup and adapts pretrained architectures with specialized pooling, losses, and localization steps.
- Framework: The framework detects multiple pathologies and generates bounding boxes around corresponding disease regions.It combines weakly supervised multi-label classification with pathology localization in a unified DCNN pipeline.
- Framework: The model first predicts whether each X-ray contains one or more pathologies, then uses network activations and weights to localize them.This establishes the sequential detection-and-localization objective of the method.
- Network Architecture: The method modifies ImageNet-pretrained AlexNet, GoogLeNet, VGGNet-16, and ResNet-50 by replacing final layers with transition, global pooling, prediction, and loss layers.The transition layer produces a uniform spatial feature representation across architectures.
- Multi-label Setup: Each image is represented by an 8-dimensional binary label vector, while an all-zero vector denotes “Normal.”Each component indicates the presence of one of the eight disease categories.
- Multi-label Classification Loss: The framework evaluates Hinge Loss, Euclidean Loss, and Cross Entropy Loss, addressing sparse labels and difficulty learning positive pathology instances.The label imbalance arises because batches contain extensively more zeros than ones.
- Localization: Disease heatmaps are converted into bounding boxes by normalizing intensities to [0, 255] and thresholding at 60 and 180.Heatmap peaks indicate likely disease patterns, and simple thresholding is sufficient for box generation.
4 Experiments
Experiments evaluate weakly supervised eight-disease classification and localization on ChestX-ray8, using dataset splits, pretrained CNN initialization, pooling strategies, and balanced losses. ResNet-50 performs best overall, while LSE pooling and weighted loss further improve classification performance.
- Data: ChestX-ray8 contains 108,948 frontal-view images, including 24,636 with pathologies and 84,312 normal cases, split into 70% training, 10% validation, and 20% testing.Eight-disease recognition is reported on the test set; 983 images with 1,600 annotated pathology boxes are used only to evaluate localization.
- CNN Setting: The unified DCNN is initialized with ImageNet-pretrained AlexNet, GoogLeNet, VGGNet-16, or ResNet-50 models, while transition and prediction layers are trained from scratch.The architecture is implemented in Caffe.
- Multi-label Disease Classification: ResNet-50 achieves the best multi-label classification results, including AUC=0.8141 for Cardiomegaly and AUC=0.7891 for Pneumothorax.The four initialization choices produce substantially different quantitative performance.
- Multi-label Disease Classification: LSE pooling with r = 10 reports the best overall classification performance, whereas average and max pooling achieve approximately equivalent performance.LSE pooling declines first as r increases, reaches its lowest performance at r = 5, and performs best around r = 10.
- Multi-label Disease Classification: Weighted loss improves overall performance over CEL, increasing Cardiomegaly AUC from 0.7262 to 0.8141 and Pneumonia AUC from 0.5164 to 0.6333.The gains are especially noted for classes with relatively fewer positive instances.
- Disease Localization: Disease heatmaps and pathology bounding boxes are generated from fine-tuned DCNN activations and prediction-layer weights, then evaluated against hand-annotated ground-truth boxes.The localization annotations comprise 1,600 instances across 983 images.
5 Conclusion
This work constructs a hospital-scale, machine-human annotated chest X-ray database addressing realistic clinical and methodological challenges across tens of thousands of patients. It also provides extensive quantitative benchmarking for eight common thoracic pathologies.
- Database construction: The study constructs a comprehensive, hospital-scale chest X-ray database with machine-human annotations.The database is designed to reflect realistic clinical and methodological challenges in radiology data.
- Database construction: The database addresses challenges involving at least tens of thousands of patients, paralleling the scale of ImageNet in natural images.The work frames this as a realistic clinical and methodological setting for radiology image analysis.
- Performance benchmarking: The study conducts extensive quantitative performance benchmarking on eight common thoracic pathologies.This benchmarking accompanies the construction of the hospital-scale database.
A Supplementary Materials · A.1 SNOMED-CT Concepts
The supplementary materials define the SNOMED-CT concept mappings used for target diseases. The work considers only the semantic types Diseases or Syndromes and Findings, with mappings developed through UMLS searches and radiologist verification.
- A.1 SNOMED-CT Concepts: The analysis considers only SNOMED-CT semantic types Diseases or Syndromes and Findings.These semantic types are identified as ‘dsyn’ and ‘fndg’, respectively.
- A.1 SNOMED-CT Concepts: Table 5 lists SNOMED-CT concepts relevant to the target diseases.The table provides the corresponding concepts used for the disease mappings.
- A.1 SNOMED-CT Concepts: The disease-concept mappings were developed by searching disease names in the UMLS terminology service.The UMLS terminology service was used to identify relevant SNOMED-CT concepts.
- A.1 SNOMED-CT Concepts: A board-certified radiologist verified the disease-concept mappings.Radiologist verification was applied after the mappings were developed through terminology-service searches.
- A.1 SNOMED-CT Concepts: The supplementary mapping procedure links target diseases to corresponding SNOMED-CT concepts.The mappings connect disease names with concepts relevant to the paper’s target diseases.
- A.1 SNOMED-CT Concepts: The concept selection combines terminology-based searching with expert radiological verification.UMLS searches generated the mappings, and a board-certified radiologist checked them.
A.2 Rules of Negation/Uncertainty
The work addresses negation and uncertainty by hand-crafting novel syntactic-level rules. These rules use dependency graphs, including dependency labels and directions, to capture semantic relationships between words.
- Rule design: The authors hand-crafted novel rules of negation and uncertainty at the syntactic level.The rules were introduced to overcome a complication in text processing.
- Rule design: The rules are defined on dependency graphs using dependency labels and directions between words.This representation provides the structural basis for identifying negation and uncertainty.
- Motivation: Dependency information was selected because it is close to semantic relationships between words and prevalent in biomedical text processing.The method therefore connects syntactic structure with semantic relationships in biomedical text.
A.3 More Disease Localization Results
The paper evaluates disease localization across eight thoracic disease classes using localization accuracy and average false positives over multiple IoU thresholds. Qualitative heatmap examples and a simple threshold-based bounding-box generator demonstrate localization behavior while leaving room for improved box generation.
- Quantitative localization results: Table 7 reports localization accuracy and average false positive (AFP) numbers for each disease type at IoU thresholds from 0.1 to 0.7.The evaluation includes only detections with IoU > T(IoU), where T(IoU) ∈ {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7}.
- Qualitative localization results: Tables 8–15 show localization results for all eight disease classes alongside radiology reports, mined disease keywords, and heatmap overlays.The examples cover Atelectasis, Cardiomegaly, Effusion, Infiltration, Mass, Nodule, Pneumonia, and Pneumothorax.
- Qualitative localization results: The visualizations distinguish correct bounding boxes, false positives, and ground truth using green, red, and blue overlays, respectively.These annotations are plotted over the original chest X-ray images, while heatmaps are shown as overlays on the right.
- Bounding-box generation: A two-level thresholding method detects heatmap peaks and generates approximately 1–3 bounding boxes per heatmap for evaluation against ground truth.The authors suggest that more sophisticated methods, such as selective search, could further optimize localization accuracy and AFP.
B ChestX-ray14 Dataset
ChestX-ray14 expands the disease categories beyond ChestX-ray8 by adding six common thorax diseases and updating the NLP-mined labels. Its dataset statistics are presented in Table 16 and Figure 8, while the pathology bounding boxes remain unchanged.
- Dataset expansion: Six additional thorax diseases expand ChestX-ray8 into ChestX-ray14: Consolidation, Edema, Emphysema, Fibrosis, Pleural Thickening, and Hernia.The NLP-mined labels were also updated.
- Dataset statistics: ChestX-ray14 corpus statistics are reported in Table 16 and Figure 8, including total counts and overlap counts for ChestX-ray8 and ChestX-ray14.Table 16 identifies Pleural Thickening as PT.
- Annotations: Pathology bounding boxes remain unchanged in the ChestX-ray14 update.The expansion changes disease categories and NLP-mined labels, not the existing bounding-box annotations.
B.1 Evaluation of NLP Mined Labels
The method was evaluated against the OpenI dataset and a newly annotated corpus of clinical reports, achieving higher precision, recall, and F1-scores than MetaMap with NegEx. The improvement indicates that syntactic-level negation and uncertainty detection reduces false positives.
- Evaluation data: 900 clinical reports were independently annotated by two annotators for 14 finding types, with conflicts resolved through agreement.The annotated reports were used to evaluate finding-recognition systems.
- Evaluation data: The method was evaluated on both OpenI and the proposed dataset using precision, recall, and F1-score.These datasets provided alternative and newly annotated evaluation resources.
- Results: Higher precision, recall, and F1-scores were achieved than with MetaMap using NegEx.The result suggests that syntactic-level negation and uncertainty detection successfully removes false-positive cases.
B.2 Benchmark Results
The unified disease classification and localization framework was evaluated on ChestX-ray14 using patient-level training, validation, and testing splits. Overall, both classification and localization performance were equivalent to ChestX-ray8.
- Dataset and evaluation: ChestX-ray14 evaluation used 112,120 frontal-view X-ray images, including 51,708 with one or more pathologies and 60,412 without the listed 14 findings.The dataset supports 14-disease recognition and localization evaluation.
- Dataset and evaluation: Images were randomly divided at the patient level into approximately 70% training, 10% validation, and 20% testing sets.Images from the same patient appeared in only one split, and testing used the published test set.
- Localization evaluation: Localization was evaluated only on the original eight categories because the annotated pathology bounding boxes were unchanged, using the Intersection over the detected B-Box area ratio (IoBB).IoBB is described as similar to Area of Precision or Purity.