Source-linked AI summary
PadChest: A large chest x-ray image dataset with multi-label annotated reports
Aurelia Bustos, Antonio Pertusa, Jose-Maria Salinas, Maria de la Iglesia-Vayá
TL;DR
Chest-x-ray research needs large, high-quality datasets that capture clinically relevant labels despite uncertainty, annotation difficulty, and confounding factors. PadChest constructs a large Spanish-report dataset using physician-labeled ground truth and supervised neural-network annotation, achieving 0.93 MicroF1 on an independent test set. Its breadth, localization, metadata, and language-mapped labels support supervised chest-radiograph research, while external-institution generalization remains to be confirmed.
Problem
Medical imaging datasets need large, high-quality, clinically relevant annotations while handling uncertainty, incomplete reports, and confounding factors in radiology data.
Method
PadChest combines physician manual labeling with supervised deep-neural-network multi-label classification and labeling-resolution rules to annotate Spanish chest-x-ray reports.
Results
0.93 MicroF1 was achieved on an independent test set, and the best model labeled 73% of the PadChest dataset.
Takeaways & Limitations
PadChest provides more than 160K high-resolution chest-x-ray images with Spanish reports, 299 UMLS-mapped medical entities, anatomical labels, and metadata for supervised model development.
Takeaways & Limitations
All data came from a single institution, so generalization must be confirmed using reports from other Spanish institutions.
Abstract
from arXiv · showhide
We present a labeled large-scale, high resolution chest x-ray dataset for the automated exploration of medical images along with their associated reports. This dataset includes more than 160,000 images obtained from 67,000 patients that were interpreted and reported by radiologists at Hospital San Juan Hospital (Spain) from 2009 to 2017, covering six different position views and additional information on image acquisition and patient demography. The reports were labeled with 174 different radiographic findings, 19 differential diagnoses and 104 anatomic locations organized as a hierarchical taxonomy and mapped onto standard Unified Medical Language System (UMLS) terminology. Of these reports, 27% were manually annotated by trained physicians and the remaining set was labeled using a supervised method based on a recurrent neural network with attention mechanisms. The labels generated were then validated in an independent test set achieving a 0.93 Micro-F1 score. To the best of our knowledge, this is one of the largest public chest x-ray database suitable for training supervised models concerning radiographs, and the first to contain radiographic reports in Spanish. The PadChest dataset can be downloaded from http://bimcv.cipf.es/bimcv-projects/padchest/.
1 Introduction
Chest-x-ray datasets face challenges in obtaining high-quality, clinically relevant labels, handling uncertainty, and controlling confounders. PadChest addresses these needs with broad, localized annotations, metadata, physician-labeled ground truth, and supervised expansion to Spanish reports.
- Open challenges: Radiology datasets must define clinically relevant labels while annotating large image collections with sufficient quality despite medical expertise requirements.Unstructured medical text further complicates semantic interpretation and knowledge extraction.
- Open challenges: Medical radiology data contain uncertainty and incompleteness, requiring decision-support systems to accommodate imperfect patient records and observer variability.The paper highlights risks when systems impose assumptions of accuracy and completeness on clinical data.
- Open challenges: Models can exploit tubes, catheters, image quality, patient position, and entity imbalance as confounders instead of learning clinically relevant radiological patterns.Projection type must be identified before interpretation because it affects how findings should be read.
- PadChest contribution: PadChest provides a large, exhaustively labeled public dataset with Spanish report excerpts, UMLS mappings, anatomical labels, high-resolution images, and acquisition and demographic metadata.Its labels cover a broader spectrum of thoracic entities than previous datasets and are usable regardless of language through UMLS identifiers.
- PadChest contribution: Physicians manually labeled the baseline ground truth, after which deep neural networks and labeling-resolution rules annotated the remaining 73% of Spanish reports.Manual annotation was chosen to maximize baseline-ground-truth reliability, while supervised annotation addressed large-scale coverage and medical NLP challenges.
- PadChest contribution: PadChest broadens trainable image labels to respiratory, cardiac, infectious, neoplastic, bone, soft-tissue, device, and anatomical-localization entities.The dataset also supplies metadata intended to help control predictive-model confounders.
2 Material and methods
PadChest was constructed from chest x-rays and reports, combining DICOM-derived acquisition metadata with manual and automatic hierarchical label annotation. The workflow addressed projection effects, report redundancy, and uncertainty in labels during dataset preparation.
- Dataset construction: 109,931 studies and 168,861 images were collected from examinations interpreted by 18 radiologists at Hospital Universitario de San Juan from 2009 to 2017.
- Dataset construction: The dataset was de-identified, made downloadable, and distributed with 1 TB of images, a 33-field study CSV, and documentation for retrieval.
- Annotation workflow: The construction workflow included image and DICOM preprocessing, report preprocessing, hierarchical manual annotation, and automatic labeling of remaining studies.
- Image preprocessing: Six projection classes were defined because projection and positioning information were heterogeneous and clinically relevant, with imbalance risking projection-specific misclassification.The classes were standard PA, standard L, AP vertical or erect, AP horizontal or supine, pediatric, and rib views.
- Image preprocessing: 20,367 images lacking projection metadata were labeled with a fine-tuned ResNet-50 model, and those automatic projection labels were excluded from subsequent model training.
- Report annotation: 46% of 500,000 report sentences belonged to the 1,000 most repeated sentences, motivating sentence-level manual labeling and topic-based organization.Doc2Vec with k-means produced 20 topics selected by physician consensus, making batches of semantically similar sentences easier to label.
- Report annotation: “Unchanged” labels required replacement with prior-study labels when available or removal when the finding could not be learned from the image.
- Label organization: Differential diagnoses were separated from directly observable radiographic findings because diagnosis requires clinical information or additional tests.
2.4 Automatic labeling of the remaining reports
The remaining reports were automatically annotated as a multi-label text-classification task using four neural architectures, including CNN and RNN models with label-specific attention. The attention models learn label-specific text representations and produce probabilities for each label.
- 75% of previously unlabeled reports were automatically tagged using a deep neural-network classifier trained on manually labeled reports.The annotations were intended to provide training outputs for an image classifier.
- Each sentence is treated as a multi-label classification instance that predicts binary labels for radiographic findings and differential diagnoses.
- The evaluated architectures were CNN, RNN, CNN-ATT, and RNN-ATT models, with the first two serving as baselines for attention comparisons.
- The CNN processes concatenated word embeddings into sentence representations, while the RNN processes variable-length embedding sequences into hidden-state representations.
- Attention model: Attention vectors assign label-specific distributions over text locations, producing weighted feature vectors that are converted into label probabilities with sigmoid outputs.
- Training minimized binary cross-entropy loss with L2 weight regularization, using Adam for CNN models and RMSprop for RNN models.
3 Evaluation of automatic labeling
The automatic labeling models were evaluated with multi-label metrics using held-out validation, cross-validation, and an independent manually labeled test set. The RNN outperformed the CNN, attention improved both architectures, and the best model achieved a MicroF1 score of 0.93 on the independent test set.
- MicroF1 counts true positives, false negatives, and false positives globally, while MacroF1 averages per-label metrics without accounting for label imbalance.
- WeightedF1 averages per-label metrics according to support, altering MacroF1 to account for label imbalance.
- The four models were trained and validated on the same random partition, with early stopping and MicroF1-based hyperparameter selection summarized in the evaluation tables.
- 11-fold cross-validation compared accuracy curves and learning patterns through 150 epochs while measuring variation caused by training and validation data distributions.
- The RNN outperformed the CNN, and attention increased performance for both model families.The overall MicroF1 gain was more pronounced for CNN-ATT, whereas RNN-ATT learned faster in early epochs.
- 0.93 MicroF1 was obtained by the best architecture on an independent random sample of 500 manually labeled sentences from a different time period.The result was similar to validation performance, and RNN-ATT was used to annotate the remaining dataset.
4 Dataset overview
PadChest contains 160,868 labeled chest x-ray images from 69,882 patients collected at one institution between 2009 and 2017, with multiple views, reports, demographic and acquisition fields, and extensive hierarchical annotations. The dataset includes manually and automatically labeled studies covering findings, differential diagnoses, and anatomic locations.
- 160,868 labeled chest x-ray images from 69,882 patients were acquired at a single institution between 2009 and 2017.Patients had a mean of 1.62 studies, and each study contained one or more position-view images associated with a radiography report.
- The dataset includes original DICOM fields alongside additional processed information for each chest x-ray image.
- Patient ages ranged from 0 to 105 years, with a mean of 58.5 and a median of 62, and the final dataset had 80,923 images from women and 79,923 from men.
- Radiation exposure varied by modality, projection, and year, with higher exposure in computer radiography and lateral views and a decreasing trend over time.
- 68,855 studies contained radiographic findings and/or differential diagnoses, while 37,871 were normal; 17,513 and 9,404 studies, respectively, were manually annotated.
- Differential diagnoses comprised 19 labels across 27,726 reported diagnoses, led by COPD with 14,557 studies and pneumonia with 5,934.
- Radiographic findings used 170 labels, while anatomic locations used 104 labels across 249,469 specified locations, most commonly in the lung fields.
- Mean and standard-deviation images from 500 training examples showed relevant differences among x-ray projections.PA images appeared centered, while AP supine images showed an enlarged mean heart silhouette.
5 Conclusions
PadChest is a large, multilingual-ready chest x-ray dataset combining diverse views, metadata, hierarchical UMLS-linked labels, and physician-validated automatic annotation. Its scope supports pathology prediction research, while single-institution sourcing and report-based ground truth constrain generalization and completeness.
- More than 160K images and Spanish reports comprise one of the largest public labeled chest x-ray datasets.
- PadChest is the first large-scale exploitable dataset available in Spanish, with standardized medical codes supporting use regardless of language.
- The dataset includes multiple projection views and acquisition metadata considered essential for adequately training models.
- Trained physicians manually annotated 27% of samples, while an RNN-ATT method labeled the remaining reports.
- The hierarchical taxonomies organize radiographic findings, differential diagnoses, and anatomical locations using UMLS CUI standard codes.
- The automatic annotation model labeled 73% of PadChest and achieved a 0.93 MicroF1 score on an independent test set.
- Single-institution sourcing and report-retrieval selection bias limit conclusions about cross-institution generalization and pathology frequencies.
- The dataset’s ground truth is limited because reports may omit abnormalities, and severity descriptors are not captured in the labels.
A.1 Counts of annotated labels in x-ray studies
The dataset includes physician-reviewed report annotations, with extracted medical entities mapped to UMLS controlled-vocabulary identifiers when available.
- 27,593 study reports were manually labeled and reviewed by trained physicians.Extracted medical entities were mapped to UMLS controlled biomedical vocabulary identifiers, with relevant unmatched entities retained without CUIs.
A.1.1 Hierarchy of labels of radiographic findings
Radiographic findings are organized hierarchically, with labels covering normal or study-quality categories, imaging findings, devices, calcifications, nodules, and other thoracic abnormalities.
- 177,628 radiographic findings were annotated across all study reports, including separately counted normal, exclude, and suboptimal study labels.
- The hierarchy includes top-level labels for normal, exclude, suboptimal study, and radiological finding.
- The taxonomy further represents detailed pulmonary patterns, atelectasis subtypes, pleural abnormalities, masses, fractures, devices, and calcified lesions.
- Common hierarchical findings include cardiomegaly, pleural effusion, aortic elongation, scoliosis, and chronic changes.
A.1.2 Differential Diagnoses Hierarchy
The differential-diagnosis hierarchy contains manually annotated and automatically represented diagnostic categories spanning infectious, neoplastic, chronic pulmonary, cardiovascular, and vascular conditions.
- 27,726 differential diagnoses were recorded, including 5,472 manually annotated from 27,593 study reports.
- The hierarchy includes pneumonia, tuberculosis, lung metastasis, pulmonary fibrosis, emphysema, COPD signs, heart insufficiency, and pulmonary edema.
- Additional diagnostic categories include atypical pneumonia, tuberculosis sequelae, lymphangitis carcinomatosa, pulmonary hypertension, and bone metastasis.
A.1.3 Anatomical Locations
Anatomical localization labels span thoracic structures and regions, with hierarchical subdivisions for lung fields, mediastinum, cardiac areas, soft tissues, and bones.
- The localization hierarchy includes a broad localization category and detailed labels for cervical, axillary, pectoral, soft-tissue, nipple, and bone regions.
- Pulmonary locations are subdivided into upper, middle, and lower lung fields, lobes, fissures, bronchi, peribronchial regions, and costophrenic angles.
- Additional localization labels identify cardiac, vascular, diaphragmatic, gastric, hypochondrial, and bilateral or side-specific regions.
- Mediastinal labels distinguish superior, anterior, middle, lower, posterior, retrocardiac, paratracheal, and paramediastinal regions.
B Regular expressions used to extract spatial concepts from Spanish reports
The section lists regular expressions that map Spanish report terms to spatial concepts, including anatomy, laterality, and bilateral or diffuse distributions.
- The expressions are presented across repeated regular-expression and spatial-concept columns, with continuation markers between columns.
- Regular expressions map Spanish report terms to spatial concepts such as tracheal, esophageal, soft-tissue, pectoral, nipple, gallbladder, and hemithorax locations.
- The mappings include right and left laterality terms.
- Additional expressions identify diffuse bilateral, bilateral, and basal bilateral distributions.
C Dataset example
The dataset example shows a study containing two x-ray projections whose associated fields are shared across both images in the study report.
- A study example contains two x-ray images with two projections.
- The fields associated with the study are shared by both projections included in the same study report.