COVID-19 Image Data Collection: Prospective Predictions Are the Future

Joseph Paul Cohen; Paul Morrison; Lan Dao; Karsten Roth; Tim Q Duong; Marzyeh Ghassemi

Source-linked AI summary

COVID-19 Image Data Collection: Prospective Predictions Are the Future

Joseph Paul Cohen, Paul Morrison, Lan Dao, Karsten Roth, Tim Q Duong, Marzyeh Ghassemi

arXiv:2006.11988v3Jun 22, 2020q-bio.QMcs.CVcs.LGeess.IV

TL;DR

COVID-19 imaging research lacked carefully curated, annotated public data for diagnosis and management. This paper constructs a machine-learning-ready chest X-ray dataset with clinical metadata and explores prognostic tasks, finding performance varies substantially by task.

Problem
COVID-19 diagnosis and management require carefully curated, annotated imaging data, but existing large public chest X-ray datasets mainly represent typical cases.
Method
The paper manually aggregates public COVID-19 chest X-rays with clinical metadata into a machine-learning-ready dataset and explores diagnostic, prognostic, and trajectory-prediction uses.
Results
Performance varied by task: ICU stay was predicted reasonably well, whereas COVID classification and survival were near random and intubation prediction was poor.
Takeaways & Limitations
The dataset provides a public resource for testing COVID-19 diagnostic, prognostic, and management tools using images linked to prospective clinical metadata.
Takeaways & Limitations
The dataset is biased and unbalanced because publicly available images may not represent the real-world distribution of cases.

Abstract

from arXiv · show

Across the world's coronavirus disease 2019 (COVID-19) hot spots, the need to streamline patient diagnosis and management has become more pressing than ever. As one of the main imaging tools, chest X-rays (CXRs) are common, fast, non-invasive, relatively cheap, and potentially bedside to monitor the progression of the disease. This paper describes the first public COVID-19 image data collection as well as a preliminary exploration of possible use cases for the data. This dataset currently contains hundreds of frontal view X-rays and is the largest public resource for COVID-19 image and prognostic data, making it a necessary resource to develop and evaluate tools to aid in the treatment of COVID-19. It was manually aggregated from publication figures as well as various web based repositories into a machine learning (ML) friendly format with accompanying dataloader code. We collected frontal and lateral view imagery and metadata such as the time since first symptoms, intensive care unit (ICU) status, survival status, intubation status, or hospital location. We present multiple possible use cases for the data such as predicting the need for the ICU, predicting patient survival, and understanding a patient's trajectory during treatment. Data can be accessed here: https://github.com/ieee8023/covid-chestxray-dataset

1. Introduction

The paper argues that curated public COVID-19 imaging data are needed to streamline diagnosis and management, addressing the absence of a computationally usable public COVID-19 CXR/CT collection. It introduces a growing COVID-19 CXR dataset centered on prospective clinical metadata, proposes prognostic and management tasks, and describes LOCO evaluation for domain shift.

Motivation: COVID-19 outbreaks make streamlined diagnosis and management crucial, while AI systems could help process imaging data and support patient triage and resource allocation.Hospitals had deployed AI-driven CT interpreters and triage initiatives in China and Italy.
Data gap: Carefully curated and annotated data are presented as the first step toward diagnostic or management tools, but no public COVID-19 CXR or CT collection was designed for computational analysis.The dataset complements a larger BIMCV COVID-19+ collection through prospective metadata from multiple medical centers and countries.
Dataset contribution: 679 frontal chest X-ray images from 412 people in 26 countries constitute the first public COVID-19 CXR image data collection, with prospective clinical attributes for prognostic and management tools.Attributes include survival, ICU stay, intubation events, blood tests, location, and freeform clinical notes; the collection is growing.
Clinical use cases: The paper proposes machine-learning tasks for pneumonia severity, survival outcome, and ICU-need prediction, and includes transfer-learning neural-network benchmark results.These tasks are framed as clinical use cases for the dataset.
Evaluation and aggregation: Location information enables Leave-One-Country/Continent-Out evaluation to simulate domain shift and provide more robust evaluation.The project aims to aggregate publicly available radiographs from papers and other datasets, using manual extraction for research articles and partially automated website collection with hand-reviewed clinical events.

2. Background and Related Work

Prior work established imaging’s role in COVID-19 care while recommending against standalone or systematic diagnosis; CXRs remain preferred initially because they are lower-dose, cheaper, and portable. Existing imaging-based prediction studies often relied on small or private datasets and inadequate evaluation, motivating reproducible public data.

Clinical imaging guidance: Clinical recommendations advise against using CXR or CT alone, systematically, or to decide whether suspected patients should receive COVID-19 testing.Normal imaging does not exclude infection, and abnormal findings are non-specific.
Clinical imaging guidance: CXR and CT can support patients at risk of progression, worsening respiratory status, or triage in resource-constrained environments.
Advantages of chest radiography: CXRs remain the preferred initial modality because a PA film delivers 0.02 mSv versus 7 mSv for chest CT.CXR is also preferred when pneumonia is suspected.
Advantages of chest radiography: CXRs are cheaper than CT and portable units can be wheeled into ICUs and emergency rooms, then cleaned to reduce patient-flow and infection risks.
Related machine-learning work: Existing COVID-19 imaging prediction studies often used small or private datasets, lacked appropriate evaluation, and risked overfitting or performance overestimation.Raj [2020] reported 90% AUC on a large private dataset that was not reproducible.

3. Cohort Details

The cohort comprised 761 images from 230 male and 139 female unique patients, with diagnosis, pneumonia, microorganism, clinical-status, and imaging metadata collected. Images and metadata were manually and programmatically aggregated from public databases and publications, with alignment and processing limitations.

Cohort statistics: As of September 18, 2020, the collection contained 761 images from 230 male and 139 female unique patients.The reported unique-patient male/female ratio was 230/139.
Cohort statistics: Metadata were collected for each image, while cohort statistics were organized by diagnosis, sub-region, and projection/view.The passage states that attributes in Table 2 were collected for each image and that Tables 1, 3, and 4 summarize these distributions.
Data collection: Data were compiled manually and with scrapers from public databases, websites, online publications, and PDFs.Sources included Radiopaedia.org, the Italian Society of Medical and Interventional Radiology, Figure1.com, and Hannover Medical School.
Data collection: Image extraction used online sources and the pdfimages tool, but processing between PAC systems and PDFs or websites could not be controlled.The authors believed radiological-finding information was otherwise maintained.
Metadata alignment: Metadata alignment required clinical measurements to be taken on the same day as the corresponding image, which was difficult to determine automatically outside captions.The scripts for scraper design were made publicly available.

4. Experimental Setup

The experiments address dataset bias and imbalance through leave-one-country/continent-out evaluation and restrict inputs to AP and PA views. Models use DenseNet-derived or pixel features with linear, logistic, or multilayer-perceptron classifiers.

Evaluation design: The dataset is highly biased and unbalanced, so experiments use AP and PA views to avoid artifacts from AP Supine images.These construction biases can produce misleading outcomes if the dataset is treated as a typical benchmark.
Evaluation design: Leave-one-country/continent-out evaluation tests on data from a single continent to address training bias in small, unbalanced datasets.Country-level separation was not possible because of the current data distribution, and incomplete labels may allow same-continent training and evaluation.
Feature representations: A pre-trained DenseNet model supplies image features after training on 88,079 non-COVID-19 images from seven large chest X-ray datasets.Labels were manually aligned across datasets so equivalent findings, such as pleural effusion and effusion, were treated as equal.
Feature representations: Experiments compare 1024-dimensional intermediate features, 18 outputs, four pneumonia-related outputs, a lung-opacity output, and 224×224=50176 image pixels.The four-output subset contains Lung Opacity, Pneumonia, Infiltration, and Consolidation; the lung-opacity feature differs from the opacity score being predicted.
Model training: Linear or logistic regression uses default Sci-kit learn parameters with an L2 penalty and LBFGS, while pixel inputs use an MLP with 100 hidden units and ReLU activations.The MLP uses full-batch Adam optimization with learning rate 0.001 and 10% early stopping data.

5. Task Ideas with Baseline Evaluations

The section proposes classification, severity, ICU, intubation, survival, and patient-trajectory tasks using chest X-rays, with evaluations showing strong limitations for several predictions. ICU-stay prediction reaches 0.81 AUROC, whereas COVID classification and survival prediction perform near random, and embeddings illustrate clinically plausible trajectories.

Classification tasks: The dataset supports classifying COVID-19 against other pneumonia agents and distinguishing viral from bacterial pneumonia.These tasks are enabled by the dataset’s hierarchy of labels.
Classification tasks: The best COVID/non-COVID performance is only slightly above random guessing, whose AUROC is 0.5 under LOCO evaluation.The authors note that prior evaluations often used other datasets as non-COVID cases, whereas balanced datasets reported lower performance.
Severity and critical-care prediction: Severity prediction uses Cohen et al.’s scores: Geographic Extent ranges from 0-8, and Opacity ranges from 0-6.The task excludes images showing intubation and, for ICU prediction, images marked as already in the ICU.
Severity and critical-care prediction: 0.81 AUROC is achieved for ICU-stay prediction using all 18 outputs, while intubation is predicted poorly.The improvement over the four pneumonia-associated outputs may indicate that ICU stay is predicted by features beyond pneumonia.
Survival prediction: Survival prediction obtains almost random performance of 0.55 AUROC using lung opacity and four hand-picked pneumonia features, and no method works well generally.The proposed use case is to estimate survival probability from a series of chest X-rays over time.
Patient trajectories: Pre-sigmoid DenseNet embeddings separate good and bad states and produce clinically plausible patient trajectories, including deterioration, recovery, and rapid improvement.The trajectories are visualized using 18 outputs and state definitions based on intubation, ICU status, and discharge.

6. Future Task Ideas

The section identifies saliency-map evaluation and out-of-distribution modeling as promising future tasks not explored in this work. It also highlights transfer-learning methods that may be useful for this setting.

Saliency maps: Saliency-map evaluation could help assess methods intended to explain model predictions.The dataset includes lung bounding boxes for 167 images and 209 generated lung segmentations.
Saliency maps: Lung bounding boxes and generated segmentations provide resources for evaluating saliency maps.The bounding boxes annotate the left and right lungs, while the generated segmentations came from a model trained on an external dataset.
Out-of-distribution modeling: Out-of-distribution tools and unpaired distribution-matching models could identify shifts in COVID-19 image distributions.Comparing COVID-19 with viral or bacterial pneumonias could support disease study and analysis of model representations.
Transfer learning: Few-shot or zero-shot learning, meta-learning, deep metric learning, and domain adaptation may be useful transfer-learning approaches.These methods are described as being actively developed in the machine-learning community.

7. Conclusion

The paper presents a COVID-19 image dataset with clinical metadata for varied tasks and for testing existing machine-learning algorithms. It also identifies COVID-19-related tasks as a relevant challenge and notes selection bias from publicly available images.

Conclusion: The dataset combines COVID-19 images with clinical metadata for a variety of tasks.The paper presents the collection as a resource for testing machine-learning algorithms.
Conclusion: Existing machine-learning algorithms can be tested on the dataset.
Conclusion: Novel COVID-19-related tasks remain a relevant challenge despite existing large chest-X-ray datasets.
Conclusion: A major limitation is selection bias from gathering publicly available images, which are likely released for educational reasons.

Broader Impact

The project makes publicly available, clinically linked COVID-19 imaging data to support safe research and parallel development of diagnosis and management tools. Resulting tools could provide physicians with digital second opinions and quantitative CXR scores for broader analysis.

The project makes data from patients with a life-threatening disease accessible to researchers while avoiding patient risk through public-data collection.The stated aim is to enable tools that aid future patient care.
A public ML-designed dataset linking image data with clinically relevant attributes enables parallel development, rapid local validation, and varied research tasks.Its structure supports multiple uses beyond a single diagnostic application.
Tools built from the data could give physicians a digital second opinion while they await radiologist analysis, increasing confidence in patient assessment.The digital assessment is intended to confirm a physician’s judgment of a patient’s condition.
Quantitative scores from these tools could enable large-scale CXR analysis without costly or time-consuming manual annotations.

Ethical Standards · Appendix

The work states that it follows appropriate ethical standards and complies with applicable laws and regulations for research involving animals or human subjects. The project was approved by the University of Montreal’s Ethics Committee.

Ethical Standards: The research follows appropriate ethical standards for conducting the work and writing the manuscript.
Ethical Standards: The manuscript follows applicable laws and regulations governing the treatment of animals or human subjects.
Ethical Standards: The project received approval from the University of Montreal’s Ethics Committee.
Ethical Standards: The ethics approval is identified by committee reference CERSES-20-058-D.
Ethical Standards: The stated ethical compliance covers research involving animals or human subjects.
Ethical Standards: The paper explicitly documents its ethical oversight within the manuscript’s ethical standards information.

Appendix A. Extra dataset statistics · Appendix B. Example images · Appendix C. Scraper details

The appendices detail dataset composition, provide representative patient-image examples, and describe the scraping pipeline, metadata limitations, and resource-conserving practices. The collection includes multiple image formats and bit depths, while clinically important ICU and intubation information requires human annotation.

Appendix A. Extra dataset statistics: 367 JPEG files and 171 PNGs comprise the PA, AP, and AP Supine views, with all but one image stored as 8-bit.One image is 16-bit.
Appendix A. Extra dataset statistics: Image-size distributions are summarized as pixel histograms.Figure 4 presents the histograms.
Appendix B. Example images: Example images from patient #19 show a 55-year-old female who survived COVID-19 infection.The images were extracted from Cheng et al. [2020].
Appendix B. Example images: Example images from patient #318 document ICU intubation on days 2, 7, and 12.The images were extracted from Nakamura et al. [2020].
Appendix C. Scraper details: Scrapers search each site for relevant radiographs, save images and metadata, download the highest available resolution, and convert each case to interoperable JSON plus CSV metadata.Selenium visits search-result pages and records only new cases.
Appendix C. Scraper details: Structured webpage fields such as Age, Sex, Imaging Notes, and Clinical Notes are scraped automatically, whereas ICU admission and intubation are extracted by human annotators.The automated process currently retrieves only metadata from easily detected structured fields.
Appendix C. Scraper details: The scrapers conserve website resources through lazy crawling and by keeping each page open in an individual browser instance during a scraping session.Pages are not requested until they can be used, reducing repeated resource requests.

C.1 Papers where images and clinical data are sourced · C.2 Output of models on each split

The section documents the papers and image-view counts underlying the collection, then presents model outputs across geographic, radiographic, diagnostic, and clinical prediction splits.

C.1 Papers where images and clinical data are sourced: Table 7 lists papers and counts for specific radiographic views.The associated view categories include PA, AP, AP Supine, and L.
C.1 Papers where images and clinical data are sourced: The source-view notation distinguishes PA, AP, AP Supine, and lateral images.These view labels accompany citation information for the sourced papers.
C.2 Output of models on each split: The model-output section includes a geographic-extent analysis for each split.Geographic extent is presented as a dedicated table.
C.2 Output of models on each split: Separate outputs address opacity and COVID-19 classification.These analyses are reported in dedicated tables for opacity and COVID-19.
C.2 Output of models on each split: Another model-output table evaluates viral-or-bacterial classification.The table is specifically labeled “Viral or Bacterial.”
C.2 Output of models on each split: Clinical prediction outputs cover survival, ICU stay, and intubation status.Dedicated tables report survival prediction, ICU stay, and intubated outcomes.
C.2 Output of models on each split: Figure 7 provides a montage of all frontal-view images in the collection.The montage includes PA, AP, and AP Supine images.