Source-linked AI summary
Predicting COVID-19 Pneumonia Severity on Chest X-ray with Deep Learning
Joseph Paul Cohen, Lan Dao, Paul Morrison, Karsten Roth, Yoshua Bengio, Beiyi Shen, Almas Abbasi, Mahsa Hoshmand-Kochi, Marzyeh Ghassemi, Haifang Li, Tim Q Duong
TL;DR
COVID-19 care needs scalable ways to monitor pneumonia severity, and chest X-rays offer a non-invasive, potentially bedside tool. The paper uses representations learned from large chest X-ray datasets to predict severity-related scores, with the single lung-opacity feature achieving the best reported correlation of 0.80.
Problem
COVID-19 strained healthcare systems, motivating chest X-ray tools to monitor disease progression and support patient-management decisions.
Method
The model uses representations and feature-extraction layers pre-trained on seven large chest X-ray datasets before learning from the smaller COVID-19 image set.
Results
0.80 correlation was achieved using the single “lung opacity” output as a feature, the best reported result among the compared feature sets.
Takeaways & Limitations
The predicted severity scores could support escalation or de-escalation of care and quantitative monitoring of disease progression and treatment response, especially in the ICU.
Takeaways & Limitations
The small sample prevents proper cohort selection and exposes the evaluation to sample bias; generalization requires new cohorts with the same severity labels.
Abstract
from arXiv · showhide
Purpose: The need to streamline patient management for COVID-19 has become more pressing than ever. Chest X-rays provide a non-invasive (potentially bedside) tool to monitor the progression of the disease. In this study, we present a severity score prediction model for COVID-19 pneumonia for frontal chest X-ray images. Such a tool can gauge severity of COVID-19 lung infections (and pneumonia in general) that can be used for escalation or de-escalation of care as well as monitoring treatment efficacy, especially in the ICU. Methods: Images from a public COVID-19 database were scored retrospectively by three blinded experts in terms of the extent of lung involvement as well as the degree of opacity. A neural network model that was pre-trained on large (non-COVID-19) chest X-ray datasets is used to construct features for COVID-19 images which are predictive for our task. Results: This study finds that training a regression model on a subset of the outputs from an this pre-trained chest X-ray model predicts our geographic extent score (range 0-8) with 1.14 mean absolute error (MAE) and our lung opacity score (range 0-6) with 0.78 MAE. Conclusions: These results indicate that our model's ability to gauge severity of COVID-19 lung infections could be used for escalation or de-escalation of care as well as monitoring treatment efficacy, especially in the intensive care unit (ICU). A proper clinical trial is needed to evaluate efficacy. To enable this we make our code, labels, and data available online at https://github.com/mlmed/torchxrayvision/tree/master/scripts/covid-severity and https://github.com/ieee8023/covid-chestxray-dataset
1. Introduction
The study addresses the need for practical CXR-based tools to monitor COVID-19 progression and support patient-care management. It develops a severity-prediction model intended to assist escalation, de-escalation, and longitudinal treatment monitoring.
- Chest X-rays offer a non-invasive, potentially bedside method for monitoring COVID-19 disease progression.
- Practically deployable CXR-based predictive models remain lacking despite recent clinical-data tools.
- The model predicts COVID-19 pneumonia severity from CXRs as an assistive tool for managing patient care.The proposed uses include escalation or de-escalation of care, particularly in the ICU.
- Automated scoring can quantitatively track disease progression and treatment response over time.
2. Materials and Methods
The study retrospectively scores public COVID-19 CXRs using two severity dimensions, then applies linear regression to features from a pre-trained non-COVID-19 chest-X-ray model. Evaluation uses patient-separated repeated train/test splits and saliency analysis.
- Data and scoring: The dataset contains 94 de-identified PA CXRs from COVID-19-positive patients reported by multiple hospitals worldwide.The images were collected from December 2019 to March 2020, with no missing data reported.
- Data and scoring: Three blinded experts scored lung involvement extent and opacity degree using separate bilateral scoring systems.Extent ranges from 0–8 and opacity from 0–6 when right and left lung scores are combined.
- Data and scoring: Inter-rater agreement was 0.45 for opacity and 0.71 for extent using Fleiss Kappa.
- Pre-training: A DenseNet was pre-trained on seven public non-COVID-19 datasets aligned across 18 radiological finding tasks.The pre-training data included 88,079 non-COVID-19 images.
- Feature extraction: COVID-19 images were processed through frozen feature-extraction and task-prediction layers to generate representations for downstream regression.The convolutional feature extractor produces a 1024-dimensional representation.
- Regression and evaluation: Linear regression predicted extent and opacity scores from intermediate features or selected pre-trained outputs.Compared feature sets included 18 outputs, four pneumonia-related outputs, and the single lung-opacity output.
- Regression and evaluation: Repeated roughly 50/50 patient-grouped train/test splits provided mean and standard-deviation performance estimates.Saliency maps used input-output gradients smoothed with a 5x5 Gaussian kernel, but offer only a restricted view of model reasoning.
3. Results
The single lung-opacity feature performed best among the tested feature sets, and its predictions generally aligned with expert scores on held-out data. Additional analyses examined representation structure and image regions influencing predictions.
- Quantitative performance: 0.80 correlation was achieved using the single lung-opacity output, compared with 0.79 using four selected outputs.The four outputs were lung opacity, pneumonia, infiltration, and consolidation.
- Quantitative performance: The single lung-opacity model was selected for subsequent qualitative analysis because it was the top-performing model.
- Qualitative analysis: Most held-out predictions fell near the line of unity when compared with blinded-expert ground-truth scores.The model overestimated scores from 1 to 3 and underestimated scores above 4.
- Learned representations: t-SNE projected learned features into two dimensions, where survival and deceased cases tended to form separate clusters.The authors state that this clustering indicates alignment between score predictions and clinical outcomes.
- Saliency analysis: Saliency maps showed that the model usually focused on opaque lung regions in unseen images.One opacity-free example instead focused on the heart and diaphragm, likely as a color reference, while other examples showed erroneous predictions.
4. Discussion
The model uses CXR-derived representations to predict severity, but evaluation is constrained by limited data and uncertain generalization to new cohorts.
- CXR-based severity models could complement existing clinical-indicator models and support decisions from CXR rather than CT.
- Limited labelled COVID-19 data creates challenges for labelling, inter-rater agreement, and learning representations that generalize to new images.
- t-SNE projects pneumonia-specific features into 2D, placing images with similar model outputs near one another while displaying geographic extent predictions.Features were extracted for all 208 dataset images.
- The model produced both correct and incorrect geographic-extent predictions, with saliency maps showing which image regions influenced outputs.Examples include assigned scores of 5 and 0 with predictions of 5.3 and -0.8, respectively.
- The small sample prevents proper cohort selection, exposes evaluation to sample bias, and limits robust large-scale evaluation.
- New cohorts with the same severity labels are needed to assess model generalization, while the radiographic scores lack corresponding functional or clinical outcomes.
5. Conclusion
COVID-19 prediction tools require robust evaluation because reported performance may be optimistic and AI systems do not replace clinical judgment. The model’s severity score may support care escalation, de-escalation, and treatment monitoring, particularly in the ICU.
- COVID-19 prediction models require robust evaluation across practical axes before premature hospital implementation.
- The model’s severity estimates could support escalation or de-escalation of care and treatment-efficacy monitoring, especially in the ICU.