Source-linked AI summary
Finding Covid-19 from Chest X-rays using Deep Learning on a Small Dataset
Lawrence O. Hall, Rahul Paul, Dmitry B. Goldgof, Gregory M. Goldgof
TL;DR
Limited and imperfect COVID-19 testing motivates evaluating chest X-rays as a rapid diagnostic source. The paper uses transfer learning with pretrained convolutional networks on balanced COVID-19 and pneumonia datasets, evaluating cross-validation and unseen cases. Results indicate preliminary feasibility, but the small dataset and missing disease-stage information limit the conclusions and motivate collecting more images.
Problem
Limited testing capacity, delays, and a projected false negative rate of up to 30% motivate investigating chest X-rays for COVID-19 diagnosis.
Method
The study applies transfer learning with pretrained VGG16 and Resnet50 convolutional networks to chest X-rays from COVID-19 and other pneumonia cases.
Results
The model achieved 89.2% overall accuracy in cross-validation and, on unseen cases, correctly identified 26 of 33 COVID-19 cases with a 93.12% true negative rate.
Takeaways & Limitations
The results indicate that deep-learning models applied to chest X-rays may, with more data, help diagnose COVID-19.
Takeaways & Limitations
The study is limited by its small number of cases, unknown COVID-19 disease stage, reduced image resolution, and missing outcome information.
Abstract
from arXiv · showhide
Testing for COVID-19 has been unable to keep up with the demand. Further, the false negative rate is projected to be as high as 30% and test results can take some time to obtain. X-ray machines are widely available and provide images for diagnosis quickly. This paper explores how useful chest X-ray images can be in diagnosing COVID-19 disease. We have obtained 122 chest X-rays of COVID-19 and over 4,000 chest X-rays of viral and bacterial pneumonia. A pretrained deep convolutional neural network has been tuned on 102 COVID-19 cases and 102 other pneumonia cases in a 10-fold cross validation. The results were all 102 COVID-19 cases were correctly classified and there were 8 false positives resulting in an AUC of 0.997. On a test set of 20 unseen COVID-19 cases all were correctly classified and more than 95% of 4171 other pneumonia examples were correctly classified. This study has flaws, most critically a lack of information about where in the disease process the COVID-19 cases were and the small data set size. More COVID-19 case images will enable a better answer to the question of how useful chest X-rays can be for diagnosing COVID-19 (so please send them).
1 Introduction
Limited COVID-19 testing motivates investigation of chest X-rays as a faster, widely available diagnostic approach. The paper evaluates whether transfer learning can distinguish COVID-19 from other pneumonia using chest X-rays.
- Motivation: Testing shortages, delays, and a projected false negative rate of up to 30% motivate timely alternatives for identifying COVID-19.Nasopharyngeal-swab testing is described as invasive and constrained by shortages of swabs, transport media, and reagents.
- Motivation: Chest X-rays provide relatively immediate, low-cost information using machines widely available in emergency and hospital settings.They can be interpreted without expert radiologists and do not increase aerosolization risk through respiratory probing.
- Related work: Prior chest-X-ray studies used relatively small datasets but reported good sensitivity to COVID-19.These findings provide preliminary evidence that chest X-rays may have diagnostic utility.
- Study aim: The study applies transfer learning with a pretrained deep CNN to distinguish COVID-19 from viral or bacterial pneumonia.The COVID-19 disease stage was unknown, although all COVID-19 images had abnormal chest X-rays.
- Study scope: The dataset and model results require caution because the COVID-19 images were few and mostly reduced-quality JPEGs.The authors state that feasibility may be greater when patients have breathing issues.
2 Data
The study assembled COVID-19 and pre-existing viral or bacterial pneumonia chest X-rays from multiple sources, separating cases acquired at different times for training and testing.
- Data sources: 135 COVID-19 cases came from three sources, while 320 pneumonia cases came from the NIH chest X-ray dataset.COVID-19 cases included 92 from source A, 33 from source B, and 10 from source C.
- Data split: The 92 source-A cases and 10 source-B cases available by March 23, 2020 were used for training.The remaining source-C cases and 23 later source-B cases formed the testing data.
- Data composition: The pneumonia cases were selected because viral and bacterial pneumonia were thought to be difficult to differentiate from COVID-19.The training set was balanced by using 102 COVID-19 cases and randomly selecting 102 pneumonia cases.
- Image characteristics: COVID-19 images were JPEGs with typical quality 0.9 and some lossy compression, whereas pneumonia images were PNGs without known reduced resolution.This difference motivated later preprocessing to reduce image-quality effects.
3 Experimental setup
The experiments use pretrained VGG16 and Resnet50 convolutional layers adapted to grayscale chest X-rays, with snapshot ensembles used to address limited data for validation and stopping.
- Model architecture: VGG16 and Resnet50 pretrained on Imagenet color images supplied unchanged convolutional layers for transfer learning.Each grayscale X-ray was resized to 224x224 and replicated across the three RGB channels.
- Model architecture: The VGG16 model replaced its final layers with global average pooling and a trainable classification component.Its convolutional blocks use 3x3 filters, while max-pooling layers use 2x2 windows.
- Optimization: RMSprop optimized binary cross-entropy with a maximum learning rate of 0.001.These settings define the reported training configuration.
- Validation: Resnet50 was evaluated with 10-fold cross-validation because it achieved the highest accuracy on unseen data.Horizontal flipping was the only augmentation, and seven stored models were averaged in a snapshot ensemble.
- Unseen-data experiment: A separate unseen-data experiment reserved 18 images from each class for validation before training snapshot ensembles.The experiment used later COVID-19 cases and remaining pneumonia cases with both Resnet50 and VGG16.
4 Results
Cross-validation and unseen-data experiments show preliminary ability to distinguish COVID-19 from other pneumonia using chest X-rays, with performance varying across evaluation settings.
- Cross-validation: 89.2% overall accuracy, 80.39% COVID-case identification, 0.99 TNR, and 0.95 AUC were obtained in 10-fold cross-validation.The evaluation used 102 COVID-19 and 102 other-pneumonia cases and produced one false positive.
- Unseen-data evaluation: 208 unseen pneumonia cases and 33 newly acquired COVID-19 cases were evaluated after matching pneumonia image compression to COVID-19 images.This preprocessing aimed to prevent image sharpness from unduly influencing predictions.
- Unseen-data evaluation: 26 of 33 COVID-19 cases were correctly identified, with a 93.12% true negative rate and 0.94 AUC.The evaluation used a 21-model ensemble combining Resnet50, VGG16, and a small CNN.
5 Discussion and Conclusions
The study finds chest X-rays may help diagnose COVID-19, but performance estimates remain preliminary because of limited and incomplete data. More representative, higher-quality COVID-19 images are needed to determine clinical usefulness.
- 6% false positive rate and 83.3 to 96% true positive rate were reported for the current CXR approach.The authors suggest changing the ratio of COVID-19 to non-COVID-19 examples could reduce false positives, but experiments require more data.
- Prior work also found diagnostic utility for chest X-rays, while CT studies used larger datasets and reported AUC=0.96 for COVID-19 identification.Related CXR studies used different formulations and datasets, including anomaly detection and transfer learning.
- 31% of COVID-19 patients in the cited Hong Kong study had no human-observable CXR findings on presentation, limiting this dataset's coverage.The algorithm was trained only on patients with CXR findings and therefore cannot detect disease in patients without such findings.
- 89.2% overall accuracy was obtained in 10-fold cross validation using 102 COVID-19 and 102 other-pneumonia cases.COVID-19 cases were correctly identified at 80.39% in this experiment.
- 78.79% true positive rate and 93.12% true negative rate were obtained on 33 unseen COVID-19 and 208 pneumonia cases.The model was built from all training data before evaluation on the unseen cases.
- The study's limitations include few cases, unknown disease stage and outcomes, reduced COVID-19 image resolution, and unknown sources of confusion.The authors identify collecting more COVID-19 images as the most important next step.