Source-linked AI summary
A Critic Evaluation of Methods for COVID-19 Automatic Detection from X-Ray Images
Gianluca Maguolo, Loris Nanni
TL;DR
The paper examines whether commonly used testing protocols for automatic COVID-19 diagnosis from X-ray images fairly measure disease-related information. It evaluates protocols using multiple datasets and lung-masked images, finding that dataset-specific patterns can support recognition and motivating a fairness metric and improved evaluation methods.
Problem
Reportedly high COVID-19 X-ray classification performance may reflect dataset-specific patterns rather than medically relevant information.
Method
The paper evaluates multiple testing protocols across four datasets, including experiments that mask the lungs and protocols using held-out datasets or cross-validation.
Results
AlexNet recognized datasets without using the lungs, with the lowest binary ROC-AUC still reaching 0.92 for COV versus NIH.
Takeaways & Limitations
Testing protocols for automatic COVID-19 diagnosis can be biased toward source-dataset features, so protocol fairness should be evaluated explicitly.
Takeaways & Limitations
The preprocessing removed nearly all lung-health information as well as a large portion of general dataset information.
Abstract
from arXiv · showhide
In this paper, we compare and evaluate different testing protocols used for automatic COVID-19 diagnosis from X-Ray images in the recent literature. We show that similar results can be obtained using X-Ray images that do not contain most of the lungs. We are able to remove the lungs from the images by turning to black the center of the X-Ray scan and training our classifiers only on the outer part of the images. Hence, we deduce that several testing protocols for the recognition are not fair and that the neural networks are learning patterns in the dataset that are not correlated to the presence of COVID-19. Finally, we show that creating a fair testing protocol is a challenging task, and we provide a method to measure how fair a specific testing protocol is. In the future research we suggest to check the fairness of a testing protocol using our tools and we encourage researchers to look for better techniques than the ones that we propose.
1. INTRODUCTION
COVID-19 diagnosis from X-ray images has attracted substantial research, but reported high performance raises concerns about generalization and dataset-related bias.
- COVID-19 became a worldwide health problem after spreading globally in 2020.
- RT-PCR is the usual diagnostic method, while researchers have increasingly explored automatic diagnosis using X-ray images.
- Convolutional neural networks had already achieved high performance in diagnosing lung diseases before COVID-19 classification was studied.
- Many COVID-19 classification papers report very high performance, motivating scrutiny of whether their testing protocols measure medically relevant information.
2. RELATED WORK
Prior chest X-ray studies established strong classification performance but also raised concerns about dataset bias and limited cross-dataset generalization.
- Chest X-ray classification is an established deep-learning task with many datasets and high-performing neural networks.Rajpurkar et al. reported a 0.76 ROC-AUC for pneumonia-versus-healthy classification on one dataset.
- Potential biases in chest X-ray classification had already been reported before this study.
- Training and testing on the same chest X-ray dataset produced consistently higher performance than training on other datasets and testing on that dataset.
- COVID-19 studies commonly merged Cohen’s dataset with other chest X-ray datasets to create classification tests.
- Narin et al. used 50 COVID-19 and 50 healthy cases with five-fold cross-validation and obtained 98% accuracy using ResNet-50.
3. DATASETS
The study uses four publicly available X-ray datasets that differ substantially in size, labeling, patient population, and COVID-19 content.
- The study analyzes four publicly available datasets: NIH, CHE, KAG, and COV.
- NIH contains 108,948 images from 32,717 patients across eight potentially overlapping categories.
- CHE contains 224,316 radiographs from 65,240 patients divided into 14 classes, with the validation set used because its test set is unavailable.
- KAG contains 5,863 pediatric images from a Kaggle competition on viral and bacterial pneumonia classification.
- COV contains 144 frontal X-ray images potentially positive for COVID-19, with patient identifiers and occasional location or uploader metadata.
4. METHODS
The methods preprocess X-ray images by masking their centers and evaluate dataset recognition and COV recognition under controlled cross-validation protocols.
- Preprocessing: Images are resized, centrally masked with black squares of 240, 270, or 300 pixels, then converted to 227×227 inputs for AlexNet.The transformation hides most lung regions while retaining outer image areas.
- Preprocessing: The transformed samples shown in Figure 5 illustrate the original and centrally masked images for COV, NIH, CHE, and KAG.The figure uses a 300-sized black square and presents the datasets from left to right in that order.
- Dataset splitting: COV is divided into 11 cross-validation folds using either patient-disjoint PAT-OUT or doctor-grouped DOC-OUT protocols.DOC-OUT uses metadata to place scans uploaded by the same doctor in one fold, although upload information is incomplete.
- Experiments: Dataset recognition trains AlexNet to classify four source-dataset labels using ten COV folds and subsets of NIH, CHE, and KAG, testing on the held-out fold and dataset test sets.The protocol is repeated once for each COV fold and results are merged.
- Experiments: COV recognition leaves one of NIH, CHE, or KAG out for testing, trains on the other two datasets plus ten COV folds, and evaluates COV versus non-COV classification.The experiment is repeated for each left-out dataset and under both PAT-OUT and DOC-OUT protocols, using ROC-AUC.
5. RESULTS
Removing most lung information did not prevent strong dataset recognition, indicating that dataset-specific patterns can dominate COVID-19 classification results. The COVID-recognition results were lower and did not clearly favor DOC-OUT over PAT-OUT, while constructing a fair protocol remained difficult.
- Dataset recognition: 0.92 ROC-AUC was the lowest value for COV versus NIH dataset recognition despite removing most lungs.The authors report that AlexNet remained highly capable of recognizing datasets from outer-image features.
- Dataset recognition: All datasets were accurately predicted in the 300-square confusion matrix, except that COV was recognized less reliably.Large AUCs involving COV appear to reflect non-COV samples rarely being classified as COV.
- Dataset recognition: The 270-square experiment produced little difference from the previous experiment, while the larger information amount allowed better performance.The corresponding confusion matrix is reported in Table IV.
- Dataset recognition: The 240-square experiment produced a more promising confusion matrix, with COV appearing to be the most confused dataset.The preprocessing used a smaller central square while cutting the upper and lower image parts, leaving more black pixels.
- COVID recognition: COVID recognition produced much lower ROC-AUC values, with no clear evidence that DOC-OUT outperformed PAT-OUT.KAG versus COV performed worse than random, and nearly all samples in this protocol were labeled COV.
- Protocol fairness: The authors conclude that dataset differences are large enough to make a fair protocol based on the considered merged datasets very difficult.They suggest validating other repositories and using left-out datasets where feasible, while noting that their preprocessing also removes general dataset information.
6. CONCLUSION
The paper argues that common COVID-19 X-ray testing protocols may be biased toward source-dataset features rather than relevant medical information. It proposes solutions and a metric for evaluating protocol bias, while leaving broader dataset integration for future work.
- Conclusion: The authors report that usual COVID-19 X-ray testing protocols may learn source-dataset features rather than relevant medical information.They discuss protocol validity and characterize these protocols as potentially biased.
- Conclusion: The paper suggests solutions for developing new testing protocols and introduces a method to evaluate their bias.The authors state that, to their knowledge, they are the first to provide such a metric.
- Conclusion: Future work will include new COVID-19 datasets and examine whether merging many datasets can produce a more diverse, less biased dataset.