Source-linked AI summary
COVIDGR dataset and COVID-SDNet methodology for predicting COVID-19 based on Chest X-Ray images
S. Tabik, A. Gómez-Ríos, J. L. Martín-Rodríguez, I. Sevillano-García, M. Rey-Area, D. Charte, E. Guirado, J. L. Suárez, J. Luengo, M. A. Valero-González, P. García-Villanova, E. Olmedo-Sánchez, F. Herrera
TL;DR
COVID-19 CXR triage is constrained by limited access to CT and RT-PCR and by heterogeneous, severity-biased datasets. The paper constructs COVIDGR-1.0 and proposes COVID-SDNet, which combines smart-data processing with CNN inference. The approach reports strongest accuracy on severe and moderate cases, while performance is lower for mild and Normal-PCR+ cases.
Problem
Existing COVID-19 CXR datasets are heterogeneous, lack representative severity spectra, and are biased toward severe positive cases, limiting high-clinical-value diagnosis systems.
Method
The paper builds the homogeneous, balanced COVIDGR-1.0 dataset and proposes COVID-SDNet, combining segmentation, data augmentation, data transformation, CNN inference, and prediction fusion.
Results
97.72% ± 0.95% accuracy was obtained for severe cases, 86.90% ± 3.20% for moderate cases, and 61.80% ± 5.49% for mild cases.
Takeaways & Limitations
COVID-SDNet can be used in a triage system to detect especially moderate and severe patients.
Takeaways & Limitations
COVIDGR-1.0 is being enriched with images from more hospitals, and additional clinical information is planned for improving prediction performance.
Abstract
from arXiv · showhide
Currently, Coronavirus disease (COVID-19), one of the most infectious diseases in the 21st century, is diagnosed using RT-PCR testing, CT scans and/or Chest X-Ray (CXR) images. CT (Computed Tomography) scanners and RT-PCR testing are not available in most medical centers and hence in many cases CXR images become the most time/cost effective tool for assisting clinicians in making decisions. Deep learning neural networks have a great potential for building COVID-19 triage systems and detecting COVID-19 patients, especially patients with low severity. Unfortunately, current databases do not allow building such systems as they are highly heterogeneous and biased towards severe cases. This paper is three-fold: (i) we demystify the high sensitivities achieved by most recent COVID-19 classification models, (ii) under a close collaboration with Hospital Universitario Clínico San Cecilio, Granada, Spain, we built COVIDGR-1.0, a homogeneous and balanced database that includes all levels of severity, from normal with Positive RT-PCR, Mild, Moderate to Severe. COVIDGR-1.0 contains 426 positive and 426 negative PA (PosteroAnterior) CXR views and (iii) we propose COVID Smart Data based Network (COVID-SDNet) methodology for improving the generalization capacity of COVID-classification models. Our approach reaches good and stable results with an accuracy of $97.72\% \pm 0.95 \%$, $86.90\% \pm 3.20\%$, $61.80\% \pm 5.49\%$ in severe, moderate and mild COVID-19 severity levels (Paper accepted for publication in Journal of Biomedical and Health Informatics). Our approach could help in the early detection of COVID-19. COVIDGR-1.0 along with the severity level labels are available to the scientific community through this link https://dasci.es/es/transferencia/open-data/covidgr/.
1 Introduction
CXR is a fast, accessible assessment tool for COVID-19 triage, but existing datasets and reported model sensitivities are affected by heterogeneity and severity bias. The paper introduces COVIDGR-1.0 and COVID-SDNet to address these data and modeling challenges.
- Clinical motivation: CXR equipment is more available and transportable than CT or RT-PCR infrastructure, and testing takes about 15 seconds per patient.CT equipment may be unavailable and require more than 15 minutes per patient plus decontamination; RT-PCR requires at least 12 hours.
- Data and model gap: Existing COVID-19 CXR databases are highly heterogeneous and biased toward severe cases, limiting construction of clinically valuable triage systems.The paper also analyzes the reliability, potential, and limitations of commonly used datasets and models.
- Contributions: COVID-SDNet combines segmentation, data augmentation, data transformation, a suitable CNN, and an inference approach for COVID-19 classification.The methodology includes segmentation-based cropping, class-inherent transformations, and fusion of predictions from four transformed classes.
- Results: 97.72% ± 0.95% accuracy was achieved for severe cases, while moderate and mild cases reached 86.90% ± 3.20% and 61.80% ± 5.49%, respectively.Normal-PCR+ cases reached 28.42% ± 2.58% accuracy.
2 Related works
Existing COVID-19 CXR datasets combine heterogeneous sources, inconsistent annotation practices, and limited severity coverage, challenging clinically useful triage. Related CNN studies therefore use varied datasets and experimental setups, limiting direct comparison.
- Datasets: COVID-19 Image Data Collection contains 76 positive and 26 negative PA views from heterogeneous equipment worldwide.
- Datasets: COVIDx 2.0 reorganizes COVIDx 1.0 into Normal, Pneumonia, and COVID-19 classes using 201 COVID CXR images, including PA and AP views.
- Datasets: Existing datasets do not guarantee useful triage because annotation protocols are unclear, data are heterogeneous, and positive cases lack representative severity coverage.
- Datasets: CNN models can obtain similar results even after most lung regions are removed from input X-ray images, indicating reliance on non-COVID visual cues.
- Related CNN studies: Related studies are not directly comparable because they use different public-dataset combinations and experimental setups.
- Related CNN studies: COVIDNet and COVID-CAPS reported strong results on prior datasets, including accuracies of 92.6% and 98.7%, respectively.
- Explanation approaches: Explanation approaches combine CNN ensembles or classifiers with Grad-CAM++, layer-wise relevance propagation, saliency maps, and uncertainty estimates.
3 COVIDGR-1.0: Data acquisition, annotation and organization
COVIDGR-1.0 was constructed through radiologist–AI collaboration using a shared protocol for CXR selection and annotation. The resulting dataset contains balanced positive and negative PA images acquired under consistent imaging conditions.
- Data acquisition and annotation: COVIDGR-1.0 uses a small, smart dataset that can be augmented to improve model performance amid costly annotation and protected medical data.
- Data acquisition and annotation: Four trained radiologists established the selection and annotation protocol, defining positivity through RT-PCR and expert-radiologist confirmation within 24 hours.
- Data acquisition and annotation: Normal-PCR+ images are CXR views with positive PCR results that expert radiologists annotated as normal.
- Dataset organization: COVIDGR-1.0 contains 852 images divided equally into 426 positive and 426 negative cases.
- Dataset organization: All COVIDGR-1.0 images come from the same equipment and X-ray regime, use only the PA view, and include severity labels available to researchers.
4 COVID-SDNet methodology
COVID-SDNet produces smart CXR data through segmentation-based cropping and class-inherent transformations, then classifies four transformed classes with a CNN and fuses predictions for the original label.
- COVID-SDNet combines segmentation, data transformation, data augmentation, a CNN classifier, and an inference approach to improve COVID-19 classification from CXR images.
- CNN learning and inference: ResNet-50, initialized with ImageNet weights and fine-tuned, predicts four transformed classes: P+, P−, N+, and N−.
- Segmentation-based cropping: Segmentation-based cropping removes irrelevant image information by segmenting both lungs, adding a 2.5% border, and cropping the resulting bounding rectangle.
- Class-inherent transformations: Class-inherent transformations use two generators: GP maps images toward the positive class domain, while GN maps them toward the negative class domain.
- Class-inherent transformations: The generator loss combines pixel-wise mean-square error, perceptual mean-square error weighted by 0.006, and classifier loss weighted by λ.
- CNN learning and inference: The inference process fuses predictions from the transformed images to produce a binary P-or-N prediction for the original image.
5 Experiments and Results
Experiments evaluate COVIDNet, COVID-CAPS, FuCiTNet, baseline models, and COVID-SDNet on COVIDGR-1.0 using repeated cross-validation. COVID-SDNet generally provides the strongest and most stable overall performance, while severity-level accuracy varies substantially.
- Experimental setup: 25 executions average sensitivity, specificity, precision, F1, and accuracy using five 5-fold cross-validations on COVIDGR-1.0.Each experiment uses 80% for training, 20% for testing, and 10% of each training set for validation.
- COVIDNet and COVID-CAPS: COVIDNet and COVID-CAPS trained on COVIDx overestimate the COVID-19 class, producing high sensitivity but low positive predictive value on COVIDGR-1.0.Retraining on COVIDGR-1.0 slightly improves overall accuracy and balances sensitivity and specificity more closely, but the models still do not perform adequately.
- Results and analysis: COVID-SDNet achieves the highest balance between specificity and sensitivity, with 76.94 ± 2.82 F1 in the negative class and 75.71 ± 3.35 F1 in the positive class.It also achieves sensitivity of 72.59 ± 6.77 and accuracy of 76.18 ± 2.70; segmentation substantially improves baseline sensitivity.
- Analysis per severity level: 97.72% accuracy is achieved for Severe cases and 86.90% for Moderate cases, whereas Mild and Normal-PCR+ cases reach 61.80% ± 5.49% and 28.42% ± 2.58%, respectively.Moderate and Severe images contain more visual features, while Mild and Normal-PCR+ cases contain few or no visible features.
- Analysis per severity level: Removing Normal-PCR+ improves overall results across approaches but decreases COVID-SDNet accuracy for Mild and Moderate cases by 15.8% and 1.52%, respectively.The authors conclude that although Normal-PCR+ is hardest to predict, its presence improves accuracy for lower-severity levels, especially Mild.
6 Inspection of model’s decision
The model’s decision inspection uses Grad-CAM heatmaps and counterfactual explanations to identify image regions supporting the predicted or opposite class. The reported visualizations highlight clinically relevant CXR patterns and show complementary positive and negative interpretations.
- Interpretability method: Grad-CAM heatmaps highlight regions or pixels that led to the actual prediction, while counterfactual heatmaps highlight regions supporting the opposite class.Higher heatmap intensity indicates greater importance of the corresponding pixel in the decision.
- Interpretation of decisions: Positive and negative interpretations are complementary, with regions supporting the correct decision usually opposing regions supporting the negative decision.The heatmaps identify opaque regions associated with infiltrates, consolidations, and osteoarthritis across severity levels.
- Example explanations: Fig. 5(b) highlights right-lung infiltrates and spinal osteoarthritis, while Fig. 6(b) identifies moderate lower-lung infiltrates and aortic dilation.The examples connect model attention to specific radiographic findings in the illustrated CXR cases.
7 Conclusions
COVID-SDNet combines a clinically stratified COVIDGR-1.0 dataset with segmentation, augmentation, and transformation, supporting triage particularly for moderate and severe cases. The authors note that mild and Normal-PCR+ cases offer fewer visual features for improvement and propose expanding the dataset across hospitals.
- COVIDGR-1.0 covers four COVID-19 severity levels, while COVID-SDNet combines segmentation, data augmentation, and data transformation.
- Mild and Normal-PCR+ cases provide fewer opportunities for improvement because they contain few or no visual features.
- COVID-SDNet can be used in a triage system to detect especially moderate and severe patients.
- More robust and accurate triage may be built by fusing this approach with other approaches.
- Future work includes adding CXR images from different hospitals and incorporating additional clinical information.
Acknowlegments
The work received support from multiple research funding programs and institutions.
- The work was supported by DeepSCOP-Ayudas, COVID19 RX-Ayudas, and a Spanish Ministry of Science and Technology project.