PaperlayerView exact revision on arXiv

Source-linked AI summary

A Critic Evaluation of Methods for COVID-19 Automatic Detection from X-Ray Images

Gianluca Maguolo, Loris Nanni

arXiv:2004.12823v4eess.IVcs.CVcs.LG

TL;DR

The validity of testing protocols for automatic COVID-19 diagnosis from X-rays is a concern because they may be biased. The paper compares protocols and proposes a way to evaluate bias, concluding that some may rely more on dataset source than relevant medical information.

  • Problem

    The paper asks whether usual testing protocols for automatic COVID-19 diagnosis from X-rays are valid, given concerns that they may be biased.

    Source: §6 · p. 8

  • Method

    The paper compares testing protocols for COVID-19 diagnosis and proposes solutions and a method for evaluating protocol bias.

    Source: §6 · p. 8

  • Results

    The authors report that usual testing protocols may be biased, with models predicting features tied more to dataset source than relevant medical information.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The authors propose evaluating protocol bias and plan to investigate whether merging many datasets could produce a more diverse, less biased dataset.

    Source: §6 · p. 8

Abstract

from arXiv · show

In this paper, we compare and evaluate different testing protocols used for automatic COVID-19 diagnosis from X-Ray images in the recent literature. We show that similar results can be obtained using X-Ray images that do not contain most of the lungs. We are able to remove the lungs from the images by turning to black the center of the X-Ray scan and training our classifiers only on the outer part of the images. Hence, we deduce that several testing protocols for the recognition are not fair and that the neural networks are learning patterns in the dataset that are not correlated to the presence of COVID-19. Finally, we show that creating a fair testing protocol is a challenging task, and we provide a method to measure how fair a specific testing protocol is. In the future research we suggest to check the fairness of a testing protocol using our tools and we encourage researchers to look for better techniques than the ones that we propose.

1. INTRODUCTION

The introduction argues that COVID-19 X-ray classifiers may exploit dataset-source cues rather than disease signals, making testing protocols potentially misleading. It proposes testing for this risk by predicting image sources after masking the lungs.

1. INTRODUCTION

  • Prior work warned that X-ray classifiers may learn dataset-specific rather than disease-specific features, despite high reported COVID-19 classification performance.The introduction situates this concern within the broader problem of generalizing across datasets and learning features independent of image content.Source: §1 · p. 1
  • With image centers blacked out, AlexNet distinguished images by source dataset at confidence much higher than reported for pneumonia detection when training and test sets shared datasets.The authors say this masks the lungs, or most of them, preventing disease detection from lung features; they note that COVID-19 samples in most papers come from one dataset.Source: §1 · p. 1
  • The authors recommend that COVID-19 dataset results in include a source-dataset baseline and caution against merging datasets with different labels.The baseline is intended to clarify how much information comes from the lung area.Source: §1 · p. 1

2. RELATED WORK

The reviewed literature spans conventional chest-X-ray classification and COVID-19-specific protocols, while raising concerns that dataset recognition and test design can affect reported performance.

2. RELATED WORK

  • Prior chest-X-ray studies raised bias concerns, and Cohen et al. found higher performance when training and testing on dataset A than when training on B, C, or D and testing on A.The datasets had the same labels; earlier work also reported 0.76 ROC-AUC for pneumonia-versus-healthy classification on.Source: §2 · p. 1
  • COVID-19 studies often combined the Cohen repository with other sources and used varied datasets and testing protocols.Narin et al. reported 98% accuracy with 5-fold cross-validation; Apostolopoulos et al. reported 93.48% accuracy on an unbalanced dataset using 10-fold cross-validation. Hemdan et al. used 5-fold cross-validation, while Pereira et al. introduced hierarchical classification and reported a 0.89 F1-Score.Source: §2 · p. 2
  • Wang et al. reported 92% accuracy for Covid-Net using a test set with 31 COVID-19 images and 100 each of pneumonia and healthy-lung images, with no patient overlap between test and training.Source: §2 · p. 2
  • Castiglioni et al. reported 0.80 ROC-AUC on an independent test set, while CORDA results ranged from random performance to 0.97 ROC-AUC across training and test datasets.Castiglioni et al. used a private dataset and argued their protocol avoided the dataset-recognition problem despite lower performance than other reported studies. Tartaglione et al. used a private dataset and explicitly tested networks across different training and test datasets.Source: §2 · p. 2
  • COVIDGR divided cases by severity and reported accuracies of 97%, 88%, and 66% for severe, moderate, and mild cases, respectively.Tabik et al. used a GAN-based classifier intended to prevent learning source-dataset features; the network could not distinguish asymptomatic patients.Source: §2 · p. 2

3. DATASETS

The study uses four publicly available datasets.

3. DATASETS

  • The study uses four publicly available datasets.The section introduces these datasets for description.Source: §3 · p. 2

A. NIH dataset

The NIH dataset is a large public labeled chest X-ray collection with 108,948 images from 32,717 patients across eight potentially overlapping categories.

A. NIH dataset

  • NIH contains 108,948 images from 32,717 patients, classified into eight potentially overlapping categories.The Chestx-ray8 dataset was labeled using natural language processing applied to radiologists’ annotations.Source: A. NIH dataset · p. 2

B. CHE dataset

The CHE dataset comprises 224,316 chest radiographs from 65,240 patients across 14 classes, with labels produced using a natural-language-processing tool. Because its test set is unavailable, the study uses the validation set, which the authors say is sufficient for their purposes because they do not compare against previous papers.

B. CHE dataset

  • CHE contains 224,316 chest radiographs from 65,240 patients, divided into 14 classes.The authors refer to the dataset as CHE.Source: B. CHE dataset · p. 3
  • Chexpert’s natural-language-processing labeling tool achieves higher performance than the tool used for Chestx-ray8.The authors identify this as a strength of the dataset.Source: B. CHE dataset · p. 3
  • Because Chexpert’s test set is not publicly available, the study uses its validation set, which the authors consider sufficient since they do not compare with previous papers.The paper refers to Chexpert as CHE.Source: B. CHE dataset · p. 3

C. KAG dataset

The paper describes KAG as a pediatric pneumonia dataset and COV as its source of COVID-19 X-ray images.

C. KAG dataset

  • KAG contains 5,863 pediatric images from a Kaggle competition on viral and bacterial pneumonia classification started in 2017.The paper says this makes KAG very different from the other datasets.Source: C. KAG dataset · p. 3

D. COV dataset

  • COV draws on Cohen et al.’s repository, which contained 144 frontal X-ray images of potentially COVID-19-positive patients when the paper was written.Each sample has metadata including patient ID and, most of the time, image location and uploader-related notes.Source: C. KAG dataset · p. 3

4. METHODS

The methods use two AlexNet experiments—dataset recognition and COVID-19 recognition—to examine testing protocols after obscuring most lung information. Both use combinations of four datasets, with COV split into folds for cross-validation.

4. METHODS

  • Images were resized with their smallest dimension set to 360, had a central 240-, 270-, or 300-sized square blacked out, and were then resized to 227 × 227 for AlexNet.The authors state that this hid most of the lungs; only Pneumonia, No Finding, and COVID-19 samples were included, except that all classes were used for the CheXpert test set.Source: §4 · p. 4
  • COV was divided into 11 folds because it lacked a train-test split, using PAT-OUT to avoid patient overlap or DOC-OUT to group scans by uploading doctor.DOC-OUT metadata were incomplete, and the authors note that the protocol could not be guaranteed unbiased; CheXpert’s validation set supplied its test images.Source: §4 · p. 4
  • For dataset recognition, AlexNet classified images into four source-dataset labels using DOC-OUT, with three runs varying the black-square size and image preprocessing.The third run used a 240-sized square, contrast limited adaptive histogram equalization, and cropping; class-vs-class ROC-AUC was computed across predictions from 11 networks because non-COV test samples appeared 11 times.Source: §4 · p. 5
  • Across both experiments, AlexNet was fine-tuned for 12 epochs with data augmentation, and training used 10 COV folds plus NIH, CHE, and KAG subsets at a two-to-one ratio of other-dataset to COV samples.The learning rate was 0.0001 except for 0.0002 in the last fully connected layer; augmentation used random vertical flips, translations from −5 to 5, and rotations from −5 to 5.Source: §4 · p. 5
  • For COVID-19 recognition, the authors left one NIH, CHE, or KAG dataset out for testing, trained on the other two plus 10 COV folds, and tested with the remaining COV fold under PAT-OUT and DOC-OUT.They repeated the experiment for each COV fold and each choice of left-out dataset, classifying samples as COV or non-COV and evaluating ROC-AUC; the protocol was similar to Cohen’s.Source: §4 · p. 5

5. RESULTS

Even with most lung information removed, AlexNet distinguished the datasets, while COVID recognition was weaker and dataset differences made fair evaluation difficult. The results suggest that dataset-specific features can drive classification independently of lung health.

Dataset recognition

  • 0.92 was the lowest ROC-AUC, for COV vs. NIH, showing that AlexNet could distinguish datasets without using the lungs.All datasets were accurately predicted, though COV was less well recognized; large AUCs involving COV appeared to reflect other datasets rarely being classified as COV. CHE’s strong contrast may be a recognizable dataset-specific feature.Source: §5 · pp. 5–6, Table I · p. 6, Table II · p. 6
  • The 270-square experiment showed little difference from the 300-square case, although the larger amount of information allowed AlexNet to perform better.The confusion matrix also showed this improvement.Source: §5 · p. 6, Table III · p. 6, Table IV · p. 6
  • With preprocessing at size 240, the confusion matrix was more promising than in prior experiments, and COV was the most confused dataset.The upper and lower image portions were cut, leaving more black pixels than in the larger-square experiments.Source: §5 · p. 6, Table VI · p. 7, Table V · p. 7

COVID recognition

  • COVID-recognition ROC-AUC values were much lower than in the dataset-recognition experiments, with little evidence that DOC-OUT outperformed PAT-OUT.KAG vs. COV performed worse than random, possibly because KAG differed substantially from NIH and CHE, the non-COV training samples.Source: §5 · p. 7, Table VII · p. 7
  • Nearly all samples were labelled COV in this protocol, and t-SNE showed COV more similar to other datasets than those datasets were to one another.The authors hypothesize that COV lacks distinctive learnable features while other datasets have them; this helps explain why leaving KAG out performed worse than random.Source: §5 · p. 7, Figure 7 · p. 8

Implications and limitations

  • Fair evaluation across the considered datasets may be difficult; a held-out dataset can be far from the non-COVID training data, and removing lung information also removes dataset information.The experiments therefore set a minimum ability to recognize datasets. The authors suggest finding a dataset with features similar to COV or preprocessing away dataset-dependent features; more image sources may enable left-out-dataset testing.Source: §5 · p. 7

6. CONCLUSION

The paper argues that common COVID-19 X-ray testing protocols may be biased, learning source-dataset features more than relevant medical information. It proposes ways to find a new protocol and evaluate its bias, and identifies broader dataset integration as future work.

6. CONCLUSION

  • Usual testing protocols may be biased, predicting features tied more to the source dataset than to relevant medical information.The authors discuss the validity of protocols used in papers on automatic COVID-19 diagnosis.Source: §6 · p. 8
  • The authors suggest solutions for finding a new testing protocol and propose a metric to evaluate its bias, which they say is the first of its kind to their knowledge.Source: §6 · p. 8
  • Future work will add COVID-19 datasets and investigate whether merging many datasets can produce a new, diverse, less-biased dataset.Source: §6 · p. 8

Related papers

Loading 2004.12823v4…