PaperlayerView exact revision on arXiv

Source-linked AI summary

Unveiling COVID-19 from Chest X-ray with deep learning: a hurdles race with small data

Enzo Tartaglione, Carlo Alberto Barbano, Claudio Berzovini, Marco Calandri, Marco Grangetto

arXiv:2004.05405v1eess.IVcs.CVcs.LG

TL;DR

The paper asks whether widely available CXR can support COVID-19 screening despite limited labelled data and potentially confounding dataset biases. It evaluates a deep-learning pipeline across available datasets, including a medium-sized hospital dataset, and concludes that apparent COVID-related CXR performance requires careful interpretation because generalization can remain poor.

  • Problem

    Limited labelled COVID-CXR data and possible hidden dataset biases constrain reliable evaluation of automated COVID classification.

  • Method

    The study evaluates a deep-learning CXR pipeline using preprocessing, lung segmentation, transfer learning, dataset balancing, and testing across different datasets.

  • Results

    CXR severity peaked 10–12 days after symptom onset, while experiments showed that COVID classification can achieve apparently strong test results yet generalize poorly across data sources.

  • Takeaways & Limitations

    Extracting a COVID feature from CXR is not easy, so strong test performance should not be interpreted as reliable generalization to new data in the same domain.

Abstract

from arXiv · show

The possibility to use widespread and simple chest X-ray (CXR) imaging for early screening of COVID-19 patients is attracting much interest from both the clinical and the AI community. In this study we provide insights and also raise warnings on what is reasonable to expect by applying deep-learning to COVID classification of CXR images. We provide a methodological guide and critical reading of an extensive set of statistical results that can be obtained using currently available datasets. In particular, we take the challenge posed by current small size COVID data and show how significant can be the bias introduced by transfer-learning using larger public non-COVID CXR datasets. We also contribute by providing results on a medium size COVID CXR dataset, just collected by one of the major emergency hospitals in Northern Italy during the peak of the COVID pandemic. These novel data allow us to contribute to validate the generalization capacity of preliminary results circulating in the scientific community. Our conclusions shed some light into the possibility to effectively discriminate COVID using CXR.

I. INTRODUCTION

The introduction frames CXR as accessible but intrinsically limited for COVID-19 assessment, motivating deep learning while emphasizing that reliable conclusions require larger, better-controlled datasets.

  • Early diagnosis can support patient treatment, disease monitoring, and prevention of COVID-19 spread.
  • Imaging findings can appear early but are nonspecific and overlap with other viral infections, while routine CT use is logistically challenging.
  • CXR can be repeated, performed bedside with portable devices, and used to assess lung involvement while limiting exposure to healthcare workers and other patients.
  • 69% sensitivity was reported for X-ray, with CXR severity peaking 10–12 days after symptom onset.
  • CXR findings may be subtle and affected by patient characteristics, positioning, projection, and medical devices, challenging radiological interpretation.
  • The study investigates whether deep learning can identify COVID-related CXR fingerprints despite few labelled images and introduces the 386-patient CORDA dataset.

II. RELATED WORKS

Related work shows growing interest in COVID classification from CXR, but small datasets, hidden acquisition biases, and tiny test sets undermine confidence in apparent performance.

  • Prior SARS and pneumonia studies used segmentation, feature extraction, classical classifiers, texture features, or neural networks for radiographic classification.
  • Recent COVID-CXR studies explored CNN architectures and transfer learning, including feature extraction followed by SVM classification.
  • Transfer learning requires caution when the feature extractor’s source task is highly specific or contains dataset biases.
  • Limited COVID-positive data often lack metadata needed to identify biases related to age, gender, comorbidities, and acquisition settings.
  • Models may exploit scan settings or textual labels associated with datasets instead of learning lung-content features.
  • Extremely small test sets provide little statistical certainty about learning performance.

III. METHODOLOGY

The methodology uses a standard deep-learning pipeline that combines CXR preprocessing, lung segmentation, and transfer-learning-based classification, with preprocessing intended to remove dataset bias.

  • The proposed pipeline performs chest-image preprocessing and lung segmentation before classification with a transfer-learning model.
  • Preprocessing is treated as fundamental because neural networks can recognize dataset biases and use them to drive classification.
  • Larger datasets used for pretraining are especially important because COVID datasets are small.

A. Datasets

The study combines a newly collected hospital dataset with publicly available COVID and non-COVID CXR datasets, each serving distinct experimental purposes.

  • Three datasets are used in the experiments, with different purposes despite all containing chest X-ray images.
  • COVID-ChestXRay provides 137 PA CXRs, including 108 COVID-positive and 29 COVID-negative images, from a broader 287-image mixed-imaging collection.
  • CORDA contains 447 CXRs from 386 patients screened at a Piedmont reference hospital during 16–30 March 2020, including 297 positive and 150 negative images.
  • Montgomery County and Shenzhen datasets provide open-source abnormality or tuberculosis-control CXR data with manually or externally provided ground truths.
  • ChestXRay contains 5,857 images labeled normal, bacterial pneumonia, or viral pneumonia.
  • RSNA contains 26,684 scans, comprising 20,672 normal CXRs and 6,012 pneumonia cases.

B. Pre-processing

The pipeline preprocesses CXRs to reduce acquisition and non-lung biases before classification. It equalizes intensity, segments and blurs lung regions, and normalizes intensities to [0, 1].

  • B. Pre-processing: The preprocessing strategy targets biases that can arise when different classes belong to different datasets.The authors warn that a model might otherwise learn dataset distinctions instead of the intended classification task.
  • B. Pre-processing: Histogram equalization reduces bias from differing image dynamic ranges caused by acquisition and calibration factors.Radiographic contrast varies with subject contrast, receptor contrast, scatter radiation, and calibration.
  • B. Pre-processing: Lung segmentation discards non-lung content, potentially removing bias sources such as medical devices and embedded text.A U-Net is trained on Montgomery County and Shenzhen Hospital X-ray datasets, and masks are blurred with a 3 pixel radius.
  • B. Pre-processing: Image intensity is normalized to the range [0, 1].

C. Training

Training combines related-task pre-training, COVID-data fine-tuning, appropriate encoder sizing, balanced data, and independent testing. These choices address limited COVID data, overfitting, class imbalance, and patient-specific artifacts.

  • C. Training: Pre-training the convolutional feature extractor on a related CXR task can exploit larger datasets when COVID data are limited.The encoder is then fine-tuned on COVID data because similar-task features may not be optimal for COVID classification.
  • C. Training: Larger encoders are more prone to over-fit small datasets, making network complexity an important performance choice.
  • C. Training: Balancing training data is delicate because adding non-COVID images can make the model distinguish healthy from unhealthy lungs or dataset-specific biases.Negative cases should include varied conditions; selecting them poorly may prevent learning COVID features.
  • C. Training: Testing on data distinct from training data helps evaluate performance correctly and prevents learning a patient’s lung-shape feature.
  • C. Training: Validation data and regularization are also identified as training considerations.The paper treats these as general issues discussed extensively elsewhere.

IV. DISCUSSION

The experiments vary feature-extractor pre-training, training-set composition, and cross-dataset testing. Dataset composition and evaluation design are used to examine performance and possible hidden biases.

  • IV. DISCUSSION: The experiments evaluate pre-training, training-set composition, and testing on different datasets.Performance is reported across Tables II, III, and IV using multiple experiments.
  • IV. DISCUSSION: The source code is available in the project repository.
  • IV. DISCUSSION: Training uses 70% of each dataset for training and 30% for testing, with the training portion split into 80% training and 20% validation.COVID-ChestXRay instead uses 15 testing samples to match partitions used by other works.
  • IV. DISCUSSION: Training data are balanced between COVID-positive and COVID-negative cases by adding negatives where possible or subsampling the more populated class.The CORDA dataset is imbalanced toward positive COVID cases, motivating borrowing samples from public non-COVID datasets.
  • IV. DISCUSSION: Cross-dataset testing is used to observe possible hidden biases, with AUC, balanced accuracy, and diagnostic odds ratio evaluated.AUC aggregates performance across thresholds, while the other metrics use a 0.5 classification threshold.

A. To pre-train or not to pre-train?

The paper compares models with and without feature-extractor pre-training. Pre-training generally improves specificity and related performance measures, but the encoder still requires fine-tuning for the COVID task.

  • A. To pre-train or not to pre-train?: Pre-training on ChestXRay or RSNA yields higher balanced accuracy and diagnostic odds ratio than no pre-training when training on CORDA.
  • A. To pre-train or not to pre-train?: 0.80 specificity with RSNA pre-training exceeds 0.58 without pre-training on the CORDA test set.Sensitivity remains very similar, and specificity improvements generally extend to other test sets except ChestXRay.
  • A. To pre-train or not to pre-train?: Fine-tuning is typically required because a related-task pre-trained encoder is not guaranteed to produce optimal features for COVID classification.

B. Pre-training on different datasets

The study compares ChestXRay- and RSNA-pre-trained encoders for COVID classification, finding that the preferable pre-training dataset depends on the evaluation dataset and task.

  • ChestXRay and RSNA are evaluated as alternative pre-training datasets for COVID classification encoders.
  • For CORDA training, RSNA pre-training typically produces higher specificity than ChestXRay pre-training at the same sensitivity.
  • On COVID-ChestXRay, ChestXRay pre-training yields higher sensitivity and specificity than RSNA pre-training.
  • With little training data, pre-training introduces feature-selection priors and can substantially change performance across classification tasks.

C. Augmenting COVID- data with different datasets

Augmenting COVID data with different non-COVID datasets can produce very high same-dataset scores but much weaker cross-dataset generalization, indicating strong dataset-related bias.

  • 0.90 BA and 122.67 DOR on the matched CORDA&ChestXRay test set fall to 0.56 BA and 2.26 DOR on CORDA&RSNA.
  • 0.90 BA and 122.64 DOR on the matched CORDA&RSNA test set fall to 0.59 BA and 2.47 DOR on CORDA&ChestXRay.
  • Same-dataset specificity reaches 0.95 for CORDA&RSNA and 0.94 for CORDA&ChestXRay, while specificity on other datasets is extremely low.
  • t-SNE shows CORDA samples separating from ChestXRay samples regardless of COVID label, whereas CORDA and RSNA samples do not form clear clusters.
  • 0.56 BA and 1.64 DOR using CORDA alone improve to 0.62 BA and 2.93 DOR when COVID-ChestXRay is added, although specificity remains problematic.

D. How deep should we go?

With very small COVID datasets, increasing network depth does not reliably improve CXR classification and can worsen generalization. The comparison with COVID-Net further shows that strong same-dataset results may collapse on a separate dataset.

  • D. How deep should we go?: Deeper ResNet-50 can overfit the small available dataset, making the smaller ResNet-18 architecture safer.On CORDA&ChestXRay, DOR drops from 122.67 with ResNet-18 to 73.35 with ResNet-50.
  • D. How deep should we go?: The smaller Conv8 architecture offers no relevant training advantage over ResNet-18.On CORDA, Conv8 reaches BA 0.61 and DOR 2.38, versus BA 0.67 and DOR 4.78 for pretrained ResNet-18.
  • E. Comparison between deep networks trained on COVID-ChestXRay: COVID-Net achieves BA 0.85 and DOR 36.0 on its training dataset but falls to BA 0.55 and DOR 6.68 on CORDA.The corresponding sensitivity and specificity indicate that the model classifies almost all data as COVID on CORDA.
  • V. CONCLUSIONS: The study concludes that extracting a COVID feature from CXR is difficult and that excellent test-set performance may generalize poorly to new data.The authors argue that larger shared CXR datasets are needed to investigate whether CNN results can aid the COVID-19 response.
Loading 2004.05405v1…