PaperlayerView exact revision on arXiv

Source-linked AI summary

Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans

Michael Roberts, Derek Driggs, Matthew Thorpe, Julian Gilbey, Michael Yeung, Stephan Ursprung, Angelica I. Aviles-Rivero, Christian Etmann, Cathal McCague, Lucian Beer, Jonathan R. Weir-McCall, Zhongzhao Teng, Effrossyni Gkrania-Klotsas, James H. F. Rudd, Evis Sala, Carola-Bibiane Schönlieb

arXiv:2008.06388v4cs.LGcs.CVeess.IVstat.ML

TL;DR

The clinical utility of machine-learning models for COVID-19 diagnosis and prognosis from chest radiographs and CT remained unclear. This systematic review assessed the literature and found that none of the reviewed models met the robustness and reproducibility threshold for clinical use.

  • Problem

    Despite rapid development of imaging-based COVID-19 machine-learning models, their methodological quality and potential clinical utility remained unclear.

    Source: Introduction · p. 2

  • Method

    The review assessed imaging-model papers for risk of bias and used a documentation quality screen before detailed review.

    Source: Introduction · p. 2

  • Results

    None of the reviewed models reached the robustness and reproducibility threshold needed for clinical practice, amid widespread methodological and reporting deficiencies.

    Source: Discussion · p. 6, Conclusions · p. 9

  • Takeaways & Limitations

    The authors identify higher-quality datasets, reproducible documentation, and external validation as requirements for models to advance toward clinical trials.

    Source: Conclusions · p. 9

    For 38 of 61 papers, the review could not assess predictor bias because deep-learning predictors were unknown abstract imaging features.

    Source: Results · p. 5

Abstract

from arXiv · show

Machine learning methods offer great promise for fast and accurate detection and prognostication of COVID-19 from standard-of-care chest radiographs (CXR) and computed tomography (CT) images. Many articles have been published in 2020 describing new machine learning-based models for both of these tasks, but it is unclear which are of potential clinical utility. In this systematic review, we search EMBASE via OVID, MEDLINE via PubMed, bioRxiv, medRxiv and arXiv for published papers and preprints uploaded from January 1, 2020 to October 3, 2020 which describe new machine learning models for the diagnosis or prognosis of COVID-19 from CXR or CT images. Our search identified 2,212 studies, of which 415 were included after initial screening and, after quality screening, 61 studies were included in this systematic review. Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases. This is a major weakness, given the urgency with which validated COVID-19 models are needed. To address this, we give many recommendations which, if followed, will solve these issues and lead to higher quality model development and well documented manuscripts.

Introduction

The review assesses COVID-19 imaging models’ readiness for clinical use and recommends reducing bias and strengthening reproducibility, validation, and clinically grounded evaluation.

Introduction

  • The review extends earlier work by focusing on imaging-based machine-learning diagnosis and prognosis, assessing bias and quality, and giving recommendations across five domains.The domains cover public imaging datasets, algorithm development, reproducibility, manuscript documentation, and peer review; the review included published and preprint work for broader coverage.Source: Introduction · pp. 2–3

Results

  • Of 2,212 initially identified papers, 61 remained after screening and quality review: 37 deep-learning, 22 traditional machine-learning, and two hybrid papers.The screening retained 415 papers after title or abstract review and 319 after full-text screening.Source: Results · p. 3
  • COVID-19 diagnostic models used CXR and CT with varied architectures and tasks; reported performance varied across studies, limiting direct comparison across different datasets, partitions, and class counts.CXR model accuracies ranged from 0·88 to 0·99; CT deep-learning AUCs ranged from 0·70 to 1·00, and traditional CT machine-learning accuracies ranged from 0·76 to 0·98.Source: Results · p. 4
  • 54/61 papers had high risk of bias in at least one domain, while the remaining papers had unclear risk in at least one domain.The assessment covered participants, predictors, outcomes, and analysis.Source: Results · p. 4

Discussion

  • None of the reviewed imaging-model literature met the robustness and reproducibility threshold for clinical use, amid recurring data, methodological, reporting, and study-design deficiencies.The review assessed machine-learning diagnostic and prognostic models using medical imaging and applied specialized quality metrics for radiomics and deep learning.Source: Discussion · p. 6
  • Dataset risks included possible duplicate or unconfirmed cases in public repositories, source and demographic confounding, and overlap in repackaged “Frankenstein” datasets.The review warns that source differences can allow models to distinguish image origins rather than disease, and that shared public datasets may encourage community-wide overfitting.Source: Discussion · pp. 6–7
  • The review highlights biased or unrepresentative cohorts, weak validation and evaluation practices, and insufficiently justified transfer-learning and image-resizing choices as methodological concerns.It recommends reporting dataset age and sex distributions and cautions that accuracy, sensitivity, or specificity alone cannot show that a model outperforms RT-PCR when RT-PCR defines the ground truth.Source: Discussion · pp. 6–7
  • CXR models may learn acquisition view rather than disease severity, while undocumented imaging-to-test timing, imperfect RT-PCR negatives, and predictor-outcome overlap threaten validity.The review also cautions that models trained on overrepresented severe disease may miss the early-stage cases where diagnosis is most useful.Source: Discussion · p. 7
  • Evaluation should use adequately sized external cohorts, calibration, decision-curve analysis, patient-level splits, confidence intervals, interpretable predictions, and clinically selected thresholds assessed on validation and test data.The authors advise against comparing model performance directly with RT-PCR or other ground truths; thresholds should be chosen using development data, not test data alone.Source: Discussion · pp. 7–8
  • The authors recommend versioned datasets and code, documented preprocessing and training details, established reporting checklists, and reviewers who assess both clinical and technical risks.They also recommend sharing random seeds and data partitions, reporting robustness analyses and cohort characteristics, and seeking clarification when bias is unclear.Source: Discussion · p. 8
  • Public COVID-19 datasets were too small or poor-quality for reliable models, and studies using them had high or unclear bias risk; the authors favor systematic data review and prospective clinical evaluation.They stress that effective translation requires a clear clinical need, complementary clinical and computational expertise, and high-quality healthcare data.Source: Discussion · p. 8

Conclusions

  • None of the reviewed models was a likely candidate for clinical translation, amid methodological flaws and bias that produced overly optimistic performance.The authors call for higher-quality datasets, reproducible documentation, and external validation before models proceed to clinical trials and independent technical and clinical validation.Source: Conclusions · p. 9

Methods

  • The registered review followed PRISMA and searched COVID-19 literature databases from January 1 through October 3, 2020, with repeated updates and staged screening.Two reviewers independently screened titles and abstracts and full texts, resolving disagreements by consensus; the authors used the latest available version of duplicate or updated papers.Source: Methods · pp. 9–10
  • The review applied CLAIM or RQS documentation thresholds, assessed bias with PROBAST, and extracted study design, data, cohort sizes, validation, performance, and code availability.Papers were assessed on what was documented in the manuscript, not details inferable from online code, to judge whether methods and results were independently reproducible.Source: Methods · p. 10

average

These parts report internal-holdout validation results and performance metrics for CXR and CT diagnosis models, with CT entries also covering external validation and severity-model results.

average

  • CXR results included AUC: 0.92, Sensitivity: 0.88, Specificity: 0.79, and Wang et al. 29 reported AUC: 1.00, Accuracy: 0.99.Source: average · pp. 22–23

PPV: 1·00

  • CT diagnosis results included AUC: 0·99, Sensitivity, 1·00, Specificity, 0·99, while another result reported Sensitivity: 0·98, Specificity: 0·28.Source: PPV: 1·00 · pp. 26–27
  • Wang et al. 45 reported internal validation AUC: 0·93 [0·90,0·96] and external validation AUC: 0·81 [0·71,0·84].Source: PPV: 1·00 · p. 26
  • CT severity results included a model correlation of 0.98 and AUC: 0.91, Accuracy: 0.90, Sensitivity: 0.83, Specificity: 0.90.Source: PPV: 1·00 · p. 28
  • Sensitivity 0·89 and Specificity 0·92; AUC 0·90, Sensitivity 0·86, Specificity 0·81; and [0·84, 0·94], Sensitivity 0·79, Specificity 0·91 were reported.Source: PPV: 1·00 · pp. 28–29
  • AUC 0·94, Accuracy 0·88, Sensitivity 0·91, and Specificity 0·83 were reported.Source: PPV: 1·00 · p. 29
  • [0·77, 0·90] and Specificity 0·83 were reported; Accuracy 0·76, Sensitivity 0·74, and Specificity 0·79 were also reported.Source: PPV: 1·00 · p. 29

Wang et al. 54

The Wang et al. 54 excerpt mentions features and clinical factors and reports sample sizes for three validation cohorts.

Wang et al. 54

  • The method passage mentions features and clinical factors, but does not specify their role.Source: Wang et al. 54 · p. 30
  • 226 images (102 COVID-19), 161 images (92 COVID-19), and 709 images (560 COVID-19) are reported across three validation cohorts.Source: Wang et al. 54 · p. 30
  • The excerpt also refers to internal holdout and external validation, without providing further details.Source: Wang et al. 54 · p. 30

Yue et al. 83

The section concerns COVID-19 prognosis, including progression to a severe stage and hospital stay.

  • The prognosis topics include whether patients progress to severe COVID-19 and hospital stay.Source: Yue et al. 83 · p. 31

Zhu et al. 77

The excerpts report prognostic-model methods and validation metrics for COVID-19, including internal and external cohorts. They also describe a separate ICU-prognosis approach using radiomic features and clinical data.

Zhu et al. 77

  • Zhu et al. 77’s prognostic model targeted risk of death, need for ventilation, and a further requirement not specified in the excerpt; reported sensitivity was 1·00 and specificity was 0·89.The excerpts mention 47 patients of varying severity and internal validation, but do not fully specify how these details relate to the reported metrics.Source: Zhu et al. 77 · p. 31

NPV: 1·00,

  • The reported performance metrics were AUC 0·86±0·02, accuracy 0·86±0·02, sensitivity 0·77±0·03, and specificity 0·88±0·015.An internal validation excerpt describes 150 images, all COVID-19, including 48 with severe outcomes.Source: NPV: 1·00 · p. 31

AUC: 0·76

  • The external validation cohort comprised 135 patients, all with COVID-19, although the passage does not specify how many had the relevant outcome.Source: AUC: 0·76 · p. 31

AUC: 0·75

  • Chao et al. 79 addressed short-term prognosis of intubation and death using radiomic features and clinical data, with the excerpt also mentioning ventilation and ICU admission.Source: AUC: 0·75 · p. 32, Wu et al. 80 · p. 32

Wu et al. 80

The passage identifies prognostication for ICU admission as a task.

Zheng et al. 81

The supplied excerpts report multiple prognostic performance estimates, including weighted classification metrics, AUC, and C-index.

Chen et al. 82

The extracted results report internal-holdout performance, with metrics varying across reported evaluations, including discrimination and classification measures.

  • Accuracy was 0·88, sensitivity 0·55, and specificity 0·95 on internal holdout validation.Source: Chen et al. 82 · p. 33
  • AUC was 0·93, accuracy 0·91, sensitivity 0·81, and specificity 0·95 on internal holdout validation.Source: Chen et al. 82 · p. 33
  • Reported ranges were C-index [0.92, 0.95], accuracy [0.85, 0.87], sensitivity [0.71,0.76], and specificity [0.91,0.92].Source: Chen et al. 82 · p. 33

Appendix · 1. Search strategy

The search strategy first collected records from arXiv and Living Evidence on COVID-19, then applied topic and imaging-term filters. It differed slightly from the original PROSPERO description, and its search history and filtering code are available online.

1. Search strategy

  • For relevant date ranges, the initial arXiv extraction selected papers with “ncov” (as a complete word), “coronavirus”, “covid”, “sars-cov-2” or “sars-cov2” in their title or abstract; all Living Evidence on COVID-19 papers in the appropriate date range were downloaded.Source: §1 · p. 34
  • The refined search required title or abstract terms from both a machine-learning group—including “ai,” “deep,” “learning,” “machine,” “neural,” “intelligence,” “prognos,” “diagnos,” “classification” or “segmentation”—and an imaging group.The imaging group included “ct,” “cxr,” “x-ray,” “xray,” “imaging,” “image” or “radiograph”.Source: §1 · p. 34
  • Only “ai,” “ct” and “cxr” were required to appear as complete words in the refined search.This complete-word requirement applied to the listed terms, not to the other filter terms.Source: §1 · p. 34
  • The search differed slightly from the original PROSPERO description because it was widened to identify additional papers.Source: §1 · p. 34
  • The full history of the searching and filtering source code can be explored in the linked repository.Source: §1 · p. 34

Related papers

Loading 2008.06388v4…