Source-linked AI summary
Estimating Uncertainty and Interpretability in Deep Learning for Coronavirus (COVID-19) Detection
Biraja Ghoshal, Allan Tucker
TL;DR
Deep-learning diagnosis can be overconfident and lacks quantified uncertainty, an important gap for COVID-19 chest X-ray interpretation. The paper uses Dropweights-based Bayesian convolutional networks to estimate uncertainty and reports that uncertainty correlates with accuracy, supporting uncertainty-aware human-machine referral.
Problem
Deep-learning disease classifiers often lack quantified confidence despite the high cost of diagnostic errors, while COVID-19 detection requires timely chest X-ray assessment.
Method
The paper uses Monte-Carlo Dropweights Bayesian convolutional neural networks on publicly available COVID-19 chest X-ray datasets to estimate predictive uncertainty.
Results
Estimated uncertainty is higher for erroneous predictions and strongly correlates with classification accuracy, enabling identification of false or unknown cases.
Takeaways & Limitations
Uncertainty information can support referral of uncertain images to radiologists and improve human-machine diagnostic performance.
Abstract
from arXiv · showhide
Deep Learning has achieved state of the art performance in medical imaging. However, these methods for disease detection focus exclusively on improving the accuracy of classification or predictions without quantifying uncertainty in a decision. Knowing how much confidence there is in a computer-based medical diagnosis is essential for gaining clinicians trust in the technology and therefore improve treatment. Today, the 2019 Coronavirus (SARS-CoV-2) infections are a major healthcare challenge around the world. Detecting COVID-19 in X-ray images is crucial for diagnosis, assessment and treatment. However, diagnostic uncertainty in the report is a challenging and yet inevitable task for radiologist. In this paper, we investigate how drop-weights based Bayesian Convolutional Neural Networks (BCNN) can estimate uncertainty in Deep Learning solution to improve the diagnostic performance of the human-machine team using publicly available COVID-19 chest X-ray dataset and show that the uncertainty in prediction is highly correlates with accuracy of prediction. We believe that the availability of uncertainty-aware deep learning solution will enable a wider adoption of Artificial Intelligence (AI) in a clinical setting.
1 Introduction
Deep learning can perform strongly on medical imaging classification, but it often cannot express uncertainty in ambiguous cases. This paper evaluates uncertainty-aware Bayesian modeling for COVID-19 chest X-ray diagnosis and human-machine decisions.
- Deep learning achieves state-of-the-art performance across many medical imaging classification and reconstruction tasks, yet lacks an ability to say “I don’t know” in ambiguous cases.
- Estimating uncertainty is critical for adding insight to point predictions and improving reliability in medical-imaging decisions.
- Limited expert-labelled image data and time-consuming manual detection constrain existing COVID-19 screening infrastructure.
- The study evaluates Dropweights-based Bayesian convolutional neural networks for uncertainty estimation rather than pursuing state-of-the-art predictive performance.
- Monte-Carlo Dropweights uncertainty strongly correlates with classification accuracy and can identify false predictions or unknown cases.
2 Related Research
Prior work frames uncertainty estimation as an important but unresolved problem in deep learning. Bayesian and Monte-Carlo approximations provide several approaches, including methods based on dropout and alternative confidence measures.
- Uncertainty estimation in deep neural networks remains a challenging and unsolved problem with multiple proposed measures.
- Bayesian neural networks offer a natural framework for modeling uncertainty, but computing the posterior over network parameters is often intractable.
- Existing approaches include Monte-Carlo Dropout, Laplace approximation, MCMC variants, stochastic batch normalization, maximum softmax probability, heteroscedastic classifiers, and deep ensembles.
3 Approximate Bayesian Convolutional Neural Networks (BCNN) and Model Uncertainty
The paper approximates Bayesian convolutional inference with stochastic Dropweights passes, using predictive distributions to produce classifications and uncertainty estimates. It compares predictive entropy and BALD, and relates uncertainty to prediction error through distributional measures.
- The method models predictive distributions by marginalizing neural-network parameters over the training data and test input.
- Because the posterior over parameters is computationally intractable, MC Dropweights approximates model uncertainty by averaging stochastic forward passes during inference.
- The predictive mean across MC iterations supplies the final test-sample prediction, with the largest predictive mean selecting the output class.
- Predictive uncertainty includes epistemic uncertainty from model parameters and limited training data, and aleatoric uncertainty from irreducible observation noise.
- The study estimates epistemic uncertainty using predictive entropy and compares Predictive Entropy with BALD for COVID-19 image classification.
- Predictive entropy measures uncertainty in the predictive distribution, whose value range depends on datasets, architectures, and MC-sampling configurations before normalization.
- BALD identifies inputs with high average model uncertainty by comparing predictive-distribution entropy with mean entropy across stochastic samples.
- The analysis evaluates how uncertainty relates to true error using Spearman correlation and 1-Wasserstein distance, which compares real and estimated distributions.
4 Dataset
The dataset combines COVID-19 posterior-anterior chest X-rays with pneumonia and normal chest radiographs. The resulting collection contains 5,941 images across four classes, with relatively few COVID-19 examples.
- Radiologists use X-rays to assess lung inflammation, enlarged lymph nodes, and pneumonia, but interpretation requires expertise and significant time.
- The dataset contains 5,941 posterior-anterior chest radiographs across Normal, Bacterial Pneumonia, non-COVID-19 Viral Pneumonia, and COVID-19 classes.
- COVID-19 cases contribute 68 images, compared with 1,583 Normal, 2,786 Bacterial Pneumonia, and 1,504 non-COVID-19 Viral Pneumonia images.
5 Experiment
The experiment uses transfer learning with a pretrained ResNet50V2, data augmentation, and uncertainty-aware asymmetric-cost training for COVID-19 X-ray classification. The loss heavily prioritizes avoiding false negatives, especially missed COVID-19 cases.
- Model and training: A pretrained ResNet50V2 was fine-tuned with added fully connected layers and drop-weights rather than training a deep model from scratch.This transfer-learning setup was chosen for limited labeled data in the experimental domain.
- Data preparation: Images were resized to 224 x 224 pixels, standardized using dataset statistics, split 80%–20% into training and testing sets, and augmented during training.Augmentation included preprocessing intended to prevent overfitting and improve learning.
- Asymmetric cost: The experiment treats false negatives as more costly than false positives because failing to detect COVID-19 is considered the more serious error.The stated objective is to avoid false-negative detection even when some false positives result.
- Asymmetric cost: A utility function with class weights was used to encode asymmetric mistake costs while approximating the posterior distribution over neural-network weights.The resulting weighted cross-entropy loss dictates predictions under the specified cost structure.
- Asymmetric cost: The class weights were Normal: 2, Bacterial: 2, Viral: 1, and COVID-19: 50 for the described misdiagnosis condition.The highest weight is assigned when a true COVID-19 case is misdiagnosed as non-COVID-19 with low uncertainty.
6 Results and Discussions
The Bayesian model estimates predictive uncertainty alongside COVID-19 chest X-ray classifications. Higher uncertainty is associated with erroneous predictions, and uncertainty-aware referral improves diagnostic performance for the remaining cases.
- Uncertainty-aware predictions: Predictive entropy was low for a correctly classified image (PH=0.0086) and higher for a misclassified image (PH=0.55332), with wider posterior distributions.Figure 1 illustrates the relationship between predictive uncertainty and individual prediction correctness.
- Bayesian model performance: Bayesian ResNet50V2 inference improved detection accuracy over standard ResNet50V2 inference on the sample X-ray dataset.The confusion matrix summarizes prediction accuracy for the implemented models.
- Bayesian model uncertainty: Estimated uncertainty was higher for erroneous than correct predictions across test images and dropweights settings.Predictive uncertainty was estimated from output distributions generated using stochastic feed-forward passes with dropweights enabled during testing.
- Bayesian model uncertainty: Predictive entropy was favored over BALD as an uncertainty measure, while incorrect classifications consistently exhibited higher uncertainty.The comparison varied the dropweights rate and the number of Monte Carlo samples, with multiple forward passes adding computational cost.
- Uncertainty and predictive accuracy: The correlation between predictive entropy and prediction errors was strong, with Spearman’s ρ = 0.99.The analysis evaluated the relationship between estimated uncertainty measures and prediction error.
- Uncertainty-aware classification and referral: For experienced radiologists, combined performance reached almost 90% after rejecting almost 40% of the most uncertain samples for further inspection.Prediction accuracy increased as more uncertain images were referred, while accuracy was computed only on non-referred images.
7 Visualizing Uncertainty and Interpretability
The paper presents uncertainty estimates alongside visual explanations to make deep-learning predictions easier to understand. It qualitatively compares saliency maps generated by several methods.
- Reliable uncertainty estimates and feature visualizations provide complementary insight into deep-learning predictions.The authors connect these tools with more informed decision-making.
- Figure 8 qualitatively compares saliency maps produced by CAM, Guided Backpropagation, Guided Gradient CAM, and Gradient-based methods.
8 Conclusion and Future work
The study trains a Bayesian deep-learning classifier with transfer learning on COVID-19 X-ray images to estimate model uncertainty. It reports a strong uncertainty–accuracy correlation and suggests future integration with multi-omics and treatment-response data.
- Transfer learning trains a Bayesian deep-learning classifier on COVID-19 X-ray images to estimate model uncertainty.
- The experiments show a strong correlation between model uncertainty and prediction accuracy.
- Correlating imaging findings with multi-omics datasets and treatment responses could provide further insights for COVID-19 diagnosis and treatment.