PaperlayerView exact revision on arXiv

Source-linked AI summary

A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks

Angelos Filos, Sebastian Farquhar, Aidan N. Gomez, Tim G. J. Rudner, Zachary Kenton, Lewis Smith, Milad Alizadeh, Arnoud de Kroon, Yarin Gal

arXiv:1912.10481v1stat.MLcs.LGeess.IV

TL;DR

Realistic evaluation of Bayesian deep learning is difficult because existing benchmarks often omit real-world demands. The paper builds a diabetic-retinopathy benchmark with multiple tasks and compares tuned methods, finding that some UCI-successful techniques underperform against simpler baselines on the new benchmark.

  • Problem

    BDL lacks realistic benchmarks that let researchers evaluate robustness, scalability, and uncertainty methods against the demands of real-world applications.

    Source: §1 · p. 2

  • Method

    The paper builds a diabetic-retinopathy benchmark with expert-referral evaluation, distribution-shift and out-of-distribution tasks, tuned baselines, and a simple evaluation API.

    Source: §1 · p. 2

  • Results

    Some techniques that perform well on UCI fail to scale, solve the new benchmark, or provide good uncertainty estimates; ensembles and MC Dropout rank above MFVI in the diabetic-retinopathy task.

    Source: §1 · p. 2, §4 · p. 8

  • Takeaways & Limitations

    The benchmark provides a way to rank BDL techniques for scalability and effectiveness in a real-world diagnostic application.

    Source: §1 · p. 2

Abstract

from arXiv · show

Evaluation of Bayesian deep learning (BDL) methods is challenging. We often seek to evaluate the methods' robustness and scalability, assessing whether new tools give `better' uncertainty estimates than old ones. These evaluations are paramount for practitioners when choosing BDL tools on-top of which they build their applications. Current popular evaluations of BDL methods, such as the UCI experiments, are lacking: Methods that excel with these experiments often fail when used in application such as medical or automotive, suggesting a pertinent need for new benchmarks in the field. We propose a new BDL benchmark with a diverse set of tasks, inspired by a real-world medical imaging application on \emph{diabetic retinopathy diagnosis}. Visual inputs (512x512 RGB images of retinas) are considered, where model uncertainty is used for medical pre-screening---i.e. to refer patients to an expert when model diagnosis is uncertain. Methods are then ranked according to metrics derived from expert-domain to reflect real-world use of model uncertainty in automated diagnosis. We develop multiple tasks that fall under this application, including out-of-distribution detection and robustness to distribution shift. We then perform a systematic comparison of well-tuned BDL techniques on the various tasks. From our comparison we conclude that some current techniques which solve benchmarks such as UCI `overfit' their uncertainty to the dataset---when evaluated on our benchmark these underperform in comparison to simpler baselines. The code for the benchmark, its baselines, and a simple API for evaluating new BDL tools are made available at https://github.com/oatml/bdl-benchmarks.

1 Introduction

Deep models can fail to communicate uncertainty, complicating trust in high-stakes decisions, while Bayesian deep learning lacks realistic benchmarks that reflect application demands. The paper introduces a diabetic-retinopathy benchmark and comparison showing that UCI rankings may not transfer to real-world tasks.

1 Introduction

  • Models that cannot communicate uncertainty leave experts unsure whether to trust automated diagnoses and can prevent drivers from knowing when to take control.These are examples of the high-stakes implications described by the authors.Source: §1 · p. 1
  • Existing BDL benchmarks often use small, low-dimensional UCI datasets and omit real-world application constraints, development cost, and scalability.The authors call for varied tasks that test robustness, performance, accuracy, and development effort.Source: §1 · p. 2
  • The benchmark comparison finds that methods successful on UCI can fail to scale, solve the new benchmark, or provide good uncertainty estimates.The authors warn that over-reliance on UCI could distort research priorities.Source: §1 · p. 2
  • The paper extends diabetic-retinopathy diagnosis with out-of-distribution and distribution-shift tasks, expert-oriented metrics, and an API that abstracts domain expertise.Test data for distribution shift come from different equipment and patient populations.Source: §1 · p. 2

2 Diabetic Retinopathy Benchmark

The benchmark uses retinal images to assess diabetic-retinopathy diagnosis with uncertainty-guided expert referral. It also evaluates performance under distribution shift and measures diagnostic quality as referral rates vary.

2.1 Dataset

  • The Kaggle dataset is recast as binary classification of sight-threatening DR, with grades 2–4 positive; shift robustness uses a disjoint Indian dataset collected with different equipment and from a different population.The Kaggle data include 35,126 training images and 53,576 test images, and positive labels account for 19.6% and 19.2%, respectively.Source: §2.1 · p. 3

2.2 Data Processing

  • Images are cropped and resized to 512 × 512, channel-wise normalized using training-set statistics, and augmented with affine transformations and flips.The affine augmentations include zooms up to ±10%, translations up to ±25 pixels, and rotations up to ±π.Source: §2.2 · p. 3

2.3 Downstream Task

  • Uncertain diagnoses are referred to experts; uncertainty can also prioritize cases according to a hospital’s available resources.The task reflects unreliable images, including those affected by camera artefacts.Source: §2.3 · p. 3
  • For MC Dropout, predictive entropy is higher on misclassified images, making it an indicator that can drive referral.The figure relates predictive entropy to sigmoid disease probabilities.Source: §2.3 · p. 4, Figure 4 · p. 4

2.4 Metrics

  • Diagnostic accuracy and ROC AUC are measured as referral rate varies, with well-calibrated uncertainty expected to improve performance as less-confident cases are referred.Accuracy is the correctly classified fraction; AUC summarizes the ROC curve.Source: §2.4 · p. 4

3 A Systematic Comparison of BDL Methods

The comparison evaluates tuned Bayesian deep learning methods and control baselines on the diabetic-retinopathy benchmark. Ensemble MC Dropout performs best at lower referral rates, while all methods score equally within standard-error bounds with no referrals.

3 A Systematic Comparison of BDL Methods

  • Nine independently seeded models were trained for each separately tuned method, with implementations showing consistency and robustness across seeds.The models use VGG variants with around 2.5 million parameters.Source: §3 · p. 5
  • Predictive entropy estimates uncertainty from the predictive distribution, capturing both input ambiguity and uncertainty about a model’s knowledge.The estimate uses stochastic forward passes through probabilistic networks.Source: §3 · p. 5

3.1 Bayesian Neural Networks

  • Mean-field variational inference uses a fully factorized Gaussian posterior and Flipout to reduce gradient-estimator variance.A Monte Carlo estimate of the KL term reduces forward-pass time complexity to O(D) in the number of weights.Source: §3.1 · p. 6

3.2 Model Ensembling

  • Deep Ensembles estimate predictive uncertainty by collecting predictions from independently trained deterministic models, but training resources scale linearly with ensemble size.The paper also evaluates ensembles of MC Dropout models, averaging predictions from fresh dropout-mask samples across separately trained models.Source: §3.2 · p. 6

3.3 Deterministic

  • The deterministic baseline treats sigmoid outputs nearer 0 or 1 as more confident, capturing aleatoric but not epistemic uncertainty; it performs worse than methods capturing both.The figure comparison reports that MC Dropout, mean-field variational inference, and Deep Ensembles use predictive uncertainty; MC Dropout and mean-field variational inference degrade under APTOS 2019 shift, while Ensemble MC Dropout performs consistently better.Source: §3.3 · p. 7, Figure 5 · p. 7
  • The Random baseline makes referrals without using uncertainty or input and has unchanged accuracy and AUC as the retained-data rate varies.This provides a control for the referral-based evaluation.Source: §3.3 · p. 7

3.2 Model Ensembling

  • At lower referral rates, Ensemble MC Dropout performs best, while MC Dropout matches it at the extreme 50% referral rate.With 100% of data retained, all methods score equally within standard-error bounds.Source: §3.2 · p. 6, Table 1 · p. 8

4 Results and Analysis

On diabetic-retinopathy diagnosis, ensemble methods and MC Dropout score consistently higher than MFVI, reversing the UCI ranking reported in [4]. Referral-rate AUC and accuracy assess uncertainty, while tuning on simple UCI tasks may not generalize.

4 Results and Analysis

  • As referral rates increase, stronger uncertainty estimates appear as steeper AUC and accuracy gains; all methods perform equally when all data are retained.The equal retained-data performance indicates similar overall model performance and provides a fair comparison of uncertainty.Source: §4 · p. 8
  • Ensemble methods and MC Dropout score consistently higher than MFVI on diabetic-retinopathy diagnosis, unlike the UCI ranking reported in [4].The comparison is based on performance across referral rates.Source: §4 · p. 8
  • Table 2 borrows Deep Ensembles results from [26] and MC Dropout and Mean-field VI results from [35].Source: Table 2 · p. 9
  • Tuning on simple UCI tasks might yield rankings that do not generalize, and [35] find UCI regression benchmarks insufficient for conclusions about effectiveness or scalability.The paper notes that its real-world comparison reverses the UCI ranking reported in [4].Source: §4 · pp. 8–9

5 Implications for the Field

UCI-based rankings can misrepresent large-data performance and shape research priorities: MC Dropout outperforms MFVI in uncertainty estimates on larger-scale data, while HMC would not scale. The authors present their benchmark as a way to demonstrate methods on large-scale problems.

5 Implications for the Field

  • UCI's 4–16 input dimensions and tiny single-layer models focus BDL research away from the large, high-dimensional settings where deep learning has had its biggest successes.The paper says UCI's role as the field's main benchmark has a distorting effect on research.Source: §5 · p. 9
  • On larger-scale data, MC Dropout has better uncertainty estimates than mean-field variational inference, while their retained-data performance is almost identical; HMC would not scale.The UCI comparison ranked HMC and mean-field variational inference among the best neural-network models considered, whereas MC Dropout ranked near the bottom [4].Source: §5 · p. 9
  • UCI may mislead method priorities: computationally intensive extensions to MFVI have drawn more work than parsimonious ensembles or MC Dropout, partly shaped by UCI's prominence.The authors describe this influence as likely partial, not definitive.Source: §5 · p. 9
  • The benchmark offers new methods a way to demonstrate effectiveness on large-scale problems and engage with downstream users' needs.Source: §5 · p. 9

Related papers

Loading 1912.10481v1…