PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning

Zachary Nado, Neil Band, Mark Collier, Josip Djolonga, Michael W. Dusenberry, Sebastian Farquhar, Qixuan Feng, Angelos Filos, Marton Havasi, Rodolphe Jenatton, Ghassen Jerfel, Jeremiah Liu, Zelda Mariet, Jeremy Nixon, Shreyas Padhy, Jie Ren, Tim G. J. Rudner, Faris Sbahi, Yeming Wen, Florian Wenzel, Kevin Murphy, D. Sculley, Balaji Lakshminarayanan, Jasper Snoek, Yarin Gal, Dustin Tran

arXiv:2106.04015v3cs.LG

TL;DR

Prior results can be difficult to reproduce and compare when papers provide inconsistent code, documentation, and experimental protocols. Uncertainty Baselines offers self-contained implementations and reusable artifacts across methods and benchmarks. Its ImageNet example illustrates the available evaluations, while leaving specific baseline comparisons to the full leaderboards.

  • Problem

    Inconsistent experimental protocols and code organization make it difficult to compare across papers and build jointly on their work.

    Source: §1 · p. 2

  • Method

    Uncertainty Baselines provides self-contained, extensible implementations and artifacts for benchmarks organized around base models, training datasets, and evaluation metrics.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    The collection comprises 83 baselines, covering 19 methods across 9 benchmarks.

    Source: §2 · p. 3

  • Takeaways & Limitations

    The library is designed to let researchers adapt baselines to their workflows and build on a shared set of benchmark components.

    Source: §1 · p. 2, §2 · p. 4

    The ImageNet figure omits its legend, so it does not identify which baseline performs best; the full leaderboards are needed for those comparisons.

    Source: §3 · p. 5

Abstract

from arXiv · show

High-quality estimates of uncertainty and robustness are crucial for numerous real-world applications, especially for deep learning which underlies many deployed ML systems. The ability to compare techniques for improving these estimates is therefore very important for research and practice alike. Yet, competitive comparisons of methods are often lacking due to a range of reasons, including: compute availability for extensive tuning, incorporation of sufficiently many baselines, and concrete documentation for reproducibility. In this paper we introduce Uncertainty Baselines: high-quality implementations of standard and state-of-the-art deep learning methods on a variety of tasks. As of this writing, the collection spans 19 methods across 9 tasks, each with at least 5 metrics. Each baseline is a self-contained experiment pipeline with easily reusable and extendable components. Our goal is to provide immediate starting points for experimentation with new methods or applications. Additionally we provide model checkpoints, experiment outputs as Python notebooks, and leaderboards for comparing results. Code available at https://github.com/google/uncertainty-baselines.

1. Introduction

Reproducing and comparing prior results is difficult when papers provide uneven artifacts and inconsistent protocols. Uncertainty Baselines responds with reusable implementations and artifacts, while focusing on scalable, forkable, well-tested code.

1. Introduction

  • Without community standards, differing experimental protocols and code organization make cross-paper comparisons and joint development difficult.Code availability alone may not suffice: retrospective analyses have found simpler baselines can work best because of flawed protocols or insufficient tuning.Source: §1 · p. 2
  • Uncertainty Baselines provides high-quality, self-contained implementations across uncertainty and out-of-distribution robustness tasks, designed for extension with minimal dependencies.The library also provides artifacts beyond raw code so researchers can adapt baselines to their workflows.Source: §1 · p. 2
  • The paper emphasizes scalable, forkable, well-tested code and a broader set of baselines across benchmarks, in some cases unifying earlier benchmarks.The authors characterize prior uncertainty and robustness benchmarks as unmaintained and focused on experimental insights rather than the codebase.Source: §1 · p. 2

2. Uncertainty Baselines

Uncertainty Baselines organizes experiments around models, datasets, and evaluation metrics, with a modular design intended to support extension and use across frameworks. The collection also emphasizes reproducibility, hardware utilization, and examples spanning multiple evaluation settings.

2. Uncertainty Baselines

  • Each benchmark combines a base model, training dataset, and suite of evaluation metrics, spanning architectures and datasets across multiple modalities.Examples include Wide ResNet, ResNet-50, BERT, and MLP models; datasets include CIFAR, ImageNet, UCI, CLINC Intent Detection, Diabetic Retinopathy, and Wikipedia Toxicity.Source: §2 · p. 2
  • Evaluation covers predictive and uncertainty metrics, inference latency, and performance on in- and out-of-distribution datasets.Examples include accuracy, selective prediction, and calibration error.Source: §2 · p. 3
  • The collection comprises 83 baselines: 19 methods across 9 benchmarks.The methods sweep over standard and more recent strategies.Source: §2 · p. 3
  • The implementations target CPU, GPU, or TPU hardware; the MIMO profile reports 100% TPU utilization, with runtime bound by model operations.Baselines are optimized for a default hardware configuration and may assume particular memory requirements and chip counts.Source: Figure 2 · p. 4, §2 · p. 4
  • Baselines use modular components and minimal non-standard dependencies, with trained checkpoints evaluated through Robustness Metrics for out-of-distribution evaluation.The modules include light dataset wrappers and models that can be incorporated into other libraries.Source: Figure 1 · p. 3, §2 · p. 4
  • The library is framework-agnostic, supporting TensorFlow, JAX, and PyTorch, while choosing a specific backend for each benchmark to ease code and performance comparisons.The paper gives JAX on JFT-300M and PyTorch Monte Carlo Dropout on Diabetic Retinopathy as examples.Source: §2 · p. 4
  • All modules include testing and results are reported over multiple seeds; TensorBoard dashboards provide training, tuning, and evaluation metrics.The paper notes that computing metrics and training from scratch can be prohibitively expensive.Source: §2 · pp. 4–5
  • The ImageNet examples show eight baselines across four datasets using top-1 accuracy and expected calibration error, plus deferred prediction by confidence or desired retention rate.The examples are described as a sampling of the repository's capabilities.Source: Figure 3 · p. 5, Figure 4 · p. 5

3. Results

The results section presents ImageNet as an example of the repository's capabilities, showing accuracy and calibration across baselines and an application to deferred prediction. It does not identify a best-performing baseline in the figure.

3. Results

  • The ImageNet example evaluates eight baselines on ImageNet, ImageNet-C, ImageNet-A, and ImageNetV2 using top-1 accuracy and expected calibration error.The authors describe these results as a sampling of the repository's capabilities.Source: §3 · p. 5
  • The figure omits its legend to avoid comparisons of which specific baselines perform best; the authors direct readers to the full leaderboards for those insights.The displayed figure covers accuracy and calibration error across the four ImageNet datasets.Source: §3 · p. 5
  • A further example applies an ImageNet baseline to deferred prediction, deferring by model confidence or desired data retention rate.The authors say they are excited to see new research building on the baselines.Source: §3 · p. 6

Appendix A. Dataset Details

The appendix specifies image preprocessing for CIFAR and ImageNet and identifies the dataset used for Diabetic Retinopathy benchmarks.

Appendix A. Dataset Details

  • CIFAR images receive four-pixel zero padding, random 32×32 cropping, and a 50% left-right flip; ImageNet uses ResNet preprocessing, with Inception preprocessing also supported.Preprocessing is deterministic given a random seed; Diabetic Retinopathy benchmarks use the Kaggle competition dataset as in Filos et al. (2019).Source: Appendix A · p. 11

Appendix B. Model Details

The baselines use different model families across image and text benchmarks.

Appendix B. Model Details

  • CIFAR10 and CIFAR100 use Wide ResNets, ImageNet and Diabetic Retinopathy use ResNet-50, and ImageNet also uses EfficientNet.Toxic Comments and CLINC Intent Detection use BERT-Base.Source: Appendix B · p. 11

Appendix C. Hyperparameter Tuning

The appendix describes training optimizers and schedules and provides tuning details for the benchmarks.

Appendix C. Hyperparameter Tuning

  • Image benchmarks generally use Nesterov momentum and a warmup followed by stepwise decay, while text benchmarks use AdamW and warmup followed by linear decay.EfficientNet uses RMSProp and warmup followed by exponential decay.Source: Appendix C · p. 11
  • For five benchmarks, the papers for each method contain their tuning details.The listed benchmarks are CIFAR10, CIFAR100, ImageNet, Toxic Comments, and CLINC Intent Detection.Source: Appendix C · p. 11
  • Diabetic Retinopathy tuning uses two 50-trial quasirandom searches, selecting hyperparameters by final validation AUC from the second round.The chosen setting is retrained on the combined training and validation sets.Source: Appendix C · p. 11

Appendix D. Supported Baselines

Table 1 describes the currently implemented methods across datasets, including a deterministic baseline.

Appendix D. Supported Baselines

  • Table 1 lists currently implemented methods for each dataset alongside a deterministic baseline; the repository may contain a more updated list.Source: Table 1 · p. 12

Appendix E. Open-Source Data

The section lists tuning and final-metrics data for Diabetic Retinopathy benchmarks.

Appendix E. Open-Source Data

  • Listed materials cover first and final tuning for deterministic, variational inference, and Radial BNN methods, plus 10-seed data for the latter two.Source: Appendix E · p. 13
  • Radial BNN materials include first and final tuning data and 10-seed data.Source: Appendix E · p. 13

Related papers

Loading 2106.04015v3…