Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Yarin Gal; Zoubin Ghahramani

Source-linked AI summary

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Yarin Gal, Zoubin Ghahramani

arXiv:1506.02142v6Jun 6, 2015stat.MLcs.LG

TL;DR

Deep learning models lack practical model-uncertainty estimates, while Bayesian approaches are often computationally expensive. This paper interprets dropout as approximate Bayesian inference and shows that existing dropout networks can represent uncertainty with improved predictive log-likelihood and RMSE over state-of-the-art methods.

Problem
Deep learning tools for regression and classification do not capture model uncertainty, while Bayesian approaches usually incur prohibitive computational cost.
Method
The paper casts dropout in neural networks as approximate Bayesian inference in deep Gaussian processes and estimates predictive uncertainty from sampled dropout models.
Results
Across regression and classification assessments, dropout uncertainty considerably improves predictive log-likelihood and RMSE over existing state-of-the-art methods.
Takeaways & Limitations
A probabilistic interpretation of dropout extracts model uncertainty from existing deep learning models without changing their models or optimisation.
Takeaways & Limitations
Different dropout variants produce different uncertainty estimates, trading off uncertainty quality against computational complexity.

Abstract

from arXiv · show

Deep learning tools have gained tremendous attention in applied machine learning. However such tools for regression and classification do not capture model uncertainty. In comparison, Bayesian models offer a mathematically grounded framework to reason about model uncertainty, but usually come with a prohibitive computational cost. In this paper we develop a new theoretical framework casting dropout training in deep neural networks (NNs) as approximate Bayesian inference in deep Gaussian processes. A direct result of this theory gives us tools to model uncertainty with dropout NNs -- extracting information from existing models that has been thrown away so far. This mitigates the problem of representing uncertainty in deep learning without sacrificing either computational complexity or test accuracy. We perform an extensive study of the properties of dropout's uncertainty. Various network architectures and non-linearities are assessed on tasks of regression and classification, using MNIST as an example. We show a considerable improvement in predictive log-likelihood and RMSE compared to existing state-of-the-art methods, and finish by using dropout's uncertainty in deep reinforcement learning.

1. Introduction

The introduction identifies a gap between deep learning’s need for model uncertainty and standard tools’ inability to capture it, while Bayesian methods impose high computational costs. It presents dropout as a Bayesian approximation enabling uncertainty estimation without changing models or optimization, and previews theoretical and empirical evaluation across regression and classification.

Motivation: Deep learning is increasingly used in fields where representing model uncertainty is crucial, alongside a recent shift toward Bayesian uncertainty.These needs arise in applications including physics, biology, and manufacturing.
Problem: Standard deep learning tools for regression and classification do not capture model uncertainty, and softmax probabilities can be mistakenly interpreted as model confidence.A model may remain uncertain despite a high softmax output, especially when extrapolating beyond the training data.
Motivation: Model confidence enables practitioners to handle uncertain inputs explicitly, including referring high-uncertainty classifications to human experts in critical applications.The introduction gives postal sorting and nuclear power infrastructure as examples.
Motivation: Bayesian probability theory provides mathematically grounded tools for model uncertainty, but these methods usually have prohibitive computational costs.The paper therefore seeks a Bayesian interpretation that preserves existing deep learning models and optimization.
Contribution: Dropout and its variants are interpreted as Bayesian approximations of Gaussian processes, approximately integrating over neural-network weights without changing models or optimization.Dropout is widely used to avoid over-fitting, and this interpretation extracts uncertainty information from existing models.
Contribution: The paper develops a complete theoretical link between Gaussian processes and dropout and evaluates dropout-derived uncertainty across architectures, nonlinearities, regression, and classification.The assessment includes neural networks and convolutional networks.

2. Related Research

Prior work links infinitely wide neural networks with weight distributions to Gaussian processes, while finite Bayesian neural networks provide robustness to overfitting but remain computationally challenging. Variational inference has been explored for these models, though with limited success.

Neural networks and Gaussian processes: Infinitely wide single-hidden-layer neural networks with distributions over their weights converge to Gaussian processes, but the limiting relation is difficult to transfer to finite networks.This connection follows a limit argument, limiting direct translation of Gaussian-process properties to finite neural networks.
Bayesian neural networks: Finite neural networks with distributions over their weights have been studied as Bayesian neural networks, offering robustness to overfitting but requiring challenging inference and additional computation.The passage identifies Bayesian neural networks as a major treatment of finite networks with weight distributions.
Variational inference: Variational inference has been applied to Bayesian neural networks, but with limited success.The cited work includes variational approaches by Hinton and Van Camp, Barber and Bishop, and Graves.

3. Dropout as a Bayesian Approximation

This section shows that dropout training in deep neural networks is an approximate Bayesian inference method for deep Gaussian processes. The approximation uses a variational distribution induced by dropout and minimizes its KL divergence from the deep-GP posterior.

Theoretical equivalence: Dropout before every weight layer in arbitrary-depth, nonlinear networks is mathematically equivalent to approximating a deep Gaussian process marginalized over covariance-function parameters.The result applies to any network architecture using dropout, including multiplicative Gaussian noise variants.
Variational objective: The dropout objective minimizes the Kullback–Leibler divergence between q(ω) and the posterior p(ω|X, Y) of the full deep Gaussian process.The variational distribution is defined through Bernoulli variables and variational matrices Mi.
Dropout construction: Dropout samples binary variables for each input and hidden unit, dropping units when variables equal zero and reusing the same values during backpropagation.A unit in layer i − 1 is dropped as input to layer i when its corresponding variable is zero.
Variational approximation: Because the deep-GP posterior p(ω|X, Y) is intractable, the method approximates it with q(ω), whose matrix columns are randomly set to zero.The resulting variational distribution is highly multimodal and induces strong joint correlations over weight-matrix rows.
Objective approximation: A single Monte Carlo sample bωn ∼ q(ω) provides an unbiased estimate of each negative log-likelihood term, and the resulting objective recovers the dropout objective under suitable precision and length-scale settings.The result is scaled by 1/τN for model precision τ > 0 and dataset size N.

4. Obtaining Model Uncertainty

The section derives model uncertainty from existing dropout neural networks using Monte Carlo stochastic forward passes. It estimates predictive moments and variance without changing the trained model, while noting that the resulting predictive distribution may be highly multi-modal and only approximately characterized.

Model uncertainty can be obtained from dropout neural-network models through a new derivation that also provides mathematically grounded uncertainty estimates.
MC dropout samples T Bernoulli realisations and averages the corresponding stochastic forward-pass results.In practice, this is equivalent to performing T stochastic forward passes through the network.
The predictive variance equals the sample variance of T stochastic forward passes plus the inverse model precision.Model precision can be obtained from the weight-decay λ and prior length-scale l.
Predictive log-likelihood is estimated by Monte Carlo integration to assess how well the model fits both the mean and uncertainty.For regression, the estimate uses a log-sum-exp of T stochastic forward-pass terms.
The predictive distribution q(y∗|x∗) is expected to be highly multi-modal, so the moment approximations provide only a glimpse into its properties.This follows from the bi-modal variational distribution placed on each weight-matrix column and the resulting multi-modal joint distribution over each layer’s weights.
The dropout model remains unchanged, and predictive mean and uncertainty are collected from stochastic forward passes that can run concurrently with constant running time identical to standard dropout.This enables the method to be used with existing neural networks trained with dropout.

5. Experiments

The experiments evaluate dropout-based uncertainty across regression, classification, and reinforcement learning, showing that uncertainty reflects architecture-dependent extrapolation behavior and improves predictive and decision-making performance. MC dropout also outperforms competing models on most benchmark datasets in RMSE and test log-likelihood.

Regression uncertainty: Standard dropout confidently predicts an insensible extrapolated value, whereas the Gaussian-process model expresses comparable uncertainty through increased predictive variance.None of the evaluated models captures the CO2 series’ periodicity under extrapolation.
Regression uncertainty: Dropout uncertainty increases away from observed CO2 data for ReLU networks, while remaining bounded for TanH networks.ReLU and TanH approximate different Gaussian-process covariance functions, and TanH saturates whereas ReLU does not.
Benchmark regression: Dropout significantly outperforms all other models in RMSE and test log-likelihood on every evaluated dataset except Yacht, where PBP achieves better RMSE.Results were averaged over 20 random data splits for most datasets, with fewer splits for Protein and Year.
Benchmark regression: 2.68 median RMSE and -2.34 median predictive log-likelihood are reported for dropout on Boston Housing, with IQRs of [2.45, 3.35] and [-2.54, -2.29].On Concrete Strength, dropout achieves a median RMSE of 5.15.
Reinforcement learning: Thompson sampling exceeds reward 1 within 25 post-burn-in batches, while epsilon-greedy requires 175 batches to reach the same performance.The approach stops improving after about 1K batches because it continues sampling random moves, whereas epsilon-greedy then exploits.

6. Conclusions and Future Research

The paper frames dropout as a probabilistic method for extracting model uncertainty from existing deep learning models and identifies future research on alternative dropout approximations and architecture–regularization choices.

Conclusions: A probabilistic interpretation of dropout extracts model uncertainty from existing deep learning models and supports applications combining Bayesian and deep learning models.The authors also studied the properties of this uncertainty in detail.
Future Research: Alternative dropout variants correspond to different approximate variational distributions, producing uncertainty estimates that trade off uncertainty quality against computational complexity.The authors identify these variants as a subject for follow-up work.
Future Research: Each Gaussian-process covariance function corresponds to a combination of neural-network nonlinearities and weight regularization, suggesting data-driven choices of network structure and regularization.For smooth functions with uncertainty increasing far from observed data, cosine nonlinearities and L2 regularization may be appropriate.

A. Appendix

The appendix is available online and includes a table comparing dropout uncertainty with longer training and a deeper model.

A. Appendix: The paper’s appendix is provided at the listed arXiv URL.The passage directs readers to http://arxiv.org/abs/1506.02157.
A. Appendix: Table 2 compares average test RMSE and predictive log likelihood for Dropout, 10x epochs, and 2 Layers.The alternatives use the same model with 10 times more epochs or two layers instead of one.