Source-linked AI summary
Concrete Dropout
Yarin Gal, Jiri Hron, Alex Kendall
TL;DR
The paper addresses the cost and infeasibility of grid-searching dropout probabilities for calibrated uncertainty, especially in large models and RL. It introduces Concrete Dropout, a continuous mask relaxation with a variational objective that enables gradient-based tuning, and reports improved calibration, uncertainty estimates, performance, and experimentation time. The method is compared with hand-tuned dropout rather than Gaussian dropout because optimised Gaussian dropout was reported to underperform.
Problem
Calibrated uncertainty requires adapting dropout probabilities, but grid-search is prohibitive for large models and infeasible when RL data changes continuously.
Method
Concrete Dropout continuously relaxes Bernoulli masks and combines reparameterised pathwise gradients with a variational objective to optimise dropout probabilities.
Results
Concrete Dropout shows improved calibration and uncertainty estimates, reduced experimentation-cycle time, and performance matching or marginally improving hand-tuned dropout across reported experiments.
Takeaways & Limitations
Dropout probabilities can be tuned automatically in large models and adapted as RL data increases, while learned probabilities achieve comparable performance to optimal fixed probabilities.
Takeaways & Limitations
The experiments compare Concrete Dropout with hand-tuned dropout and do not compare it with Gaussian dropout because optimised Gaussian dropout was known to underperform.
Abstract
from arXiv · showhide
Dropout is used as a practical tool to obtain uncertainty estimates in large vision models and reinforcement learning (RL) tasks. But to obtain well-calibrated uncertainty estimates, a grid-search over the dropout probabilities is necessary - a prohibitive operation with large models, and an impossible one with RL. We propose a new dropout variant which gives improved performance and better calibrated uncertainties. Relying on recent developments in Bayesian deep learning, we use a continuous relaxation of dropout's discrete masks. Together with a principled optimisation objective, this allows for automatic tuning of the dropout probability in large models, and as a result faster experimentation cycles. In RL this allows the agent to adapt its uncertainty dynamically as more data is observed. We analyse the proposed variant extensively on a range of tasks, and give insights into common practice in the field where larger dropout probabilities are often used in deeper model layers.
1 Introduction
Well-calibrated uncertainty matters for safety-critical and exploratory tasks, but tuning dropout probabilities is costly for large models and infeasible in continuously changing RL settings. Concrete Dropout addresses this by enabling gradient-based tuning and reports improved accuracy, uncertainty calibration, and experimentation speed.
- Miscalibrated uncertainty can cause over-exploration in RL or failure to detect ignorance in autonomous driving systems.
- Dropout provides a practical approximation to Bayesian inference, but calibrated uncertainty requires adapting dropout probabilities to the data.
- Grid-search over dropout probabilities is prohibitive for large vision models because it wastes computational resources or prolongs experimentation.
- In RL, increasing data requires decreasing dropout probability, making repeated retraining with grid-search infeasible.
- Existing alternatives include Gaussian-dropout optimisation and input-dependent belief networks, but reported practical difficulties limit their use.
- Concrete Dropout continuously relaxes discrete masks and uses gradient methods to tune dropout probabilities, improving calibration in large models and supporting principled RL use.
- Across tasks, the method studies uncertainty, data and model-size effects, large-model experimentation time, and dynamic uncertainty reduction in RL.
2 Background
The paper distinguishes epistemic, aleatoric, and predictive uncertainty, then relates dropout probability and weight magnitude to epistemic uncertainty. This motivates an objective for optimising dropout probability beyond predictive performance alone.
- Epistemic uncertainty represents ignorance about suitable models, while aleatoric uncertainty represents noise inherent in the environment.
- Epistemic uncertainty decreases as observed data increases and can inform avoidance of uninformed decisions in safety-critical settings.
- Predictive uncertainty combines explainable and irreducible noise in the model’s output.
- For dropout networks, masked weight matrices correspond to function draws, so dropout probability and weight configuration determine epistemic uncertainty.
- With fixed dropout probability, reducing epistemic uncertainty requires reducing weight magnitudes, potentially setting weights exactly to zero.
- Optimising dropout probability requires an objective targeting epistemic uncertainty rather than merely maximising model performance.
3 Concrete Dropout
Concrete Dropout replaces discrete Bernoulli masks with a differentiable Concrete relaxation and combines this with a variational objective. This makes dropout probabilities trainable by gradient methods while retaining Bayesian regularisation.
- Grid-search is expensive for large models and infeasible in continuous-learning RL, where epistemic uncertainty should collapse as data accumulates.
- The method uses dropout’s variational interpretation, treating qθ(ω) as an approximate posterior and optimising a likelihood term plus KL regularisation.
- The objective uses model outputs and likelihood evaluations over sampled data, while the KL term limits deviation from the prior.
- The variational parameters include mean weight matrices and dropout probabilities for each layer, with qθ(ω) defined over random weight matrices.
- When dropout probability is optimised, the Bernoulli entropy term is crucial because it acts as a probability-dependent regulariser.
- The score-function estimator was impractical because of high variance, motivating the lower-variance pathwise derivative estimator.
- Concrete relaxation approximates Bernoulli masks continuously, enabling reparameterisation and pathwise gradients for dropout-probability optimisation.
- The resulting Concrete Dropout masks permit gradient-based probability optimisation and are assessed across a broad range of tasks.
4 Experiments
Across synthetic, UCI, MNIST, and computer-vision experiments, Concrete dropout matches or improves benchmark performance while learning dropout probabilities and improving uncertainty calibration. The learned probabilities decrease with more data, vary with model size, and can reduce manual tuning time substantially.
- Synthetic data: Concrete dropout captures decreasing epistemic uncertainty as data increases, while aleatoric uncertainty approaches the known noise level and predictive uncertainty converges.The synthetic experiments use data generated with fixed noise standard deviation 1.
- Synthetic data: Optimal dropout probabilities decrease from near 0.5 with the smallest dataset to 0.2–0.4 with 10,000 data points.This supports adapting dropout probabilities to the amount of observed data.
- UCI and MNIST: Across UCI experiments, the input-layer dropout probability decreases to essentially zero across cross-validation splits and datasets.This pattern recurs in the UCI experiments.
- UCI and MNIST: Concrete dropout matches hand-tuned dropout on UCI datasets and achieves 98.6% MNIST accuracy.The MNIST result matches hand-tuned dropout, while UCI comparisons include standard-dropout Bayesian neural networks and deep Gaussian processes.
- UCI and MNIST: As model size increases, dropout probabilities remain near zero in the first layer but increase in other layers as epistemic uncertainty increases.With increasing training data, the first two layers can converge to zero dropout, indicating that removing dropout a priori may be suboptimal for insufficiently informative datasets.
- Computer vision: In a DenseNet semantic-segmentation model, Concrete dropout learns layer-dependent probabilities, improves uncertainty calibration, and can reduce training time by weeks.The learned pattern is tolerant to initialisation and differs from manual tuning, with larger probabilities in the second and last layers.
5 Conclusions and Insights
Concrete dropout tunes dropout probabilities rather than fixing them, improving uncertainty calibration and reducing experimentation time. It also supports insights about layerwise dropout and adaptive probabilities.
- Concrete dropout tunes dropout probabilities and improves calibration and uncertainty estimates while reducing experimentation cycle time.
- Small dropout probabilities in shallow layers are supported by dropout’s variational interpretation.
- Allowing dropout probabilities to adapt achieves performance comparable to optimally fixed probabilities without forcing weight magnitudes near zero.
A Concrete distribution with a Bernoulli random variable
Figure 10 presents a Concrete distribution alongside a two-dimensional discrete Bernoulli distribution, using temperature 0.1.
- Figure 10 shows the Concrete distribution.
- The comparison uses a two-dimensional discrete Bernoulli distribution.
- The Concrete distribution is configured with temperature 0.1.
B Choice or prior
The method uses a discrete quantised Gaussian prior whose KL divergence can be evaluated analytically. Under this prior, Concrete dropout has two interpretations depending on which term is approximated.
- The method uses a discrete quantised Gaussian prior, making the variational-to-prior KL divergence analytically evaluable.
- One interpretation treats the Concrete distribution as a Bernoulli approximation only in the expected log likelihood term to obtain derivatives with respect to p.
- A second interpretation treats Concrete noise as a new stochastic regularisation technique and approximates the KL divergence instead.
C Python code snippet for Concrete Dropout
The ConcreteDropout wrapper learns dropout probabilities for layers and adds weight and dropout regularisation terms. It can wrap dense or convolutional layers, although the current implementation supports only 2D inputs.
- ConcreteDropout wraps a given input layer and allows its dropout probability to be learned.
- The wrapper can be applied to Dense layers in sequential models without changing the resulting output shapes.
- ConcreteDropout can also wrap arbitrary layers such as Conv2D.
- The current implementation supports 2D inputs only.
- The implementation parameterizes dropout probability as the sigmoid of a trainable p_logit initialized uniformly from -2 to 0.
- The wrapper adds a regularizer combining weight penalties and a dropout entropy term, with weight scaling by 1 - p.
D More Results
This section reports regression results using average test log likelihood and average test RMSE across multiple datasets.
- Regression results are organized across the Boston, Concrete, Energy, Kin8nm, Power, Protein, Red Wine, Yacht, and Naval datasets.
- Average test log likelihood is reported in nats.
- Average test RMSE is reported as a separate regression metric.
D.1 MNIST
The MNIST experiments use separate training, validation, and testing sets, with prior length scale selected by grid search. The proposed model matches standard dropout on the reported test results, while ELBO indicates the best length scale for predictive log likelihood.
- MNIST: MNIST data are split into 5 · 10^4 training, 10^4 validation, and 10^4 testing observations.
- MNIST: The prior length-scale constant l is selected by grid search, with l = 10^-2 giving the best balance between predictive log likelihood and accuracy.The candidates are {10^-4, 10^-3, ..., 10^0, 2.0}.
- MNIST: The proposed model attains the same results as standard dropout in the right-hand plot of Figure 11.
- MNIST: ELBO is a good indicator for selecting the optimal length scale when predictive log likelihood is the target.The reported correlation was not observed for other hyperparameters.