Source-linked AI summary
Bayesian Uncertainty Estimation for Batch Normalized Deep Networks
Mattias Teye, Hossein Azizpour, Kevin Smith
TL;DR
Deep networks need reliable predictive uncertainty, especially outside their training distribution, but standard methods lack uncertainty modeling. This paper shows that batch-normalization training approximates Bayesian inference and uses repeated stochastic forward passes to estimate uncertainty without modifying BN networks or training. Across regression, classification, and segmentation settings, MCBN correlates uncertainty with prediction errors, improves over constant-uncertainty baselines, and performs on par with MCDO and MNF.
Problem
Standard deep-learning techniques lack methods to account for model uncertainty, which is problematic for out-of-distribution or adversarial inputs where predictions can become unpredictable.
Method
MCBN interprets batch-normalization mini-batch randomness as approximate Bayesian inference and estimates predictive uncertainty from multiple stochastic forward passes through existing BN networks.
Results
MCBN uncertainty correlates with prediction errors, significantly improves over optimized constant-uncertainty baselines, and performs on par with MCDO and MNF across reported tasks.
Takeaways & Limitations
Meaningful uncertainty estimates can be obtained from existing batch-normalized models without modifying the network or training procedure.
Takeaways & Limitations
Uncertainty measurements can be confounded on small datasets or under the Gaussian assumptions of CRPS, as illustrated by the Yacht Hydrodynamics results.
Abstract
from arXiv · showhide
We show that training a deep network using batch normalization is equivalent to approximate inference in Bayesian models. We further demonstrate that this finding allows us to make meaningful estimates of the model uncertainty using conventional architectures, without modifications to the network or the training procedure. Our approach is thoroughly validated by measuring the quality of uncertainty in a series of empirical experiments on different tasks. It outperforms baselines with strong statistical significance, and displays competitive performance with recent Bayesian approaches.
1. Introduction
Deep networks can make unpredictable errors outside their training distribution, motivating predictive uncertainty estimates. The paper exploits batch-normalization stochasticity to obtain Bayesian-style uncertainty from conventional BN networks and validates it empirically.
- Predictive uncertainty can complement point estimates when deep-network errors may cause physical or economic harm.
- Deep networks may behave unpredictably on conditions outside the training distribution, including adversarial examples.
- MCBN exploits randomly selected mini-batch members during batch-normalization training to cast BN training as approximate Bayesian inference.
- MCBN applies to any network using standard batch normalization without requiring architectural or training-procedure modifications.
- Figure 1 estimates uncertainty from the mean and variance of 500 stochastic forward passes on a toy dataset.
- Experiments across regression and image-classification datasets find MCBN outperforms a fixed-uncertainty baseline on nearly all datasets with strong statistical significance.
2. Related Work
Bayesian neural networks offer a principled uncertainty framework but are difficult to scale, while approximate methods often require architectural or training changes. Existing dropout-based inference reduces this burden, but does not cover networks without dropout.
- Bayesian neural networks model uncertainty through posterior distributions over parameters but do not scale well or compete consistently with modern deep architectures.
- Multiplicative Normalizing Flows use auxiliary variables and normalizing flows to construct a flexible posterior for variational Bayesian networks.
- Approximate Bayesian methods address some scaling difficulties but require modifications to network architectures or training procedures and specialized practitioner knowledge.
- Monte Carlo Dropout treats dropout-trained networks as approximate Bayesian models and estimates uncertainty from predictions under different dropout masks.
3. Method
The method interprets batch-normalization randomness as an approximate posterior over model parameters. It then estimates predictive uncertainty by repeated stochastic inference, with theoretical conditions connecting BN optimization to variational Bayesian objectives.
- The paper develops a variational Bayesian interpretation of batch-normalized deep networks and derives their induced parameter prior.
- The analysis assumes i.i.d. observation noise for the ELBO and studies the induced weight-decay prior for fully connected layers.
- Batch Normalized Deep Nets as Bayesian Modeling: During BN training, mini-batch means and variances make inference stochastic because each sample’s computation depends on other batch members.
- Batch Normalized Deep Nets as Bayesian Modeling: The approximate posterior qθ(ω) is induced by repeated sampling of normalization parameters through randomly sampled training batches.
- Batch Normalized Deep Nets as Bayesian Modeling: After training, the method estimates predictive uncertainty from the inherent stochasticity of batch normalization.
- Batch Normalized Deep Nets as Bayesian Modeling: With weight decay, the induced prior over BN means is wide while the prior over standard deviations is narrow, with its narrowness depending on N.
- Batch Normalized Deep Nets as Bayesian Modeling: BN and variational objectives become equivalent under conditions including large training and mini-batch sizes, BN on all layers, uncorrelated units, and no scale or shift.
4. Experiments and Results
The experiments evaluate uncertainty quality across regression and image-classification tasks using normalized PLL and CRPS, with qualitative visualizations. MCBN generally improves over fixed-uncertainty baselines and performs competitively with other approximate Bayesian methods, while performance depends on batch size and stochastic-pass count.
- Experimental setup: The evaluation covers CIFAR10 image classification, eight standard regression datasets, and qualitative uncertainty visualizations including error-sorted variance plots and image segmentation.The quantitative analysis uses standard metrics alongside qualitative experiments.
- Metrics and baselines: PLL and CRPS are normalized between constant-uncertainty lower bounds and optimized upper bounds to improve interpretability.A normalized score of 0 matches fixed uncertainty, while 1 indicates optimal performance.
- Experimental setup: MCBN is compared with MCDO and MNF using repeated stochastic forward passes and task-specific cross-validation procedures.Regression experiments use five random splits, while predictive distributions are estimated from T = 500 stochastic passes in the main setup.
- Regression results: MCBN significantly exceeds the constant-variance baseline on most regression datasets for both CRPS and PLL, while generally performing on par with MCDO and MNF.Table 1 reports normalized uncertainty metrics and one-sample t-tests against the lower bound.
- Image-classification results: On CIFAR10, MCBN improves test PLL from -0.32 for the softmax baseline to -0.28, with further gains as stochastic forward passes increase.The improvement becomes significantly better than the softmax baseline as the number of passes grows.
- Sensitivity analyses: Uncertainty performance deteriorates with batch sizes ≤16, whereas larger numbers of stochastic passes generally help and T = 50 can perform well in some cases.The appendix also reports slight predictive-accuracy improvements from multiple forward passes compared with non-stochastic counterparts.
5. Discussion
MCBN produces uncertainty estimates that correlate with prediction errors and generally match competing approximate Bayesian methods, while performance depends on batch size and evaluation context.
- PLL improves with more stochastic forward passes on CIFAR10 until MCBN significantly surpasses the softmax baseline of -0.32.ResNet32 uses a fixed training batch size of 32.
- MCBN uncertainty estimates correlate with actual prediction errors and generally perform on par with MCDO and MNF.Direct comparison is confounded because the methods learn different parameters and therefore have different predictive means.
- Yacht Hydrodynamics produces contradictory CRPS and PLL results, possibly because only 60 test samples and variable model accuracy make measurements unstable.The authors also hypothesize that the Gaussian assumption underlying CRPS contributes to the discrepancy.
- CRPS and PLL improvements rarely exceed 10% over the lower bound, but the upper bound is difficult to achieve and the lower bound is reasonably strong.The upper bound is optimized separately for each test sample.
- Small batch sizes can reduce uncertainty quality, likely because they increase the variance of the approximate posterior.Large batches may also create memory problems for large inputs and parameter-heavy networks, although storing batch statistics or Gaussian estimates can mitigate this.
- MCBN uncertainty maps are qualitatively better for PASCAL-VOC than CamVid, consistent with its larger mini-batch size of 36 versus 10.CamVid’s smaller batches were imposed by memory limits from its larger 360x480 images.
6. Conclusion
The paper interprets batch-normalized training as approximate Bayesian inference and uses that connection to estimate uncertainty from existing networks. Across regression, classification, and segmentation, MCBN improves over a constant-uncertainty baseline and performs comparably to MCDO and MNF.
- Batch-normalized deep-network training is equivalent to approximate inference in Bayesian models.
- MCBN uncertainty estimates correlate with actual prediction errors across regression, image classification, and image segmentation tasks.
- MCBN significantly improves over the optimized constant-uncertainty baseline and performs on par with MCDO and MNF.
- Meaningful uncertainty estimates can be obtained from existing batch-normalized models without modifying the network or training procedure.The procedure computes variance from multiple stochastic forward passes.
1. Appendix
This appendix develops the variational framework behind the method, represents batch-normalization stochasticity through mini-batch selection, and analyzes Gaussian approximations to batch statistics.
- The approximate posterior qθ(ω) is optimized by minimizing KL divergence to the true posterior, equivalently maximizing the ELBO.
- Mini-batch subsampling provides an unbiased estimate of the variational objective for iterative optimization.
- An auxiliary variable ϵ represents training stochasticity, with ω = g(θ, ϵ) inducing the approximate posterior qθ(ω).For batch-normalized networks, ϵ can represent mini-batch selection because batch means and variances depend on it.
- The variational distribution factorizes over stochastic parameters, and Gaussian factors decompose the KL divergence into individual terms.
- The method approximates mini-batch means and standard deviations separately with Gaussian distributions.The normality of batch statistics is empirically checked with one-sample Kolmogorov-Smirnov tests.
- The Gaussian approximation for batch statistics relies on sufficiently large mini-batches and central-limit-theorem reasoning.The text notes that sufficiently large samples are often at least 30, while practical mini-batches are sampled without replacement.
Finalizing the distribution
The paper derives the approximate Bayesian distributions induced by batch normalization and uses stochastic forward passes to estimate predictive uncertainty. It then evaluates these estimates across regression and vision tasks, including sensitivity to batch size and the number of samples.
- Approximate posterior: Batch normalization is modeled as inducing Gaussian stochastic variables, with approximate posterior qθ(ω) derived under weak assumptions and large enough batch size.The derivation uses the Central Limit Theorem and treats stochastic variables as factorized across BN units.
- Prior construction: The Bayesian objective requires priors independent of changing network weights θk, and the BN-specific derivation restricts relevant weights to those in the same layer.The stated simplifying assumptions include no scale or shift transformations, BN on each layer, and independent input units.
- Prior construction: For sufficiently large N, the prior for a BN unit mean is p(µuB) → N(0, 1/(2Nτλl)) when 1/(2Nτλl) is sufficiently small.This condition is presented as reconciling the batch-normalization objective with the variational-inference formulation.
- Uncertainty evaluation: Predictive log likelihood is used to measure uncertainty quality without assuming a particular form for the approximate predictive distribution.The paper also uses CRPS in its regression evaluation.
- Predictive distribution: The predictive mean and covariance are approximated by Monte Carlo integration over T sampled stochastic parameter sets, with covariance combining observation noise and sample covariance.For regression, the observation-noise contribution is specified through model precision τ.
- Experimental findings: Across extended experiments, MCBN performs on par or better than MCDO, while multiple stochastic forward passes provide slight predictive-accuracy improvements over non-stochastic counterparts.Uncertainty estimates also correlate with prediction errors, and performance generally improves with larger T but deteriorates for overly small batch sizes.