Source-linked AI summary
Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
Christos Louizos, Max Welling
TL;DR
The paper addresses limited and overconfident uncertainty estimates from conventional neural-network training by introducing multiplicative normalizing flows for variational Bayesian neural networks. This approach uses auxiliary random variables and normalizing flows while retaining local reparametrizations and a tractable lower bound, and it improves on mean field in predictive performance and uncertainty, though overconfidence remains on the evaluated CIFAR task.
Problem
Neural networks can overfit when labeled data are scarce and produce overconfident predictions, motivating better approximate Bayesian inference.
Method
Multiplicative normalizing flows augment variational Bayesian neural-network posteriors with auxiliary multiplicative variables and flows while retaining local reparametrizations.
Results
MNF posteriors significantly improve on mean field in predictive performance and uncertainty, achieving more realistic uncertainties with predictive capabilities on par with Dropout.
Takeaways & Limitations
The proposed approximation offers a flexible and efficient variational posterior for Bayesian neural networks with improved predictive accuracy and uncertainty.
Takeaways & Limitations
On the five-class CIFAR-10 subset, all methods were overconfident, and the authors call for further investigation.
Abstract
from arXiv · showhide
We reinterpret multiplicative noise in neural networks as auxiliary random variables that augment the approximate posterior in a variational setting for Bayesian neural networks. We show that through this interpretation it is both efficient and straightforward to improve the approximation by employing normalizing flows while still allowing for local reparametrizations and a tractable lower bound. In experiments we show that with this new approximation we can significantly improve upon classical mean field for Bayesian neural networks on both predictive accuracy as well as predictive uncertainty.
1. Introduction
The paper motivates Bayesian neural networks as a way to address overfitting with scarce data and overconfident predictions, then proposes flexible variational inference using auxiliary variables and normalizing flows.
- Limited labeled data can cause neural networks to overfit, restricting applications such as MRI classification.
- Maximum-likelihood and MAP-trained deep networks tend to be overconfident, especially for inputs far from the training distribution.
- Bayesian inference estimates parameter distributions whose integration can provide better predictive uncertainty than point estimates.
- The paper adopts stochastic gradient variational inference to estimate posterior distributions over neural-network weight matrices.
- Auxiliary random variables and normalizing flows are introduced to produce flexible approximate posteriors efficiently.
2. Multiplicative normalizing flows
The paper augments variational Bayesian neural-network posteriors with multiplicative auxiliary variables, applying normalizing flows in a way that preserves local reparametrizations. This increases posterior flexibility while retaining a tractable, auxiliary-distribution-based lower bound.
- 2.1. Variational inference for Bayesian Neural Networks: Variational inference estimates a posterior over neural-network weight matrices, with approximate-posterior flexibility determining how well true parameter uncertainty is captured.The lower bound is optimized using reparametrization and stochastic gradient techniques.
- 2.2. Improving the variational approximation: Applying flows directly to weight-matrix samples is expensive and loses the local-reparametrization benefits available with Gaussian approximate posteriors.The paper addresses both computational cost and preservation of local reparametrizations.
- 2.2. Improving the variational approximation: Multiplicative normalizing flows use auxiliary variables z to form a compound q(W), with a Gaussian q(W|z) enabling local reparametrizations and a flow-enhanced q(z) modeling richer dependencies.The resulting approximation can represent multimodality and nonlinear dependencies between weight-matrix elements.
- 2.2. Improving the variational approximation: The forward pass applies a normalizing flow to sampled multiplicative variables before computing activation means and variances, followed by Gaussian activation noise.Algorithms cover both fully connected and convolutional layers.
- 2.3. Bounding the entropy: An auxiliary distribution r(z|W) lower-bounds the otherwise intractable entropy term, making the objective tractable while introducing a potentially looser bound.Bound tightness depends on how well r(z|W) approximates q(z|W), and inverse normalizing flows provide flexibility for r.
- 2.3. Bounding the entropy: The bound generalizes the dropout variational bound, while restricting likelihood contributions to a lower range introduces small bias and loosens the bound to avoid bad local minima.The restriction is described as analogous to the free-bits objective.
3. Related work
Prior work on approximate Bayesian inference spans deterministic approximations, MCMC, and variational methods, while bootstrap and ensembles offer non-Bayesian alternatives for uncertainty estimation.
- Laplace approximation offers an easy-to-obtain Gaussian posterior but scales poorly without performance-reducing approximations.
- Hamiltonian Monte Carlo is regarded as a strong reference method, but large networks and datasets make storage and convergence assessment difficult.
- Variational inference methods progressed from biased fully factorized estimators toward improved deterministic approximations for Bayesian neural networks.
- Bootstrap and ensembles can provide reasonable confidence intervals in certain scenarios without using the Bayesian paradigm.
4. Experiments
The experiments compare MNF posteriors with dropout, ensembles, and mean-field Bayesian networks across predictive accuracy, uncertainty, adversarial examples, and toy regression. MNFs generally improve the accuracy–uncertainty trade-off and produce more realistic uncertainty, although adversarial uncertainty remains difficult on CIFAR 10.
- Predictive performance and uncertainty: MNF posteriors improve upon mean field while reaching similar accuracies to Deep Ensembles on MNIST and CIFAR 10.The comparison uses LeNet architectures and reports test errors in Table 2.
- Predictive performance and uncertainty: MNF uncertainty estimates on notMNIST are better than other approaches because low-entropy predictions occur less often.Weight decay is most overconfident, while Dropout lies between it and MNFs.
- Predictive performance and uncertainty: MNF posteriors provide a better trade-off between uncertainty and accuracy on observed CIFAR 10 classes.The log-uniform-prior network exhibits substantial pruning, with 94.9%, 99.8%, 99.9%, and 92.7% sparsity levels across its layers.
- Accuracy and uncertainty on adversarial examples: On MNIST adversarial examples, MNF accuracy drops close to random while entropy rises to almost maximum, indicating nearly uniform predictions.Dropout retains better adversarial accuracy but remains overconfident, with the dominant class receiving above 0.7 probability on average.
- Accuracy and uncertainty on adversarial examples: On CIFAR 10 adversarial examples, all methods remain overconfident, suggesting that higher dimensionality makes uncertainty harder to recover.The authors state that this setting requires further investigation.
- Regression on toy dataset: For toy regression, MNF predictive distributions are closer to the true posterior and more uncertain in regions without observed data.Fixed-rate Dropout performs better than FFLU uncertainty in those areas, while learned-rate Dropout produces the most overfitted uncertainties.
5. Conclusion
The paper introduces multiplicative normalizing flows for variational Bayesian neural networks and reports improved predictive performance and uncertainty relative to mean field. MNFs achieve predictive capabilities on par with Dropout while producing more realistic uncertainties, although optimizer choice and prior design remain open directions.
- MNF posteriors significantly improve upon mean field in predictive performance and predictive uncertainty.
- M במש?
A. Memorization capabilities
Deep neural networks can memorize random labels, fitting training data while achieving random-chance test accuracy even with Dropout or weight decay. The cited comparison reports random-chance baselines of 11% on MNIST and 20% on CIFAR 5.
- Memorization under random labels: Deep neural networks can perfectly fit training data with random labels while achieving random-chance accuracy on test data.This behavior was reported even when using Dropout or weight decay regularization.
- Memorization under random labels: Dropout and weight decay did not prevent the reported random-label memorization phenomenon.
- Evaluation baselines: Random-chance accuracy is 11% on MNIST and 20% on CIFAR 5 in the LeNet random-label evaluation.