Source-linked AI summary
Bayesian Hypernetworks
David Krueger, Chin-Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, Aaron Courville
TL;DR
Bayesian hypernetworks address approximate Bayesian inference for neural-network parameters by transforming simple noise into a flexible posterior with an invertible hypernetwork. They support multimodal, correlated parameter distributions with efficient sampling and show benefits across uncertainty-evaluation tasks, including adversarial-example defense.
Problem
Existing generative parameter models can make the entropy term needed for variational inference difficult to evaluate because many-to-one mappings require integrating over latent noise variables.
Method
Bayesian hypernetworks use an invertible hypernetwork to generate primary-network parameters from noise and enable Monte Carlo estimation of the entropy term during variational-inference training.
Results
Bayesian hypernetworks express complicated multimodal distributions, including dependent parameters, and provide benefits over simpler Bayesian deep-learning methods across regularization, active learning, anomaly detection, and adversarial-example defense.
Takeaways & Limitations
Bayesian hypernetworks provide efficient training and sampling while broadening the posterior distributions representable for Bayesian deep learning.
Takeaways & Limitations
Bayesian hypernetworks, like most Bayesian deep-learning approaches, do not capture uncertainty about which model specification is correct.
Abstract
from arXiv · showhide
We study Bayesian hypernetworks: a framework for approximate Bayesian inference in neural networks. A Bayesian hypernetwork $\h$ is a neural network which learns to transform a simple noise distribution, $p(\vecε) = \N(\vec 0,\mat I)$, to a distribution $q(\pp) := q(h(\vecε))$ over the parameters $\pp$ of another neural network (the "primary network")\@. We train $q$ with variational inference, using an invertible $\h$ to enable efficient estimation of the variational lower bound on the posterior $p(\pp | \D)$ via sampling. In contrast to most methods for Bayesian deep learning, Bayesian hypernets can represent a complex multimodal approximate posterior with correlations between parameters, while enabling cheap iid sampling of~$q(\pp)$. In practice, Bayesian hypernets can provide a better defense against adversarial examples than dropout, and also exhibit competitive performance on a suite of tasks which evaluate model uncertainty, including regularization, active learning, and anomaly detection.
1 INTRODUCTION
Bayesian hypernetworks address limitations of simple variational posteriors by learning flexible distributions over neural-network parameters. An invertible hypernetwork transforms noise into primary-network parameters, enabling efficient variational training and sampling.
- Motivation: Bayesian DNNs model parameter uncertainty to improve confidence calibration, which can support safer behavior when mistakes are costly.Applications include seeking human guidance, using safe defaults, or rejecting uncertain classifications.
- Problem: Standard variational approximations often restrict the posterior to simple families and can underestimate uncertainty by penalizing mass in low-density regions of the true posterior.The restriction can prevent the approximation from representing multiple modes of the true posterior.
- Contribution: Bayesian hypernetworks use a neural network h to learn an expressive posterior over the parameters of a primary network.The paper proposes this approach to represent complicated parameter distributions rather than a single mode.
- Method: An invertible hypernetwork maps Gaussian noise to primary-network parameter samples and makes the entropy term of the variational objective estimable by Monte Carlo.This supports variational inference while retaining sampling-based evaluation.
- Evaluation: The paper evaluates Bayesian hypernets through prior-work review, method development, and experiments on expressivity and uncertainty-related tasks.The experiments address expressivity, regularization, active learning, and anomaly detection.
2 RELATED WORK
Prior Bayesian neural-network methods use several inference strategies, with variational inference often relying on restricted posteriors. Bayesian hypernetworks instead use generative and invertible-network ideas to model richer parameter distributions while addressing scalability and entropy-estimation challenges.
- Bayesian DNNs: Bayesian neural-network research includes MCMC, variational inference, expectation propagation, and α-divergences, while this paper focuses on variational inference.Variational inference is identified as the most popular approach in the cited overview.
- Bayesian DNNs: MC dropout provides simple implementation and cheap posterior samples, whereas Gaussian-dropout approaches remain unimodal and cannot represent arbitrary parameter dependencies.These methods are important comparison points for expressive Bayesian approximations.
- Bayesian DNNs: Bayes by Backprop is a special Bayesian-hypernetwork case that applies element-wise scale and shift to noise, producing a factorial Gaussian posterior.This limits the dependencies and distributional structure that can be represented.
- Related hypernetworks: Earlier hypernetwork-based approaches generated scaling factors for factorial-Gaussian means, but scaling to large primary networks remained a challenge addressed by this work.The related approach also required an auxiliary inference network to estimate the entropy term, weakening the variational-bound estimate.
- Hypernetworks: Hypernetworks generate parameters for primary networks, but naive parameterization becomes impractical because DNN parameter counts scale quadratically with units per layer.Normalization-based techniques reduce this burden by generating scale and shift parameters or otherwise reparameterizing weights.
- Generative modeling: Bayesian hypernets use differentiable directed generators and invertible generative-model techniques to transform simple noise into complex parameter distributions.Invertibility enables change-of-variables calculations involving the inverse Jacobian.
3 METHODS
The method uses invertible hypernetworks to generate samples from flexible parameter posteriors, enabling Monte Carlo variational inference for Bayesian deep networks. A weight-normalized parametrization improves scalability while retaining multimodality and parameter dependence.
- Variational Inference: Variational inference maximizes the ELBO, a lower bound on the marginal log-likelihood, using likelihood, prior, and entropy terms.The expectation is generally intractable for deep networks and is estimated with Monte Carlo sampling.
- Bayesian Hypernets: Bayesian hypernets transform random noise into independent samples from an approximate posterior over primary-network parameters, enabling Monte Carlo estimates of the ELBO and its gradients.The estimates can be backpropagated to train the hypernetwork.
- Bayesian Hypernets: Invertibility allows the entropy term of the ELBO to be evaluated through the change-of-variables formula, avoiding likelihood-intractability issues in many-to-one generative models.RealNVP and inverse autoregressive flows are used to train invertible hypernetworks efficiently.
- Efficient Parametrization and Training of Bayesian Hypernets: The proposed scalable parametrization outputs scaling factors while learning maximum-likelihood weight directions, reducing computation from quadratic to linear in the number of primary-network units.This restricts the approximate-posterior family but still permits substantial multimodality and parameter dependence.
- Efficient Parametrization and Training of Bayesian Hypernets: Weight normalization, small initialization, and clipped softmax outputs are used to stabilize hypernetwork training and improve numerical stability.Weight normalization within the hypernetwork was found to stabilize training dramatically.
4 EXPERIMENTS
Experiments evaluate Bayesian hypernetworks across regularization, active learning, anomaly detection, and adversarial-example detection, including their ability to model dependent and multimodal posteriors. BHNs generally perform competitively or outperform comparison methods, with performance depending on architecture and training procedure.
- 4 EXPERIMENTS: BHN experiments span MNIST, CIFAR10, and 1D regression across regularization, active learning, anomaly detection, and adversarial-example detection.Uncertainty is evaluated through several tasks rather than a single metric.
- 4.1 QUALITATIVE RESULTS AND VISUALIZATION: BHNs learn dependent and multimodal approximate posteriors, including both modes a = b = 1 and a = b = −1 in an over-parameterized identity-function model.The experiment uses the primary network ŷ = a · b · x.
- 4.2 CLASSIFICATION: BHNs act as regularizers and perform on par with dropout on full MNIST and CIFAR10 datasets, while additional coupling layers improve performance.Models with zero coupling layers cannot model dependencies between parameters.
- 4.3 ACTIVE LEARNING: Warm-started BHNs initially lag but outperform other methods after moderate to large numbers of active-learning acquisitions.Warm-starting was essential for good BHN performance in the reported procedure.
- 4.4 ANOMALY DETECTION: Both BHN and MCdropout improve anomaly-detection performance over the non-Bayesian baseline, but MCdropout performs significantly better than BHN on this task.The maximum-confidence softmax score, or variation ratio, gave the best performance among tested acquisition functions.
- 4.5 ADVERSARY DETECTION: IAF-based BHNs consistently outperform RealNVP-based BHNs and dropout in detecting adversarial examples and errors.For BHNs, BALD and Mean STD uncertainty measures increase strongly with perturbation, unlike dropout.
5 CONCLUSIONS
The paper introduces Bayesian hypernets as an invertible-hypernetwork approach to variational Bayesian deep learning. BHNs model complicated multimodal parameter distributions, train and sample efficiently, and provide benefits over simpler Bayesian methods.
- Bayesian hypernets use an invertible hypernetwork as a generative model for neural-network parameters.
- BHN parameterizations scale to real-world tasks while supporting efficient training and sampling.
- Bーム BHNs express complicated multimodal distributions, addressing overconfidence issues in simpler variational approximations.
- The paper reports significant benefits over simpler methods for Bayesian deep learning.
A.1 LEARNING CORRELATED WEIGHTS
The hypernetwork posterior captures correlations between different parameters, as shown by Pearson tests and a scatter matrix of posterior samples.
- Many Pearson correlation test p-values are below .05, indicating detected correlations among sampled parameters.Figure 6 pairs the p-value histogram with a scatter matrix of samples from the hypernetwork approximate posterior.
- The scatter matrix visualizes dependencies between parameters in the hypernetwork approximate posterior.
- Figure 6 combines a Pearson p-value histogram with a scatter matrix to assess posterior parameter correlations.
A.2 UNSEEN MODE DETECTION
The paper replicates anomaly detection on MNIST with unseen classes and reports results organized by the class omitted during training and by precision-recall and ROC metrics.
- The anomaly-detection experiment is replicated on MNIST with unseen classes.
- The table’s first column identifies the class label missing from the training set.
- Results are grouped into ROC score, positive precision-recall, and negative precision-recall blocks.
A.3 STRONGER ATTACK
The stronger-attack experiment estimates the input-gradient direction from 32 samples, linking gradient quality to attack strength and adversarial-example detection.
- 32 samples are used to estimate the gradient direction with respect to the input.
- A better gradient estimate produces a stronger attack, lowering accuracy for a given step size.
- A more informative uncertainty measure can make adversarial examples easier to detect.
B DERIVATION OF TRAINING OBJECTIVE
The training objective uses a variational lower bound with weight normalization, treating only the scaling factors g as random variables. The bound is optimized over the primary network’s direction and bias parameters and the hypernetwork parameters.
- Weight normalization treats only the scaling factors g as random variables and assigns them an isotropic Gaussian prior.This prior induces an L2 weight-decay penalty on g.
- The variational lower bound is optimized with respect to the primary network parameters v and b and the hypernetwork parameters φ.