PaperlayerView exact revision on arXiv

Source-linked AI summary

Liberty or Depth: Deep Bayesian Neural Nets Do Not Need Complex Weight Posterior Approximations

Sebastian Farquhar, Lewis Smith, Yarin Gal

arXiv:2002.03704v4cs.LGstat.ML

TL;DR

Mean-field variational inference is often considered too restrictive for Bayesian neural networks, motivating more complex weight-posterior approximations. This paper develops theoretical arguments and empirical tests of whether depth can instead support rich function-space predictions with mean-field weights. It concludes that deep mean-field networks can approximate predictive distributions and can approach structured-covariance performance in sufficiently deep settings.

  • Problem

    Researchers have assumed that mean-field weight posteriors are too restrictive for Bayesian neural networks, prompting interest in more complex approximations.

    Source: §1 · p. 1

  • Method

    The paper analyzes induced covariance in deep linear networks, extends the analysis to piecewise-linear networks, proves a function-space approximation result, and tests posterior approximation with HMC and deep-network comparisons.

    Source: §1 · p. 2, §4 · p. 5, §5 · p. 6

  • Results

    Mean-field networks with at least two hidden layers can approximate any posterior distribution over predictive functions, while sufficiently deep models show a small or nonexistent performance gap against structured-covariance approximations.

    Source: §7 · p. 9

  • Takeaways & Limitations

    Depth and richer weight-posterior approximations are complementary ways to create rich predictive-function distributions, making deeper mean-field models a supported alternative within the paper's stated scope.

    Source: §7 · p. 9

    The approximation result may require arbitrarily wide models and reduced weight variance for arbitrarily small error, and it does not show that VI will find the approximate posterior.

    Source: §5 · p. 6

Abstract

from arXiv · show

We challenge the longstanding assumption that the mean-field approximation for variational inference in Bayesian neural networks is severely restrictive, and show this is not the case in deep networks. We prove several results indicating that deep mean-field variational weight posteriors can induce similar distributions in function-space to those induced by shallower networks with complex weight posteriors. We validate our theoretical contributions empirically, both through examination of the weight posterior using Hamiltonian Monte Carlo in small models and by comparing diagonal- to structured-covariance in large settings. Since complex variational posteriors are often expensive and cumbersome to implement, our results suggest that using mean-field variational inference in a deeper model is both a practical and theoretically justified alternative to structured approximations.

1 Introduction

The paper challenges the assumption that mean-field variational inference is too restrictive for deep Bayesian neural networks, arguing that depth can support rich function-space predictions with simple weight posteriors. It develops theoretical results and empirical evidence for this claim.

1 Introduction

  • Mean-field VI assumes factorized weight posteriors, yet researchers have treated this diagonal-covariance approximation as severely limiting; structured alternatives are costly.The introduction notes that even efficient structured methods can take over twice as long per epoch as comparable mean-field approaches [Osawa et al., 2019].Source: §1 · p. 1
  • Deeper models can make mean-field approximations matter less for posterior predictive distributions over functions than they do in small, shallow networks.The paper frames depth and complex covariance as complementary routes to rich function-space distributions; function-space predictions, rather than weight posteriors for their own sake, matter for measures such as accuracy and log-likelihood.Source: §1 · pp. 1–2
  • For any full-covariance BNN, the Weight Distribution Hypothesis posits a deeper mean-field BNN with a similar function-space posterior predictive.The introduction distinguishes this hypothesis from the True Posterior Hypothesis, which asks whether mean-field weight distributions can approximate the true predictive posterior.Source: §1 · p. 2
  • In deep linear networks, three weight layers suffice for nonzero correlations between any pair of product-matrix elements, and a Matrix Variate Gaussian is a special case.The paper introduces the local product matrix to partially extend these linear-network results to piecewise-linear activations such as ReLUs.Source: §1 · p. 2
  • The paper proves the True Posterior Hypothesis for BNNs with at least two hidden layers and arbitrarily wide models, and reports supporting experiments.HMC experiments find that mean-field approximation errors decrease with depth in small networks; deep convolutional experiments find no significant performance difference between a particular diagonal and full-covariance method (SWAG).Source: §1 · p. 2

2 Related Work

Prior work has regarded mean-field VI as restrictive and explored richer posterior approximations, despite their computational costs. The paper notes that evidence for the claimed restriction in deep networks was limited, while mean-field methods had also succeeded in practice.

2 Related Work

  • Full-covariance VI was introduced to capture weight correlations, but its parameter count and time complexity prompted research into structured-covariance alternatives.Structured methods still have unattractive time complexity relative to mean-field VI; Table 1 compares forward-pass complexity and notes mean-field VI avoids numerically unstable matrix inversion.Source: §2 · pp. 2–3, Table 1 · p. 2
  • Evidence that mean-field is too restrictive had typically concerned shallow networks, including single-layer regression pathologies, not demonstrations in deep networks.Foong et al. [2020] conjectured that their identified pathologies extend to deeper models and classification, but did not prove this.Source: §2 · p. 3
  • Mean-field and even more restrictive approximations have succeeded in deep models, with some work addressing gradient variance rather than posterior restrictiveness.Hinton and van Camp [1993] hypothesized that optimization can find network parameters for which the mean-field restriction is least costly.Source: §2 · p. 3

3 Emergence of Complex Covariance in Deep Linear Mean-Field Networks

Deep linear mean-field networks can induce complex covariance in their product matrices, even though each layer has factorized weights. Under restricted conditions, deeper mean-field networks can also match function-space predictions from models with structured weight distributions.

Defining a Product Matrix

  • In a deep linear model, multiplying the weight matrices produces a single product matrix that maps inputs to outputs and defines the induced function distribution.The analysis derives its covariance for two and three layers and an update rule for increasing depth; the results apply to factorized weights with finite first- and second-order moments.Source: §3 · p. 3

Covariance of the Product Matrix

  • For L ≥3, a product matrix of factorized weight matrices can have non-zero covariance between every pair of its elements.This shows that deep mean-field linear models can induce function-space distributions requiring weight covariance in a shallower model.Source: §3 · p. 3

How Expressive is the Product Matrix?

  • A product matrix can have a Matrix Variate Gaussian distribution when its outer factors are deterministic and its middle factor has fully factorized unit-variance Gaussian elements.This yields the restricted result that a deeper linear mean-field model can induce the same function-space posterior predictive as a linear model with MVG weights.Source: §3 · p. 4
  • The expressiveness result is a lower bound, not a demonstration that every possible product-matrix covariance can be induced.The analysis uses strong parameterization restrictions and is not proposed as a practical method.Source: §3 · p. 4
  • A full-covariance layer has more degrees of freedom than a three-layer mean-field product matrix, illustrating a practical cost of full covariance.The compared product matrix has 2LK^2 parameters, while one full-covariance weight matrix has K^2 mean parameters and K^2(K^2 + 1)/2 covariance parameters.Source: §3 · p. 4

4 Weight Dist. Hypothesis in Deep Piecewise-Linear Mean-Field BNNs

For piecewise-linear networks, the local product matrix extends the product-matrix analysis to a fixed input region. Its elements can develop off-diagonal covariance under mean-field weights, subject to stated activation conditions.

Defining a Local Product Matrix

  • Within each activation region, a piecewise-linear network acts linearly, allowing its weight layers and activation pattern to define a local product matrix at an input.Sampling the local product matrices across weight realizations gives the same distribution for the output at that input as multiplying the local product matrix by the input.Source: §4 · p. 5

Covariance of the Local Product Matrix

  • For L ≥3 and activations that are non-zero everywhere, all elements of the local product matrix can have non-zero off-diagonal covariance.This illustrates how deep factorized networks can emulate rich covariance in a shallower network, but does not establish exact replication of every full-covariance weight distribution.Source: §4 · p. 5
  • The result is restricted to activations that are non-zero everywhere; for activations such as ReLU, layers switched off at an input could reduce effective depth below three.The authors report that they cannot rule this out theoretically, although they say it seems unlikely in a trained network.Source: §4 · p. 5

Numerical Simulation

  • In a Leaky ReLU FashionMNIST model, 10,000 samples showed increasingly complex off-diagonal covariance in the local product matrix as depth increased.The experiments also found extra correlations tied to the activation pattern predominantly present at an input.Source: §4 · p. 5

5 True Posterior Hypothesis in Two-Hidden-Layer Mean-Field Networks

The paper uses a stochastic adaptation of the universal approximation theorem to show that two-hidden-layer mean-field BNNs can approximate true posterior predictive distributions arbitrarily closely, given arbitrary width and stated conditions. The result has practical limitations, including potentially very wide models and low weight-space ELBOs.

Proof of the True Posterior Hypothesis

  • With two hidden layers and arbitrary width, a mean-field BNN can approximate any true posterior predictive distribution over function values arbitrarily closely, under stated conditions.These include a first hidden layer of at least D + 1 units, non-polynomial non-periodic activations, and a continuous finite non-zero predictive density.Source: §5 · p. 6
  • The proof uses a mean-field distribution to induce a distribution over first-layer hidden units, then approximates the inverse cumulative density function of the true predictive posterior.The result strengthens prior work that considered only the first two moments of the predictive posterior.Source: §5 · p. 6

Limitations

  • The construction may require arbitrarily wide models and reduced weight variance for small error, potentially producing very low weight-space ELBOs.The proposition does not show that variational inference will find such an approximate posterior.Source: §5 · p. 6

6 Empirical Validation

Across small-model posterior comparisons and larger-task evaluations, the evidence suggests that deeper models can reduce the practical gap between diagonal and structured covariance. The results span HMC analyses, Iris, CIFAR-100, and ImageNet, with the large-model posterior probes carrying scope caveats.

6.1 Examining the True Posterior with Hamiltonian Monte Carlo

  • As depth increases, HMC-based Wasserstein and KL errors fall rapidly across activations, including models without activations.Models have roughly 1,000 parameters; the plotted depth trend is consistent with deeper models having mean-field modes of the true posterior.Source: Figure 2 · p. 7, §6.1 · p. 7
  • For the HMC comparison, the authors fit full-covariance and factorized Gaussians to No-U-turn HMC samples from the two-moons posterior.They deliberately target a single mode, average each plotted point over 20 restarts, and keep models near 1,000 parameters across depths.Source: §6.1 · p. 7

6.2 Factorization and Downstream Task Performance

  • On Iris, full covariance achieves lower test cross-entropy with one hidden layer, while deeper mean-field models match it.Training full-covariance VI is intractable except for very small models because of optimization difficulties.Source: Figure 3 · p. 8, §6.2 · p. 8
  • On CIFAR-100, diagonal and low-rank SWAG show no observable negative-log-likelihood difference in the evaluated PresNets, all of which have more than two hidden layers.The authors describe this as an approximate probe because they cannot compute true-posterior samples in larger models.Source: Figure 4 · p. 8, §6.2 · p. 8
  • On ImageNet, diagonal- and structured-covariance methods have similar accuracy, log-likelihood, and expected calibration error, with method-specific marginal differences.VOGN's mean-field model is marginally better on all three measures; SWAG's diagonal approximation has marginally better accuracy but marginally worse log-likelihood and ECE.Source: §6.2 · p. 8, Table 2 · p. 9

7 Discussion

The discussion argues that depth and richer weight-posterior approximations are complementary ways to obtain expressive predictive-function distributions. It connects results for deep linear and piecewise-linear networks to a theorem about sufficiently wide models.

7 Discussion

  • In deep linear models, multiplying mean-field weight layers induces a rich-covariance product matrix, with the Matrix Variate Gaussian as a special case from three weight layers onward.Source: §7 · p. 9
  • The local product matrix extends the linear analysis to piecewise-linear networks, and its covariance analysis and HMC samples suggest the linear results inform nonlinear networks.Source: §7 · p. 9
  • Mean-field weight distributions in networks with at least two hidden layers can approximate any posterior distribution over predictive functions.The claim is proved for sufficiently wide models.Source: §7 · p. 9
  • The authors argue that larger models can make some true-posterior modes approximately mean-field, leaving other challenges for MFVI at scale to address.Source: §7 · p. 9

Broader Impact

The broader-impact discussion presents cheap uncertainty approximations as relevant to scalable neural-network systems, while warning against relying on uncertainty measures uncritically in critical decisions. It also notes the computational advantages of mean-field VI.

Broader Impact

  • The work demonstrates the theoretical applicability of cheap uncertainty approximations that avoid modeling complex weight correlations in large-scale settings.The authors say this challenges an assumed tension between good uncertainty and powerful models, while recommending caution about reliance on uncertainty in critical decisions.Source: Broader Impact · p. 10
  • The authors see potential for Bayesian neural networks with robust uncertainty in a wide range of settings, but advise caution when decisions depend on uncertainty measures in critical settings.Source: Broader Impact · p. 10

A Introduction to Variational Inference in Bayesian Neural Networks

B Discussions of Foong et al. 2020

The authors distinguish their deep-network analysis from Foong et al.’s small-model regression findings, and report that depth improves mean-field networks’ ability to capture in-between uncertainty. Their accompanying experiments examine induced product-matrix covariance and use a deliberately limited Iris setting to compare covariance structures.

B Discussions of Foong et al. 2020

C.1 Full Description of Covariance Visualization

  • A three-layer Leaky ReLU mean-field model produced multimodal local product-matrix entries: 12 of 20 sampled entries were multimodal, unlike the examined linear case.Source: §C.1 · p. 17

C.2 Effect of Depth Measured on Iris Experimental Settings

  • In the full- versus diagonal-covariance experiment, some deeper-model seeds failed; authors therefore report mean and standard error for the 10 best of 100 runs, framing results as relative best-case merits, not a real-world benchmark.Source: §C.2 · p. 17
  • Iris was chosen because its small training set yields a broad posterior, a best-case setting for full-covariance approximation.Source: §C.2 · p. 17

C.3 HMC Experimental Settings

The appendix describes how the authors compare mean-field and full-covariance approximations to HMC samples and vary SWAG model depth. It also notes limits on the reliability of some deep-model HMC estimates and reports no clear CIFAR-100 performance difference between diagonal and low-rank SWAG.

C.4 Diagonal- and Structured-SWAG at Varying Depths

  • CIFAR-100 shows no clear accuracy difference between diagonal- and low-rank-covariance SWAG models above the paper’s depth threshold.Source: Figure 9 · p. 18

C.3 HMC Experimental Settings

  • The ReLU HMC estimates may be less reliable beyond about seven layers; acceptance rates were generally near the stated 65% target range.Source: §C.3 · p. 18, Table 6 · p. 18
  • The HMC comparison fits one Gaussian mode, selected from mixtures of one to four components by Bayesian information criterion, because VI is mode-seeking.Source: Figure 8 · p. 18, §C.3 · p. 19

C.4 Diagonal- and Structured-SWAG at Varying Depths

  • SWAG depth varied across seven PreResNet configurations, with 10 seeds for error bars and 30 forward-pass samples.Source: §C.4 · p. 19

C.5 Large Scale Experiments Descriptions

The large-scale comparisons report little systematic difference between mean-field and structured methods within CIFAR-10 architectures. The accompanying ImageNet standard deviations are based on three runs and understate variability across training configurations.

C.5 Large Scale Experiments Descriptions

  • ImageNet standard deviations cover three runs and underestimate method variability because they vary random seed while holding the training configuration otherwise fixed.Source: §C.5 · p. 19
  • Within the same CIFAR-10 architecture, mean-field and structured-covariance methods show little evidence of systematic accuracy differences, and observed gaps are marginal.Source: §C.5 · p. 19

D.1 Full Derivation of the Product Matrix Covariance

The derivations show that products of factorized weight matrices can induce broad covariance structures, including matrix-variate Gaussians as a special case. The appendix also examines product-entry distributions and defines a local product matrix that represents predictive function values within a shared linear region.

D.1 Full Derivation of the Product Matrix Covariance

  • For three or more factorized weight matrices, some choices yield nonzero covariance between every pair of product-matrix elements.The existence proof restricts means and standard deviations to positive values; the authors do not claim every covariance matrix is attainable in the linear case.Source: §D.1 · pp. 19, 22
  • The CIFAR-10 and ImageNet table captions describe only slight differences between comparable methods on a given architecture.Source: Table 7 · p. 20, Table 8 · p. 20

D.2 Matrix Variate Gaussian as a Special Case of Three-Layer Product Matrix

  • With deterministic outer matrices and a factorized unit-variance Gaussian middle matrix, the three-layer product matrix has a matrix-variate Gaussian distribution.Source: §D.2 · pp. 22–23

D.3 Distribution of the Product Matrix

  • For diagonal Gaussian priors with element variance 0.232, simulated product-matrix entries remain roughly similarly distributed across depths, though they are not strictly Gaussian.Source: §D.3 · p. 24, Figure 10 · p. 24

D.4.1 Proof of Local Linearity

  • With piecewise-linear activations, almost every input lies in a compact region where a realized network function is linear.Source: §D.4.1 · p. 24

D.4.2 Defining the Local Product Matrix

  • The nonzero-measure argument for the shared linear region applies to any finite collection of weight samples, but does not guarantee nonzero measure as sample count tends to infinity.Source: §D.4.2 · p. 25
  • The local product matrix incorporates activation patterns so that its product with an input has the predictive-posterior distribution within the shared linear region.Source: §D.4.2 · p. 26

D.4.3 Proof that the Local Product Matrix has Non-zero Off-diagonal Covariance

The appendix establishes when deep networks with mean-field weights can produce correlated local product matrices and approximate posterior predictive distributions. Its constructions build from a simplified uniform-input network to a two-hidden-layer Bayesian neural network.

D.4.3 Proof that the Local Product Matrix has Non-zero Off-diagonal Covariance

  • For L ≥ 3 and activations nonzero everywhere, some weight matrices give every local product-matrix element non-zero off-diagonal covariance.The proof extends the result from L = 3 to all larger L.Source: §D.4.3 · pp. 26–27
  • When activation constants vanish, nonlinearities such as ReLU can disconnect regions, reducing effective depth below 3 and preventing covariance between every element.The proof assumes the relevant activation constants are nonzero except on a measure-zero region.Source: §D.4.3 · p. 27

D.5 Existence Proof of a Two-Hidden-Layer Mean-field Approximate Posterior Inducing the True Posterior Predictive

  • Under stated activation, width, and predictive-density conditions, a two-hidden-layer mean-field BNN can make its predictive density arbitrarily close to that of a multivariate regression target.The first hidden layer needs at least D + 1 units, while the second can be arbitrarily wide.Source: §D.5 · p. 27

D.5.1 Simplified Construction

  • A sufficiently wide single-hidden-layer network taking x and uniform U as inputs can approximate a univariate predictive density within ε when the c.d.f. is continuous and the p.d.f. is everywhere nonzero.The construction approximates the inverse c.d.f. and transfers the bound to the predictive density.Source: §D.5.1 · pp. 28–29

D.5.2 Full Construction

  • The full construction uses the first BNN layer to map x to a near-copy x′ and Gaussian Z, then uses a stochastic later layer to approximate the target mapping.This yields a probabilistic bound while extending the construction to multivariate regression.Source: §D.5.2 · pp. 29–30

Step 2: Invoking a Result Similar to the Simplified Construction

The proof shows that fully factorized Gaussian weights in the later network layers can approximate the transformed target mapping and its predictive density. It uses a near-copy of the input and Gaussian noise, under continuity and nonzero-density assumptions.

Step 2: Invoking a Result Similar to the Simplified Construction

  • The Gaussian-to-target transformation gives equality in distribution when the posterior predictive c.d.f. is continuous and its p.d.f. is nonzero everywhere.The construction maps Gaussian Z through its c.d.f. to a uniform variable, then through the target inverse c.d.f.Source: Step 2: Invoking a Result Similar to… · pp. 30–32
  • The first layer maps x to a near-copy x′ and Gaussian Z; arbitrarily small first-layer variance lets the proof transfer its bound from x′ back to x.For any ε, the proof gives a δ such that the probability of an x′ deviation exceeding ε is below δ.Source: Step 2: Invoking a Result Similar to… · p. 31
  • Mean-field Gaussian weights in the later layers approximate the inverse-c.d.f. mapping, yielding a probabilistic bound on predictive-density error.The bound is over the weight distributions of the later layers and biases.Source: Step 2: Invoking a Result Similar to… · p. 31

Related papers

Loading 2002.03704v4…