PaperlayerView exact revision on arXiv

Source-linked AI summary

Model Selection in Bayesian Neural Networks via Horseshoe Priors

Soumya Ghosh, Finale Doshi-Velez

arXiv:1705.10388v1stat.ML

TL;DR

BNN model selection remains open because choosing too few or too many nodes can impair model fit or leave predictions uncertain. This paper places Horseshoe priors on weights incident to each node, shrinking unnecessary nodes. It reports avoiding under-fitting with grossly overestimated network width while learning compact networks without sacrificing predictive performance.

  • Problem

    BNN model selection receives relatively less attention, despite architecture choices involving a trade-off between insufficient flexibility and large predictive variance.

    Source: §1 · p. 1

  • Method

    The paper places Horseshoe priors on weights incident to each node, using shared scales to shrink unnecessary nodes and select network size.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    The approach avoids under-fitting when node count is grossly overestimated and learns compact networks without sacrificing—and sometimes improving—predictive performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Horseshoe shrinkage offers computationally efficient, statistically effective model selection over the number of BNN nodes.

    Source: §1 · p. 1

    Strong weight-scale correlations create a difficult posterior geometry, and fully factorized approximations can produce non-sparse solutions.

    Source: §3 · p. 3

Abstract

from arXiv · show

Bayesian Neural Networks (BNNs) have recently received increasing attention for their ability to provide well-calibrated posterior uncertainties. However, model selection---even choosing the number of nodes---remains an open question. In this work, we apply a horseshoe prior over node pre-activations of a Bayesian neural network, which effectively turns off nodes that do not help explain the data. We demonstrate that our prior prevents the BNN from under-fitting even when the number of nodes required is grossly over-estimated. Moreover, this model selection over the number of nodes doesn't come at the expense of predictive or computational performance; in fact, we learn smaller networks with comparable predictive performance to current approaches.

1 Introduction

BNN model selection remains difficult because too few nodes limit flexibility while too many can leave predictions with large variance. The paper uses Horseshoe priors over incident-weight variances to shrink unneeded nodes and select network size.

1 Introduction

  • Too few BNN nodes can limit flexibility, while too many can leave predictions with large variance, making architecture selection important.Practitioners typically search over discrete layer sizes.Source: §1 · p. 1
  • Horseshoe priors on the weights incident to each node can shrink unnecessary nodes toward zero and support model selection over node count.Small variance parameters effectively turn nodes off; large ones correspond to active nodes.Source: §1 · p. 1
  • The continuous Horseshoe relaxation keeps node selection differentiable, allowing training with variational inference methods.It mimics a spike-and-slab on-off choice for each node.Source: §1 · p. 2
  • The approach avoids under-fitting when the required node count is grossly overestimated and learns compact networks without sacrificing predictive performance.The paper reports predictive performance that is sometimes improved.Source: §1 · p. 2

2 Bayesian Neural Networks

Bayesian neural networks represent weights as random variables and infer their posterior from data. This paper uses shared, heavy-tailed priors on weights incident to a unit to induce unit-level sparsity and prune units that do not explain the data well.

2 Bayesian Neural Networks

  • A deep network maps inputs to responses by recursively transforming each layer’s input with its weight matrix and a point-wise nonlinearity.The passage gives rectified-linear activation as one example.Source: §2 · p. 2
  • A Bayesian neural network assigns distributions to weights and estimates their posterior given observed input-response pairs.The learned posterior is used to predict responses for unseen inputs.Source: §2 · p. 2
  • The paper’s shared heavy-tailed priors induce sparsity at the unit level, allowing units that do not explain the data well to be pruned.Earlier scale-mixture priors encouraged sparsity in individual weights.Source: §2 · p. 2

3 Automatic Model Selection through Horseshoe Priors

The horseshoe prior shares scale parameters across weights incident to each unit, shrinking small units while allowing sufficiently large ones to escape shrinkage. The section also explains why robust inference requires auxiliary-variable and non-centered parameterizations.

3 Automatic Model Selection through Horseshoe Priors

  • The horseshoe prior’s heavy tails let sufficiently large node weight vectors escape shrinkage, while its spike at zero severely shrinks smaller ones.Shared scales across a unit’s incident weights induce sparsity at the unit level.Source: §3 · pp. 2–3
  • Direct Half-Cauchy parameterization challenges variational learning because exponential-family approximations struggle with thick tails and Cauchy approximations yield high-variance gradients.The paper instead uses an auxiliary-variable parameterization based on inverse-gamma distributions; the output layer uses independent Gaussian priors.Source: §3 · p. 3
  • Strong correlations between node weights and their scales create a funnel-shaped posterior that is difficult to sample or approximate, and factorized approximations can yield non-sparse solutions.These issues can erase the sparsity benefits of the horseshoe prior.Source: §3 · p. 3
  • A non-centered parameterization samples scales and weights from independent prior distributions, with their coupling introduced by the likelihood after conditioning on data.The paper reports improved posterior approximation quality and better discovery of sparse solutions with this parameterization.Source: §3 · pp. 3–4

4 Learning Bayesian Neural Networks with Horseshoe priors

The section formulates variational inference for Bayesian neural networks with Horseshoe priors as ELBO optimization. It addresses intractable likelihood expectations with black-box gradients and pre-activation sampling, then combines gradient and fixed-point updates.

4 Learning Bayesian Neural Networks with Horseshoe priors

  • Variational inference approximates the posterior by minimizing KL divergence, equivalently maximizing an ELBO containing expected log joint density and variational entropy.The ELBO is a lower bound on the marginal likelihood.Source: §4 · p. 4

4.1 Black Box Variational Inference

  • The implementation uses a fully factorized family with Gaussian non-centered weights, log-Normal scales, and inverse-Gamma optimal auxiliary-variable factors.It cycles between Adam gradient updates and fixed-point updates for auxiliary variables to maximize the ELBO.Source: §4 · p. 4, §4.1 · p. 5

4 Learning Bayesian Neural Networks with Horseshoe priors

  • Neural-network nonlinearities and possible non-conjugacy make the ELBO’s first expectation intractable, ruling out traditional fixed-point updates alone.Source: §4 · p. 4

4.1 Black Box Variational Inference

  • Black-box variational inference estimates gradients with unbiased Monte Carlo expectations; reparameterization gradients typically have lower variance than score-function estimators.For Gaussian approximations, a non-centered parameterization separates randomness from the optimized parameters.Source: §4.1 · pp. 4–5
  • Sampling directly from the lower-dimensional variational posterior over pre-activations can further reduce gradient-estimator variance.The weights affect the objective through each layer’s pre-activations.Source: §4.1 · p. 5

5 Related Work

Related work spans scalable Bayesian inference, Bayesian and non-Bayesian structure learning, and sparsity-inducing approaches. The paper distinguishes node-level Horseshoe selection from edge pruning and contrasts prior-driven group shrinkage with methods that do not model weight uncertainty.

5 Related Work

  • Early BNN inference using Laplace approximation or MCMC did not scale well to modern architectures and large datasets, while newer stochastic methods enabled larger-scale inference.Source: §5 · p. 5
  • The Indian buffet process approach was described as vulnerable to poor local optima and reliant on non-scalable MCMC; the mixture prior instead turns off edges, unlike Horseshoe node selection.The Horseshoe extends the mixture approach with an infinite scale mixture, avoiding direct specification of mixture variances and mixing proportion; experiments found relative robustness to prior-scale values.Source: §5 · p. 5
  • Gaussian dropout’s sparsity comes from variational optimization under log-uniform weight priors, an approach the paper describes as orthogonal to prior-driven sparsity.Source: §5 · p. 6
  • Non-Bayesian pruning and group-sparsity methods do not model weight uncertainty and apply uniform shrinkage, whereas the Horseshoe permits large weights in active groups.Source: §5 · p. 6

6 Experiments

The experiments test whether HS-BNN handles excess network capacity and compare its predictive performance and sparsity with other BNN approaches on simulated, MNIST, and UCI regression tasks.

6.1 Experiments on simulated data

  • As network width increases from 50 to 1000 units, Gaussian-prior BNN performance deteriorates quickly, while HS-BNN shows only a marginal predictive-performance drop by pruning extra capacity.The regression experiment uses 20 noisy samples from a cubic function.Source: §6.1 · p. 6, Figure 2 · p. 7
  • On a synthetic classification task, the non-centered horseshoe model prunes all but two units, while the centered horseshoe struggles to recover the known network structure.All three compared models fit the data with high predictive accuracy.Source: §6.1 · p. 6

6.2 Classification and Regression experiments

  • Across MNIST architectures, HS-BNN performs significantly better than BNN, comparably to SM-BNN, and worse than VMG.The test-error plots and sparsity comparisons cover architectures with 400, 800, and 1200 units per hidden layer.Source: §6.2 · p. 7, Figure 3 · p. 8
  • On UCI regression datasets, HS-BNN remains competitive with VMG on RMSE and predictive log likelihood, and outperforms it on some datasets.The authors suggest careful scale-parameter tuning and structured variational approximations as possible directions for improving results.Source: §6.2 · pp. 8–9, Table 1 · p. 9

7 Discussion and Conclusion

The discussion argues that horseshoe model selection depends on combining strong shrinkage with heavy tails, while the appendix describes fixed-point updates for auxiliary variables.

7 Discussion and Conclusion

  • The horseshoe’s near-zero spike and heavy tails are both needed: exponential shrinkage lacks heavy tails, while a half-Cauchy alone does not encourage scales toward zero.The authors contrast these alternatives as lacking one of the properties needed for model selection.Source: §7 · p. 9
  • A continuous horseshoe prior is differentiable and can use existing inference methods, unlike an explicit spike-and-slab model.Source: §7 · p. 9
  • Putting a prior on the variation of z·w may be more general, but the authors found that formulation has many more local optima and is harder to optimize.Source: §7 · p. 9

A Fixed point updates

  • The appendix derives inverse-Gamma fixed-point updates for auxiliary variables and uses Monte Carlo estimates for the likelihood expectation.With the chosen variational families, the entropies are available in closed form.Source: A Fixed point updates · p. 11

B.1 Simulated Data

An additional experiment tests whether non-centered HS-BNN can recover the active units when network capacity is greatly increased.

B.1 Simulated Data

  • In a 100-unit-per-layer experiment, non-centered HS-BNN prunes the extra capacity and recovers two active nodes.Source: §B.1 · p. 11

B.2 Further Exploration of Model Selection Properties

As network capacity grows, HS-BNN shows clearer model-selection behavior: inferred node-weight norms indicate that it uses only a fraction of the available capacity.

B.2 Further Exploration of Model Selection Properties

  • HS-BNN shows clear inflection points in node-weight norms across 400-400, 800-800, and 1200-1200 networks, indicating use of only a fraction of available capacity.Figure 5 plots first-layer log norms with the x-axis normalized by unit count.Source: §B.2 · p. 11, Figure 5 · p. 13
  • A 100-unit synthetic-classification comparison contrasts Gaussian BNN with centered and non-centered Horseshoe parameterizations.Source: Figure 4 · p. 12
  • For the five smallest-norm node vectors, HS-BNN concentrates weights more tightly around zero than SM-BNN, with sharper concentration at larger capacities.The density plots cover 400-400, 800-800, and 1200-1200 networks.Source: §B.2 · pp. 11, 14, Figure 6 · p. 14

B.3 Gesture Recognition

On the NATOPS gesture-recognition task, HS-BNN significantly outperforms BNN and is competitive with VMG and SM-BNN. The experiment uses kinematic and tracking features from 9,600 gesture examples.

B.3 Gesture Recognition

  • The gesture-recognition dataset contains 9,600 examples of 24 unique aircraft-handling signals performed by 20 subjects, each repeating every signal 20 times; experiments used kinematic and tracking data.Source: §B.3 · p. 14
  • Each gesture is represented by a 600-dimensional feature vector formed by uniformly sampling frames and concatenating body, hand, and tracking features.The experiment uses kinematic and tracking data, not video data.Source: §B.3 · p. 14
  • HS-BNN significantly outperforms BNN and is competitive with VMG and SM-BNN on gesture recognition.Reported error rates average five random 75/25 dataset splits.Source: §B.3 · p. 14, Figure 7 · p. 15

Related papers

Loading 1705.10388v1…