PaperlayerView exact revision on arXiv

Source-linked AI summary

Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors

Christos Louizos, Max Welling

arXiv:1603.04733v5stat.MLcs.LG

TL;DR

Large neural networks need Bayesian posteriors that provide reliable uncertainty without the restrictive independence assumptions of fully factorized weights. The paper uses matrix variate Gaussian posteriors, connects them through local reparametrization to deep multi-output Gaussian Processes, and introduces pseudo-data for efficient sampling. Experiments report better regression RMSE and predictive log-likelihoods than VI, PBP, and Dropout uncertainty, plus better classification errors for small architectures.

  • Problem

    Large neural networks require tractable posterior inference and reliable confidence intervals, while common variational methods assume fully factorized weight posteriors despite correlations among weights.

  • Method

    The paper models each weight matrix with a matrix variate Gaussian posterior, uses local reparametrization to obtain a Gaussian Process interpretation, and adds pseudo-data for efficient sampling.

  • Results

    Experiments report better regression RMSE and predictive log-likelihoods than VI, PBP, and Dropout uncertainty, and better classification errors than state-of-the-art methods for small architectures.

  • Takeaways & Limitations

    The proposed model combines covariance-aware Bayesian neural-network posteriors with Gaussian Process structure and efficient pseudo-data-based sampling.

  • Takeaways & Limitations

    The formulation assumes that each layer input is augmented with an extra dimension of ones to account for biases.

Abstract

from arXiv · show

We introduce a variational Bayesian neural network where the parameters are governed via a probability distribution on random matrices. Specifically, we employ a matrix variate Gaussian \cite{gupta1999matrix} parameter posterior distribution where we explicitly model the covariance among the input and output dimensions of each layer. Furthermore, with approximate covariance matrices we can achieve a more efficient way to represent those correlations that is also cheaper than fully factorized parameter posteriors. We further show that with the "local reprarametrization trick" \cite{kingma2015variational} on this posterior distribution we arrive at a Gaussian Process \cite{rasmussen2006gaussian} interpretation of the hidden units in each layer and we, similarly with \cite{gal2015dropout}, provide connections with deep Gaussian processes. We continue in taking advantage of this duality and incorporate "pseudo-data" \cite{snelson2005sparse} in our model, which in turn allows for more efficient sampling while maintaining the properties of the original model. The validity of the proposed approach is verified through extensive experiments.

1. Introduction

The paper develops a fully Bayesian deep-learning framework to provide reliable confidence intervals while addressing the difficulty of approximate inference in large neural networks. It replaces fully factorized weight posteriors with covariance-aware matrix variate Gaussian distributions and connects the resulting model to Gaussian Processes for efficient sampling.

  • Deep-learning predictions can be highly accurate yet lack reliable confidence intervals, which are important when decisions depend on recognizing unreliable predictions.The paper motivates this need with autonomous driving and medical diagnosis scenarios.
  • Bayesian prediction estimates a posterior distribution over network parameters and marginalizes them through the posterior predictive distribution, helping protect against overfitting and overconfident predictions.
  • Exact parameter posterior inference is intractable for large neural networks, motivating MCMC and variational approximations.
  • Most existing variational approaches use fully factorized weight posteriors, although correlated true posteriors may provide information sharing among weights.
  • The proposed variational Bayesian neural network models each weight matrix as a matrix variate Gaussian, estimating row- and column-specific variances instead of one variance per weight.This introduces correlations among weights and is intended to make posterior-uncertainty estimation easier.
  • Applying the local reparametrization trick yields a connection to deep multi-output Gaussian Processes and enables pseudo-data for more efficient sampling while maintaining the original model's properties.

2. Beyond fully factorized parameter posteriors

The model replaces fully factorized weight posteriors with matrix variate Gaussian posteriors that represent row and column structure more efficiently. Local reparameterization exposes a deep multi-output Gaussian-process view and supports pseudo-data-based sampling.

  • 2.1. Matrix variate Gaussian distribution: Matrix variate Gaussian posteriors model each weight matrix jointly, separating covariance across rows and columns that affect input and output hidden units.The mean is matrix-valued, while row and column covariance matrices capture structured dependencies.
  • 2.2. Variational inference with matrix variate Gaussian posteriors: Variational inference optimizes the marginal-likelihood lower bound using Monte Carlo integration and reparameterization, with the Gaussian KL divergence computed in closed form.The objective combines expected log-likelihood with a complexity loss defined by the negative KL divergence.
  • 2.1. Matrix variate Gaussian distribution: Diagonal row and column covariances require (nin × nout) + nin + nout parameters per layer, versus 2(nin×nout) for fully factorized Gaussian posteriors.The approximation reduces covariance parameters while retaining input- and output-feature-specific variances.
  • 2.3. Deep matrix variate Bayesian nets as deep multi-output Gaussian Processes: The local reparameterization trick makes layer outputs approximate posterior Gaussian-process functions, yielding a deep Gaussian-process interpretation with correlated multi-output kernels.The row covariance after multiplication induces dependence among minibatch inputs, while marginalization retains Gaussian-process properties.
  • 2.4. Efficient sampling and pseudo-data: Pseudo-data condition each pre-activation distribution: nearby inputs receive lower variance and means shifted toward pseudo-data, while distant inputs depend mostly on the weight posterior.This conditioning enables more certain predictions in regions represented by pseudo-inputs without changing the underlying posterior structure elsewhere.

3. Related work

Prior Bayesian neural-network approaches include MCMC and variational inference, but related methods often use fully factorized weight posteriors or restricted approximations. Connections to deep Gaussian processes exist, although prior work does not model hidden-unit variances and samples in weight space.

  • MCMC methods for Bayesian neural networks include Hamiltonian Monte Carlo and stochastic-gradient approaches.
  • Dropout-based Bayesian neural networks connect to deep Gaussian processes but use independent column-wise Gaussians, omit hidden-unit variances, and sample in weight space.
  • Variational inference recasts posterior estimation as optimization that minimizes KL-divergence between approximate and true posteriors.
  • Probabilistic backpropagation uses closed-form approximations but is limited to rectified-linear nonlinearities and regression problems.

4. Experiments

Experiments evaluate the proposed model on UCI regression, permutation-invariant MNIST classification, and a toy regression task. The model generally improves regression metrics, performs strongly with small MNIST networks, and produces realistic toy predictive distributions.

  • Experimental setup: Experiments cover UCI regression, permutation-invariant MNIST classification, and a toy regression dataset.Regression uses datasets from prior studies; the toy task visualizes predictive distributions.
  • Regression experiments: Regression evaluation averages test RMSE and predictive log-likelihood across repeated train-test splits with reported standard errors.The split is repeated 20 times, except Protein five times and Year once.
  • Regression experiments: VMG achieves lower test RMSE than VI, PBP, and Dropout on most regression datasets.
  • Regression experiments: VMG outperforms VI and PBP on most regression datasets and exceeds Dropout on 6 out of 10 datasets in predictive log-likelihood.
  • Classification experiments: On MNIST, three hidden layers of 150 units achieve 1.18% test error, outperforming several methods with larger architectures and a same-sized variational-dropout network.
  • Classification experiments: Learned pseudo-data preserve Gaussian-Process-related flexibility, compensating for limited network size in the classification model.
  • Toy experiment: In the toy regression experiment, the matrix-Gaussian model produces a realistic predictive distribution slightly better than PBP.The comparison uses predictive distributions formed after 200 samples.
  • Toy experiment: The fully factorized Gaussian network underfits the sparse toy data, whereas the matrix-Gaussian model avoids this effect and learns fewer variance parameters.

5. Conclusions

The model combines matrix variate Gaussian posteriors, Gaussian Process structure, and pseudo-data to form a scalable variational Bayesian neural network. Experiments report stronger regression performance than several baselines, better errors for small MNIST architectures, and empirical verification of predictive distributions.

  • 5. Conclusions: The model represents neural-network parameters with matrix variate Gaussian distributions and exploits their Gaussian Process interpretation.Its pseudo-data formulation is intended to preserve Gaussian Process properties while increasing posterior flexibility.
  • 5. Conclusions: Regression experiments report better RMSE and predictive log-likelihoods than VI, PBP, and Dropout uncertainty.
  • 5. Conclusions: The model achieves better errors than state-of-the-art methods for small architectures.The paper attributes this to increased flexibility from pseudo-data despite limited neural-network capacity.
  • 5. Conclusions: The quality of the model’s predictive distribution was empirically verified on a toy experiment.

A. KL divergence between matrix variate Gaussian prior and posterior

The appendix computes the KL divergence between matrix variate Gaussian distributions by flattening the matrices and applying the multivariate Gaussian KL divergence, then using vectorization and Kronecker-product properties.

  • A. KL divergence between matrix variate Gaussian prior and posterior: Flattening a matrix variate Gaussian converts it into a multivariate Gaussian with covariance V0 ⊗ U0, enabling the corresponding KL-divergence formula.
  • A. KL divergence between matrix variate Gaussian prior and posterior: Vectorization and Kronecker-product properties are used to compute each KL term efficiently.

B. Different toy dataset

A second toy dataset is generated from inputs in two separated intervals and a nonlinear sinusoidal transformation with Gaussian noise. Four two-hidden-layer networks are compared using probabilistic backpropagation or the proposed model with different activations, and their predictive distributions are shown in Figure 2.

  • B. Different toy dataset: The toy inputs comprise 12 samples from U[0, 0.6] and 8 samples from U[0.8, 1].
  • B. Different toy dataset: The targets are generated as yi = xi + ϵi + sin(4(xi + ϵi)) + sin(13(xi + ϵi)).The noise satisfies ϵi ∼ N(0, 0.0009).
  • B. Different toy dataset: Four networks with two hidden layers of 50 units are fitted using probabilistic backpropagation or the proposed model with ReLU, cosine, or hyperbolic tangent activations.
  • B. Different toy dataset: The resulting predictive distributions are displayed in Figure 2, with grey regions representing ±{1, 2} standard deviations around the mean function.
Loading 1603.04733v5…