PaperlayerView exact revision on arXiv

Source-linked AI summary

The Variational Gaussian Process

Dustin Tran, Rajesh Ranganath, David M. Blei

arXiv:1511.06499v4stat.MLcs.LGcs.NEstat.CO

TL;DR

The paper addresses the limited expressiveness of mean-field variational families for complex posterior inference. It develops the VGP, which learns Gaussian-process nonlinear transformations within a Bayesian nonparametric variational family, and reports universal approximation and best-to-date benchmark results on deep latent Gaussian models and DRAW.

  • Problem

    Mean-field variational families enable efficient computation but limit dependencies among latent variables and the expressiveness of deep representations.

  • Method

    The VGP samples from a simple distribution and warps those samples through a learned Gaussian-process nonlinear mapping to govern latent-variable distributions, using a tractable auto-encoder-inspired objective.

  • Results

    The VGP is a universal approximator and reports the best results to date for inference in deep latent Gaussian models and DRAW.

  • Takeaways & Limitations

    The VGP provides a variational family whose shape can adapt to complex posterior distributions while supporting black-box inference across a wide class of models.

  • Takeaways & Limitations

    The optimization procedure's local optima remain insufficiently characterized, leaving the limits of the optimization procedure and variational inference for future analysis.

Abstract

from arXiv · show

Variational inference is a powerful tool for approximate inference, and it has been recently applied for representation learning with deep generative models. We develop the variational Gaussian process (VGP), a Bayesian nonparametric variational family, which adapts its shape to match complex posterior distributions. The VGP generates approximate posterior samples by generating latent inputs and warping them through random non-linear mappings; the distribution over random mappings is learned during inference, enabling the transformed outputs to adapt to varying complexity. We prove a universal approximation theorem for the VGP, demonstrating its representative power for learning any model. For inference we present a variational objective inspired by auto-encoders and perform black box inference over a wide class of models. The VGP achieves new state-of-the-art results for unsupervised learning, inferring models such as the deep latent Gaussian model and the recently proposed DRAW.

1 INTRODUCTION

The paper introduces the variational Gaussian process (VGP), a Bayesian nonparametric variational model designed to represent complex posterior distributions. It combines flexible nonlinear transformations with efficient black-box inference and establishes broad theoretical and empirical support for the approach.

  • Mean-field variational families are computationally convenient but restrict latent-variable dependencies and the expressiveness needed for deep representations.
  • The VGP is a Bayesian nonparametric variational model whose complexity can adapt toward any distribution.
  • The paper proves universal approximation, derives efficient stochastic black-box inference, and reports the best results to date on deep latent Gaussian models and DRAW.
  • The VGP draws simple-distribution inputs through a Gaussian-process random nonlinear mapping, using its outputs to govern latent-variable distributions.
  • Its variational parameters include Gaussian-process kernel parameters and variational input-output pairs that anchor the nonlinear mappings and help learn complex representations.

2 VARIATIONAL GAUSSIAN PROCESS

The VGP is a Bayesian nonparametric variational model that uses random nonlinear mappings to transform latent inputs into flexible mean-field parameters. It can capture dependent, complex posteriors and is theoretically universal under stated conditions.

  • Variational Gaussian Process: The VGP introduces a Bayesian nonparametric variational model whose complexity adapts toward the inference problem.Its Gaussian-process-based construction provides latent variables that capture downstream distributions with varying complexity.
  • Variational Models: Classical mean-field approximations are computationally convenient but limit expressiveness by assuming independent latent variables.Hierarchical variational models enrich the family by placing a variational prior over mean-field parameters.
  • Variational Gaussian Process: The VGP draws latent inputs, warps them through random nonlinear mappings, and uses the warped outputs as parameters of a mean-field distribution.The mappings are Gaussian-process draws conditioned on optimized variational data.
  • Variational Gaussian Process: Evaluating independent Gaussian-process draws at the same latent input induces correlations among mean-field parameters and therefore among latent variables.The nonlinear mappings also allow the model to represent complex discrete and continuous posteriors.
  • Universal Approximation Theorem: Theorem 1 states that, under finite-dimensional and continuous-quantile conditions, a sequence of VGP parameters can represent any posterior with strictly positive density.The construction uses an optimal transformation from standard-normal latent inputs through the posterior’s inverse CDF.

3 BLACK BOX INFERENCE

The paper develops a tractable auto-encoder-inspired objective and black-box optimization procedure for inference with the VGP across a wide class of models.

  • 3.1 VARIATIONAL OBJECTIVE: The VGP objective uses an auxiliary model to make inference tractable while minimizing divergences in both posterior and variational latent spaces.The auxiliary model r(ξ, f | z) supports inference over the VGP’s latent variables.
  • 3.1 VARIATIONAL OBJECTIVE: The objective is rewritten in auto-encoder form using tractable KL divergences and log-density terms.This connects VGP inference to nested variational auto-encoder bounds.
  • 3.2 AUTO-ENCODING THE VGP: Inference networks parameterize both the variational and auxiliary models, replacing local parameters with global neural-network outputs.The variational network maps x_n to q(z_n | x_n; θ_n), while the auxiliary network uses x_n and z_n.
  • 3.3 STOCHASTIC OPTIMIZATION: Nested reparameterization moves gradients inside expectations and enables unbiased stochastic gradients with reduced variance.Analytic KL terms and reparameterization contribute to the variance reduction.
  • 3.3 STOCHASTIC OPTIMIZATION: The algorithm supports subsampling, automatic differentiation, and black-box gradient estimators for non-differentiable or non-reparameterizable components.These choices broaden the range of models to which the inference procedure can be applied.
  • 3.4 COMPLEXITY: O(d + m3 + LH2) complexity is linear in the number of latent variables, with O(c + LH) variational and auxiliary parameter complexity.Here d is the number of latent variables, m the variational-data size, and L and H describe the neural networks.

4 RELATED WORK

The VGP differs from parametric transformation methods by placing a Bayesian nonparametric prior over continuous mappings, avoiding invertibility restrictions and Jacobian determinants.

  • Parametric transformations: Parametric transformation methods can incur O(d3) Jacobian-determinant computation, where d is the number of latent variables.Their transformation flexibility is also restricted by tractability requirements on the Jacobian.
  • Variational Gaussian process: The VGP places a Bayesian nonparametric prior over continuous mappings from latent inputs to mean-field parameters.Gaussian-process mappings can recover certain parametric transformation classes while retaining broader mapping flexibility.
  • Computational comparison: GP-based output distributions require no Jacobian determinants, instead using auxiliary inference for variational latent variables.The paper describes this auxiliary inference as fast.
  • Connections to Monte Carlo transformations: The VGP adaptively learns a transformation between tractable inputs and posterior samples rather than specifying a discretized schedule explicitly.This contrasts with transformation sequences used in Monte Carlo methods.
  • Connections to existing models: Restricting the VGP recovers discrete mixtures of mean-field distributions and a form of factor analysis in variational space.These recoveries show that the VGP contains established variational models as special cases.

5 EXPERIMENTS

Experiments evaluate the VGP on deep latent Gaussian models and DRAW, reporting improved log-likelihood bounds and more complex generated sketches than the original DRAW variational auto-encoder.

  • Experimental setup: The experiments apply VGP inference to deep latent Gaussian models and the Deep Recurrent Attentive Writer on image datasets.DLGM experiments use binarized MNIST, while DRAW is also evaluated on the Sketch dataset.
  • Binarized MNIST: The VGP achieves log-likelihood -79.88 with DRAW, compared with -80.97 for the original highest result.The reported values are test-set lower bounds or approximations to log p(x).
  • Binarized MNIST: The VGP achieves log-likelihood -81.32 with the DLGM, compared with -82.90 for the previous best non-structure-exploiting model.The comparison is reported for the class of non-structure-exploiting models.
  • Sketch: On Sketch, DRAW with the VGP achieves a significantly better lower bound than the original version.The table evaluates negative predictive log-likelihood after training on 18,000 examples.
  • Sketch: VGP-augmented DRAW generates more complex images with higher visual fidelity, including greater texture and sharpness than the original version.The figure compares VGP-generated images in the top panel with original-DRAW images in the bottom panel.

6 DISCUSSION

The VGP adapts its shape to complex posterior distributions and is presented as a universal approximator. Future work includes Monte Carlo applications and analysis of optimization limits.

  • The VGP adapts its shape to match complex posterior distributions.It draws samples from a tractable distribution and learns transformations over continuous mappings.
  • The VGP is a universal approximator that finds posterior approximations through optimization.
  • Future work will explore the VGP as a proposal distribution for importance sampling and sequential Monte Carlo.
  • Characterizing local optima is identified as important for understanding the limits of the optimization procedure and variational inference.

A SPECIAL CASES OF THE VARIATIONAL GAUSSIAN PROCESS

The VGP contains simpler variational models as special cases, including mixtures of mean-field distributions and factor analysis. Kernel choices determine whether the induced latent dependence is linear or nonlinear.

  • A mixture of mean-field distributions is recovered by removing the VGP kernel.
  • Nearest-neighbor mappings over variational inputs induce a multinomial distribution over outputs.The probabilities are proportional to the areas of the partitioned nearest-neighbor space.
  • Factor analysis is obtained in variational space under a linear-kernel construction with degenerate mean-field factors.
  • Marginalizing latent inputs induces linear dependence in z, while nonlinear kernels induce nonlinear dependence.Learning kernel hyperparameters captures variation in the latent embedding of z.

B PROOF OF THEOREM 1

Theorem 1 establishes that VGPs can approximate posterior distributions with finitely many latent variables and continuous quantile functions. The proof constructs increasingly fine variational data grids and uses Gaussian-process continuity.

  • Under finite latent dimensionality and a continuous posterior quantile function, a sequence of VGP parameters approximates the posterior.
  • The proof assumes a latent-input size matching the number of latent variables and uses a uniform distribution on a d-dimensional hypercube.
  • Variational data are formed from grid points paired with values of the inverse posterior quantile function.
  • As the grid becomes finer, the Gaussian-process mapping converges to the target quantile function by continuity.
  • A broad sufficient condition for continuous quantiles is positive density with respect to Lebesgue measure.
  • Finite-data convergence rates require stronger continuity conditions and Matérn covariance functions for the available theory to apply.

C VARIATIONAL OBJECTIVE

The variational objective is derived as a tractable lower bound by introducing an auxiliary model and recombining terms into KL divergences. Analytic Gaussian terms reduce computational difficulty, while intractable KL terms are combined with reconstruction terms.

  • The derivation starts from a tractable lower bound to model evidence and penalizes the ELBO with an expected KL term.
  • The objective combines likelihood, prior, variational, and auxiliary-model terms before applying the product rule.
  • Recombining terms as KL divergences recovers the auto-encoded variational objective.
  • For deep latent Gaussian models and DRAW, the KL between the mean-field distribution and model prior is analytically tractable because both are Gaussian.
  • When the KL is intractable, it is combined with the reconstruction term and the resulting variational objective is maximized.
  • The auxiliary-model KL and Gaussian log-density difference are simple to compute and backpropagate gradients.
  • The stochastic gradients may experience slightly higher variance during optimization.

D GRADIENTS OF THE VARIATIONAL OBJECTIVE

The variational objective is differentiated through backpropagation and nested reparameterizations, with expectations handled explicitly and computational graphs differentiated automatically.

  • Gradients of the variational objective in Eq.7 are obtained by backpropagation through analytically expressed KL terms and their computational graph.The expectation terms require care, while automatic differentiation handles the gradients of the written functions.
  • The φ-gradient combines the KL contribution between q(f | ξ; θ) and r(f | ξ, z; φ) with the gradient of log r(ξ | z; φ).The expression takes expectations over ξ and ϵ.
  • The general variational bound in Eq.8 accommodates cases where the KL divergence between q and the prior for p is not tractable.Reparameterizations from Section 3.3 rewrite the objective using the displayed KL and log-density terms.
  • Gradients for the general objective are calculated by backpropagating over nested reparameterizations.This produces the parameter gradient expressions for θ and φ shown in the appendix.

E SCALING THE SIZE OF VARIATIONAL DATA

Variational data can be scaled when cubic matrix inversion becomes a computational bottleneck by fixing variational inputs on a grid and exploiting Toeplitz structure.

  • Grid-based variational inputs enable fast m × m matrix inversion for stationary kernels by exploiting Toeplitz structure.The method embeds the Toeplitz matrix in a circulant matrix and uses conjugate gradients with fast Fourier transforms.
Loading 1511.06499v4…