PaperlayerView exact revision on arXiv

Source-linked AI summary

Variational Implicit Processes

Chao Ma, Yingzhen Li, José Miguel Hernández-Lobato

arXiv:1806.02390v2stat.MLcs.LG

TL;DR

Bayesian modeling of random functions with implicit distributions offers flexibility, but inference over these models is intractable. The paper introduces IP priors and VIP wake-sleep inference using GP approximations; it reports better uncertainty estimates and lower errors than existing methods on challenging models.

  • Problem

    The paper addresses how to perform Bayesian inference over random functions with flexible implicit models, where parameter-space inference can encounter symmetric modes in Bayesian-neural-network weight posteriors.

    Source: §1 · p. 1

  • Method

    IPs define implicit joint distributions over finite collections of function values, and VIP uses generalized wake-sleep updates with a GP approximation for inference.

    Source: §2 · p. 2, §3.1 · p. 4

  • Results

    VIPs are reported to outperform existing GP and BNN methods, with better uncertainty estimates and lower errors; in a synthetic comparison, VIP provides the best predictive uncertainty.

    Source: §1 · p. 1, §4.1 · p. 5

  • Takeaways & Limitations

    IPs combine the flexibility of implicit models with the well-calibrated uncertainty estimates of Bayesian models.

    Source: §6 · p. 8

Abstract

from arXiv · show

We introduce the implicit processes (IPs), a stochastic process that places implicitly defined multivariate distributions over any finite collections of random variables. IPs are therefore highly flexible implicit priors over functions, with examples including data simulators, Bayesian neural networks and non-linear transformations of stochastic processes. A novel and efficient approximate inference algorithm for IPs, namely the variational implicit processes (VIPs), is derived using generalised wake-sleep updates. This method returns simple update equations and allows scalable hyper-parameter learning with stochastic optimization. Experiments show that VIPs return better uncertainty estimates and lower errors over existing inference methods for challenging models such as Bayesian neural networks, and Gaussian processes.

1 Introduction

The paper extends implicit models to Bayesian modeling of random functions, using implicit processes (IPs) as flexible priors and performing posterior inference directly in function space. It introduces variational implicit processes (VIPs) for inference and reports stronger performance than existing methods on challenging models.

1 Introduction

  • Function-space inference with IP priors is intended to improve uncertainty calibration and avoid symmetric modes in Bayesian neural-network weight posteriors.The paper notes the uncertainty benefit is like that of Gaussian processes.Source: §1 · p. 1
  • IPs provide implicit priors over functions, enabling flexible models such as data simulators and Bayesian LSTMs within a Bayesian inference framework.The paper formalizes these priors and proves well-definedness in finite and infinite dimensions.Source: §1 · p. 1
  • VIP uses generalized wake-sleep updates for approximate inference, and the paper reports consistent gains over existing methods and state-of-the-art results on large-scale Bayesian LSTM inference.The proposed method also permits scalable hyper-parameter learning.Source: §1 · p. 1

2 Implicit Stochastic Processes

An implicit process (IP) generalizes the Gaussian-process sampling construction by replacing its linear latent-variable transformation with a nonlinear one. The section establishes conditions for valid processes and illustrates the framework with several model families.

2 Implicit Stochastic Processes

  • An IP assigns an implicitly defined joint distribution to function values at any finite collection of inputs by transforming latent variables through a nonlinear function.This generalizes the Gaussian-process construction that transforms latent variables linearly.Source: §2 · p. 2
  • For finite-dimensional latent variables an IP defines a unique stochastic process; an infinite-dimensional construction is also given under stated operator, kernel, and transformation conditions.The infinite-dimensional case uses a continuous latent process and a specified integral operator.Source: A function distributed according to… · p. 2

A function distributed according to the above IP is denoted

  • The examples include neural samplers, warped Gaussian processes, Bayesian neural networks, and Bayesian recurrent neural networks.Neural samplers use a neural network to transform latent noise, while the Bayesian-network examples treat model weights as random variables.Source: A function distributed according to… · p. 2

3 Variational Implicit Processes

VIP addresses intractable IP posterior and marginal-likelihood inference by using a Gaussian-process approximation within generalized wake-sleep updates. Its sleep update fits the approximation to IP-generated data, while its wake update supports scalable parameter learning and prediction.

3 Variational Implicit Processes

  • VIP approximates the IP posterior with a GP and uses generalized wake-sleep updates to address intractable posterior and marginal-likelihood inference.The approach uses the GP posterior approximation for prediction and marginal-likelihood approximation for hyper-parameter optimization.Source: §3 · p. 3

3.1 Sleep phase: GP posterior as variational distribution

  • In the sleep phase, the method fits a GP to dreamed outputs and function values, yielding an approximation whose optimal mean and covariance match those of the IP.The analytic sleep-phase solution avoids explicitly performing sleep-phase optimization.Source: §3.1 · pp. 3–4

3.2 Wake phase: a scalable approach to learning the model parameters θ

  • In the wake phase, the optimal sleep-phase GP supports marginal-likelihood approximation; stochastic optimization and mini-batches enable joint learning of IP parameters and variational parameters.For scalability, the method uses an α-energy approximation.Source: §3.2 · p. 4

3.3 Computational complexity and scalable predictive inference

  • Training costs O(CMS + MS^2 + S^3), and the reduced predictive cost is O(CKS + S^3), comparable to conventional Bayesian-neural-network prediction.Here M is mini-batch size, S is the number of sampled IP functions, and C is the cost of evaluating one sampled function value.Source: §3.3 · pp. 4–5

4 Experiments

The experiments evaluate VIP across synthetic and real-world regression, interpolation, Bayesian LSTM, and simulator-based tasks, using comparisons with GP, dropout, and other inference methods.

4.1 Synthetic example

  • In the synthetic example, VIP recovers the mean better than VDO and the full GP and provides the best predictive uncertainty, with NLL/RMSE results also favoring VIP.The full GP still overfits and estimates uncertainty poorly near |x| ≈2.5.Source: §4.1 · p. 5

4.2 Solar irradiance interpolation under missingness

  • On solar-irradiance interpolation, VIP achieves the best NLL/RMSE performance, while VDO underestimates uncertainty and GP methods predict interpolation means poorly.The interpolation task removes five segments of length 20 from the dataset.Source: §4.2 · p. 5, Figure 4 · p. 6

4.3 Predictive Performance: Multivariate regression

  • VIP methods achieve the best test NLL on 7 datasets and best test RMSE on 8 of 9 datasets; VIP-BNN ranks best among six methods.VIP-NS has the second-best average RMSE ranking; comparisons with exact GPs and fBNNs are not direct because they use smaller datasets, and fBNNs use longer training.Source: §4.3 · p. 6, Table 1 · p. 7, Table 2 · p. 7

4.4 Bayesian LSTM for predicting power conversion efficiency of organic photovoltaics molecules

  • On organic-molecule PCE prediction, VIP significantly outperforms the listed baselines and achieves state-of-the-art test likelihood and RMSE.The VIP prior is a Bayesian LSTM and directly takes raw sequential molecule structures as input.Source: §4.4 · p. 7

4.5 ABC example: the Lotka–Volterra model

  • On the Lotka–Volterra task, VIP-BNN beats the other methods in test NLL and RMSE, and is the only regression-based method to outperform ABC methods.Regression-based approaches infer predictions in function space, while ABC first infers parameters and then simulates predictions.Source: Table 3 · p. 7, §4.5 · p. 7

5 Related Research

Related work links Gaussian processes (GPs) and Bayesian neural networks (BNNs), while identifying limits in GP scalability and representation for complex data. The paper instead treats BNNs as implicit processes and uses a GP variational distribution for function-space inference.

GP limitations

  • GPs scale poorly to large datasets and commonly used kernels have limited representational power for high-dimensional data such as texts and images.Deep-GP and expressive-kernel approaches still struggle with such data, while deep-feature GP hybrids retain Bayesian treatment only at the output layer.Source: §5 · p. 8

Motivation

  • As uncertainty quantification in neural networks becomes increasingly important, uncertainty in deep learning remains an open challenge despite progress on Bayesian neural networks.GPs provide accurate uncertainty estimates on unseen data, but their computational and representational constraints limit their use for some tasks.Source: §5 · pp. 7–8

GP–BNN correspondence

  • GP–BNN correspondence work uses neural-network structure to construct compositional GP kernels or uses exact GP inference for Bayesian prediction in BNNs.The paper distinguishes its approach from these aims: it treats a BNN as an implicit process and uses a GP as a variational distribution, without requiring correspondence assumptions.Source: §5 · p. 8

Related approaches

  • Flam-Shepherd et al. (2017) encode GP-prior properties in BNN weight priors, while concurrent functional-BNN work uses GP priors to regularize BNN training through gradient estimators.Neural processes resemble the neural-sampler special case of IPs, but infer in latent z space and do not apply to other IPs such as BNNs; VIP infers in function space.Source: §5 · p. 8

6 Conclusions

The paper presents variational inference over function space using implicit-process priors, combining implicit-model flexibility with Bayesian uncertainty estimates. Its BNN-prior experiments outperform many GP/BNN methods, and it identifies several extensions for future work.

6 Conclusions

  • The proposed variational approach performs learning and Bayesian inference over function space using implicit-process priors.The framework combines the flexibility of implicit models with uncertainty estimates from parametric or nonparametric Bayesian models.Source: §6 · p. 8
  • With BNNs as the implicit-process prior, the approach outperformed many existing GP/BNN methods and improved results on molecule regression data.Source: §6 · p. 8
  • Future work includes posterior approximations beyond GP prior matching, classification with implicit-process priors, latent-variable models, and inference for other priors such as data simulators.The named simulator domains include astrophysics, ecology, and climate science.Source: §6 · p. 8

A Brief review of Gaussian processes

The review presents Gaussian processes as distributions over functions that support posterior prediction, then outlines computational and representational limits and variational alternatives. It describes BB-α as a flexible approximate-inference objective that can be optimized with minibatches.

Gaussian processes

  • A Gaussian process is specified by a mean function and covariance function, which determine joint Gaussian distributions over finite collections of function values.For regression, the posterior over test function values has a closed-form Gaussian distribution.Source: A Brief review of Gaussian processes · p. 13

Gaussian-process limitations

  • O(N^3) computation and O(N^2) storage make exact GP methods impractical for large-scale datasets.The matrix inversion for GP regression takes O(N^3) time; approximate methods are often used instead.Source: A Brief review of Gaussian processes · p. 13
  • Common local GP kernels can lack hierarchical representations for high-dimensional data, limiting their usefulness on complex tasks.Deep GPs and expressive kernels are among the proposed responses, but the review notes continued difficulty with texts and images.Source: A Brief review of Gaussian processes · p. 13

Variational inference

  • Variational inference optimizes an approximate posterior by minimizing its KL divergence to the true posterior, while BB-α generalizes this approach through α-divergence minimization.The α-divergence includes inclusive KL at α=1, Hellinger distance at α=0.5, and exclusive KL at α=0; BB-α uses tied factors and can be optimized with reparameterization, Monte Carlo estimates, and minibatches.Source: B Brief review of variational… · pp. 13–14
  • BB-α with α≠0 can yield better uncertainty estimates than VI, and its energy has been reported as a better marginal-likelihood estimate than variational free energy.Source: B Brief review of variational… · p. 14

C.1 Proof of Proposition 1 (finite dimensional case)

For a finite-dimensional latent variable, the induced finite marginals satisfy the consistency conditions needed to establish a unique stochastic process.

C.1 Proof of Proposition 1 (finite dimensional case)

  • A finite-dimensional latent variable induces a unique stochastic process when its finite marginals satisfy marginal and permutation consistency.The proof represents each finite joint distribution by integrating over the latent variable, verifies consistency, and applies the Kolmogorov extension theorem.Source: §C.1 · pp. 14–15

C.2 Proof of Proposition 2 (infinite dimensional case)

The appendix establishes conditions under which nonlinear transformations of stochastic processes define well-defined processes, then develops related inference details and comparisons. It also describes empirical Bayes and implementation choices for VIP.

C.2 Proof of Proposition 2 (infinite dimensional case)

  • With square-integrable kernels and a Borel measurable, linearly bounded activation, the transformed process is well-defined; the proof uses a Karhunen–Loève expansion.The result also extends to deep function machine operators on L2(R^d).Source: §C.2 · p. 15

C.3 Inverse Wishart process as a prior for kernel functions

  • With S fantasy functions, moment matching gives the maximum-likelihood GP mean and covariance estimates; an inverse Wishart process prior enables MAP estimation.The implementation uses a posterior-mean estimator with ν = N and Ψ(x1, x2) = ψδ(x1, x2).Source: §C.3 · p. 16

C.4 Derivation of the upper bound U(m, K)or sleep phase

  • Optimizing the sleep-phase upper bound also decreases its gap to the function-space objective when the mean and covariance functions are optimized.The passage attributes this implication to the non-negativity of KL divergence and the choice q(y|f) = p(y|f).Source: §C.4 · p. 16

C.5 Empirical Bayes approximation for VIP with a hierarchical prior on θ

  • A hierarchical prior on θ adds a variational approximation q(θ); using a Dirac-delta q(θq) gives a compact empirical Bayes solution.The alternative likelihood regularization uses −log qGP(y|X, θq), estimated on each mini-batch.Source: §C.5 · pp. 16–17

D KL divergence on function space v.s. KL divergence on weight space

  • Function-space KL divergence forms a tighter bound than weight-space KL divergence for the described finite-dimensional representations.The passage frames weight-space inference as a representation of both stochastic processes through W.Source: D KL divergence on function space v.s.… · p. 17

E Further discussions on Bayesian neural networks

  • The one-hidden-layer BNN correspondence applies when the IP is Bayesian logistic regression; the paper instead approximates the IP output with a GP, beyond that case.The compared construction treats the hidden-layer output, not the BNN output, as the IP.Source: E Further discussions on Bayesian… · p. 17

F.1 General settings for VIP

  • General VIP settings use S = 20 unless noted, and the appendix provides code for UCI experiments.When VIP uses a Bayesian NN/LSTM prior, weight-prior parameters are untied and can be tuned individually.Source: §F.1 · p. 17, F Further experimental details · p. 17

F.2 Further experimental details of synthetic example

  • For the synthetic example, the GP kernel sums RBF and Periodic kernels, with parameters optimized by marginal likelihood rather than hand-picked for smoother fit.The authors report that hand-picking smoother-fit parameters decreased test predictive likelihood and increased RMSE.Source: §F.2 · p. 18

F.3 Further implementation details for multivariate regression experiments

This section records the multivariate-regression comparison methods and the validation-based noise-variance search used for VIP and several baselines.

F.3 Further implementation details for multivariate regression experiments

  • The comparison includes VI-BNN, VIP-NS, VDO, α-dropout, variational sparse GPs, and exact GPs, with method-specific approximations and settings.Examples include a mean-field Gaussian for VI-BNN, two hidden layers for VIP-NS, and 50 inducing points for sparse GPs.Source: §F.3 · p. 18
  • The α-dropout setup fixes α = 0.5 and uses K = 10 Monte Carlo samples; VDO uses 2000 forward passes to evaluate posterior likelihood.These are implementation settings, not reported comparative outcomes.Source: §F.3 · p. 18
  • For VIP-BNN, VIP-NS, VDO, and α-dropout, noise variance is selected by optimization followed by a validation-set grid search, then the model is trained on all training data.The same procedure searches the VIP inverse-Wishart-process hyperparameter ψ and avoids repeated model training as in Bayesian optimization.Source: §F.3 · p. 18

F.4 Additional implementation details for ABC experiment

The appendix specifies the simulated L-V dataset and training configurations used for the ABC experiment.

F.4 Additional implementation details for ABC experiment

  • The simulated L-V data contain 600 measurements over [0, 30], with the first 500 for training and the remainder for testing.MCMC-ABC uses 10,000 samples and tolerance ϵ = 2.0; SMC-ABC uses 100 particles, while VIP (α = 0) trains for 10,000 iterations.Source: §F.4 · p. 18

F.5 Additional implementation details for predicting power conversion efficiency of organic photovoltaics molecules

The appendix describes implementation choices for predicting photovoltaic-molecule power-conversion efficiency and lists additional performance tables.

F.5 Additional implementation details for predicting power conversion efficiency of organic photovoltaics molecules

  • For Bayesian LSTMs, all methods use Adam at 0.001 and train for 100 epochs; the implementation shares K sample paths across training data to reduce memory use.The authors state this approximation is accurate with their small dropout rate of 0.005.Source: §F.5 · pp. 18–19

F.6 Additional Tables

  • The appendix lists additional tables for interpolation on the toy dataset and solar irradiance, and performance on the clean-energy dataset.The supplied captions do not report table values or identify a winner.Source: Table 4 · p. 19, Table 5 · p. 19, Table 6 · p. 19

Related papers

Loading 1806.02390v2…