PaperlayerView exact revision on arXiv

Source-linked AI summary

Gaussian Process Prior Variational Autoencoders

Francesco Paolo Casale, Adrian V Dalca, Luca Saglietti, Jennifer Listgarten, Nicolo Fusi

arXiv:1810.11738v2cs.LGstat.ML

TL;DR

VAEs assume iid latent representations, limiting their specification for data with structured sample correlations. GPPVAE replaces this assumption with a GP prior and introduces efficient inference strategies, outperforming CVAEs and linear-interpolation VAE baselines in two image applications.

  • Problem

    VAEs assume iid latent representations, although many datasets contain structured sample correlations that matter for model specification.

  • Method

    GPPVAE places a GP prior on latent encodings, uses structured covariance representations, and applies low-memory stochastic backpropagation while retaining VAE encoder-decoder components.

  • Results

    GPPVAE outperforms CVAEs and linear interpolations in the VAE latent space when predicting out-of-sample test images across two image applications.

  • Takeaways & Limitations

    Modeling correlations in the latent prior supports disentangling object and view covariances and enables out-of-sample prediction of high-dimensional data.

  • Takeaways & Limitations

    The reported setting intentionally ignores observed pose-angle features, making the task more challenging but limiting the evaluation to unobserved object and view factors.

Abstract

from arXiv · show

Variational autoencoders (VAE) are a powerful and widely-used class of models to learn complex data distributions in an unsupervised fashion. One important limitation of VAEs is the prior assumption that latent sample representations are independent and identically distributed. However, for many important datasets, such as time-series of images, this assumption is too strong: accounting for covariances between samples, such as those in time, can yield to a more appropriate model specification and improve performance in downstream tasks. In this work, we introduce a new model, the Gaussian Process (GP) Prior Variational Autoencoder (GPPVAE), to specifically address this issue. The GPPVAE aims to combine the power of VAEs with the ability to model correlations afforded by GP priors. To achieve efficient inference in this new class of models, we leverage structure in the covariance matrix, and introduce a new stochastic backpropagation strategy that allows for computing stochastic gradients in a distributed and low-memory fashion. We show that our method outperforms conditional VAEs (CVAEs) and an adaptation of standard VAEs in two image data applications.

1 Introduction

VAEs provide expressive dimensionality reduction but assume iid latent representations, which is unsuitable when samples have structured correlations. GPPVAE replaces that prior with a GP prior while addressing the resulting computational and stochastic-gradient challenges.

  • Motivation: VAEs combine neural-network expressiveness with probabilistic generative modeling, but assume latent encodings are iid across dimensions and samples.The VAE framework uses variational inference because exact inference is intractable.
  • Motivation: Structured sample correlations are crucial for correct model specification in applications such as temporally correlated image sequences.Closer-in-time images should generally have more similar latent representations than images farther apart.
  • Proposed model: GPPVAE replaces the VAE's iid latent prior with a GP prior that specifies sample correlations through a kernel function.The approach targets correlations across multiple factors, including time and object identity.
  • Computational challenges: GPPVAE focuses on low-rank covariance factorizations to mitigate the GP prior's cubic computational complexity in the number of samples.The model also faces a loss of the iid assumption that normally supports unbiased mini-batch gradient estimates.

2 Related work

Related VAE extensions enrich latent representations using auxiliary information, structured posteriors, or richer priors. GPPVAE instead models the relationship between latent space and auxiliary information with a GP prior while retaining standard VAE encoder and decoder networks.

  • Auxiliary information: Prior work enriches VAEs with auxiliary data such as time, pose, or lighting, using grouping, semi-supervised models, or conditional inputs.These approaches incorporate observed auxiliary information in different ways.
  • Flexible posteriors: Other methods make variational posteriors more flexible through auxiliary dependence, structured encoders, invertible transformations, or hierarchical parameter dependencies.These methods primarily modify the posterior family rather than the latent prior.
  • Richer priors: Richer latent priors address limitations of overly simple priors, including excessive regularization that can constrain model success.Examples include conditional-independence-based latent graphical models with message-passing inference.
  • GPPVAE distinction: GPPVAE uses a GP prior to model latent-space relationships with auxiliary information while leaving the encoder and decoder independent of that information.The framework supports complex sample structure, latent auxiliary-information estimation, and out-of-sample high-dimensional prediction.

3 Gaussian Process Prior Variational Autoencoder

GPPVAE replaces the VAE’s iid latent prior with a GP prior that models sample correlations from object and view features. It combines this structured latent model with variational inference and low-memory methods for scalable prediction and optimization.

  • Model: GPPVAE models latent representations as functions of object and view feature vectors, enabling correlations between samples through a GP prior.The decoder maps latent representations back to the high-dimensional sample space.
  • Model: The model factorizes covariance into view and object kernels, allowing separate correlations induced by views and objects.Unobserved feature vectors can be treated as latent variables and estimated from data.
  • Inference: Variational inference approximates the posterior over latent representations with neural-network-based variational parameters and stochastic reparameterized sampling.The resulting objective contains reconstruction, latent-space GP, and regularization terms.
  • Computation: Efficient GP computation assumes a low-rank overall kernel, exploiting the relatively small number of views and a low-rank object kernel to achieve linear scaling in samples.The method also supports low-rank approximations of arbitrary kernels through fully independent training conditional approximations.
  • Inference: Low-memory stochastic backpropagation computes encoder representations in mini-batches, accumulates GP-term gradients across low-dimensional representations, and updates parameters from a proxy loss.This strategy addresses the coupling between samples that prevents ordinary mini-batch gradient descent.
  • Prediction: GPPVAE predicts an unseen sample by encoding training data, using the GP predictive posterior for its latent representation, and decoding that representation.The predictive posterior uses object and view feature vectors for the new sample.

4 Experiments

The experiments evaluate GPPVAE on predicting unseen object-view combinations in rotated MNIST and face images. Jointly optimized GPPVAE generally achieves lower MSE than CVAE and LIVAE, while learning structured object and view covariances.

  • Experimental task: The experiments predict an object’s image in an unseen view when the object and view were each observed elsewhere.This setup targets interpolation across object-view combinations, such as predicting an unseen pose of a known person.
  • Methods: The comparisons include jointly optimized GPPVAE, disjointly optimized GPPVAE, conditional VAE, and latent-space linear interpolation.All methods use the same encoder and decoder architecture and are evaluated with pixel-wise MSE.
  • Rotated MNIST: The rotated-MNIST experiment uses 400 handwritten digits across 16 rotation angles, with observed rotation features and learned object feature vectors.A periodic squared exponential view kernel and linear object kernel define the GP covariance structure.
  • Rotated MNIST: On rotated MNIST, GPPVAE-joint achieved lower interpolation MSE than CVAE and LIVAE and significantly outperformed GPPVAE-dis: 0.0280 ± 0.0008 versus 0.0306 ± 0.0009, p < 0.02.GPPVAE-joint also learned variational parameters different from those of a standard VAE.
  • Face dataset: The face experiment contains 4,835 images of 542 people across nine poses and evaluates predictions for orientations left unobserved during training.The model uses a full-rank view covariance and a low-rank linear object covariance.
  • Face dataset: On the face dataset, GPPVAE-joint and GPPVAE-dis yielded lower MSE than CVAE and LIVAE, with no significant difference between the two GPPVAE variants.GPPVAE-joint learned separate people and pose covariances through joint kernel learning.

5 Discussion

GPPVAE combines a GP prior over latent space with efficient inference for large, high-dimensional datasets. It outperforms CVAE and linear-interpolation baselines when predicting out-of-sample images in specified views.

  • GPPVAE incorporates a Gaussian Process prior over the latent space.
  • A low-memory and computationally efficient inference strategy makes GPPVAE applicable to large, high-dimensional datasets.
  • GPPVAE outperforms CVAE and linear-interpolation baselines when predicting out-of-sample test images of objects in specified views.
  • The face-dataset experiments report mean squared error, out-of-sample predictions in unseen poses, and learned object and view covariances.
  • The method’s implementation is publicly available.

Derivation of the ELBO

The derivation assumes a variational posterior over latent variables and uses it to formulate the evidence lower bound. The supplied passages identify ψ as the variational parameters.

  • The latent variables are assigned a posterior distribution parameterized by ψ.
  • The evidence lower bound is derived from the assumed posterior distribution.
  • The posterior is denoted qψ(Z | Y) in the ELBO derivation.

Derivation of the loss function

The loss derivation approximates the ELBO by sampling reparameterized latent representations and separates reconstruction from latent-space GP regularization. The trade-off parameter λ is selected using validation performance.

  • The expectation in the ELBO is approximated by sampling a latent representation Zψ.
  • The sampling procedure uses εn drawn from a standard multivariate normal distribution.
  • The reparameterization trick separates noisy sample generation from model parameters in the approximate ELBO.
  • The training cost contains a reconstruction term and a latent-space GP model term.
  • λ balances data reconstruction against latent-space prior regularization.
  • λ is selected by maximizing the standard VAE ELBO on a validation set and reused across compared models.

Fast low-rank computations

The method accelerates Gaussian-model inference by exploiting a covariance matrix formed from a low-rank component plus the identity. Woodbury and determinant-lemma computations reduce the required costs.

  • Gaussian-model parameter inference normally scales cubically with the number of observations.
  • The covariance structure is expressed as a low-rank matrix plus the identity matrix, with rank O much smaller than N.
  • The Woodbury identity and determinant lemma compute K^-1M and logdet K using the low-rank structure.
  • The resulting complexities are O(NO^2 + O^3 + ONK) for the linear system and O(NO^2 + O^3) for the log determinant.

Implementation of low-memory stochastic backpropagation

GPPVAE uses a low-memory stochastic backpropagation strategy to optimize GP-coupled latent representations without loading the full high-dimensional dataset into memory. The procedure combines mini-batch encoder computation, low-dimensional full-sample operations, and a local Taylor approximation, while the predictive posterior combines GP and decoder components.

  • Low-memory stochastic backpropagation: Full-matrix operations are restricted to the low-dimensional representation space, avoiding the infeasible memory demands of naive full-dataset gradient descent.The latent representation depends on image-space inputs, so naive differentiation would require loading the entire high-dimensional dataset.
  • Low-memory stochastic backpropagation: The GP term is locally approximated with a first-order Taylor expansion so its gradients can be accumulated using mini-batches.The approximation locally preserves gradients at each optimization step and linearizes the high-memory nested derivatives.
  • Low-memory stochastic backpropagation: The four-step procedure stores reparameterization noise, computes latent variables by mini-batch forward propagation, evaluates low-dimensional quantities across samples, and accumulates gradients mini-batchwise.The low-memory operations evaluate a, B, and c across all samples before using the Taylor proxy for optimization.
  • Low-memory stochastic backpropagation: Computing the low-dimensional quantities has linear complexity in the number of samples because the covariance matrix has low-rank structure.This structure supports scalable evaluation of the quantities used in the gradient procedure.
  • Predictive posterior: The predictive posterior for a new image combines the decoder likelihood, the GP predictive posterior for its latent representation, and the approximate posterior over training latents.The GP prior is defined over training latent variables, while the new latent representation is conditioned on object and view representations and the training latents.
  • Experimental architecture: The MNIST and face experiments use the same encoder and decoder architectures for VAE and GPPVAE, while CVAE additionally receives view information in both networks.MNIST CVAE inputs encode rotation angles, whereas face CVAE inputs use one-hot representations of nine poses.
Loading 1810.11738v2…