PaperlayerView exact revision on arXiv

Source-linked AI summary

Amortized Structured Stochastic Variational Inference for Gaussian Process Latent Variable Models

Maksym Tretiakov, Sarah Lucie Filipp, Vincent Fortuin, Ruth Misener, Ruby Sedgwick, James Odgers

arXiv:2610.03647v1stat.MLcs.LG

TL;DR

GPLVM uncertainty estimates are limited by a mean-field approximation that separates latent variables from GP inducing points. The paper uses amortized structured variational inference to model their conditional dependence. Across four datasets, the method improves several reconstruction and predictive-distribution metrics, with results varying by baseline and metric.

  • Problem

    The mean-field variational approximation limits GPLVM uncertainty estimation by separating latent variables from GP inducing points.

    Source: §1 · p. 1, §3 · p. 4

  • Method

    A-S-GPLVM applies structured variational inference to couple the latent-variable posterior to GP inducing variables.

    Source: §3 · p. 4, §7 · p. 10

  • Results

    Across four datasets, A-S-GPLVM significantly improves train and test CRPS over A-MF-GPLVM, with RMSE and NLPD improvements also significant except for Oilflow RMSE.

    Source: §7 · p. 10

  • Takeaways & Limitations

    Against IWVI-GPLVM, results are stronger on MNIST and FMNIST, while smaller-dataset results and empirical 95% coverage are more mixed.

    Source: §7 · p. 10

    A-S-GPLVM has a slight wall-clock slowdown and significantly higher memory usage than amortized mean-field GPLVM.

    Source: §6 · p. 10

Abstract

from arXiv · show

Many machine learning methods aim to approximate the lower-dimensional manifold on which the data lives. A desirable feature of such methods is that they should capture the epistemic uncertainty of this learned manifold. One model that achieves this is the Gaussian Process Latent Variable Model, in which a Gaussian Process (GP) mapping from the latent space provides an estimate of the uncertainty of the manifold. However, the effectiveness of this uncertainty estimation is limited by the mean-field variational approximation between the GP inducing points and the latent variables. In this work, we apply Amortized Structured Stochastic Variational Inference to allow the variational posterior for the latent space to be conditionally dependent on the value of the inducing points. We demonstrate that this more flexible variational posterior improves several metrics relating to the reconstruction of points on the data manifold.

1 Introduction

High-dimensional data may lie on a lower-dimensional manifold, and noisy, limited datasets make estimating uncertainty about that manifold important. The paper addresses a mean-field limitation in GPLVMs with a structured variational posterior that links latent variables to GP inducing points.

1 Introduction

  • Noisy, limited datasets make it important to estimate epistemic uncertainty about a learned data manifold, not just its point estimates.GPLVMs provide a non-parametric Bayesian approach for learning complex manifolds while remaining robust to overfitting.Source: §1 · p. 1
  • The mean-field variational approximation makes GPLVM latent positions and the GP mapping independent, resulting in underestimated epistemic uncertainty.The paper identifies this as a shared shortcoming of GPLVM approaches.Source: §1 · p. 1
  • Structured Stochastic Variational Inference lets the latent-variable posterior depend on sparse GP inducing points, relaxing the mean-field restriction.The paper presents this as an improved approximation to the true posterior.Source: §1 · p. 1
  • The experiments compare reconstruction and uncertainty metrics, while later sections discuss limitations and further work.The paper also provides background, derives the ELBO, and relates the approach to existing methods.Source: §1 · p. 2

2 Background

GPLVMs model high-dimensional observations through lower-dimensional latent variables and Gaussian Process mappings, while variational inference makes posterior inference tractable. The background identifies the mean-field independence between latent variables and GP mappings as a limitation that can lead to underestimated manifold uncertainty.

2 Background

  • GPLVMs represent each high-dimensional observation y_n with a lower-dimensional latent variable x_n and a mapping from latent space to observations.The setup assumes P ≫ Q for observations in R^P and latent variables in R^Q.Source: §2 · p. 2
  • Independent zero-mean Gaussian Processes model the output dimensions, with an ARD kernel controlling function covariance and fit through marginal likelihood.The latent variables have a spherical Normal prior, and observation noise is assumed Gaussian.Source: §2 · p. 2
  • Variational inference approximates the posterior by minimizing KL divergence, with the ELBO as its optimization objective; GP inducing points make the approach practical.The inducing points are additional latent-space inputs and random outputs, shared across output dimensions for computational reasons.Source: §2 · p. 3
  • The mean-field variational family factorizes latent-variable and GP posteriors, so every likely latent value is combined with every likely inducing-point value.The paper says this trade-off tends to underestimate epistemic uncertainty in the manifold.Source: §2 · p. 3
  • The MF-GPLVM ELBO drops the KL term for GP function values because their conditional distribution is shared by the variational posterior and prior.The section describes amortized inference as predicting latent-posterior parameters from observations with a learned function.Source: §2 · p. 3

3 Structured Stochastic Variational Inference for GPLVMs

The method relaxes GPLVM mean-field independence by making each latent-variable posterior depend on the global inducing variables. Its structured ELBO retains per-observation contributions, enabling stochastic mini-batch optimization.

3 Structured Stochastic Variational Inference for GPLVMs

  • The variational posterior mirrors the full posterior’s conditional structure, allowing latent variables and inducing points to be dependent.The standard mean-field assumption is described as strong and incorrect because conditioning on observations makes the posterior over inducing points and latent variables dependent.Source: §3 · p. 4
  • The latent posterior q(x_n|U) uses parameters produced by h(y_n,U), while q(U) retains Gaussian distributions with learned constant parameters.The variational procedure learns both the inducing-point parameters and the mapping h.Source: §3 · p. 4
  • The structured ELBO uses a local objective over q(x_n|U), and Monte Carlo samples of U approximate its otherwise intractable expectation.The per-observation terms allow stochastic optimization by mini-batching; special cases may permit analytic evaluation.Source: §3 · p. 4
  • Across datasets, A-S-GPLVM and A-MF-GPLVM converge at similar rates, but A-S-GPLVM reaches a better training NLPD optimum.Figure 1 shows the training NLPD progression across all datasets.Source: Figure 1 · p. 5

3.1 Local and Global Variables for Stochastic Variational Inference

  • Treating inducing variables as global and latent variables as local yields per-data-point ELBO contributions that support mini-batch gradient estimates.The method is designed for datasets too large to fit into memory at once.Source: §3.1 · p. 5

4 Related work

Related approaches couple latent states with GP posteriors in sequential models or tighten variational bounds through importance weighting. This work instead conditions GPLVM latent variables on the GP mapping.

4 Related work

  • Ialongo et al. [2019] couple latent-state trajectories and GP transition posteriors, improving calibration at similar complexity in sequential GP state-space models.Their setting differs from this paper's latent-variable setting.Source: §4 · pp. 5–6
  • Importance-weighted variational inference offers a sparse-GPLVM alternative to explicit posterior coupling, trading additional computation for a tighter lower bound.The approach was developed for deep Gaussian processes by Salimbeni et al. [2019] and adapted as a relevant baseline for this comparison.Source: §4 · p. 6
  • The authors report no prior method conditioning GPLVM latent variables on the GP mapping, distinguishing their approach from related RFLVM and AIS work.RFLVM replaces sparse variational GPs with random Fourier features; later work used variational inference, while Xu et al. [2024] used AIS.Source: §4 · pp. 6–7

5 Experiments

The experiments compare A-S-GPLVM with mean-field and importance-weighted GPLVM variants on four datasets. A-S-GPLVM generally improves reconstruction and predictive-uncertainty metrics, while latent-space and uncertainty plots show additional qualitative differences.

5 Experiments

  • Experiments use Oilflow and qPCR at small scale and subsampled MNIST and Fashion MNIST at larger scale, with 90/10 splits for small datasets and 1,000 training plus 500 test points per class for the larger ones.The comparisons include SVI and amortized MF variants; larger-dataset comparisons omit non-amortized MF-GPLVM.Source: §5 · pp. 7–8
  • Evaluation uses negative ELBO, RMSE, NLPD, CRPS, and empirical coverage of marginal 95% predictive intervals.Lower CRPS is better; coverage closer to 0.95 indicates better calibration.Source: §5 · p. 8

5.1 Results.

  • Across four datasets, A-S-GPLVM lowers train/test RMSE and NLPD versus A-MF-GPLVM; versus IWVI-GPLVM, NLPD is lower on all datasets, with significant RMSE gains on MNIST and FMNIST.On Oilflow, IWVI-GPLVM has lower RMSE; on qPCR, A-S-GPLVM's lower RMSE is not statistically significant.Source: §5.1 · pp. 8–9
  • A-S-GPLVM has significantly lower train/test CRPS than A-MF-GPLVM on all four datasets, but empirical 95% coverage shows no uniform winner.Against IWVI-GPLVM, A-S-GPLVM's coverage is closer to 95% on Oilflow and qPCR.Source: §5.1 · p. 9
  • A-S-GPLVM converges at a rate similar to A-MF-GPLVM but reaches a better final training NLPD; non-amortized MF-GPLVM test performance suffers from per-point non-convex optimization.That optimization can converge to poor local optima.Source: §5.1 · p. 9
  • On FMNIST, A-S-GPLVM forms a crescent-shaped latent space separating footwear from clothing, with a clear data-free gap between the groups.The latent projections show tighter clusters than the (A-)MF-GPLVM; the qPCR and MNIST plots show similar behavior.Source: §5.1 · p. 9
  • A-S-GPLVM reconstructions are reasonable and mostly improve statistically over A-MF-GPLVM, yet their visual similarity suggests other GPLVM improvements are needed for an effective generative model.The examples show means of q(y_n) for eight data points.Source: §5.1 · p. 9
  • Away from high-probability latent inputs, roughly [−2, 2], A-S-GPLVM's epistemic uncertainty rises faster than MF models, though its GP approximation prevents perfect posterior-uncertainty estimation.The plotted slice varies the most informative latent coordinate while fixing the others at zero.Source: Figure 5 · p. 9, §5.1 · p. 10

6 Limitations

The structured approximation adds computational cost and still restricts the variational posterior, so it is expected not to match the exact posterior.

Computational overhead

  • The structured objective requires Monte Carlo inducing-variable samples and conditional local posteriors, causing a slight wall-clock slowdown but a significant memory increase.The K sampled states can be processed in parallel; informative sample selection is suggested as a possible way to reduce cost.Source: §6 · p. 10

Restrictive variational family

  • The posterior remains restricted to diagonal-Gaussian q(xn | U) and factorized q(U), and is expected not to match the exact posterior.The authors suggest relaxing these approximations as a direction that may yield similar performance improvements.Source: §6 · p. 10

7 Conclusion

The paper introduces SSVI for GPLVMs by conditioning latent-variable posteriors on inducing variables. Across four datasets, it reports improvements over mean-field and IWVI baselines, with mixed results on smaller datasets and nonuniform empirical coverage.

7 Conclusion

  • Across Oilflow, qPCR, MNIST, and FMNIST, A-S-GPLVM improves train and test CRPS over A-MF-GPLVM; RMSE and NLPD improve significantly except Oilflow RMSE.Compared with IWVI-GPLVM, it has significantly lower train and test NLPD on all four datasets, and lower RMSE and CRPS on MNIST and FMNIST.Source: §7 · p. 10
  • On smaller datasets, comparisons with IWVI-GPLVM are more mixed, and empirical 95% coverage does not improve uniformly.The authors interpret the findings as quantitative evidence that structured posterior dependence improves predictive-distribution quality.Source: §7 · p. 10
  • Future work will explore using improved data-manifold uncertainty for information sharing and uncertainty estimates in multiobjective Bayesian optimization.The paper gives Sedgwick et al. [2025] as an example.Source: §7 · p. 10

B SSVI ELBO derivation details

This appendix details the ELBO calculation, amortized encoder choices, selected training settings, and computational and statistical setup.

B SSVI ELBO derivation details

  • The local ELBO combines expected log-likelihood terms across the P output dimensions with a KL divergence, which is closed form for the parametric local posterior.The derivation approximates the ELBO using K Monte Carlo samples of U; the ARD-kernel expectations have closed-form solutions.Source: B SSVI ELBO derivation details · pp. 14–15

C Amortized Encoder

  • The encoder uses separate two-hidden-layer MLPs for mean and log-variance, and reparameterized inducing samples provide well-spread inputs while retaining sample-specific coupling information.Reparameterization also allows gradients to propagate through samples of the global inducing variables.Source: C Amortized Encoder · p. 15

D Model Configuration

  • For Oilflow and qPCR, the listed settings include 10 latent dimensions, 64 inducing points per output, 200,000 iterations, and K = 4 samples.The configuration also lists MLP hidden dimensions (512, 256).Source: D Model Configuration · p. 16

E Computational setup

  • Experiments used similar GPU-cluster nodes with slight hardware variation; Table 3 reports Welch-test p-values from five independent runs comparing A-MF-GPLVM and A-S-GPLVM.The passage points to Appendix G for runtime and memory measurements.Source: E Computational setup · p. 17, Table 3 · p. 17

F Statistical significance of the observed improvements

Welch tests over five runs indicate statistically significant improvements on probabilistic metrics across datasets, while Oilflow RMSE differences are not significant. The Oilflow results distinguish probabilistic quality from point-estimate accuracy.

F Statistical significance of the observed improvements

  • The comparison uses two-sided Welch’s t-tests over five independent runs of A-MF-GPLVM and A-S-GPLVM on each dataset.The p-values are reported in Table 3.Source: F Statistical significance of the… · p. 17
  • Across datasets, probabilistic metrics improve significantly; qPCR differences are significant at 10^-3 or smaller and MNIST and FMNIST differences at 10^-4 or smaller.On Oilflow, both NLPD improvements have p < 10^-3 and the ELBO difference has p < 10^-4.Source: F Statistical significance of the… · p. 17
  • On Oilflow, RMSE differences are not significant (p = 0.12 training; p = 0.55 test), despite significant NLPD and ELBO improvements.The authors interpret this as improved posterior approximation and predictive uncertainty rather than merely a shift in predictive mean.Source: F Statistical significance of the… · p. 17

G Runtime and memory overhead

The structured approximation raises runtime and GPU memory use in the measured configuration. Its additional operations are parallelizable, making memory the expected practical trade-off, while the reported overhead depends partly on the implementation.

Runtime and memory overhead

  • The structured method enlarges encoder input by concatenating sampled inducing states with observations and uses K Monte Carlo replicas per ELBO evaluation; both operations are parallelizable.The expected practical trade-off is increased memory rather than an inherently sequential increase in computation.Source: G Runtime and memory overhead · p. 17
  • 8.36 s versus 5.03 s in wall-clock time, and 598.7 MB versus 29.0 MB in peak GPU memory, for A-S-GPLVM versus A-MF-GPLVM on Oilflow.Measurements covered 100 iterations with a minibatch of 32, 32 inducing points, and 32 Monte Carlo samples.Source: G Runtime and memory overhead · pp. 17–18
  • The measurements characterize the current implementation, not an unavoidable algorithmic constant; the method retains sparse minibatched stochastic variational structure while trading memory for a richer posterior.Some wall-clock overhead is attributed to implementation details in a shared layered design across models.Source: G Runtime and memory overhead · p. 18

H Ablation studies

The ablations support the structured variational family, rather than increased encoder capacity, as the source of the main gains; they also show performance depends on inducing-point and inducing-state sample counts.

Encoder architecture

  • Across five two-hidden-layer MLP widths, A-S-GPLVM consistently outperforms A-MF-GPLVM on ELBO and NLPD, even with a small encoder versus a larger mean-field encoder.The tested widths were [16, 8], [32, 16], [64, 32], [128, 64], and [256, 128].Source: H Ablation studies · p. 18, Figure 7 · p. 19

Number of inducing points

  • At 16, 32, and 64 inducing points, A-S-GPLVM performs better on ELBO and NLPD, often matching or exceeding A-MF-GPLVM with twice as many points.The performance margin narrows at larger inducing-point counts; the authors suspect the growing encoder input becomes too large for the small MLP in this ablation.Source: H Ablation studies · p. 18, Figure 8 · p. 20

Number of inducing-variable samples

  • Larger K improves results; when K is small, lowering the learning rate can recover performance but slows convergence, while memory constraints may prevent increasing K.The authors attribute larger-K improvements to reduced gradient-estimator variance; the ablation compares K = 4 and K = 64, and learning rates 10^-3 and 3 × 10^-4.Source: H Ablation studies · p. 18, Figure 9 · p. 20

Ablation studies

  • Across the ablations, gains track the structured variational family rather than encoder capacity, while performance also depends on inducing-point and inducing-state sample counts.The ablations cover encoder architecture, inducing-point counts, and sampled inducing-variable states.Source: H Ablation studies · p. 18

Additional uncertainty visualization

  • In the most informative latent-dimension slice, the A-S-GPLVM GP mapping has significantly larger predictive uncertainty away from the data.Source: Figure 10 · p. 21

I Additional Results Plots

The appendix adds plots of latent spaces, reconstructions, and training trajectories across qPCR, MNIST, and Fashion-MNIST.

qPCR latent spaces

  • The qPCR latent-space comparison presents MF-GPLVM, A-MF-GPLVM, and A-S-GPLVM side by side.Source: Figure 11 · p. 22

qPCR training dynamics

  • On qPCR, A-S-GPLVM converges to a lower final NLPD than MF-GPLVM and A-MF-GPLVM; the plots track ELBO, RMSE, and NLPD over training.Source: Figure 12 · p. 22

Sample reconstructions

  • The sample reconstructions show posterior predictive means from A-MF-GPLVM.Source: Figure 13 · p. 23

MNIST results

Fashion-MNIST results

  • The Fashion-MNIST training-dynamics plots track ELBO, RMSE, and NLPD across inference methods.Source: Figure 16 · p. 25

Related papers

Loading 2610.03647v1…