PaperlayerView exact revision on arXiv

Source-linked AI summary

Variational inference for Monte Carlo objectives

Andriy Mnih, Danilo J. Rezende

arXiv:1602.06725v2cs.LGstat.ML

TL;DR

Simple variational posteriors can limit model capacity, while multi-sample objectives create difficult high-variance gradients for discrete latent variables. The paper introduces VIMCO, an unbiased estimator with low-variance per-sample learning signals, and finds that it consistently outperforms NVIL and competes with biased estimators across generative and structured output prediction models.

  • Problem

    Insufficiently expressive variational posteriors can restrict model capacity, while multi-sample objectives are difficult to optimize for discrete latent variables because their gradient estimates have high variance.

  • Method

    VIMCO is an unbiased gradient estimator for multi-sample objectives that uses low-variance per-sample learning signals without learned baselines or additional computational cost.

  • Results

    VIMCO consistently outperformed NVIL and was competitive with currently used biased estimators on generative and structured output prediction models.

  • Takeaways & Limitations

    Multi-sample objectives offer a way to trade computation for quality of fit by increasing the number of samples, and VIMCO makes them usable with discrete latent variables.

  • Takeaways & Limitations

    The estimator can have very high variance when most latent configurations explain an observation poorly because sampling does not account for that observation.

Abstract

from arXiv · show

Recent progress in deep latent variable models has largely been driven by the development of flexible and scalable variational inference methods. Variational training of this type involves maximizing a lower bound on the log-likelihood, using samples from the variational posterior to compute the required gradients. Recently, Burda et al. (2016) have derived a tighter lower bound using a multi-sample importance sampling estimate of the likelihood and showed that optimizing it yields models that use more of their capacity and achieve higher likelihoods. This development showed the importance of such multi-sample objectives and explained the success of several related approaches. We extend the multi-sample approach to discrete latent variables and analyze the difficulty encountered when estimating the gradients involved. We then develop the first unbiased gradient estimator designed for importance-sampled objectives and evaluate it at training generative and structured output prediction models. The resulting estimator, which is based on low-variance per-sample learning signals, is both simpler and more effective than the NVIL estimator proposed for the single-sample variational objective, and is competitive with the currently used biased estimators.

1. Introduction

Multi-sample objectives address the capacity limitations of simple variational posteriors, but extending their gradient-based optimization to discrete latent variables remains difficult because gradient estimates can have high variance. The paper develops an unbiased estimator using low-variance per-sample learning signals.

  • Neural variational inference approximates an intractable posterior with a neural network and estimates lower-bound gradients from posterior samples.This provides an efficient feedforward alternative to traditional per-datapoint iterative variational inference.
  • Insufficiently expressive variational posteriors can cause powerful models to use only a small fraction of their capacity.One direct response is to construct more expressive but still tractable variational posteriors.
  • Averaging multiple likelihood samples produces a tighter lower bound that increasingly approaches the true log-likelihood as sample count grows.The paper calls objectives based on such averaged independent samples Monte Carlo objectives.
  • Multi-sample objectives have been applied to generative modeling, structured output prediction, and hard-attention models, with prior work reporting better likelihoods and greater latent-variable use.These objectives are a better proxy for log-likelihood than single-sample objectives.
  • For discrete latent variables, differentiating a multi-sample objective yields a high-variance gradient estimator, so prior methods generally did not optimize the proposal distribution with that gradient.This creates a mismatch between proposal and model-parameter updates.
  • The paper introduces an unbiased multi-sample gradient estimator that replaces one learning signal with lower-variance per-sample signals and requires no additional variance-reduction parameters.The estimator is designed to make discrete-latent-variable models easier to integrate into end-to-end systems.

2. Multi-sample stochastic lower bounds

The paper constructs multi-sample stochastic lower bounds from unbiased likelihood estimators and explains why their proposal-distribution gradients are difficult to estimate. It then introduces per-sample learning signals to reduce variance without bias or additional variance-reduction parameters.

  • Likelihood estimation: The likelihood estimator averages K independent samples from a proposal distribution Q(h|x), using importance sampling to incorporate the observation.Importance sampling can reduce variance relative to sampling from the prior because Q(h|x) can favor latent configurations with high joint probability with x.
  • Lower bound construction: Taking the logarithm of an unbiased likelihood estimator produces a stochastic lower bound on the log-likelihood by Jensen’s inequality.This construction applies to any unbiased likelihood estimator based on random sampling.
  • Multi-sample objective: Increasing K makes the multi-sample bound tighter and a better proxy for log-likelihood, while reducing the need for every individual sample to explain the observation well.Multi-sample objectives have been used for variational training, generative modelling, structured prediction, and hard attention.
  • Gradient analysis: The naive proposal-distribution gradient uses one learning signal for all K samples, so it lacks within-set credit assignment and can have high variance.The learning signal can also have unbounded magnitude, especially early in training when the proposal samples explain the data poorly.
  • Variance reduction: VIMCO replaces the shared learning signal with lower-variance per-sample signals based on separate baselines that depend on the other samples, while remaining unbiased.The per-sample signals have the same computational order as the global signal, require no additional learned variance-reduction parameters, and require more than one sample.

3. Structured output prediction

The paper treats structured output prediction as conditional latent-variable modeling, using contexts and observations to capture dependencies. It contrasts this with prior-based training, which can require many samples because the prior ignores the observation.

  • Structured output prediction models high-dimensional outputs with rich structure while capturing dependencies among output variables and between outputs and inputs.
  • The paper formulates structured output prediction as conditional probabilistic modeling with latent variables.
  • Given context/observation pairs (c, x), the model fits P(x, h|c) to capture context–observation and observed-dimension dependencies.
  • The conditional model factorizes as P(x, h|c) = P(x|h, c)P(h|c), with both likelihood and prior conditioned on context.
  • Prior-based training ignores observation information, so such methods tend to require many samples to perform well.

4. Related work

Related work includes multi-sample objectives, Reweighted Wake-Sleep, and black-box gradient methods. The paper positions VIMCO as an unbiased black-box estimator for multi-sample objectives, contrasting it with methods limited to continuous variables or based on biased or separate updates.

  • Multi-sample objectives: Multi-sample objectives have been applied to structured prediction and evaluated with gradient estimators, including a simple biased estimator that performed best in one study.
  • Multi-sample objectives: The unbiased gradient estimator associated with that prior structured-prediction method applies only to models with continuous latent variables.
  • Reweighted Wake Sleep: Reweighted Wake-Sleep uses a model-parameter estimator identical to the multi-sample objective's estimator, while its proposal update is motivated as a biased KL-gradient estimator whose bias vanishes with infinitely many samples.
  • Reweighted Wake Sleep: RWS combines wake and sleep updates, but the passage states that these updates do not appear related to optimizing the lower bound and therefore do not optimize a well-defined objective.
  • Black Box Methods: BBVI and NVIL are black-box methods for classical variational objectives, typically built around unbiased gradient estimators for the proposal distribution.
  • Black Box Methods: VIMCO resembles local expectations because both use multiple latent-variable values and can work without baselines, but local expectations optimizes a single-sample objective and requires exact expectations for each variable.

5. Results

VIMCO was evaluated against NVIL and RWS on generative modeling and structured output prediction, with performance generally improving as multi-sample objectives used more samples. VIMCO was consistently strong, especially with learned proposals and larger sample counts.

  • Experimental setup: VIMCO was compared with NVIL and RWS on generative modeling and structured output prediction using multi-sample objectives.The experiments used sigmoid belief networks with hundreds of binary latent variables.
  • 5.1. Generative modelling: On MNIST generative modeling, VIMCO optimized the multi-sample validation objective more effectively than NVIL and benefited more from additional samples.RWS performed similarly overall, with VIMCO better at 2 samples and RWS learning slightly faster with more samples.
  • 5.1. Generative modelling: As sample counts increased, VIMCO and RWS test performance steadily improved, whereas NVIL remained nearly unchanged before becoming markedly worse at K = 50.Overall, VIMCO and RWS performed similarly, with VIMCO showing a slight edge across the considered sample counts.
  • 5.1. Generative modelling: For the 10-sample objective, VIMCO’s average learning-signal magnitude was 3 times lower than NVIL’s.The methods were comparable with 2 samples, but VIMCO benefited substantially more from increasing the sample count.
  • 5.2. Structured output prediction: In structured output prediction with prior proposals, increasing samples improved performance dramatically, and VIMCO surpassed NVIL at 20 and 50 samples.The strong sample-count effect suggested that prior-generated samples rarely explained the observations well.
  • 5.2. Structured output prediction: Learned proposals drastically improved every method and sample-count combination; VIMCO performed better than NVIL and captured multimodal digit completions.The worst learned-proposal result exceeded the best prior-proposal result.
  • 5.2. Structured output prediction: 46.1 nats was achieved with a learned proposal, producing the first sub-50 nat result on the structured output prediction task.The same two-layer VIMCO model achieved 56.5 nats with the prior proposal, while the earlier best result was 53.8 nats.

6. Discussion

The paper introduces VIMCO as an unbiased estimator for multi-sample objectives and demonstrates its effectiveness on generative and structured prediction models. It provides variance reduction without extra computational cost and performs competitively with biased estimators.

  • Contribution: VIMCO is an unbiased general gradient estimator designed for multi-sample objectives that generalize the classical variational lower bound.It uses the objective’s structure for per-sample variance reduction without additional computational cost.
  • Empirical conclusion: VIMCO consistently outperformed NVIL and was competitive with currently used biased estimators in generative and structured output prediction.The evaluation covered both model classes.
  • Discussion: Multi-sample objectives provide a way to trade computation for quality of fit by increasing the number of samples used inside the objective.The paper suggests combining these objectives with black-box variational inference methods.

A. Algorithm for computing VIMCO gradients

The VIMCO algorithm samples K latent configurations, computes the multi-sample stochastic bound and importance weights, then accumulates proposal and model gradient contributions using leave-one-out learning signals.

  • Inputs and sampling: For one observation, the algorithm requires K ≥2 samples h_i drawn from Q(h|x).Each sample’s log f(x, h_i) is stored before computing the bound.
  • Bound computation: The multi-sample stochastic bound is computed as LogSumExp(l) − log K from the sampled log-likelihood terms.The sum of log f values is precomputed for subsequent leave-one-out baselines.
  • Variance reduction: For each sample, VIMCO replaces its log f value with the average of the other K−1 terms to form a leave-one-out bound.The original value is restored after computing each leave-one-out bound.
  • Gradient accumulation: The algorithm computes importance weights with SoftMax(l) and sums weighted model-gradient contributions across samples.Proposal gradients use the difference between the full and leave-one-out bounds multiplied by ∇θ log Q(h_i|x).

B. Details of the experimental protocol

The experiments used Adam, minibatches of 24, centered proposal-network inputs, multiple learning rates, and validation-based model selection. VIMCO used geometric-mean per-sample learning signals, while NVIL used learned baselines and variance normalization.

  • Optimization: All models were trained with Adam using minibatches of size 24.Proposal-network inputs were centered by subtracting their mean.
  • Model selection: For each method and sample-count combination, models were trained repeatedly with different learning rates and selected by their best validation score.The reported plots and scores came from the run with the highest validation score.
  • Variance analysis: Figure 3 tracks the root-mean-square learning-signal magnitude for VIMCO and NVIL across sample counts and parameter updates.The learning-signal magnitude is related to the variance of the resulting gradient estimator.
  • Optimization: The learning-rate candidates were {3×10^-4, 1×10^-3, 3×10^-3} for VIMCO and RWS and {1 × 10^-4, 3 × 10^-4, 1 × 10^-3} for NVIL.Different candidate sets were used for the estimator methods.
  • Variance reduction: NVIL used constant and input-dependent baselines plus variance normalization, while VIMCO used geometric means for per-sample learning signals.The input-dependent NVIL baseline was a one-hidden-layer neural network with 100 tanh units.

C. Effect of variance reduction on the learning signal

VIMCO and NVIL reduce gradient-estimator variance by subtracting a baseline from the original learning signal while preserving unbiasedness.

  • VIMCO and NVIL reduce estimator variance by subtracting a baseline from the original learning signal ˆL(h1:K).The reduction works by decreasing the learning-signal magnitude without changing estimator unbiasedness.

D. Gradient derivation for the multi-sample objective

The multi-sample gradient is derived by differentiating the proposal distribution and objective with standard product- and chain-rule identities, then substituting the resulting expressions.

  • The derivation begins by applying the product rule to the multi-sample objective.
  • The identity ∇θg(x) = g(x)∇θ log g(x) rewrites the proposal-gradient term using a log-gradient form.
  • The chain rule and the same identity are used to compute the gradient of ˆL(h1:K).
  • The normalized importance weight is defined as ˜wj ≡ f(x,hj) PK i=1 f(x,hi).
  • Substituting the preceding expressions into Eq. 13 yields the multi-sample gradient expression.

E. Structured output prediction: digit completions

The structured output experiment generates conditional digit completions from observed top halves using a VIMCO-trained three-layer SBN. Variability across completions indicates that the model captured multimodality in the data distribution.

  • The model generates conditional digit completions by combining each original top half with a sampled bottom half.The completions use a three-layer SBN trained with VIMCO and the 20-sample objective.
  • Variability among completions for the same observed top halves shows that the model captured multimodality in the data distribution.
Loading 1602.06725v2…