PaperlayerView exact revision on arXiv

Source-linked AI summary

Filtering Variational Objectives

Chris J. Maddison, Dieterich Lawson, George Tucker, Nicolas Heess, Mohammad Norouzi, Andriy Mnih, Arnaud Doucet, Yee Whye Teh

arXiv:1705.09279v3cs.LGcs.AIcs.NEstat.ML

TL;DR

Latent-variable maximum likelihood is difficult when marginalization is intractable, and restricted ELBO posteriors can limit model capacity. The paper introduces FIVOs, log-transformed particle-filter likelihood objectives whose tightness is linked to estimator variance. On sequential data, FIVO-trained models significantly outperform ELBO- and IWAE-trained models across four polyphonic music tasks and one speech waveform task.

  • Problem

    Marginal likelihood optimization is generally intractable in latent-variable models, while restricted variational posteriors can reduce model capacity.

  • Method

    The paper defines FIVOs as Monte Carlo objectives formed from particle-filter marginal-likelihood estimators and studies their convergence and optimization.

  • Results

    FIVO-trained models significantly outperformed ELBO- and IWAE-trained models across four polyphonic music modeling tasks and one speech waveform modeling task.

  • Takeaways & Limitations

    Particle filters can provide tighter training objectives for sequential latent-variable models by exploiting more favorable estimator-variance scaling than naive importance sampling.

  • Takeaways & Limitations

    The sharpness analysis relies on an independence assumption that most models do not satisfy, and optimal variational proposals are complicated by resampling dynamics.

Abstract

from arXiv · show

When used as a surrogate objective for maximum likelihood estimation in latent variable models, the evidence lower bound (ELBO) produces state-of-the-art results. Inspired by this, we consider the extension of the ELBO to a family of lower bounds defined by a particle filter's estimator of the marginal likelihood, the filtering variational objectives (FIVOs). FIVOs take the same arguments as the ELBO, but can exploit a model's sequential structure to form tighter bounds. We present results that relate the tightness of FIVO's bound to the variance of the particle filter's estimator by considering the generic case of bounds defined as log-transformed likelihood estimators. Experimentally, we show that training with FIVO results in substantial improvements over training the same model architecture with the ELBO on sequential data.

1 Introduction

Latent-variable maximum likelihood is often intractable, motivating tractable lower-bound objectives. The paper introduces FIVOs, which use particle-filter likelihood estimators to exploit sequential structure and can produce tighter bounds than ELBO- or IWAE-style objectives.

  • Latent-variable maximum likelihood is difficult because marginal log-likelihood objectives generally involve intractable integration.
  • The ELBO provides a tractable lower bound whose joint optimum in the model and variational posterior recovers maximum likelihood.
  • Restricted variational posteriors can reduce model capacity by encouraging the model posterior to match the variational family.
  • FIVOs treat the log of a particle filter’s likelihood estimator as an objective, with tightness related to the estimator’s relative variance.
  • Experiments report that sequential models trained with FIVO outperform models trained with ELBO or IWAE in practice.

2 Background

The background reviews ELBO optimization for differentiable latent-variable models and explains why restricted variational families can limit model capacity. It then motivates objectives whose tightness need not depend solely on posterior approximation quality.

  • Expectation-maximization alternates optimization over the variational distribution and model parameters, but generally requires intractable inference.
  • Stochastic ELBO gradients can be estimated by sampling from q and updating model and variational parameters, though variance reduction is usually necessary.
  • Reparameterizable variational distributions provide lower-variance unbiased ELBO-gradient estimators when the model, variational distribution, and transformation are differentiable.
  • A restricted variational family can reduce model capacity, motivating ELBO generalizations whose tightness is controlled independently of q’s closeness to the model posterior.

3 Monte Carlo Objectives (MCOs)

Monte Carlo objectives generalize the ELBO by taking the expected log of positive unbiased marginal-likelihood estimators. Their bound tightness and convergence are linked to estimator variance, motivating particle-filter objectives for sequential models.

  • Monte Carlo objectives are defined as logarithms of positive unbiased estimators of the marginal likelihood and remain lower bounds usable for maximum likelihood.
  • The tightness of a Monte Carlo objective is closely related to the variance of its defining estimator.
  • The ELBO uses one importance weight, whereas IWAE uses an average of N independent importance weights.
  • Consistent likelihood estimators can drive their Monte Carlo objectives toward log p(x) as computation N increases.
  • Relative variance can compare convergence rates, but convergence of the bound is not generally monotonic.
  • Particle-filter estimators can have more favorable relative-variance scaling than naive importance sampling for sequential models.

4 Filtering Variational Objectives (FIVOs)

FIVOs apply Monte Carlo objectives to particle-filter likelihood estimators, exploiting sequential structure to obtain potentially tighter bounds than IWAE. Their optimization uses particle-filter trajectories and reparameterized gradients, while sharpness at the marginal likelihood requires a specific independence structure.

  • 4 Filtering Variational Objectives (FIVOs): In some sequential models, particle-filter relative variance scales linearly rather than exponentially with sequence length, making FIVOs potentially tighter than IWAE.Resampling refocuses particles toward higher-posterior-mass regions, although it introduces immediate variance and can greedily discard useful trajectories.
  • 4 Filtering Variational Objectives (FIVOs): Particle filters propagate N weighted particles through sequential importance sampling and resampling, using q to propose latent-state extensions.Resampling may occur according to criteria such as effective sample size, with particles inherited in proportion to their weights.
  • 4 Filtering Variational Objectives (FIVOs): FIVOs are Monte Carlo objectives defined as the expected log of a particle filter’s unbiased marginal-likelihood estimator.The estimator’s unbiasedness makes the expected log a lower-bound objective.
  • 4 Filtering Variational Objectives (FIVOs): FIVO converges to the marginal log-likelihood as N increases when the particle-filter log estimator is uniformly integrable.The convergence follows from strong consistency of the particle-filter estimator.
  • 4.1 Optimization: The practical gradient estimator keeps gradients through log ˆpN(x1:T), while omitting resampling-event terms because those terms contribute most of the estimator’s variance.The retained objective gradients are the solid red arrows in Figure 1; the full gradient also includes parameter-dependent sampling and resampling terms.
  • 4.2 Sharpness: At the optimal q, FIVO equals log p(x1:T) only under an independence structure, and most models do not satisfy that assumption.The authors report that future-observation conditioning does not reliably improve FIVO in richer models, while tighter-bound optimization remains advantageous.

5 Related Work

Related work has studied marginal-likelihood estimation, importance sampling, and sequential Monte Carlo, while FIVO treats an inference procedure’s likelihood-estimator output as a training objective. Concurrent work also considered particle-filter log estimators, but this paper emphasizes MCO convergence with particle count.

  • 5 Related Work: Prior Monte Carlo-objective work primarily studied estimating marginal likelihoods, while IWAE directly influenced FIVO as an extended bound.The paper positions FIVO within literature using Jensen’s inequality and importance-sampling estimators.
  • 5 Related Work: Other approaches use importance sampling or sequential Monte Carlo to approximate posterior gradients, with inference remaining their ultimate goal rather than the objective itself.FIVO instead treats the output of inference as an objective for learning.
  • 5 Related Work: Concurrent work also considered optimizing particle-filter log estimators, whereas this paper studies Monte Carlo-objective convergence as particle count N grows.The cited concurrent work characterized FIVO differently through a particle-filter approximation or a KL-based bias expression.

6 Experiments

Experiments compare ELBO, IWAE, and FIVO training for VRNNs on speech and polyphonic music, including resampling gradients, smoothing, and stochastic-state usage. FIVO significantly outperformed ELBO and IWAE across the evaluated sequential tasks, while omitting resampling-gradient terms improved optimization.

  • Experimental setup: VRNNs were trained with ELBO, IWAE, and FIVO bounds on natural speech waveforms and four polyphonic music datasets.The music datasets were Nottingham, JSB chorales, MuseData, and Piano-midi.de.
  • Speech: FIVO significantly outperformed IWAE and ELBO on TIMIT test log-likelihood bounds.TIMIT results were reported as average log-likelihood bounds per sequence relative to models trained with ELBO.
  • Resampling Gradients: Models trained without resampling gradients outperformed models trained with them on both training and held-out data.Many runs with resampling gradients failed to improve beyond random initialization.
  • Resampling Gradients: Gradients without the resampling term produced earlier convergence and a better solution in representative TIMIT training curves.The comparison used LFIVO 128 on TIMIT with N = 4.
  • Smoothing: FIVO is not sharp at its optimal variational posterior because that posterior does not condition on future observations.A smoothing VRNN variant supplied future information through a backward deterministic RNN.
  • Use of Stochastic State: ELBO-trained models showed KL collapse toward the prior, whereas FIVO-trained models retained high KL while achieving a higher log-likelihood bound.The KL divergence was evaluated on JSB chorales with N = 16.

7 Conclusions

The paper introduces FIVOs as lower bounds extending the ELBO and evaluates them against ELBO and IWAE. FIVO training significantly outperformed the alternatives across four polyphonic music tasks and a speech waveform task.

  • Contributions: FIVOs are lower bounds on log marginal likelihood that extend the evidence lower bound.They are designed for maximum likelihood estimation in neural latent variable models.
  • Empirical findings: FIVO-trained models significantly outperformed ELBO- and IWAE-trained models across four polyphonic music tasks and one speech waveform task.Future work includes control variates for resampling gradients and more sophisticated filtering algorithms.

Other Examples of MCOs.

The paper places FIVOs within a broader class of Monte Carlo objectives formed by log-transforming unbiased likelihood estimators. It illustrates this perspective with annealed and multiple importance sampling objectives.

  • Monte Carlo objectives: Marginal likelihood estimators can define Monte Carlo objectives by applying a logarithm to the estimator.The related-work discussion considers annealed importance sampling and multiple importance sampling in more detail.
  • Annealed Importance Sampling MCO: The AIS objective uses a variational posterior, a sequence of inverse temperatures, and Markov transitions targeting intermediate distributions.The initial latent state is sampled from q(z|x).
  • Annealed Importance Sampling MCO: AIS does not require intermediate states to be at equilibrium or transitions to preserve the joint target distribution.The resulting sequential Monte Carlo procedures provide unbiased marginal-likelihood estimators.
  • Annealed Importance Sampling MCO: The AIS objective is sharp when q equals the true posterior and transitions are identity copy operators.Finding scalable, easily optimized transition operators is the main practical difficulty identified.
  • Multiple Importance Sampling MCO: The MIS objective combines samples from possibly distinct variational posteriors using nonnegative weights summing to one.Each sampled latent state is evaluated under all proposal distributions, yielding a Rao-Blackwellized view.
  • Multiple Importance Sampling MCO: MIS is sharp when any proposal equals the true posterior and its weight is one, but useful optimization must distinguish proposals and assign appropriate weights.The objective is based on an unbiased estimator.

Proof of Proposition 1.

The proof establishes properties of Monte Carlo objectives built from unbiased likelihood estimators. Under consistency and integrability conditions, the expected log estimator converges to the true log marginal likelihood.

  • Definition: The Monte Carlo objective is defined as the expectation of the logarithm of an unbiased positive likelihood estimator.The estimator satisfies E[ˆp_N(x)] = p(x).
  • Lower-bound property: Concavity of the logarithm and Jensen’s inequality show that the Monte Carlo objective lower-bounds the log marginal likelihood.This is the foundational lower-bound property used by the proof.
  • Convergence: Strong consistency of the estimator implies almost-sure convergence to the marginal likelihood as the particle count grows.The stated condition is ˆp_N(x) → p(x) almost surely as N →∞.
  • Convergence: Uniform integrability of the log estimators allows convergence of their expectations to the expected log marginal likelihood.The proof invokes continuity of log and Vitali’s convergence theorem.
  • Bias analysis: The bound’s bias is expressed as log p(x) − L_N(x,p) = −E[log(1 + ∆)].The proof then Taylor-expands log(1 + ∆) around zero and bounds the resulting terms.

Controlling the first inverse moment.

The section gives a sufficient near-zero-tail condition for controlling the inverse moment of particle-filter estimators, then uses unbiasedness to support FIVO optimization and gradient derivations.

  • Inverse-moment condition: A bound on P(w_i < w) near zero suffices to bound the inverse moment E[ˆp_N(x)−1] for averages of positive i.i.d. weights.The condition is P(w_i < w) ≤ Cw^(1+ϵ) for w ∈ [0,M), yielding a finite bound involving C, M, and ϵ.
  • Estimator unbiasedness: The particle filter estimator ˆp_N(x_1:T) is unbiased for the marginal likelihood p(x_1:T), established by representing it as an importance weight on an extended space.The proof uses conditional SMC to define a target density whose importance-sampling ratio matches the particle-filter estimator.
  • Gradient estimation: FIVO gradients are formulated from particle-filter simulation, covering reparameterized and non-reparameterized sampling as well as adaptive resampling decisions.In practice, the work considers reparameterized proposals and drops gradient terms arising from resampling.
  • Gradient estimation: Adaptive resampling partitions the simulation domain into differentiable regions separated by discontinuities, producing additional boundary terms in the full gradient.These surface-integral terms are dropped in practice.

Proof of Proposition 2.

The section establishes when FIVO is exact and describes how the bounds are compared experimentally. It reports that FIVO often appears tighter in practice, while evaluations use different bound-reporting conventions.

  • Proof of Proposition 2: Under the posterior proposal condition p(z_1:t−1|x_1:t)=p(z_1:t−1|x_1:t−1), FIVO equals log p(x_1:T) when q(z_t|z_1:t−1,x_1:t) matches the filtering posterior.The proof proceeds by induction, showing equal particle weights and an estimator equal to the marginal likelihood.
  • Evaluation and Comparison of Bounds: FIVO is not provably tighter than ELBO or IWAE, but experiments suggest it tends to be tighter, with same-model values generally ordered FIVO, IWAE, then ELBO.For FIVO-trained models, ELBO and IWAE evaluations can diverge, whereas the FIVO bound exceeds corresponding FIVO evaluations on the other models.
  • Evaluation and Comparison of Bounds: Evaluations report the maximum of ELBO, IWAE_128, and FIVO_128 for ELBO- and IWAE-trained models, but only FIVO_128 for FIVO-trained models.This convention was chosen as the strongest comparison to existing bounds.
  • Evaluation and Comparison of Bounds: TIMIT log-likelihoods are reported relative to an ELBO baseline because standardized-data likelihoods differ from unstandardized values by a constant offset.Absolute standardized-data values are also reported in Tables 3–5 for comparison with other papers.
Loading 1705.09279v3…