PaperlayerView exact revision on arXiv

Source-linked AI summary

Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images

Rewon Child

arXiv:2011.10650v2cs.LGcs.CV

TL;DR

Autoregressive models had historically led image likelihood benchmarks, raising whether their inductive bias was superior to that of VAEs. This paper formalizes how sufficiently deep hierarchical VAEs can represent autoregressive models, introduces a scalable very deep architecture, and finds higher likelihoods with greater efficiency across natural-image benchmarks.

  • Problem

    Autoregressive models historically achieved higher image likelihoods than VAEs, motivating whether sufficiently improved VAEs could match or outperform them.

  • Method

    The paper formalizes the equivalence between autoregressive models and sufficiently deep hierarchical VAEs, then introduces an architecture scaling beyond previously explored stochastic depths.

  • Results

    Very deep VAEs outperform PixelCNN models on all benchmarks, using fewer parameters, sampling thousands of times faster, and scaling to larger images.

  • Takeaways & Limitations

    The results support stochastic depth as an explanation for the historical likelihood gap and indicate that efficient hierarchical representations can make VAEs both effective and efficient.

  • Takeaways & Limitations

    The study restricts VAEs to an independent pθ(x|z), excluding autoregressive conditional decoders because their generation is slow.

Abstract

from arXiv · show

We present a hierarchical VAE that, for the first time, generates samples quickly while outperforming the PixelCNN in log-likelihood on all natural image benchmarks. We begin by observing that, in theory, VAEs can actually represent autoregressive models, as well as faster, better models if they exist, when made sufficiently deep. Despite this, autoregressive models have historically outperformed VAEs in log-likelihood. We test if insufficient depth explains why by scaling a VAE to greater stochastic depth than previously explored and evaluating it CIFAR-10, ImageNet, and FFHQ. In comparison to the PixelCNN, these very deep VAEs achieve higher likelihoods, use fewer parameters, generate samples thousands of times faster, and are more easily applied to high-resolution images. Qualitative studies suggest this is because the VAE learns efficient hierarchical visual representations. We release our source code and models at https://github.com/openai/vdvae.

1 INTRODUCTION

The paper asks whether sufficiently deep VAEs can overcome autoregressive models’ likelihood advantage while retaining practical benefits. It formalizes why depth may enable this and tests a scalable architecture that outperforms PixelCNN models across image benchmarks.

  • Motivation: Autoregressive models historically achieved the highest log-likelihoods, despite modeling observed-variable dependencies rather than latent scene structure.VAEs can in principle learn simpler latent-variable models, but had not matched autoregressive likelihoods on key benchmarks.
  • Motivation: The central question is whether the autoregressive assumption is genuinely better for images or whether improved VAEs can outperform it.The answer matters because autoregressive models are large and compute-intensive, while latent-variable models may support faster synthesis and larger data with smaller architectures.
  • Theory: Hierarchical VAEs can at least match autoregressive models because autoregressive models are equivalent to VAEs with powerful priors and restricted approximate posteriors.If useful latent representations exist, hierarchical VAEs may instead achieve better modeling with fewer layers.
  • Contributions: The paper formalizes why greater stochastic depth may improve VAE performance and introduces an architecture scaling beyond 70 layers, compared with 30 previously explored.The theoretical justification allows depth up to the data dimension D, or potentially a much smaller value K.
  • Results: VAEs with greater depth outperform PixelCNN models on all benchmarks, while using fewer parameters, sampling thousands of times faster, and scaling to larger images.The authors also present evidence that efficient hierarchical image representations may underlie these advantages.
  • Resources: The authors release source code and trained models for the very deep VAE.The release is available at https://github.com/openai/vdvae.

2 PRELIMINARIES

This section introduces VAEs, independent-decoder generation, and hierarchical latent-variable structures. It then describes top-down VAEs, whose bottom-up inference and top-down latent processing provide the base architecture used here.

  • Learned generative structures: Figure 2 contrasts an identity-encoder VAE that reproduces autoregressive modeling with a learned encoder that can expose conditionally independent latent variables for parallel sampling.The right-hand hierarchy illustrates why learned latent structure may reduce sampling time.
  • Variational autoencoders: VAEs combine a generator pθ(x|z), a prior pθ(z), and an approximate posterior qφ(z|x), trained by maximizing the evidence lower bound.The networks are optimized end-to-end using backpropagation and the reparameterization trick.
  • Variational autoencoders: The paper studies VAEs with independent pθ(x|z), so each observed variable is generated without conditioning on other observed variables.This avoids generation time increasing linearly with data dimensionality but requires a rich latent distribution.
  • Hierarchical VAEs: Hierarchical VAEs use several stochastic latent-variable layers, emitting groups z0, z1, ..., zN that are conditionally dependent.For images, these groups commonly form feature maps across resolutions, from low-resolution top variables to high-resolution bottom variables.
  • Top-down VAEs: A top-down VAE generates latent variables in the same order for both the prior and approximate posterior.The architecture is presented as a conditioning structure for hierarchical latent variables.
  • Top-down VAEs: Inference first performs a deterministic bottom-up pass, then processes latent groups from top to bottom using shared feedforward features.The shared features support the approximate posterior, prior, and reconstruction network pθ(x|z).
  • Architecture: Figure 3 depicts the top-down architecture with residual blocks, GELU-preceded convolutions, diagonal Gaussian distributions, and pooling or nearest-neighbor unpooling.Latents are sampled from qφ during training and pθ during sampling.

3 WHY DEPTH MATTERS FOR HIERARCHICAL VAES

The paper argues that sufficiently deep hierarchical VAEs can generalize autoregressive models and potentially learn shorter, more efficient latent generation procedures.

  • 3 WHY DEPTH MATTERS FOR HIERARCHICAL VAES: Hierarchical VAEs can generalize autoregressive models when their stochastic depth reaches the data dimension, while shorter depths can learn lower-dimensional latent densities.The paper states that N-layer VAEs generalize autoregressive models at N=D and are universal approximators of N-dimensional latent densities.
  • 3 WHY DEPTH MATTERS FOR HIERARCHICAL VAES: A VAE may require up to D stochastic layers when images truly require D sequential generation steps.For 32x32 images, D equals 3072.
  • 3 WHY DEPTH MATTERS FOR HIERARCHICAL VAES: If K<D latent variables efficiently compress the data, a K-layer VAE can learn their posterior and prior distributions.This provides a shorter generative procedure when an effective K-dimensional representation exists.
  • 3 WHY DEPTH MATTERS FOR HIERARCHICAL VAES: Conditional independence lets a hierarchical VAE generate groups of latent variables in parallel within one layer.The paper identifies parallel generation and low-dimensional representations as two sources of shorter generative paths.
  • 3 WHY DEPTH MATTERS FOR HIERARCHICAL VAES: Earlier hierarchical VAE work explored substantially fewer stochastic layers, ranging from 3 layers to at most 30 layers.The paper motivates deeper exploration because images contain thousands of observed variables.

4 AN ARCHITECTURE FOR VERY DEEP VAES

The architecture is a minimal hierarchical VAE redesigned for substantially greater stochastic depth, with residual scaling, nearest-neighbor upsampling, and gradient skipping supporting stable training.

  • 4 AN ARCHITECTURE FOR VERY DEEP VAES: The very deep VAE uses convolutions, nonlinearities, and Gaussian stochastic layers, and is defined by having more layers than prior implementations explored.The authors do not define “very deep” by a fixed layer count.
  • 4 AN ARCHITECTURE FOR VERY DEEP VAES: Residual scaling improves stability and performance with many layers by scaling each bottleneck block’s final convolution according to network depth.The scaling is applied by a factor involving N, the depth.
  • 4 AN ARCHITECTURE FOR VERY DEEP VAES: Nearest-neighbor upsampling prevents low-resolution layers from being ignored and removes the need for free-bits and KL-warming-up terms.The paper reports no posterior collapse in networks trained with nearest-neighbor interpolation.
  • 4 AN ARCHITECTURE FOR VERY DEEP VAES: Skipping updates with gradient norms above a threshold almost entirely eliminates divergence while affecting fewer than 0.01% of updates.Very large gradient norms are identified as a source of VAE optimization difficulty.
  • 4 AN ARCHITECTURE FOR VERY DEEP VAES: On ImageNet-32, increasing stochastic depth up to 48 layers improves loss, and layers at higher resolutions provide greater benefit.The comparison holds total layers at 48 when varying their resolution distribution.

5 EXPERIMENTS

Experiments show that stochastic depth improves likelihood independently of parameter count, while very deep VAEs outperform autoregressive baselines, sample rapidly, and learn hierarchical image representations.

  • 5 EXPERIMENTS: Stochastic depth correlates clearly with performance through 48 layers, even when parameter count is unchanged.The experiment reduces stochastic depth by grouping consecutive layers without changing the number of parameters.
  • 5 EXPERIMENTS: 2.87 bits per dim is achieved on CIFAR-10 with 45 stochastic layers and 39M parameters.On ImageNet-32 and ImageNet-64, models with 78 and 75 stochastic layers achieve likelihoods of 3.80 and 3.52 using approximately 120M parameters.
  • 5 EXPERIMENTS: Across all tasks, the results outperform GatedPixelCNN, PixelCNN++, and nonautoregressive models with similar or fewer parameters.The authors attribute the gap to stochastic depth rather than other factors.
  • 5 EXPERIMENTS: On FFHQ-256, fewer than 1% of latent variables at low resolution almost wholly determine global image features.Higher-resolution variables appear spatially independent and can therefore be emitted in parallel.
  • 5 EXPERIMENTS: Very deep VAEs use roughly 30% fewer parameters than PixelCNN models across tested datasets.The paper offers fewer long-range dependencies or a simpler learned generation procedure as possible explanations.
  • 5 EXPERIMENTS: The VAE scales from 32x32 to 1024x1024 images with similar training resources and few hyperparameter changes.Both resolutions were trained for 1.5M steps using 32 GPUs for 2.5 weeks.

6 RELATED WORK AND DISCUSSION

The discussion positions depth as a complementary, overlooked factor in hierarchical VAEs and relates the approach to autoregressive models, diffusion models, and other latent-variable methods.

  • 6 RELATED WORK AND DISCUSSION: The authors emphasize depth as a key overlooked factor rather than claiming a universally superior VAE method.They state that many components of prior hierarchical VAE work could be combined with their approach.
  • 6 RELATED WORK AND DISCUSSION: Table 2 compares benchmark likelihood, parameter count, stochastic depth, and network evaluations per sample across VAE and autoregressive models.The table defines D as data dimensionality and marks estimated parameter counts with an asterisk.
  • 6 RELATED WORK AND DISCUSSION: Diffusion models are described as deep VAEs with analytical posteriors, while the presented VAEs use a single network evaluation per sample.The comparison is framed as a sampling-efficiency difference in the current methods discussed.
  • 6 RELATED WORK AND DISCUSSION: Inverse autoregressive flows are closely related, but the paper discusses their differences from hierarchical models separately.The authors also address work that may appear to contradict their findings.

7 CONCLUSION

The paper argues that deeper VAEs can outperform PixelCNN-based autoregressive models in likelihood while also being more efficient.

  • Deeper VAEs outperform all PixelCNN-based autoregressive models in likelihood.
  • The introduced architecture is more efficient than the PixelCNN-based alternatives.
  • The authors encourage further improvement of VAEs and latent variable models.

A.1 ABLATIONS OF ARCHITECTURAL COMPONENTS

The ablations identify architectural and optimization choices that improve training stability and reduce posterior collapse in very deep VAEs.

  • Upsampling layers and residual connections affect posterior collapse, and these architectural choices may remove the need for free bits or KL warmups.
  • Residual initialization leads to smoother and better training of very deep VAEs.Without it, very deep VAEs encounter many unstable updates and higher losses.

A.2 PROPOSITION 1: N-LAYER VAES GENERALIZE AUTOREGRESSIVE MODELS WHEN N IS THE DATA DIMENSION

Proposition 1 shows that an autoregressive model over N observed variables can be represented by an N-layer VAE with a restricted posterior and identity generator.

  • An autoregressive model with any ordering over x ∈R^N is equivalent to an N-layer VAE.
  • The equivalent VAE uses an approximate posterior that outputs observed variables in the given order and a generator implementing the identity function.
  • The construction sets q(z_i = x_i|z<i, x) = 1 and p(x_i = z_i|z) = 1, yielding p(z|x) = q(z|x).
  • Under this construction, the ELBO becomes the autoregressive log-likelihood over observed variables.

A.3 PROPOSITION 2: N-LAYER VAES ARE UNIVERSAL APPROXIMATORS OF N-DIMENSIONAL LATENT DENSITIES

The paper characterizes deep hierarchical VAEs as expressive latent-density models, while noting practical limits from Gaussian elementwise distributions and sampling assumptions.

  • Hierarchical VAEs learn depthwise autoregressive flows and can express any N-dimensional latent density given sufficient capacity under stated conditions.
  • The universality argument applies to the prior, approximate posterior, and posterior distributions because each can be represented by an autoregressive flow.
  • Using Gaussian elementwise distributions restricts the densities the VAE can express in practice.The paper identifies more expressive elementwise densities as future work.
  • For high-dimensional images, Gibbs sampling from the last stochastic layer may take unacceptably long to converge, whereas a latent hierarchy supports efficient and tractable sampling.
  • Generative models trained on datasets such as ImageNet, CIFAR-10, and FFHQ may propagate or reinforce harmful dataset biases if deployed without careful consideration.
Loading 2011.10650v2…