PaperlayerView exact revision on arXiv

Source-linked AI summary

Variational Lossy Autoencoder

Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, Pieter Abbeel

arXiv:1611.02731v2cs.LGstat.ML

TL;DR

Representation learning seeks global structure while discarding irrelevant detail, but VAEs do not automatically produce such codes. The paper combines VAEs with autoregressive priors and decoders to control latent information and improve density estimation, while noting that the resulting lossy code depends on decoder constraints and dataset statistics.

  • Problem

    VAEs do not always autoencode, and representation learning from generative models is ill-posed without assumptions controlling which information enters latent variables.

  • Method

    The paper designs a Variational Lossy Autoencoder by combining VAE latent structure with autoregressive priors and decoders to control latent information.

  • Results

    The proposed model is reported to achieve strong density-estimation performance, including state-of-the-art log-likelihood results on MNIST, OMNIGLOT, and Caltech-101.

  • Takeaways & Limitations

    By designing decoder constraints, the global latent code can discard irrelevant information such as image texture and represent data in a lossy fashion.

  • Takeaways & Limitations

    The lossy code does not always capture the desired global information and depends on decoder constraints and the statistics relevant to each dataset or task.

Abstract

from arXiv · show

Representation learning seeks to expose certain aspects of observed data in a learned representation that's amenable to downstream tasks like classification. For instance, a good representation for 2D images might be one that describes only global structure and discards information about detailed texture. In this paper, we present a simple but principled method to learn such global representations by combining Variational Autoencoder (VAE) with neural autoregressive models such as RNN, MADE and PixelRNN/CNN. Our proposed VAE model allows us to have control over what the global latent code can learn and , by designing the architecture accordingly, we can force the global latent code to discard irrelevant information such as texture in 2D images, and hence the VAE only "autoencodes" data in a lossy fashion. In addition, by leveraging autoregressive models as both prior distribution $p(z)$ and decoding distribution $p(x|z)$, we can greatly improve generative modeling performance of VAEs, achieving new state-of-the-art results on MNIST, OMNIGLOT and Caltech-101 Silhouettes density estimation tasks.

1 INTRODUCTION

The paper targets representations that preserve global image structure while discarding local texture, but argues that generative modeling alone does not uniquely determine such representations. It therefore asks whether a model can combine strong density estimation with hierarchical representation learning.

  • Representation learning seeks disentangled factors that support understanding, classification, and other downstream tasks.For images, the desired representation may capture object identity and style while ignoring textures and white-noise patterns.
  • Probabilistic latent-variable models can encode the same observed-data density with different latent structures, making representation learning ill-posed without additional assumptions.Architectural and modeling choices strongly influence the learned representation.
  • The paper asks whether one model can provide both powerful density estimation and the hierarchical structure needed for representation learning.
  • Combining a VAE with an autoregressive model is proposed as a way to exploit both latent-variable structure and powerful recurrence.Earlier hybrid attempts often allowed the autoregressive component to explain the data while leaving latent variables unused.

2 VAES DO NOT AUTOENCODE IN GENERAL

This section explains why VAEs need not autoencode: sufficiently powerful autoregressive decoders can model the data without latent information, making latent usage inefficient. A Bits-Back Coding view identifies this information preference and connects it to posterior mismatch and coding cost.

  • VAEs are not guaranteed to reconstruct datapoints closely, and their latent codes may be unused unless the decoder is weakened.The section frames this as an essential issue for understanding when a VAE actually autoencodes.
  • Technical background: A VAE introduces q(z|x) to optimize a variational lower bound when flexible latent-variable models make marginal likelihood intractable.For continuous z, reparameterization enables efficient optimization.
  • Technical background: The VAE loss can be interpreted as a regularized autoencoder, with q(z|x) as encoder and p(x|z) as decoder.Its reconstruction term and KL term correspond to negative reconstruction error and regularization.
  • Information preference: Autoregressive decoders can represent any joint distribution in theory, even without dependence on z, because they factorize distributions through prior variables.RNNs are described as universal function approximators in this setting.
  • Bits-Back Coding: Bits-Back Coding measures VAE code length by subtracting information transmitted through q(z|x), making expected code length equal to negative variational lower bound.The coding perspective also shows that posterior mismatch adds at least DKL(q(z|x)||p(z|x)) nats.
  • Information preference: If p(x|z) models pdata(x) without z, the latent code is ignored and q(z|x) can match p(z), avoiding posterior-mismatch cost.This explains why expressive RNN decoders often regress to unconditional autoregressive models.

3 VARIATIONAL LOSSY AUTOENCODER

The paper uses autoregressive structure to control which information enters a VAE’s latent representation and to improve coding efficiency. It constrains decoder receptive fields for lossy representations and uses autoregressive-flow priors to reduce posterior mismatch without added training cost.

  • 3 VARIATIONAL LOSSY AUTOENCODER: Autoregressive decoders can explicitly control information placement by constraining their receptive fields or factorization.The decoder can model information intended to be discarded while forcing desired information through the latent code.
  • 3.1 LOSSY CODE VIA EXPLICIT INFORMATION PLACEMENT: Small local receptive fields tend to model image texture, while long-range structure such as object shape is communicated through the latent code.This design targets global representations that discard detailed texture.
  • 3.1 LOSSY CODE VIA EXPLICIT INFORMATION PLACEMENT: The information-preference analysis is asymptotic: it applies when the variational lower bound can be optimized well, so KL annealing and free bits remain useful.These optimization techniques are still used in the paper’s experiments.
  • 3.2 LEARNED PRIOR WITH AUTOREGRESSIVE FLOW: Autoregressive-flow priors reduce Bits-Back coding inefficiency by making the learned prior more expressive and improving its match to the posterior.The paper frames this as equivalent to using an inverse autoregressive-flow approximate posterior under a reparameterized latent code.
  • 3.2 LEARNED PRIOR WITH AUTOREGRESSIVE FLOW: An autoregressive-flow prior has the same expected training cost as an inverse autoregressive-flow posterior while providing a deeper decoder-path generative model.The equivalence holds under the expectation over z sampled from q(z|x).

4 EXPERIMENTS

Experiments show that VLAE can learn lossy representations retaining global structure while regenerating local statistics, and can improve density estimation across binary and natural-image benchmarks. Its effectiveness depends on designing the decoder and latent code to control which information is preserved.

  • Experimental setup: VLAE uses AF priors and small-receptive-field PixelCNN decoders to separate global latent information from local image statistics.The evaluation covers binary image datasets with a unified architecture and tests whether autoregressive priors and decoders improve density estimation.
  • Lossy compression: 13.3 nats, or 19.2 bits, are used on average to encode one statically binarized MNIST image, versus 37.3 bits for an otherwise identical VAE with factorized decoding.The result indicates that VLAE learns a lossier compression than a VAE with a regular factorized conditional distribution.
  • Lossy compression: VLAE decompressions preserve global image structure while changing local statistics such as binary masks and stroke width.The decompressed images are not exact reconstructions; they regenerate likely local statistics instead of preserving the original ones.
  • Density estimation: AF priors improve over equivalent IAF posteriors, reducing statically binarized MNIST train NLL by 0.8 nat and test NLL by 0.6 nat when an autoregressive decoder is used.A VLAE with AF prior and PixelCNN conditional also achieves new state-of-the-art results on statically binarized MNIST.
  • Density estimation: With hyperparameters tuned on statically binarized MNIST, VLAE significantly exceeds prior state-of-the-art results on dynamically binarized MNIST and Caltech-101 Silhouettes and ties the best prior OMNIGLOT result.On CIFAR10, DenseNet VLAE attains state-of-the-art performance among variational latent-variable models and slightly trails PixelCNN++.
  • Lossy compression: Larger PixelCNN receptive fields make the decoder capture more structural information, leaving the latent code with progressively rougher shape information.A smaller receptive field preserves more detailed shape information in the lossy code, whereas a larger one retains only rough shape.
  • Natural images: CIFAR10: Changing the decoder’s receptive field can alter which information the latent code preserves: a grayscale receptive field retains color information while preserving roughly the same structural content.The resulting decompressions generally produce more recognizable objects, illustrating how decoder design can specify what the lossy code encodes.

5 RELATED WORK

The paper situates VLAE among efforts combining latent-variable models with expressive autoregressive components, while emphasizing its distinct control over lossy representations and autoregressive latent modeling.

  • VLAE combines variational autoencoders with neural autoregressive models, including PixelCNN-based architectures for images.The related work discusses recurrent, convolutional, and flow-based autoregressive components in latent-variable models.
  • Earlier hybrid models often allowed expressive autoregressive decoders to explain the data without using latent variables.Dropout and other decoder-weakening strategies were proposed to encourage latent-code usage.
  • Prior work explored autoregressive priors for discrete and continuous latent variables, but some architectures required variational inference for intractable autoregressive data likelihoods.VLAE instead uses autoregressive flows in its modeling framework.
  • VLAE provides a complementary approach to lossy compression by using autoregressive models to explicitly control which information is discarded.This differs from hierarchical latent-variable compression approaches such as ConvDRAW.

6 CONCLUSION

The paper analyzes when VAEs use latent codes and designs VLAE as a controllable lossy compressor. Its autoregressive components improve modeling but make generation slower, while extensions beyond images remain future work.

  • The paper analyzes conditions for latent-code usage and uses them to design a VAE that compresses observed data lossily.This frames VLAE as a model for controllable representation learning rather than exact reconstruction alone.
  • VLAE combines controllable representation learning with improved density-estimation performance.The conclusion characterizes these as two complementary modeling improvements.
  • Autoregressive generation makes VLAE slower than VAEs with simple priors and decoders because generation is sequential.This is the stated cost of the model's autoregressive design.
  • The paper proposes extending lossy-code learning to temporal data such as audio and video and using task-focused representations for semi-supervised learning.These are identified as future research directions rather than evaluated results.

A DETAILED EXPERIMENT SETUP FOR BINARY IMAGES

The binary-image experiments use a ResNet VAE with a conditional PixelCNN decoder and an autoregressive-flow prior, trained with stabilization techniques including free bits and Polyak averaging.

  • The decoder transforms the ResNet VAE output and original binary image into a 28x28x5 feature map consumed by a masked PixelCNN.The VAE decoder outputs 28x28x4 features, which are concatenated channel-wise with the original image channel.
  • The latent code has dimension 64, while the autoregressive prior uses four mean-only MADE flows with three layers and 640 hidden units per flow.Mean-only flows were selected for numerical stability.
  • Training uses Adamax with learning rate 0.002, 0.01 nats/data-dim free bits, and Polyak averaging with α = 0.998.Free bits were effective for preventing early neglect of the latent code.
  • All experiments are implemented using TensorFlow.

B ADDITIONAL EXPERIMENT SETUP FOR CIFAR10

The CIFAR10 setup represents latents as spatial feature maps, uses PixelCNN-based autoregressive flows and conditional PixelCNN++ decoding, and reports the best density-estimation result with a 7x4 receptive-field configuration.

  • Latent codes use 16 feature maps of size 8x8, with a factorized Gaussian prior transformed by six PixelCNN-based autoregressive flows.Each flow has two hidden layers and 128 feature maps, with stochastic-unit ordering reversed between alternating flows.
  • The ResNet and DenseNet VLAE variants use downsampling convolutional encoder structures with symmetric or similar decoders.The ResNet encoder has staged residual blocks and stride-2 convolutions; DenseNet replaces some residual blocks with three-step DenseNet blocks.
  • Conditional PixelCNN++ serves as the decoder, using a channel-autoregressive variant to retain capacity with small receptive fields.The setup includes multiple blocks of 64 feature maps and separate horizontal and vertical convolutional stacks.
  • The best density-estimation result is obtained with the 7x4 receptive-field experiment.The grayscale experiment converts RGB images using (0.299∗R)+(0.587G)+(0.114B).

C SOFT FREE BITS

Soft free bits retains a target of at least λ nats while changing the optimization objective gradually rather than using the sharp transition of standard free bits.

  • “Free bits” encourages K groups of stochastic units to be used through a surrogate objective.The technique requires choosing a minimum information level, λ, for the stochastic units.
  • Choosing λ is easier than setting a fixed KL annealing schedule because the minimum number of bits or nats is usually easy to determine.
  • KL annealing changes smoothly toward the variational lower bound, whereas free bits has a sharp transition at its boundary.
  • The method changes γ online: it increases when KL is more than λ and decreases when KL is below λ to encourage information flow.The authors experimented with thresholds from 3% to 30% and tended to use 5%.
  • Increasing or decreasing γ by 10% was sufficient without further tuning.

D AUTOREGRESSIVE DECODER WITHOUT AUTOREGRESSIVE PRIOR

The experiment compares an unconditional PixelCNN, a VAE with Gaussian latents, and a VAE with autoregressive-flow latents, all using an autoregressive decoder. Each modification improves density estimation, while autoregressive latents transmit more information.

  • The ablation compares three configurations with the same model: unconditional small-receptive-field PixelCNN, Gaussian-latent VAE, and autoregressive-flow-latent VAE.
  • Table 1 reports an ablation on dynamically binarized MNIST.
  • Each successive modification improves density estimation performance.
  • Autoregressive latent codes transmit more information, as indicated by the difference in E[DKL(q(z|x)||p(z))].
Loading 1611.02731v2…