PaperlayerView exact revision on arXiv

Source-linked AI summary

BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling

Lars Maaløe, Marco Fraccaro, Valentin Liévin, Ole Winther

arXiv:1902.02102v3stat.MLcs.CVcs.LG

TL;DR

VAEs have lagged behind autoregressive and flow-based models in likelihood and generated-image quality, motivating more expressive latent-variable models. BIVA addresses this with skip-connected generation and bidirectional stochastic inference across a deep hierarchy, improving benchmark performance and supporting sharp images, anomaly detection, and semi-supervised classification.

  • Problem

    VAEs offer latent representations but have commonly shown blurry generations and poor likelihood performance, while prior hierarchical and covariance-structured models remained inferior to autoregressive counterparts.

  • Method

    BIVA combines a deep hierarchy of stochastic variables with skip-connections in the generative model and bottom-up and top-down stochastic paths in the inference network.

  • Results

    BIVA improves over previously introduced probabilistic latent variable and flow-based models, generates sharp and coherent natural images, and improves semi-supervised classification benchmarks.

  • Takeaways & Limitations

    BIVA's hierarchical latent representations support anomaly detection by extracting high-level semantic features, while its performance reaches or improves on reported state-of-the-art results within the evaluated tasks.

Abstract

from arXiv · show

With the introduction of the variational autoencoder (VAE), probabilistic latent variable models have received renewed attention as powerful generative models. However, their performance in terms of test likelihood and quality of generated samples has been surpassed by autoregressive models without stochastic units. Furthermore, flow-based models have recently been shown to be an attractive alternative that scales well to high-dimensional data. In this paper we close the performance gap by constructing VAE models that can effectively utilize a deep hierarchy of stochastic variables and model complex covariance structures. We introduce the Bidirectional-Inference Variational Autoencoder (BIVA), characterized by a skip-connected generative model and an inference network formed by a bidirectional stochastic inference path. We show that BIVA reaches state-of-the-art test likelihoods, generates sharp and coherent natural images, and uses the hierarchy of latent variables to capture different aspects of the data distribution. We observe that BIVA, in contrast to recent results, can be used for anomaly detection. We attribute this to the hierarchy of latent variables which is able to extract high-level semantic features. Finally, we extend BIVA to semi-supervised classification tasks and show that it performs comparably to state-of-the-art results by generative adversarial networks.

1 Introduction

Explicit density models offer tractable or approximate likelihoods, but VAEs have lagged behind autoregressive and flow-based models in likelihood and image quality. BIVA addresses this gap with a deep stochastic hierarchy, skip-connected generation, and bidirectional inference.

  • Explicit density models include autoregressive, flow-based, and probabilistic latent variable models, with tractable or approximate likelihood estimation.
  • Autoregressive models achieve strong likelihoods but generate slowly as input-distribution complexity grows, while flow-based models generate compelling images yet do not outperform autoregressive models from the learned distribution.
  • VAEs provide posterior-based latent representations but have commonly suffered from blurry natural-image generations and poor likelihood performance.
  • Prior efforts combining expressive covariance structures and deeper latent hierarchies improved latent models but remained inferior to autoregressive counterparts.
  • BIVA combines skip-connections, a deep stochastic hierarchy, and bidirectional stochastic inference to improve likelihoods, generated samples, anomaly detection, and semi-supervised classification.

2 Variational Autoencoders

VAEs use hierarchical stochastic latent variables and approximate their intractable posterior with a variational distribution. Standard deep hierarchies can suffer variable collapse, motivating inference and generative pathways that preserve data-dependent information.

  • A hierarchical VAE models observed x through stochastic latent variables z and optimizes its parameters by maximizing the evidence lower bound.
  • The standard variational posterior factorizes bottom-up, conditioning each latent variable on the variable below in the hierarchy.
  • Deep VAE hierarchies are more expressive, but their upper stochastic variables can collapse into the prior.
  • LVAE adds a stochastic top-down inference path so every latent variable has a deterministic dependency on x, reducing noisy inputs and making collapse less likely.
  • BIVA extends LVAE with a deterministic top-down generative path and a bottom-up stochastic inference path to address collapse in very deep hierarchies.

3 Bidirectional-Inference Variational Autoencoder

BIVA extends hierarchical VAEs with skip-connected generation and bidirectional stochastic inference, aiming to preserve expressive latent structure while improving posterior flexibility. Its hierarchy supports high-level semantic representations that are used for anomaly detection.

  • Deterministic top-down path: The deterministic top-down path summarizes information from higher stochastic layers and propagates it to lower generative layers.At each hierarchy level, a deterministic variable receives the next higher latent variable and carries information downward.
  • Bidirectional inference: The bidirectional inference network computes posterior approximations sequentially through bottom-up and top-down stochastic paths.Bottom-up variables depend on the data and lower bottom-up variables, while top-down variables also incorporate higher-level variables and shared generative parameters.
  • Model architecture: BIVA extends LVAE with a deterministic top-down path and a bidirectional inference network.The generative path uses skip-connections, while inference combines bottom-up and top-down stochastic paths.
  • Bidirectional inference: Stochastic bottom-up variables make the marginal posterior over higher top-down variables non-factorized, enabling complex covariance structure.This covariance structure is described as fundamental for extracting high-level semantic features from the data distribution.
  • Anomaly detection: BIVA’s anomaly-detection objective emphasizes higher latent layers instead of relying primarily on lower-level statistics.The alternative bound uses the prior for the first k layers and the variational approximation for the remaining higher layers; larger k forces evaluation toward high-level semantics but generally weakens likelihood approximation.

4 Experiments

Experiments show that BIVA's skip-connected generative model and bidirectional stochastic inference improve latent-variable utilization and likelihood across image modeling tasks. BIVA also produces sharp CelebA samples, supports semi-supervised classification, and improves anomaly detection when higher-level latent information is emphasized.

  • Ablation Study: BIVA adds skip-connected generative dependencies and a bottom-up stochastic inference path to the LVAE architecture.The ablation setup compares LVAE, LVAE+, and BIVA using matched architectural variants.
  • Ablation Study: A roughly 0.2 bits/dim ELBO gain comes from skip connections, while BIVA improves ELBO by more than 0.3 bits/dim over LVAE+.A deeper LVAE+ hierarchy performs worse than the shallower LVAE+ despite having significantly more parameters.
  • Binary Images: BIVA achieves state-of-the-art results on statically binarized MNIST and reports L1e3 = 78.41 on dynamically binarized MNIST.On dynamically binarized OMNIGLOT, it reports L1e3 = 91.34.
  • Binary Images: BIVA achieves comparable semi-supervised MNIST classification performance to recent state-of-the-art generative adversarial networks using 100 labeled samples.The evaluation averages test error over five runs with different randomly chosen, evenly distributed labeled subsets.
  • Natural Images: On natural images, BIVA outperforms other state-of-the-art non-autoregressive models on CIFAR-10 and outperforms flow-based models on 32x32 ImageNet.It performs slightly worse than state-of-the-art autoregressive models on CIFAR-10, while the ImageNet gap to autoregressive models remains large.
  • Natural Images: BIVA generates visually sharp CelebA samples, and higher values of k reverse the standard ELBO's incorrect anomaly ranking between CIFAR-10 and SVHN.The authors interpret this reversal as evidence that higher-level semantics help differentiate the distributions.

5 Conclusion

BIVA improves on prior probabilistic latent-variable and flow-based models, producing sharp, coherent natural images, stronger semi-supervised classification results, and anomaly detection through high-level latent semantics.

  • BIVA significantly improves performance over previously introduced probabilistic latent variable models and flow-based models.
  • BIVA generates natural images that are both sharp and coherent.
  • BIVA improves semi-supervised classification benchmarks and supports anomaly detection using extracted high-level semantics.

A Deep Learning and Variational Inference

VAEs model data with hierarchical stochastic latent variables and approximate their intractable posterior using an inference network. Deep hierarchies increase expressiveness but can suffer latent-variable collapse, motivating richer inference structures such as BIVA’s bidirectional paths and skip connections.

  • A VAE models an observed variable x through a hierarchy of stochastic latent variables z = z1, ..., zL.
  • The VAE posterior is typically intractable, so variational inference introduces an encoder qφ(z|x) and optimizes the ELBO.
  • Standard VAE inference uses a bottom-up factorization in which each latent variable is conditioned on the variable below it, commonly with Gaussian factors.
  • A deep latent hierarchy increases expressiveness but may cause top stochastic variables to collapse into the prior when lower-level generative paths are sufficiently powerful.
  • BIVA combines a skip-connected generative model with bottom-up and top-down stochastic inference paths to use a deeper latent hierarchy.

B Detailed Model Description

BIVA uses a top-down generative process with skip connections and an inference model containing complementary bottom-up and top-down stochastic paths. The top-down inference path shares parameters with the generative model and combines bottom-up and top-down information.

  • The generative model proceeds top-down from zL through intermediate stochastic variables to x, with ResNet blocks between stochastic layers.
  • The generative process samples zL from N(0, I), iteratively generates lower latent variables, and expresses p(x|z) using z1 and d1 through a final ResNet block.
  • BIVA’s inference model has bottom-up and top-down paths: bottom-up units receive bottom-up information, while top-down units receive both directions.
  • The top-down inference path shares parameters with the generative model and uses ResNet blocks associated with each stochastic latent variable.
  • The top-down latent variables are inferred using bottom-up latents and the generative path to determine intermediate feature maps.

C Experimental Setup

The experiments use fixed BIVA architectures and optimization procedures across binary and natural image settings. Training uses Adamax, specified batch and GPU configurations, exponential moving averages, and architectures with progressively deeper hierarchies for larger images.

  • All experiments follow the detailed BIVA model description in Appendices B and F.
  • Models are trained with Adamax using batch-size 48, GPU-specific configurations, and exponential moving averages for evaluation.
  • Binary image architecture: Binary image experiments use BIVA with L = 6 stochastic layers and densely connected latent variables of dimensions 48, 40, 32, 24, 16, 8.
  • Natural image architecture (32x32): The 32x32 natural-image architecture uses L = 15 layers with convolutional latent variables containing 38 through 10 feature maps.
  • Natural image architecture (64x64): The 64x64 natural-image architecture uses L = 20 layers with convolutional networks transitioning from 64 to 128 kernels.

D Modeling Complex 2D Densities

This section evaluates BIVA on complex two-dimensional target densities, focusing on posterior approximation quality and optimization robustness. BIVA generally learns posteriors closer to the targets than baseline models, although optimization remains challenging.

  • Optimizing sufficiently expressive latent-variable models remains challenging despite their ability to represent complex non-Gaussian densities.The section identifies optimization difficulty as distinct from representational capacity.
  • BIVA is evaluated on target densities defined by potentials p(z) = e−U(z).The experiments use an isotropic Gaussian p(x) and model the target density through the top stochastic variable.
  • The study compares BIVA with VAE and LVAE models using matched stochastic-variable counts and a deliberately minimal architecture.All models use five stochastic layers, one-hidden-layer MLPs, and residual connections.
  • BIVA tends to learn posterior distributions closer to the target density than the baseline models.The comparison estimates KL(q(zL)||p(zL)) using 1e6 posterior samples, with each model trained 100 times per density.
  • BIVA often learns more complex posterior features, while baseline posteriors remain closer to the modes.LVAE can occasionally match the best BIVA solutions, but BIVA exhibits more robust optimization behavior.

E Initial Results on Text Generation Tasks

BIVA is tested for sentence modeling with autoregressive decoders, where latent-variable optimization is difficult because autoregressive components can cause posterior collapse. Replacing the LSTM inference model with BIVA activates the latent space more effectively, yielding higher measured KL than the compared VAE baselines.

  • Motivation: Autoregressive generative models can cause posterior collapse, making latent variables ignored during optimization.The experiment investigates BIVA without weakening the decoder using techniques such as word dropout.
  • Dataset: The BookCorpus experiment trains on 56M sentences and tests on 10,000 sentences with a 20,000-word vocabulary.Sentences are limited to 40 words and tokenized by whitespace.
  • Results: The text-generation results are reported on BookCorpus using one importance-weighted sample.The corresponding table compares test performance for sentences limited to 40 words.
  • Models: BIVA is coupled with an LSTM decoder and compared against an LSTM language model, RNN-VAE, Hybrid VAE, and non-autoregressive variants.The models use a convolutional architecture as input to the autoregressive decoder.
  • Results: Replacing the LSTM inference model with BIVA yields a higher measured KL than the RNN-VAE and Hybrid VAE baselines.BIVA without the autoregressive decoder is not competitive with an LSTM language model, but the BIVA inference model exploits the latent space more actively.

F Semi-Supervised Learning

For semi-supervised classification, BIVA augments its latent-variable model with a categorical class variable and a separate classification network. Training combines labeled and unlabeled-data objectives, including a classification loss for labeled examples.

  • Model: BIVA is extended for semi-supervised classification by introducing qφ(y|x,zBU<L) and a categorical dependency in the generative model.Here y denotes the class variable.
  • Inference model: The classification inference model uses another deterministic hierarchy and ends with a Softmax layer sized to the number of categories.The hierarchy is parameterized equivalently to the deterministic variables ˜di,1, ..., ˜di,M.
  • Generative model: The generative model conditions each stochastic latent variable and the observed variable x on the categorical variable y.The conditioning is also added along the top-down inference path.
  • Objective: Semi-supervised training uses separate ELBO formulations for labeled and unlabeled data.For unlabeled examples, the labels are marginalized.
  • Objective: A categorical cross-entropy classification term is added because the classification model is not included in the basic labeled-data ELBO.Its weight is controlled by the hyperparameter α.

G Additional Results

Additional experiments assess BIVA across binarized image datasets, CIFAR-10 training behavior, and CelebA generation. The results illustrate likelihood evaluation, hierarchical stochasticity, and layer-dependent control over generated image variation.

  • Likelihood results: Tables 10–13 report test log-likelihoods on dynamically binarized MNIST and OMNIGLOT, statically binarized Fashion MNIST, and ImageNet 32x32.The evaluations vary the number of importance-weighted samples; finetuned models use no free bits.
  • CIFAR-10: On CIFAR-10, the convergence plot compares LVAE and BIVA models using ELBO and reconstruction-loss trajectories.The discrepancy between these curves is attributed to the KL cost of stochastic latent variables.
  • CelebA generation: BIVA generates 64x64 CelebA samples with adjustable stochasticity across 20 stochastic latent layers.Latent variances are scaled from 0.1 to 1.0, while images in each row share Gaussian noise.
  • CelebA generation: On CelebA, increasing latent variance moves generations from low-variance samples near the mode toward samples from the full distribution.The reported test performance is L1 = 2.48 bits/dim.
  • Hierarchical representations: Fixing higher-level latent variables from an input image gives BIVA more freedom to augment that image as the fixed layer increases.The experiment varies fixed layers i = 12, 14, 16, 17, 18, and 19 while sampling lower variables from the prior.
  • CIFAR-10 generation: On CIFAR-10, BIVA achieves 3.08 bits/dim and produces coherent category-resembling structure, though samples are less sharp and coherent than PixelCNN++ at 2.92 bits/dim.The model uses 15 stochastic latent variables.
Loading 1902.02102v3…