PaperlayerView exact revision on arXiv

Source-linked AI summary

Glow: Generative Flow with Invertible 1x1 Convolutions

Diederik P. Kingma, Prafulla Dhariwal

arXiv:1807.03039v2stat.MLcs.AIcs.LG

TL;DR

Existing flow models relied on fixed channel permutations, limiting the design of generative flows for image modeling. Glow replaces this operation with an invertible 1×1 convolution and improves log-likelihood while efficiently synthesizing realistic high-resolution images.

  • Problem

    Existing flow architectures used fixed channel permutations, motivating a learned alternative for improving generative image modeling.

    Source: §3.2 · p. 5

  • Method

    Glow replaces fixed channel permutations with a learned invertible 1×1 convolution in a generative flow.

    Source: §3.2 · p. 5, §3.3 · p. 6

  • Results

    Glow achieves lower negative log-likelihood and faster convergence than comparable coupling models on CIFAR-10, with approximately 7% higher wall-clock time.

    Source: §5 · p. 7

  • Takeaways & Limitations

    Glow improves log-likelihood on standard image benchmarks and efficiently synthesizes realistic high-resolution natural images.

    Source: §7 · p. 10

    The computational-cost difference of invertible 1×1 convolutions becomes significant for large channel counts.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Flow-based generative models (Dinh et al., 2014) are conceptually attractive due to tractability of the exact log-likelihood, tractability of exact latent-variable inference, and parallelizability of both training and synthesis. In this paper we propose Glow, a simple type of generative flow using an invertible 1x1 convolution. Using our method we demonstrate a significant improvement in log-likelihood on standard benchmarks. Perhaps most strikingly, we demonstrate that a generative model optimized towards the plain log-likelihood objective is capable of efficient realistic-looking synthesis and manipulation of large images. The code for our model is available at https://github.com/openai/glow

1 Introduction

Generative modeling aims to capture dependencies in high-dimensional data, with potential applications including speech synthesis, text analysis, semi-supervised learning, and model-based control. The paper proposes Glow, a generative flow with new elements, and evaluates it quantitatively against previous flows and qualitatively on high-resolution datasets.

Motivation

  • Generative models may address data-efficiency and generalization by learning realistic world models and meaningful, task-independent features from unlabeled data.These capabilities could support planning before real-world interaction and make downstream solutions more robust and data efficient.Source: §1 · pp. 1–2
  • Generative modeling seeks to model all dependencies in high-dimensional input data through a full joint probability distribution.Accurate generative models could support diverse applications, including speech synthesis, text analysis, semi-supervised learning, and model-based control.Source: §1 · p. 2

Related methods

  • Likelihood-based generative models include autoregressive models, variational autoencoders, and flow-based models, each offering different tradeoffs in optimization and parallelizability.Autoregressive synthesis has limited parallelizability, VAEs optimize a lower bound on log-likelihood, and flow-based models were introduced by NICE and extended by RealNVP.Source: §1 · p. 2

Flow-based models

  • Flow-based generative models provide exact latent-variable inference and exact log-likelihood evaluation rather than approximations or lower bounds.They also support efficient parallelization of inference and synthesis, unlike the difficult-to-parallelize synthesis of autoregressive models.Source: §1 · p. 2

Contribution

  • The paper proposes Glow, a new generative flow with various new elements, and studies it quantitatively against previous flows and qualitatively on high-resolution datasets.The paper also notes potential flow-model benefits including useful latent spaces for downstream tasks and memory savings in reversible neural networks.Source: §1 · pp. 2–3

2 Background: Flow-based Generative Models

Flow-based generative models represent data through an invertible mapping to a tractable latent distribution, enabling exact latent inference and likelihood computation via the change-of-variables formula. Glow builds on this framework by composing invertible transformations whose log-determinants can be tractably computed.

Training objective

  • For continuous data, training minimizes a dequantized log-likelihood objective whose terms measure expected compression cost in nats or bits.The dequantized input is ˜x(i) = x(i) + u with u ∼ U(0, a), and optimization uses stochastic gradient descent with minibatches.Source: §2 · p. 3

Invertible latent-variable modeling

  • Flow-based models define x through an invertible transformation gθ of a latent variable z with a typically simple, tractable density.A common choice is a spherical multivariate Gaussian, pθ(z) = N(z; 0, I), and inference uses z = gθ^-1(x).Source: §2 · p. 3

Normalizing flows

  • The inverse mapping is composed as a sequence of invertible transformations, f = f1 ◦ f2 ◦ · · · ◦ fK, forming a normalizing flow.The intermediate representations are defined by h0 ≜ x and hK ≜ z.Source: §2 · p. 3

Likelihood computation

  • The model density is computed with change of variables by summing the log-determinants of each transformation’s Jacobian.The log-determinant measures the change in log-density between consecutive representations, and suitable transformations can make it surprisingly simple to compute.Source: §2 · p. 3

3 Proposed Generative Flow

Glow builds a multi-scale generative flow from repeated actnorm, invertible 1 × 1 convolution, and coupling-layer steps. It replaces fixed channel permutations with learned invertible 1 × 1 convolutions and introduces actnorm to support training with minibatch size 1 per processing unit.

Overall architecture

  • Each flow step applies actnorm, an invertible 1 × 1 convolution, and a coupling layer within a multi-scale architecture.The architecture has depth K and number of levels L.Source: §3 · p. 4

Actnorm

  • Actnorm applies a learned per-channel affine scale and bias, avoiding batch-normalization degradation with minibatch size 1 per processing unit.Large images impose memory constraints that lead to minibatch size 1 per processing unit.Source: §3.1 · pp. 4–5

Invertible 1 × 1 convolution

  • LU parameterization reduces invertible-convolution determinant cost from O(c3) to O(c), although experiments found no large wall-clock difference.The computational-cost difference becomes significant for large c.Source: §3.2 · p. 5

Coupling layers

  • Affine coupling layers provide computationally efficient reversible transformations, while Glow splits and concatenates tensors only along the channel dimension.The final convolution in each coupling-layer network is zero-initialized so the layer initially performs the identity function.Source: §3.3 · p. 5

4 Related Work

Glow builds on earlier flow-based models, including NICE and RealNVP, while excluding MAF from comparisons because its synthesis is non-parallelizable and inefficient. GANs synthesize large realistic images but have limitations in latent encoding, data support, optimization, and evaluating generalization.

Flow-based models

  • Glow builds on the flow models NICE and RealNVP, with comparisons made throughout the paper.The paper identifies NICE and RealNVP as foundational ideas and flows for this work.Source: §4 · p. 6
  • MAF is omitted from comparisons because its synthesis is non-parallelizable and therefore inefficient.MAF is described as a generative flow based on IAF.Source: §4 · p. 6

GANs

  • GANs are known for synthesizing large, realistic images but generally lack latent-space encoders and full data support, are difficult to optimize, and complicate assessment of overfitting and generalization.These limitations distinguish GANs from likelihood-based methods.Source: §4 · p. 6

5 Quantitative Experiments

The experiments evaluate Glow against RealNVP and prior generative models on standard image datasets, while isolating the contribution of invertible 1 × 1 convolutions. Glow’s invertible convolution lowers negative log-likelihood and converges faster than alternative channel permutations, while the model also supports efficient image generation and latent-space manipulation.

Synthesis and latent manipulation

  • Generating a 256 × 256 image at batch size 1 takes about 130ms on a single 1080 Ti and about 550ms on a K80.These timings demonstrate efficient synthesis for large images.Source: §5 · p. 6
  • The experiments include random sampling, linear interpolation between real-image latent codes, and face-attribute manipulation through latent-code interpolation.The reported visualizations use temperature 0.7 for random samples, and attribute rows place the original image in the middle.Source: Figure 4 · p. 7, Figure 5 · p. 8, Figure 6 · p. 8

Gains using invertible 1 × 1 Convolution

  • On CIFAR-10, the invertible 1 × 1 convolution achieves lower negative log likelihood and faster convergence than reversing and fixed-random channel permutations for both additive and affine coupling.All training conditions are held constant, and results are averaged across three random seeds.Source: §5 · p. 7
  • Affine coupling models converge faster than additive coupling models in the CIFAR-10 experiments.The comparison uses average negative log-likelihood in bits per dimension under otherwise constant training conditions.Source: §5 · p. 7

Comparison with RealNVP on standard benchmarks

  • Glow is compared with RealNVP on CIFAR-10, ImageNet, and LSUN using the preprocessing from Dinh et al. (2016).The comparison is intended to assess whether Glow’s overall architecture is competitive with RealNVP beyond the permutation operation.Source: §5 · p. 7

6 Qualitative Experiments

Glow scales to 256^2 CelebA-HQ images and produces extremely high-quality samples with smooth latent interpolations. Post-training latent directions enable semantic manipulation, while temperature 0.7 balances sample diversity and quality.

High-resolution datasets

  • Glow is evaluated on 30,000-image CelebA-HQ at 256^2 resolution using K = 32, L = 6, and 5-bit images to improve visual quality.The 5-bit training choice slightly reduces color fidelity.Source: §6 · p. 8

Synthesis and Interpolation

  • Random samples are extremely high quality for a non-autoregressive likelihood-based model, and latent interpolations produce almost entirely realistic intermediate faces.Real image pairs are encoded, linearly interpolated in latent space, and decoded.Source: §6 · p. 8

Semantic Manipulation

  • Semantic attributes can be manipulated after training by shifting latents along the difference between average vectors for labeled positive and negative examples.This requires only a small amount of post-training supervision, and labels are not used during model training.Source: §6 · pp. 8–9

Effect of temperature and model depth

  • Temperature 0.7 is selected as a sweet spot balancing sample diversity and quality, whereas the highest temperatures produce noisy images.Reduced-temperature sampling can improve sample quality; in additive coupling layers, temperature scales the standard deviation of pθ(z).Source: §6 · pp. 8–9

7 Conclusion

Glow is a new flow model that improves log-likelihood on standard image-modeling benchmarks and efficiently synthesizes realistic high-resolution natural images. The authors describe it as the first likelihood-based model, to their knowledge, with this capability.

7 Conclusion

  • Glow improves quantitative log-likelihood performance on standard image-modeling benchmarks.The paper presents Glow as a new type of flow.Source: §7 · p. 10
  • When trained on high-resolution faces, Glow synthesizes realistic images efficiently.The conclusion reports this result for high-resolution face modeling.Source: §7 · p. 10
  • Glow is, to the authors’ knowledge, the first likelihood-based model that efficiently synthesizes high-resolution natural images.This claim concerns the literature as characterized by the authors.Source: §7 · p. 10

A Additional quantitative results

The section presents quantitative test-set results measured in bits per dimension.

Additional quantitative results

  • Table 3 reports quantitative results.The results are evaluated on the test set and measured in bits per dimension.Source: Table 3 · p. 11
  • The reported metric is bits per dimension.This metric is used for the quantitative results in Table 3.Source: Table 3 · p. 11
  • The evaluation uses the test set.Table 3 specifies the test set as the evaluation split.Source: Table 3 · p. 11

B Simple python implementation of the invertible 1 × 1 convolution · C Optimization details

The implementation uses an invertible 1 × 1 convolution initialized with a random orthogonal matrix, with explicit forward and reverse transformations and log-determinant updates. Optimization uses Adam with α = 0.001 and reports separate hyperparameter settings for quantitative and qualitative experiments.

B Simple python implementation of the invertible 1 × 1 convolution

C Optimization details

  • Optimization uses the Adam optimizer with α = 0.001 and default β1 and β2.These settings are stated for the model optimization procedure.Source: C Optimization details · p. 12
  • Quantitative experiments in Section 5 and Table 2 use hyperparameters specified in Table 4.The passage introduces Table 4 as the source of those experimental hyperparameters.Source: C Optimization details · p. 12, Table 4 · p. 12
  • Qualitative experiments in Section 6 use hyperparameters specified in Table 5.The passage introduces Table 5 as the source of those experimental hyperparameters.Source: C Optimization details · p. 12, Table 5 · p. 12

D Extra samples from qualitative experiments

The qualitative experiments generated class-conditional CIFAR-10 and 32×32 ImageNet samples using the quantitative setup with a class-dependent top-level prior and an auxiliary classification loss.

Experimental setup

E Extra samples from the quantitative experiments

The section presents extra samples from Glow’s quantitative experiments, including class-conditional CIFAR-10 and ImageNet samples, large LSUN images, and unconditional CIFAR-10 samples. The figures specify the datasets, resolutions, model setting, and sampling temperatures.

Experimental setup

Class-conditional samples

  • Figure 10 shows class-conditional samples from 5-bit CIFAR-10 and 32 × 32 ImageNet at temperature 0.75.The figure covers both datasets under class-conditional sampling.Source: Figure 10 · p. 13

Large-image samples

  • Figure 11 shows samples from 8-bit, 64×64 LSUN bedrooms, church, and towers at temperature 1.0.The examples cover three LSUN scene categories at 64×64 resolution.Source: Figure 11 · p. 14

Unconditional samples

  • Figure 12 shows samples from an unconditional model with affine coupling layers trained on CIFAR-10 at temperature 1.0.This figure isolates an unconditional CIFAR-10 model using affine coupling layers.Source: Figure 12 · p. 15

Related papers

Loading 1807.03039v2…