PaperlayerView exact revision on arXiv

Source-linked AI summary

Glow: Generative Flow with Invertible 1x1 Convolutions

Diederik P. Kingma, Prafulla Dhariwal

arXiv:1807.03039v2stat.MLcs.AIcs.LG

TL;DR

Existing flow models relied on fixed channel permutations, limiting the design of generative flows for image modeling. Glow replaces this operation with an invertible 1×1 convolution and improves log-likelihood while efficiently synthesizing realistic high-resolution images.

  • Problem

    Existing flow architectures used fixed channel permutations, motivating a learned alternative for improving generative image modeling.

  • Method

    Glow replaces fixed channel permutations with a learned invertible 1×1 convolution in a generative flow.

  • Results

    Glow achieves lower negative log-likelihood and faster convergence than comparable coupling models on CIFAR-10, with approximately 7% higher wall-clock time.

  • Takeaways & Limitations

    Glow improves log-likelihood on standard image benchmarks and efficiently synthesizes realistic high-resolution natural images.

  • Takeaways & Limitations

    The computational-cost difference of invertible 1×1 convolutions becomes significant for large channel counts.

Abstract

from arXiv · show

Flow-based generative models (Dinh et al., 2014) are conceptually attractive due to tractability of the exact log-likelihood, tractability of exact latent-variable inference, and parallelizability of both training and synthesis. In this paper we propose Glow, a simple type of generative flow using an invertible 1x1 convolution. Using our method we demonstrate a significant improvement in log-likelihood on standard benchmarks. Perhaps most strikingly, we demonstrate that a generative model optimized towards the plain log-likelihood objective is capable of efficient realistic-looking synthesis and manipulation of large images. The code for our model is available at https://github.com/openai/glow

1 Introduction

Generative modeling aims to capture dependencies in high-dimensional data, with potential applications including speech synthesis, text analysis, semi-supervised learning, and model-based control. The paper proposes Glow, a generative flow with new elements, and evaluates it quantitatively against previous flows and qualitatively on high-resolution datasets.

  • Motivation: Generative models may address data-efficiency and generalization by learning realistic world models and meaningful, task-independent features from unlabeled data.These capabilities could support planning before real-world interaction and make downstream solutions more robust and data efficient.
  • Motivation: Generative modeling seeks to model all dependencies in high-dimensional input data through a full joint probability distribution.Accurate generative models could support diverse applications, including speech synthesis, text analysis, semi-supervised learning, and model-based control.
  • Related methods: Likelihood-based generative models include autoregressive models, variational autoencoders, and flow-based models, each offering different tradeoffs in optimization and parallelizability.Autoregressive synthesis has limited parallelizability, VAEs optimize a lower bound on log-likelihood, and flow-based models were introduced by NICE and extended by RealNVP.
  • Flow-based models: Flow-based generative models provide exact latent-variable inference and exact log-likelihood evaluation rather than approximations or lower bounds.They also support efficient parallelization of inference and synthesis, unlike the difficult-to-parallelize synthesis of autoregressive models.
  • Contribution: The paper proposes Glow, a new generative flow with various new elements, and studies it quantitatively against previous flows and qualitatively on high-resolution datasets.The paper also notes potential flow-model benefits including useful latent spaces for downstream tasks and memory savings in reversible neural networks.

2 Background: Flow-based Generative Models

Flow-based generative models represent data through an invertible mapping to a tractable latent distribution, enabling exact latent inference and likelihood computation via the change-of-variables formula. Glow builds on this framework by composing invertible transformations whose log-determinants can be tractably computed.

  • Training objective: For continuous data, training minimizes a dequantized log-likelihood objective whose terms measure expected compression cost in nats or bits.The dequantized input is ˜x(i) = x(i) + u with u ∼ U(0, a), and optimization uses stochastic gradient descent with minibatches.
  • Invertible latent-variable modeling: Flow-based models define x through an invertible transformation gθ of a latent variable z with a typically simple, tractable density.A common choice is a spherical multivariate Gaussian, pθ(z) = N(z; 0, I), and inference uses z = gθ^-1(x).
  • Normalizing flows: The inverse mapping is composed as a sequence of invertible transformations, f = f1 ◦ f2 ◦ · · · ◦ fK, forming a normalizing flow.The intermediate representations are defined by h0 ≜ x and hK ≜ z.
  • Likelihood computation: The model density is computed with change of variables by summing the log-determinants of each transformation’s Jacobian.The log-determinant measures the change in log-density between consecutive representations, and suitable transformations can make it surprisingly simple to compute.

3 Proposed Generative Flow

Glow builds a multi-scale generative flow from repeated actnorm, invertible 1 × 1 convolution, and coupling-layer steps. It replaces fixed channel permutations with learned invertible 1 × 1 convolutions and introduces actnorm to support training with minibatch size 1 per processing unit.

  • Overall architecture: Each flow step applies actnorm, an invertible 1 × 1 convolution, and a coupling layer within a multi-scale architecture.The architecture has depth K and number of levels L.
  • Actnorm: Actnorm applies a learned per-channel affine scale and bias, avoiding batch-normalization degradation with minibatch size 1 per processing unit.Large images impose memory constraints that lead to minibatch size 1 per processing unit.
  • Invertible 1 × 1 convolution: LU parameterization reduces invertible-convolution determinant cost from O(c3) to O(c), although experiments found no large wall-clock difference.The computational-cost difference becomes significant for large c.
  • Coupling layers: Affine coupling layers provide computationally efficient reversible transformations, while Glow splits and concatenates tensors only along the channel dimension.The final convolution in each coupling-layer network is zero-initialized so the layer initially performs the identity function.

4 Related Work

Glow builds on earlier flow-based models, including NICE and RealNVP, while excluding MAF from comparisons because its synthesis is non-parallelizable and inefficient. GANs synthesize large realistic images but have limitations in latent encoding, data support, optimization, and evaluating generalization.

  • Flow-based models: Glow builds on the flow models NICE and RealNVP, with comparisons made throughout the paper.The paper identifies NICE and RealNVP as foundational ideas and flows for this work.
  • Flow-based models: MAF is omitted from comparisons because its synthesis is non-parallelizable and therefore inefficient.MAF is described as a generative flow based on IAF.
  • GANs: GANs are known for synthesizing large, realistic images but generally lack latent-space encoders and full data support, are difficult to optimize, and complicate assessment of overfitting and generalization.These limitations distinguish GANs from likelihood-based methods.

5 Quantitative Experiments

The experiments evaluate Glow against RealNVP and prior generative models on standard image datasets, while isolating the contribution of invertible 1 × 1 convolutions. Glow’s invertible convolution lowers negative log-likelihood and converges faster than alternative channel permutations, while the model also supports efficient image generation and latent-space manipulation.

  • Synthesis and latent manipulation: Generating a 256 × 256 image at batch size 1 takes about 130ms on a single 1080 Ti and about 550ms on a K80.These timings demonstrate efficient synthesis for large images.
  • Synthesis and latent manipulation: The experiments include random sampling, linear interpolation between real-image latent codes, and face-attribute manipulation through latent-code interpolation.The reported visualizations use temperature 0.7 for random samples, and attribute rows place the original image in the middle.
  • Gains using invertible 1 × 1 Convolution: On CIFAR-10, the invertible 1 × 1 convolution achieves lower negative log likelihood and faster convergence than reversing and fixed-random channel permutations for both additive and affine coupling.All training conditions are held constant, and results are averaged across three random seeds.
  • Gains using invertible 1 × 1 Convolution: Affine coupling models converge faster than additive coupling models in the CIFAR-10 experiments.The comparison uses average negative log-likelihood in bits per dimension under otherwise constant training conditions.
  • Comparison with RealNVP on standard benchmarks: Glow is compared with RealNVP on CIFAR-10, ImageNet, and LSUN using the preprocessing from Dinh et al. (2016).The comparison is intended to assess whether Glow’s overall architecture is competitive with RealNVP beyond the permutation operation.

6 Qualitative Experiments

Glow scales to 256^2 CelebA-HQ images and produces extremely high-quality samples with smooth latent interpolations. Post-training latent directions enable semantic manipulation, while temperature 0.7 balances sample diversity and quality.

  • High-resolution datasets: Glow is evaluated on 30,000-image CelebA-HQ at 256^2 resolution using K = 32, L = 6, and 5-bit images to improve visual quality.The 5-bit training choice slightly reduces color fidelity.
  • Synthesis and Interpolation: Random samples are extremely high quality for a non-autoregressive likelihood-based model, and latent interpolations produce almost entirely realistic intermediate faces.Real image pairs are encoded, linearly interpolated in latent space, and decoded.
  • Semantic Manipulation: Semantic attributes can be manipulated after training by shifting latents along the difference between average vectors for labeled positive and negative examples.This requires only a small amount of post-training supervision, and labels are not used during model training.
  • Effect of temperature and model depth: Temperature 0.7 is selected as a sweet spot balancing sample diversity and quality, whereas the highest temperatures produce noisy images.Reduced-temperature sampling can improve sample quality; in additive coupling layers, temperature scales the standard deviation of pθ(z).

7 Conclusion

Glow is a new flow model that improves log-likelihood on standard image-modeling benchmarks and efficiently synthesizes realistic high-resolution natural images. The authors describe it as the first likelihood-based model, to their knowledge, with this capability.

  • 7 Conclusion: Glow improves quantitative log-likelihood performance on standard image-modeling benchmarks.The paper presents Glow as a new type of flow.
  • 7 Conclusion: When trained on high-resolution faces, Glow synthesizes realistic images efficiently.The conclusion reports this result for high-resolution face modeling.
  • 7 Conclusion: Glow is, to the authors’ knowledge, the first likelihood-based model that efficiently synthesizes high-resolution natural images.This claim concerns the literature as characterized by the authors.

A Additional quantitative results

The section presents quantitative test-set results measured in bits per dimension.

  • Additional quantitative results: Table 3 reports quantitative results.The results are evaluated on the test set and measured in bits per dimension.
  • Additional quantitative results: The reported metric is bits per dimension.This metric is used for the quantitative results in Table 3.
  • Additional quantitative results: The evaluation uses the test set.Table 3 specifies the test set as the evaluation split.

B Simple python implementation of the invertible 1 × 1 convolution · C Optimization details

The implementation uses an invertible 1 × 1 convolution initialized with a random orthogonal matrix, with explicit forward and reverse transformations and log-determinant updates. Optimization uses Adam with α = 0.001 and reports separate hyperparameter settings for quantitative and qualitative experiments.

  • B Simple python implementation of the invertible 1 × 1 convolution: The convolution weights are initialized by sampling a random orthogonal matrix with QR decomposition.The implementation computes np.linalg.qr(np.random.randn(c,c))[0] and uses the resulting matrix to initialize W.
  • B Simple python implementation of the invertible 1 × 1 convolution: The forward log-determinant contribution is h * w * tf.log(abs(tf.matrix_determinant(w))).This quantity is computed as dlogdet before being added to logdet in the forward pass.
  • B Simple python implementation of the invertible 1 × 1 convolution: The forward pass reshapes the weight matrix into a 1 × 1 convolution kernel and applies it to z.The convolution uses stride [1,1,1,1] and ’SAME’ padding.
  • B Simple python implementation of the invertible 1 × 1 convolution: The reverse pass applies the matrix inverse as a 1 × 1 convolution and subtracts the same log-determinant contribution.The inverse weight is reshaped to [1,1,c,c] before applying the convolution.
  • C Optimization details: Optimization uses the Adam optimizer with α = 0.001 and default β1 and β2.These settings are stated for the model optimization procedure.
  • C Optimization details: Quantitative experiments in Section 5 and Table 2 use hyperparameters specified in Table 4.The passage introduces Table 4 as the source of those experimental hyperparameters.
  • C Optimization details: Qualitative experiments in Section 6 use hyperparameters specified in Table 5.The passage introduces Table 5 as the source of those experimental hyperparameters.

D Extra samples from qualitative experiments

The qualitative experiments generated class-conditional CIFAR-10 and 32×32 ImageNet samples using the quantitative setup with a class-dependent top-level prior and an auxiliary classification loss.

  • Experimental setup: The experiments evaluated class-conditional samples on CIFAR-10 and 32×32 ImageNet.The results are reported in Figure 10.
  • Experimental setup: The CIFAR-10 and ImageNet experiments used the same hyperparameters as the quantitative experiments.They changed the top-most level to use a class-dependent prior.
  • Experimental setup: An auxiliary classification loss predicted the class label from the encoder’s second-last layer with weight λ = 0.01.This loss was added for the class-conditional qualitative experiments.

E Extra samples from the quantitative experiments

The section presents extra samples from Glow’s quantitative experiments, including class-conditional CIFAR-10 and ImageNet samples, large LSUN images, and unconditional CIFAR-10 samples. The figures specify the datasets, resolutions, model setting, and sampling temperatures.

  • Experimental setup: The datasets are preprocessed exactly as in Dinh et al. (2016) for direct comparison with other work.The corresponding results are reported in Figures 11 and 12.
  • Class-conditional samples: Figure 10 shows class-conditional samples from 5-bit CIFAR-10 and 32 × 32 ImageNet at temperature 0.75.The figure covers both datasets under class-conditional sampling.
  • Large-image samples: Figure 11 shows samples from 8-bit, 64×64 LSUN bedrooms, church, and towers at temperature 1.0.The examples cover three LSUN scene categories at 64×64 resolution.
  • Unconditional samples: Figure 12 shows samples from an unconditional model with affine coupling layers trained on CIFAR-10 at temperature 1.0.This figure isolates an unconditional CIFAR-10 model using affine coupling layers.
Loading 1807.03039v2…