Source-linked AI summary
Emerging Convolutions for Generative Normalizing Flows
Emiel Hoogeboom, Rianne van den Berg, Max Welling
TL;DR
1 × 1 convolutions suffer from limited flexibility, and using standard convolutions is not straightforward. The paper introduces emerging d × d convolutions built from autoregressive convolutions, periodic convolutions decoupled in the frequency domain, and QR-parameterized 1 × 1 convolutions. Generalized invertible convolutions significantly improve results on galaxy images, CIFAR10 and ImageNet, even when correcting for increased parameters, with consistent improvements especially in size-constrained models.
Problem
1 × 1 convolutions suffer from limited flexibility, and using standard convolutions is not straightforward.
Source: §1 · p. 1Method
The paper introduces emerging d × d convolutions built from autoregressive convolutions, periodic convolutions decoupled in the frequency domain, and QR-parameterized 1 × 1 convolutions.
Source: §1 · p. 2, §3.2 · p. 5, §3.3 · p. 6Results
Takeaways & Limitations
Emerging convolutions provide invertible d × d standard zero-padded convolutions, while periodic convolutions are intended for periodic data or data with minimal boundary variation.
Source: §6 · p. 8Periodic convolutions may be non-invertible if any frequency-domain matrix has zero determinant, though initialization and the likelihood objective push determinants away from zero.
Source: §3.2 · p. 6
Abstract
from arXiv · showhide
Generative flows are attractive because they admit exact likelihood optimization and efficient image synthesis. Recently, Kingma & Dhariwal (2018) demonstrated with Glow that generative flows are capable of generating high quality images. We generalize the 1 x 1 convolutions proposed in Glow to invertible d x d convolutions, which are more flexible since they operate on both channel and spatial axes. We propose two methods to produce invertible convolutions that have receptive fields identical to standard convolutions: Emerging convolutions are obtained by chaining specific autoregressive convolutions, and periodic convolutions are decoupled in the frequency domain. Our experiments show that the flexibility of d x d convolutions significantly improves the performance of generative flow models on galaxy images, CIFAR10 and ImageNet.
1. Introduction
Normalizing flows offer exact likelihood optimization, but existing flow transformations trade flexibility against invertibility: 1 × 1 convolutions are limited, while standard convolutions are costly to invert. The paper proposes invertible convolutions with standard-convolution receptive fields and reports improved results on galaxy images, CIFAR10 and ImageNet.
1. Introduction
- Normalizing flows optimize continuous-data log likelihood exactly, while 1 × 1 convolutions have limited flexibility and standard convolutions are computationally expensive to invert.Flow models also allow sampling at computational cost comparable to inference.Source: §1 · pp. 1–2
- Generalized invertible convolutions significantly improve results on galaxy images, CIFAR10 and ImageNet, even when correcting for increased parameters.The proposed convolutions operate across channel and spatial axes while retaining receptive fields identical to standard convolutions.Source: §1 · p. 2
- QR parameterization addresses flexibility limitations of the PLU decomposition used for 1 × 1 convolutions.Kingma & Dhariwal (2018) proposed the PLU decomposition.Source: §1 · p. 2
- Emerging convolutions chain autoregressive convolutions, and periodic convolutions use frequency-domain decoupling to create invertible d × d transformations.The paper also proposes QR-parameterized 1 × 1 convolutions and an accelerated inversion module for autoregressive convolutions.Source: §1 · p. 2
2. Background
Normalizing flows build tractable likelihoods by composing invertible transformations, while image flows must also address discrete pixels and keep inverses and Jacobians practical. The section introduces convolutional and autoregressive layers, whose matrix structure makes their determinant and inversion properties explicit.
2.1. Change of variables formula
- A bijection expresses pX(x) through the tractable density pZ(z) and a Jacobian determinant, while practical flow design also requires an efficient inverse for sampling.Composing layers adds their log-Jacobian determinants to the log-likelihood.Source: §2.1 · p. 2, §2.1.1 · p. 2
2.1.1. COMPOSITION OF FUNCTIONS
- A normalizing flow composes change-of-variables transformations, with intermediate representations whose layerwise log-Jacobian determinants sum to the data log-likelihood.The supplied text begins the definition of a normalizing flow and then describes the intermediate representations and summed determinants.Source: §2.1.1 · p. 2
2.1.2. DEQUANTIZATION
- For discrete-valued pixels, dequantization relates the discrete model to a continuous density by integrating over each pixel bin and uses additive uniform noise for a Jensen lower bound.The noise is added to integer-valued pixels from the data distribution.Source: §2.1.2 · p. 2
2.2. Generative flows
- Generative-flow layers are designed to have tractable inverses and determinants; coupling layers split inputs, while Actnorm and 1 × 1 convolutions provide additional transformations.Actnorm is initialized to give activations mean zero and standard deviation one; 1 × 1 convolutions are easy to invert and improve coupling-layer effectiveness.Source: §2.2 · p. 2, Table 1 · p. 3
2.3. Convolutions
- A convolution is equivalent to multiplying a vectorized input by a sparse matrix with repeated filter parameters, but naively computing its determinant and inverse is expensive.The stated naive matrix-operation complexity is O(h^3 · w^3 · n_c^3).Source: §2.3 · p. 3
2.4. Autoregressive Convolutions
- Autoregressive convolutions use masked filters to produce a triangular matrix, making the determinant depend only on diagonal elements and allowing inversion by forward substitution.The inverse traverses the feature map in autoregressive order, with complexity O(h · w · n_c^2) and parallelism across minibatch examples.Source: §2.4 · pp. 3–4
3. Method
The method generalizes invertible 1 × 1 convolutions with emerging and periodic d × d convolutions, and also introduces a stable, flexible QR parameterization for 1 × 1 convolutions.
3. Method
- The method combines emerging convolutions built from autoregressive layers, periodic convolutions decoupled in the frequency domain, and a stable, flexible QR parameterization for invertible 1 × 1 convolutions.These approaches generalize the transformations available in flow modules.Source: §3 · p. 4
3.1. Emerging convolutions
- Chaining specific autoregressive convolutions yields flexible, invertible emerging convolutions whose composite need not remain autoregressive.Their inverse uses autoregressive inverses; an accelerated parallel inversion module reduces the added sequential cost.Source: §3.1 · p. 4
3.1.1. SQUARE EMERGING CONVOLUTIONS
- Square emerging convolutions combine off-center square autoregressive filters to match a standard convolution’s receptive field with fewer masked values.For computation, the two filters can be applied consecutively or combined into an equivalent filter using convolution associativity.Source: §3.1.1 · p. 5
3.2. Invertible Periodic Convolutions
- Periodic convolutions assume wraparound boundaries and use Fourier-domain channel transformations, where each frequency component contributes to the layer’s log determinant.The Fourier transform decouples the convolution by frequency, making its determinant and inverse tractable; filters are unconstrained.Source: §3.2 · pp. 5–6
3.3. QR 1 × 1 convolutions
- QR parameterization addresses instability and limited PLU flexibility with W = Q(R + diag(s)), where Q is orthogonal, R strictly triangular, and s has nonzero entries.The number of Householder reflections used to construct Q trades computational complexity against flexibility.Source: §3.3 · p. 6
4. Related Work
The related work situates generative flows among autoregressive, variational, and adversarial approaches, and the model architecture builds on Glow.
4. Related Work
- The paper situates its flow approach among work by Rippel & Adams (2013), Dinh et al. (2014; 2017), Papamakarios et al. (2017), and Kingma & Dhariwal (2018).Normalizing flows are also used for variational-autoencoder inference and Bayesian neural networks, while continuous-time variants connect flows to ordinary differential equations.Source: §4 · p. 6
- PixelCNNs impose a dimension ordering and tend to sample slowly, while VAEs optimize a likelihood lower bound whose estimate depends on the proposal and sample count.Autoregressive convolutions in Papamakarios et al. (2017) also make sampling expensive because of model depth and channel count.Source: §4 · p. 6
- GANs can generate high-quality images but do not directly optimize likelihood, making likelihood evaluation and dataset-coverage measurement difficult.The experiments use the Glow architecture as their starting point.Source: §4 · p. 7, Figure 7 · p. 7
5. Results
The experiments evaluate Glow-based architectures in bits per dimension across galaxy images, CIFAR10, and ImageNet. The reported comparisons favor the proposed convolutions, with sampling speed and QR parameterization also examined.
5.1. Galaxy density modeling
- On the galaxy images dataset, periodic and emerging convolutions significantly outperform 1 × 1 convolutions; performance is measured in bits per dimension, with results from 3 runs using different random seeds (± standard deviation).Their performance is less sensitive to initialization.Source: Table 2 · p. 7, §5.1 · p. 7
5.2. Emerging convolutions
- Emerging convolutions perform on par with or better than Glow3 on CIFAR10 and ImageNet at different architectural sizes.The authors note that overparameterization of these large models may contribute to the results.Source: §5.2 · p. 7, Table 3 · p. 8
- For smaller models, emerging convolutions become more important, as evidenced by an increasing performance gap in the reported CIFAR10 and ImageNet 32x32 experiments.Source: §5.2 · p. 8, Table 4 · p. 8
5.3. Modeling and sample time comparison with MAF
- 39 ms per CIFAR-10 sample with accelerated emerging inverses is 46 times faster than the naive inverse, while 1 × 1 convolutions take 5 ms.MAFs achieve 3.33 versus 3.34 bits per dimension for emerging convolutions, but take 3000 ms per sample naively or 650 ms with the inversion module; emerging convolutions are 17 times faster to invert.Source: §5.3 · p. 8, Table 5 · p. 8
5.4. QR 1 × 1 convolutions
- QR 1 × 1 convolutions match standard parameterization performance on CIFAR10, while PLU performs worse in the reported comparison.The authors attribute PLU's lower performance to its fixed permutation matrix.Source: §5.4 · p. 8, Table 6 · p. 8
6. Conclusion
The paper introduces invertible emerging and periodic d × d convolutions, alongside QR-parameterized 1 × 1 convolutions, and reports consistent improvements across datasets under the same parameter budget, especially for size-constrained models.
6. Conclusion
- The paper introduces emerging convolutions as invertible standard zero-padded convolutions, periodic convolutions for periodic or minimally varying boundaries, and stable, flexible QR-parameterized 1 × 1 convolutions.Across datasets, the methods show consistent improvements under the same parameter budget, especially in size-constrained models.Source: §6 · p. 8