PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual Flows for Invertible Generative Modeling

Ricky T. Q. Chen, Jens Behrmann, David Duvenaud, Jörn-Henrik Jacobsen

arXiv:1906.02735v6stat.MLcs.LG

TL;DR

Prior invertible residual networks relied on biased log-density estimates, limiting faithful maximum-likelihood training as expressiveness increased. Residual Flows introduce unbiased estimation and memory-efficient gradients, alongside non-saturating activations and generalized Lipschitz constraints, and achieve competitive or better performance across density estimation, sample quality, and hybrid modeling.

  • Problem

    Prior invertible residual networks used fixed-truncation log-density estimates whose bias increased with expressiveness, undermining maximum-likelihood density estimation.

  • Method

    Residual Flows use an unbiased log-density estimator, an alternative infinite series for memory-efficient gradients, non-saturating activations, and induced mixed norms.

  • Results

    Residual Flows achieve competitive or better performance than alternative flow-based models in density estimation, sample quality, and hybrid modeling.

  • Takeaways & Limitations

    Residual Flows open a design space for expressive Lipschitz-constrained architectures beyond models relying on exact log-determinant computations.

  • Takeaways & Limitations

    Memory-efficient backpropagation still requires memory comparable to one residual-network evaluation for path-wise derivatives, and reduced-entropy sampling does not converge to the distribution mode.

Abstract

from arXiv · show

Flow-based generative models parameterize probability distributions through an invertible transformation and can be trained by maximum likelihood. Invertible residual networks provide a flexible family of transformations where only Lipschitz conditions rather than strict architectural constraints are needed for enforcing invertibility. However, prior work trained invertible residual networks for density estimation by relying on biased log-density estimates whose bias increased with the network's expressiveness. We give a tractable unbiased estimate of the log density using a "Russian roulette" estimator, and reduce the memory required during training by using an alternative infinite series for the gradient. Furthermore, we improve invertible residual blocks by proposing the use of activation functions that avoid derivative saturation and generalizing the Lipschitz condition to induced mixed norms. The resulting approach, called Residual Flows, achieves state-of-the-art performance on density estimation amongst flow-based models, and outperforms networks that use coupling blocks at joint generative and discriminative modeling.

1 Introduction

Flow-based models use invertible transformations for tractable density evaluation and exact sampling, but scalable designs often impose structured Jacobians and architectural constraints. Residual Flows address biased density estimation in invertible residual networks with unbiased estimation and more flexible Lipschitz-constrained architectures.

  • Flow-based generative models construct flexible distributions through invertible transformations, enabling exact sampling and efficient density evaluation.
  • Existing scalable flows use sparse or structured Jacobians, coupling blocks, or ordinary differential equations, imposing architectural engineering and strong inductive biases.
  • Invertible residual networks require only a Lipschitz constraint, but prior fixed-truncation log-density estimates introduced bias that increased with network expressiveness.
  • Residual Flows provide unbiased log-density estimates, memory-efficient backpropagation, non-saturating activations, and induced mixed norms for Lipschitz-constrained networks.
  • Residual Flows are presented as a flow-based model that supports expressive architectures while training with maximum likelihood.

2 Background

Flow-based models apply the change of variables theorem to invertible transformations, making likelihood evaluation tractable for maximum-likelihood training. Invertible residual networks achieve invertibility through contractive residual functions, but fixed truncation of their log-determinant series introduces expressiveness-dependent bias.

  • Maximum likelihood trains a model distribution to align with an unknown data distribution, using samples from the data distribution.
  • The change of variables theorem relates densities through an invertible transformation and permits tractable likelihood evaluation for flow-based models.
  • Residual networks use transformations y = f(x) = x + g(x), which are invertible when g is contractive with Lipschitz constant strictly below one.
  • Fixed truncation approximates the residual-network power series but incurs bias that grows with input dimensionality and g's Lipschitz constant.

3 Residual Flows

Residual Flows make maximum-likelihood training tractable for expressive invertible residual networks by combining unbiased Russian roulette estimation with memory-efficient gradient computation. They also address derivative saturation using LipSwish activations.

  • 3.1 Unbiased Log Density Estimation for Maximum Likelihood Estimation: The Russian roulette estimator gives finite-time, unbiased estimates of the infinite log-density power series under stated support and invertibility conditions.The estimator randomizes how many series terms are evaluated, while requiring the sampling distribution to support all indices.
  • 3.1 Unbiased Log Density Estimation for Maximum Likelihood Estimation: Unbiased log-density gradients enable maximum-likelihood training with more expressive networks, avoiding the bias–expressiveness trade-off of fixed truncation.Fixed truncation bias grows with dimensionality and the Lipschitz constant, whereas Residual Flows can backpropagate through the unbiased estimator.
  • 3.2 Memory-Efficient Backpropagation: Naive differentiation stores every computed power-series term, producing O(n · m) memory cost and occasional out-of-memory failures.Here n is the number of computed terms and m is the number of residual blocks.
  • 3.2 Memory-Efficient Backpropagation: The Neumann gradient series avoids differentiating through the power series, reducing memory by a factor of n and making memory constant with respect to sampled term count.The alternative estimator remains unbiased under the Lipschitz condition and positive-integer sampling support.
  • 3.2 Memory-Efficient Backpropagation: Backward-in-forward computation releases each residual block’s graph during the forward pass, reducing additional memory by a factor of m to O(1) with negligible overhead.The method computes parameter derivatives for each residual block early and combines them later using the scalar log-determinant derivative.
  • 3.3 Avoiding Derivative Saturation with the LipSwish Activation Function: LipSwish preserves bounded first derivatives while maintaining non-vanishing second derivatives near maximal Lipschitz regions, addressing saturation during training.The design targets |φ′(z)| ≤ 1 and avoids asymptotically vanishing φ′′ when |φ′| is close to one.

4 Related Work

Prior work on invertible and residual networks established flow-based density estimation, memory-efficient gradient methods, and randomized estimation of infinite series. These approaches motivate comparisons across architectures and evaluation settings.

  • Randomized truncation has been applied across fields, including stochastic differential equations, light-path rendering, and optimization-limit estimation.
  • Figure 5 presents real and random 5bit 64×64 CelebA samples, with the most visually appealing outputs selected from five random batches.
  • Prior memory-efficient backpropagation work studied residual networks and continuous residual-network analogues, whereas this work targets log-determinant gradients for generative modeling.
  • Flow-based models use invertibility as a design principle for density estimation, while Continuous Normalizing Flows use Jacobian traces instead of determinants.

5 Experiments

Experiments evaluate Residual Flows on density estimation, sample quality, ablations, and hybrid generative-discriminative modeling. The results show competitive density estimates, strong sample quality, benefits from unbiased estimation and LipSwish, and advantages of residual blocks for joint tasks.

  • Experimental setup: Residual Flow models train with standard batch size 64 and roughly 300–350 epochs on MNIST and CIFAR-10; most settings fit on one GPU.On CelebA-HQ, the authors used batch size 3 per GPU across 4 GPUs, compared with Glow’s batch size 1 per GPU across 40 GPUs.
  • Density estimation: Residual Flows achieve competitive bits-per-dimension performance across MNIST, CIFAR-10, downsampled ImageNet, and CelebA-HQ.Evaluation used 20 exact power-series terms followed by the unbiased estimator, yielding negligible standard deviation in test bits per dimension.
  • Density estimation: Learning norm orders jointly yields a 0.003 bits/dim gain on CIFAR-10 over spectral normalization, while p = ∞ constraints are more suited to lower-dimensional data.The Lipschitz condition can be generalized to arbitrary p-norms and mixed matrix norms.
  • Sample quality: Residual Flows significantly improve FID over i-ResNets and PixelCNN and slightly outperform an official Glow model with twice as many layers.Lower FID indicates better sample quality; CIFAR-10 samples are also described as more globally coherent than PixelCNN and Flow++.
  • Sample quality: Naïve entropy reduction does not produce proper temperature annealing for general flow-based models, because samples fail to converge to the distribution mode.For Residual Flows, the mode of p(x) and p(z) need not coincide, so reducing latent entropy can yield black hair and background artifacts.
  • Ablation experiments: The unbiased estimator improves training even when Lipschitz constants and bias are relatively low, while LipSwish converges faster and outperforms softplus and ELU at high Lipschitz settings.For a larger CIFAR-10 i-ResNet, the biased estimate fell below 0.8 bits/dim while actual bits/dim oscillated above 3.66.
  • Hybrid modeling: Residual blocks outperform coupling blocks for joint generative and discriminative representations, whereas coupling blocks perform poorly across both objectives simultaneously.The architecture also outperforms Nalisnick et al. (2019) on pure classification and hybrid modeling.

6 Conclusion

Residual Flows turn invertible residual networks into powerful generative models with competitive or better performance across density estimation, sample quality, and hybrid modeling. Their stochastic estimation recipe enables expressive Lipschitz-constrained architectures beyond methods requiring exact log-determinants.

  • Conclusion: Residual Flow achieves competitive or better performance than alternative flow-based models in density estimation, sample quality, and hybrid modeling.The conclusion presents this as evidence that invertible residual networks can serve as powerful generative models.
  • Conclusion: The proposed stochasticity-based recipe constructs tractable flow-based models under architectural constraints different from exact log-determinant methods.The authors identify expressive Lipschitz-constrained architectures as a design space for future exploration.

A Random Samples

Appendix A presents random samples from Residual Flow models across MNIST, ImageNet, and 5-bit CelebA-HQ at multiple resolutions.

  • Random samples: Random samples are shown for MNIST, ImageNet at 32×32 and 64×64, and 5-bit CelebA-HQ at 64×64 and 256×256.For 5-bit CelebA-HQ 256×256, the most visually appealing batch out of five was selected.

B Proofs

The proofs establish unbiasedness for randomized truncated infinite-series estimators under support and convergence conditions. They use convergence theorems, trace estimation, and Lipschitz-based bounds to verify the required assumptions.

  • Unbiased randomized series: The proof framework begins with a lemma characterizing when a randomized truncated series is an unbiased estimator.The specific Residual Flow estimator is then shown to satisfy the lemma’s assumption.
  • Assumptions: The randomized truncation variable must have support over the positive integers, while the series increments must satisfy summability conditions.These conditions permit exchanging expectations, limits, and summations using convergence theorems.
  • Theorem 1: Theorem 1 is verified by combining the Skilling-Hutchinson trace estimator with bounds based on eigenvalues and spectral radius.The contraction condition Lip(g) < 1 ensures the required power-series bounds, completing the assumption verification.
  • Theorem 2: Theorem 2 follows analogously to Theorem 1, using Fubini’s theorem, trace estimation, and similar bounds.The proof is abbreviated because the same structure applies.

C Memory-Efficient Gradient Estimation of Log-Determinant

The method replaces biased fixed-truncation gradients with an alternative infinite-series formulation whose computation graph is not tracked, reducing memory during log-determinant backpropagation.

  • The two infinite-series formulations agree for exact trace computation at a given truncation, but Hutchinson trace estimates generally differ because their terms differ.
  • Differentiating an alternative power series avoids tracking its computational graph, enabling gradient computation with constant memory relative to truncation n.

D Generalizing Lipschitz Constraints to Induced Mixed Norms

Residual Flows generalize spectral normalization from one vector norm to induced mixed norms while preserving the invertibility argument, and learn norm orders during training.

  • Any induced matrix norm can enforce the Jacobian bound through sub-multiplicativity, including mixed norms mapping inputs measured by p-norms to outputs measured by q-norms.
  • The Banach fixed point theorem still guarantees residual-block invertibility when g maps back to the original complete normed vector space.
  • Generalized power iteration estimates induced mixed norms, while backpropagation can optimize the norm orders during training.
  • 0.003 bits/dim improvement on CIFAR-10 was obtained by learned norms, while the infinity norm performed worse than spectral normalization for convolutional networks.
  • On Checkerboard 2D, the infinity norm matched or outperformed p = 2, producing more evenly spread density estimates than the spectral-norm model.The comparison used 200 residual blocks.

E Experiment Setup

Experiments use flow architectures built from normalized residual blocks, squeeze operations, dataset-specific preprocessing, and adaptive spectral normalization, with an added multiscale classifier for hybrid modeling.

  • Density experiments use an unsquashing logit transform followed by alternating residual blocks and squeeze layers, with activation normalization before and after every block.
  • MNIST and CIFAR-10 density models add four fully connected residual blocks with hidden dimension 128 at the network end.
  • Datasets larger than 32×32 factor out half the variables after each squeeze operation except the first.
  • Hybrid CIFAR-10 modeling replaces the logit transform with standard mean-and-standard-deviation normalization, while MNIST and SVHN retain their density-model architectures.
  • Hybrid models add classifier branches after every squeeze layer and at the network end, concatenate pooled outputs across scales, and feed them to a linear softmax classifier.
  • Spectral normalization uses adaptive power iterations that stop when the estimated norm's relative change is sufficiently small.
  • Optimization uses Adam with learning rate 0.001 and weight decay 0.0005, with Polyak averaging decay 0.999 for evaluation.
Loading 1906.02735v6…