PaperlayerView exact revision on arXiv

Source-linked AI summary

Normalizing Flows for Probabilistic Modeling and Inference

George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, Balaji Lakshminarayanan

arXiv:1912.02762v2stat.MLcs.LG

TL;DR

Normalizing flows address the need for richer, well-specified probabilistic models and a unified understanding of a rapidly developing field. This review organizes flow design through probabilistic modeling and inference, covering expressive power, computational trade-offs, generalized probability transformations, and applications. It concludes that flows support broad applications, while their construction remains bounded by computational asymmetries and topological constraints.

  • Problem

    Probabilistic modeling needs richer and better-specified distributions, while normalizing-flow research lacks a unifying perspective connecting its advances and applications.

  • Method

    The review develops formal foundations, analyzes finite and infinitesimal flows, explains design trade-offs, generalizes probability transformations, and surveys applications.

  • Results

    Under stated regularity conditions, flows can represent any target distribution from a suitable base, and autoregressive flows are universal approximators when their components are sufficiently flexible.

  • Takeaways & Limitations

    Flows provide a general framework spanning probabilistic modeling, inference, simulation, density estimation, likelihood-free inference, and classification.

  • Takeaways & Limitations

    Flow constructions require topological compatibility between spaces, and continuous-time flows preserve the topology of all intermediate spaces unless auxiliary dimensions are introduced.

Abstract

from arXiv · show

Normalizing flows provide a general mechanism for defining expressive probability distributions, only requiring the specification of a (usually simple) base distribution and a series of bijective transformations. There has been much recent work on normalizing flows, ranging from improving their expressive power to expanding their application. We believe the field has now matured and is in need of a unified perspective. In this review, we attempt to provide such a perspective by describing flows through the lens of probabilistic modeling and inference. We place special emphasis on the fundamental principles of flow design, and discuss foundational topics such as expressive power and computational trade-offs. We also broaden the conceptual framing of flows by relating them to more general probability transformations. Lastly, we summarize the use of flows for tasks such as generative modeling, approximate inference, and supervised learning.

1. Introduction

Normalizing flows are presented as a flexible tool for constructing richer probabilistic models, addressing the difficulty of specifying distributions that accurately describe data. The review organizes the field around enduring design principles, formal foundations, flow variants, general probability transformations, and applications.

  • Motivation: The review addresses the statistical need for tools that support increasingly rich and better-specified probabilistic descriptions.This need arises because correctly describing data-generating processes is achievable only in the simplest settings.
  • Motivation: Normalizing flows push a simple density through successive transformations to produce richer, potentially multimodal distributions.Repeated simple transformations can yield highly complex models suitable for modeling, inference, and simulation.
  • Scope and perspective: The paper provides a unifying, tutorial-oriented perspective on an active research area lacking a common lens for relating recent advances to earlier work.It complements existing treatments with broader coverage and deeper discussion of topics such as discrete variables and Riemannian manifolds.
  • Organization: The review develops formal foundations, discusses finite and infinitesimal flow construction, presents a general probability-transformation perspective, and surveys applications.Applications are discussed in Section 6, following the foundational and construction-focused sections.

2. Normalizing Flows

A normalizing flow constructs a flexible distribution by transforming samples from a base distribution through an invertible, differentiable map. Its change-of-variables formulation supports both sampling and exact density evaluation, while composition enables complex transformations and implementation choices trade off these operations.

  • Definition: A flow expresses a D-dimensional variable x as an invertible transformation T of a sample u from a base density p_u(u).The transformation and base distribution can each have parameters, inducing a family of distributions over x.
  • Definition: The defining requirement is that T and T^-1 are differentiable, making T a diffeomorphism and ensuring a well-defined transformed density.The transformation must preserve dimensionality between the base and target spaces.
  • Density evaluation: The density is computed by change of variables as p_x(x) = p_u(u)|det J_T(u)|^-1, with u = T^-1(x).The Jacobian determinant accounts for the local volume change induced by T.
  • Composition: Composing invertible differentiable transformations produces complex flows while retaining invertibility, differentiability, and density computability.A typical flow chains T_1 through T_K, transforming z_0 = u into z_K = x.
  • Computational trade-offs: Sampling requires the forward map T, whereas density evaluation requires T^-1, its Jacobian determinant, and the base density.Applications determine which direction and operation must be implemented efficiently.

2.2 Expressive Power of Flow-Based Models

The review shows constructively that flows can represent broad classes of target distributions from suitable base distributions. The construction uses conditional cumulative distribution functions to build a differentiable, invertible map to a uniform distribution, then composes transformations to reach the target.

  • Universal representation: For well-behaved target and base distributions, there exists a diffeomorphism that transforms the base distribution into the target distribution.The argument is constructive and establishes universal representation under stated regularity conditions.
  • Assumptions: The construction assumes p_x(x) is positive everywhere and conditional probabilities are differentiable with respect to the relevant variables.These conditions ensure the conditional cumulative-distribution construction is differentiable and invertible.
  • Construction: The map F sends each coordinate to the conditional cumulative distribution function, producing z in the open unit cube (0, 1)^D.Its inverse is obtained element by element using inverse conditional cumulative distribution functions.
  • Construction: Because the Jacobian of F is lower triangular with positive diagonal entries, its determinant is nonzero and F is a diffeomorphism.The triangular structure follows because each transformed coordinate depends only on preceding coordinates and itself.
  • General base distributions: Any suitable base distribution can be handled by first mapping it to uniform z and then composing T = F^-1 ◦ G to obtain the target distribution.Thus, universal representation is not restricted to a uniform base.

2.3 Using Flows for Modeling and Inference

Flows can be fitted with divergences chosen according to which densities, samples, transformations, and inverses are available. Forward and reverse KL formulations connect maximum likelihood, variational inference, and model distillation through complementary computational requirements.

  • Forward KL Divergence and Maximum Likelihood Estimation: Forward KL fitting is equivalent to maximum-likelihood estimation on samples from the target distribution.The Monte Carlo objective uses the inverse flow, its Jacobian determinant, and the base density.
  • Reverse KL Divergence: Reverse KL fitting is suitable when the target density can be evaluated, even if target samples are unavailable or the density is known only up to a constant.Training requires sampling from the base and computing the forward transformation and its Jacobian determinant.
  • Applications: Reverse KL is used in variational inference, where the target is a posterior formed from a likelihood and prior.The review also identifies model distillation as another reverse-KL application when the target density is evaluable but sampling is inconvenient.
  • Relationship Between Forward and Reverse KL Divergence: Forward and reverse KL objectives are equivalent under a change of variables: fitting the model to the target corresponds to fitting the induced target distribution to the base with the opposite KL direction.The paper presents both equivalences as alternative perspectives on the same flow model.
  • Alternative Divergences: Flow learning also supports f-divergences and integral probability metrics, with the appropriate choice depending on simulation, density, and transformation access.These alternatives connect flow-based models to the broader class of implicit probabilistic models.

2.4 Brief Historical Overview

Normalizing flows emerged from earlier probability transformations and density-estimation ideas, then gained expressive deep architectures and broader conceptual connections. Their history includes whitening, Gaussianization, composition, change of measure, copulas, and optimal transport.

  • Early Precursors: Whitening transformations provided an early precursor by converting data into white noise and later supporting density estimation through Gaussianization.Gaussianization treated whitening as density estimation rather than feature preprocessing.
  • Deep Compositions: Deep composition made flow parameterizations more expressive, while subsequent architectures improved scalability and computational efficiency for image modeling and inference.Rippel and Adams highlighted expressive distribution classes; Dinh et al. introduced a scalable architecture.
  • Broader Connections: Change-of-measure theory connects normalizing flows to statistical mechanics and other probability transformations.Liouville’s theorem is cited as a historical example from statistical mechanics.
  • Optimal Transport: Triangular maps connect autoregressive flows to optimal transport, where they arise as limiting solutions to a class of Monge–Kantorovich problems.The review places these maps within a longer history beginning with Rosenblatt’s work.

3. Constructing Flows Part I: Finite Compositions

Finite-composition flows build expressive transformations from simple invertible components with tractable Jacobian determinants. Increasing the number of components raises complexity linearly in depth, while inverse efficiency and Jacobian computation determine practical usability.

  • Composition: A flow composes simple transformations T_k into a transformation T, with forward evaluation proceeding from z_0 = u to z_K = x.The inverse applies the component inverses in reverse order.
  • Composition: Simple building blocks yield greater expressive power while localizing forward, inverse, and Jacobian-determinant computations to sub-flows.This modularity is the central computational rationale for finite compositions.
  • Computational Trade-offs: O(K) computational growth accompanies increasing depth, making additional composed transformations a practical route to greater expressivity.The stated complexity is with respect to the number K of composed sub-flows.
  • Design Requirements: Each component must be invertible with a tractable Jacobian determinant, and the chosen direction of implementation depends on intended usage.Sampling uses the forward transformation, whereas density evaluation uses the inverse and its Jacobian determinant.
  • Design Requirements: For large input dimension D, flow constructions generally target Jacobian-determinant computation that is at most O(D).Direct determinant computation can be intractable when the input dimension is large.

3.1 Autoregressive Flows

Autoregressive flows use monotonic, conditionally parameterized transformations with triangular Jacobians, making them invertible and enabling tractable determinant computation. Their main design trade-off is between expressive power and computational symmetry: masked autoregressive flows are universal approximators but asymmetric, whereas coupling layers are symmetric but require composition for comparable expressivity.

  • Autoregressive construction: Each autoregressive transformation applies a strictly monotonic transformer to z_i, with parameters produced by a conditioner using only earlier dimensions z_{<i}.The transformer is invertible, while the conditioner need not be bijective.
  • Autoregressive construction: Autoregressive inversion proceeds iteratively because each conditioner requires previously recovered dimensions, although forward outputs can be computed independently or in parallel.This creates a directional computational asymmetry between the forward and inverse transformations.
  • Jacobian structure: The Jacobian is lower triangular, so its log-absolute-determinant is computed from diagonal transformer derivatives in O(D) time.The lower-triangular entries do not affect the determinant.
  • Expressivity: Autoregressive flows are universal approximators when the conditioner and transformer are sufficiently flexible, but this representational guarantee does not ensure good practical behavior.The result follows from the autoregressive construction based on conditional cumulative distribution functions.
  • Masked autoregressive flows: Masked autoregressive flows evaluate efficiently in one conditioner pass with parallel dimension-wise transformations, yet exact inversion requires D sequential conditioner calls and can be prohibitively expensive for high-dimensional data.A Newton-like approximate inversion can be more efficient when it converges in substantially fewer than D iterations, but convergence is guaranteed only locally.
  • Coupling layers: Coupling layers make sampling and density evaluation computationally symmetric, but a single layer is not universal; D suitably ordered layers recover universality while requiring D sequential computations.It remains open whether strictly fewer than O(D) coupling layers can achieve universal approximation.

3.2 Linear Flows

Linear flows generalize permutations through invertible matrix transformations, but direct parameterization creates invertibility, inversion-cost, determinant-cost, and expressivity trade-offs.

  • Interaction with autoregressive flows: Autoregressive restrictions make flow behavior depend on input ordering at finite capacity, although infinite-capacity flexibility is not limited by that ordering.Permuting variables between layers helps address these practical limitations.
  • Linear flows: Linear flows apply an invertible matrix transformation, with Jacobian determinant det W; permutations are a special case.Alternating linear transformations with autoregressive or coupling layers is used in practice.
  • Computational trade-offs: Directly learning W does not guarantee invertibility, while inversion and determinant computation are costly for high-dimensional data.The cited passages identify these as separate practical problems with unconstrained matrix parameterization.
  • Structured parameterizations: Restricting W to structured matrices can guarantee invertibility and reduce computational costs; triangular matrices are one example.Positive diagonal elements can ensure triangular-matrix invertibility.
  • Representational limitation: Continuous parameterizations that guarantee invertibility cannot cover both determinant-sign components of the invertible-matrix set.The positive- and negative-determinant regions are disconnected by singular matrices.

3.3 Residual Flows

Residual flows use transformations of the form z′ = z + gφ(z), gaining dense interactions and flexibility when invertibility is enforced, but trading computation against expressivity.

  • Residual-flow construction: Residual flows compose transformations z′ = z + gφ(z), whose invertibility can be enforced through contractive maps or determinant-lemma constructions.The review presents both approaches as general designs for invertible residual transformations.
  • Contractive residual flows: Contractivity guarantees invertibility and enables iterative inversion, with convergence becoming faster as the Lipschitz constant decreases.The Banach fixed-point theorem supplies uniqueness and convergence of the iterative inverse.
  • Contractive residual flows: A smaller Lipschitz constant improves inversion efficiency but constrains the residual transformation and reduces its flexibility.At L = 0, inversion takes one iteration and the transformation reduces to adding a constant.
  • Density estimation: Power-series and trace-estimation methods provide unbiased or approximate log-determinant estimates for training, while determinant evaluation remains difficult in high dimensions.The Hutchinson estimator and Russian-roulette estimator are used to estimate the required quantities.
  • Computational trade-offs: Contractive residual flows have dense Jacobians, allowing all inputs to affect all outputs, but exact density evaluation and sampling are computationally expensive.This contrasts with one-pass density evaluation and sampling in coupling-based flows.
  • Matrix determinant lemma: Planar, Sylvester, and radial flows compute Jacobian determinants in O(D) and can be made invertible, but lack analytical inverses and use relatively simple individual transformations.These properties explain their predominant use for approximate posteriors in variational autoencoders.

3.4 Practical Considerations when Combining Transformations

Deep flow compositions require stabilization and computational strategies, including normalization and multi-scale architectures that reduce cost while organizing information across dimensions.

  • Practical motivation: Deep flows can require hundreds of sub-transformations, creating practical optimization and computational challenges.Glow is cited as using 320 sub-transformations distributed across 40 GPUs.
  • Normalization: Batch normalization can stabilize gradients and be inserted between flow transformations because its Jacobian determinant is easy to compute.Its elementwise action gives it a diagonal Jacobian.
  • Normalization: During training, batch normalization is not invertible unless batch statistics are cached, and its usual determinant expression is not mathematically valid for whole-batch dependence.Activation normalization avoids batch statistics through data-dependent initialization followed by parameter optimization.
  • Multi-scale architectures: Multi-scale architectures reduce the increasing cost of preserving dimensionality through every flow step by factoring out dimensions at regular intervals.The approach addresses the tension between dimensionality-dependent cost and using many transformations.
  • Multi-scale architectures: Multi-scale flows align with granular data and encode more global semantic information in dimensions transformed throughout the flow, while earlier-factored dimensions represent lower-level information.The review discusses pixels, waveforms, image shapes, and textures as examples.

4. Constructing Flows Part II: Continuous-Time Transformations

Continuous-time flows define transformations through ODE dynamics, enabling symmetric forward and inverse computation and adaptive numerical implementation while introducing solver and trace-estimation costs.

  • 4.1 Definition: Continuous-time flows parameterize infinitesimal dynamics with an ODE and integrate those dynamics from a base state u at t0 to x at t1.The vector field gφ takes time and the current state as inputs and outputs the state derivative.
  • 4.1 Definition: Uniform Lipschitz continuity in the state and continuity in time ensure a unique ODE solution under the stated conditions.Many neural-network layers satisfy these requirements.
  • Inversion: Continuous-time flows have the same computational complexity in both directions, so choosing the forward direction is less consequential than in autoregressive flows.The inverse is obtained by reversing integration limits.
  • Log-density computation: Log-density change uses the trace of the vector-field Jacobian, but exact trace computation requires O(D) backpropagation passes; Hutchinson estimation is about D times more efficient.The estimator computes a Jacobian-vector product in a single backpropagation pass.
  • Density evaluation: Forward transformation and log-density evaluation can be computed through numerical integration, but general dynamics are not analytically tractable.The review describes combined integrals and the need for numerical integrators.
  • Numerical solvers: Euler discretization connects continuous-time flows to contractive residual flows when the step size satisfies ε < 1/L.The approximation becomes exact as ε approaches zero.
  • Adjoint method: The adjoint sensitivity method expresses gradient computation as a separate ODE, avoiding backpropagation through the solver’s computational graph.This reduces computational and memory demands and permits black-box solvers with instance-dependent tolerances.

5. Generalizations

The review generalizes normalizing flows through a conservation-of-probability framework, covering piecewise-invertible, discrete, manifold, and symmetry-aware transformations. These extensions broaden the standard Euclidean, bijective formulation while exposing computational and topological constraints.

  • General probability transformations: A general probability-transformation relation unifies standard flows with transformations between broader spaces, sets, and integration measures.Standard flows arise when the transformation is a diffeomorphism on R^D with Lebesgue measure; the density formula then follows from change of variables.
  • Piecewise-invertible transformations and mixtures of flows: Piecewise-invertible transformations induce mixtures of flows, with each component restricted to a subset and weighted by its base-distribution probability.The many-to-one construction partitions U into subsets whose restrictions are invertible.
  • Piecewise-invertible transformations and mixtures of flows: Real and discrete transformations provide non-overlapping mixture components by mapping one base sample to multiple possible outputs and selecting among them.Each component maps U into a distinct subset of X, with conditional base distributions and mixture weights determined by the selection probabilities.
  • Discrete flows: Discrete flows use counting measure and bijections over discrete sets, so their density calculation does not include a Jacobian term.Affine autoregressive constructions implement discrete transformations using rounding or argmax, with straight-through gradient estimators for backpropagation.
  • Bypassing topological constraints: Continuous-time flows cannot represent transformations that require topology-changing intersections, while lifted-space augmentation broadens representable functions but removes analytic density evaluation.The auxiliary variables introduced by augmentation must be numerically integrated in the original space.
  • Symmetric densities and equivariant flows: Flows with equivariant transformations and invariant base densities produce densities invariant to a prescribed symmetry group, including rotations, reflections, and axis permutations.Gradients of invariant functions provide a general way to construct equivariant transformations for orthogonal group representations.

6. Applications

Normalizing flows support probabilistic modeling and inference because they combine expressive distributions with tractable density calculation and sampling. The review covers density estimation, generation across data types, importance and rejection sampling, MCMC, and flexible approximate posteriors.

  • Overview: Flows provide density calculation and sampling, making them applicable to probabilistic modeling, inference, supervised learning, and reinforcement learning.Their applications are organized around the two primitive operations: density calculation and sampling.
  • Probabilistic modeling: For probabilistic modeling, flows fit an approximation to an unknown data-generating distribution, commonly using maximum likelihood and supporting density estimation or generation.Maximum likelihood exploits the forward KL divergence, while other divergences and integral probability metrics can also be used.
  • Probabilistic modeling: Continuous flows are not directly applicable to discrete data, so discrete observations are often dequantized by adding fixed or learned continuous noise.Variational dequantization is one example of learning the noise distribution.
  • Generation: Flows have been applied to generation of images, video, audio, text, graphs, molecules, point clouds, and motion-related structured objects.Applications include raw RGB video synthesis, efficient audio sampling, discrete character-level language modeling, and structured-object generation.
  • Inference: Importance sampling and rejection sampling use flows as proposals because both require tractable sampling and density evaluation.Flow parameters can be optimized through divergences or by directly minimizing the variance of the importance-sampling estimator.
  • Inference: Flows reparameterize unnormalized-target sampling for MCMC and provide flexible approximate posteriors that are readily reparameterizable by design.A sufficiently powerful transformation can make exploration easier by mapping the target toward a simpler base distribution, although Hamiltonian proposals may remain useful.

7. Conclusions

The review synthesizes normalizing flows for modeling and inference, emphasizing expressive power, construction principles, generalized probability transformations, and applications. It argues that foundational principles remain useful despite changing implementations, while tractable flexible transformations and finite-sample theory remain open challenges.

  • The review covers flow-based modeling and inference, including expressive power, discrete and continuous-time construction, generalized probability transformations, and applications from density estimation to classification.
  • Foundational principles, including the probability chain rule and triangular Jacobians, underlie autoregressive flows’ expressive power and efficient implementation.
  • Autoregressive flows suit applications prioritizing either sampling or density evaluation, whereas coupling-based conditioners make both operations efficient at a cost in expressive power.
  • Normalizing flows typically provide analytical calculations and exact sampling in high dimensions, but flexibility must be balanced against tractable transformation construction.
  • Finite-sample and finite-depth approximation capabilities require further study to help practitioners select appropriate flow classes.

Appendix A. Proof of KL Dualities

This appendix introduces distributions induced by a flow and its inverse, then invokes the flow density formula and change of variables to state the corresponding dual relationships.

  • The appendix defines p_x(x) as the distribution induced by transformation T and base distribution p_u(u).
  • It likewise defines p*_u(u) through the inverse transformation T^-1 and base distribution p*_x(x).
  • The stated dualities follow by combining the flow-based density formula with a change-of-variables argument.

Appendix B. Constructing Linear Flows

The appendix constructs linear flows through structured matrix decompositions and orthogonal parameterizations that preserve invertibility while making determinants and inverses tractable. These choices involve trade-offs among computational cost, coverage, optimization, and expressive interaction.

  • QR flows: QR flows decompose matrices into orthogonal and upper-triangular factors, enabling efficient determinant computation and inversion through orthogonal multiplication and triangular solves.
  • PLU flows: PLU flows decompose W into permutation, lower-triangular, and upper-triangular matrices, with positive diagonals ensuring invertibility.
  • PLU flows: PLU determinants and inverses are computed through structured triangular operations, while P is typically fixed and L and U are learned.
  • PLU flows: PLU multiplication corresponds to composing two autoregressive affine flows with opposite order, followed by a permutation.
  • Orthogonal flows: Orthogonal flows are volume preserving and trivially invertible, generalize permutations, and can compose with other structured matrices.
  • Orthogonal parameterizations: Exponential and Cayley parameterizations have dimensional or coverage restrictions, while Householder products cost O(KD) and may create optimization saddle points.
Loading 1912.02762v2…