Source-linked AI summary

Normalizing Flows for Probabilistic Modeling and Inference

George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, Balaji Lakshminarayanan

arXiv:1912.02762v2stat.MLcs.LG

TL;DR

Normalizing flows need a unified perspective for building richer probabilistic models and applying them across statistical tasks. This review organizes flow design, expressive power, computational trade-offs, probability transformations, and applications, showing that flows can represent broad distribution classes under stated conditions while imposing practical and topological constraints.

  • Problem

    Developing richer, well-specified probabilistic models remains difficult, motivating unified tools and principles for normalizing-flow construction and application.

    Source: §1 · pp. 1–2
  • Method

    The review frames normalizing flows through probabilistic modeling and inference, covering finite and infinitesimal constructions, general probability transformations, and applications.

    Source: §1 · p. 2
  • Results

    Under stated conditions, flow-based models and autoregressive variants can represent any target distribution, while coupling layers provide computationally symmetric evaluation and inversion.

    Source: §2.2 · p. 5, §3.1 · p. 13, §3.1.2 · pp. 18–19
  • Takeaways & Limitations

    Flow design should match computational priorities: autoregressive flows suit one-sided speed requirements, whereas coupling-based implementations suit frequent fast sampling and density evaluation.

    Source: §7 · p. 51

    Continuous-time flows cannot realize transformations that violate the required topological equivalence of intermediate spaces without augmentation.

    Source: §5.5 · p. 40

Abstract

from arXiv · show

Normalizing flows provide a general mechanism for defining expressive probability distributions, only requiring the specification of a (usually simple) base distribution and a series of bijective transformations. There has been much recent work on normalizing flows, ranging from improving their expressive power to expanding their application. We believe the field has now matured and is in need of a unified perspective. In this review, we attempt to provide such a perspective by describing flows through the lens of probabilistic modeling and inference. We place special emphasis on the fundamental principles of flow design, and discuss foundational topics such as expressive power and computational trade-offs. We also broaden the conceptual framing of flows by relating them to more general probability transformations. Lastly, we summarize the use of flows for tasks such as generative modeling, approximate inference, and supervised learning.

1. Introduction

The review presents normalizing flows as a tool for constructing increasingly rich probabilistic models and develops a unified, tutorial perspective on their principles, extensions, and applications. It covers formal foundations, finite and infinitesimal constructions, broader probability transformations, structured domains, and common applications.

Motivation

  • The review addresses the need for tools that enable richer probabilistic descriptions and better-specified models beyond simple statistical settings.Source: §1 · p. 1
  • Normalizing flows transform a simple density through successive bijections into richer, potentially multimodal distributions suitable for modeling, inference, and simulation.Repeated simple transformations can produce highly complex distributions.Source: §1 · pp. 1–2

Paper structure

  • The paper develops formal foundations, finite and infinitesimal flow constructions, general probability transformations, extensions to structured domains and geometries, and applications.These topics correspond to Sections 2–6 of the review.Source: §1 · p. 2

2. Normalizing Flows · 2.1 Definition and Basics

Normalizing flows construct flexible continuous distributions by transforming samples from a base distribution through differentiable, invertible maps. Their change-of-variables density evaluation, compositional structure, and distinct sampling and inference operations define the basic framework.

2. Normalizing Flows

  • This section introduces normalizing flows, their basic properties, practical construction, expressive power, and historical background for readers without prior familiarity.It frames the material as an introduction to the field.Source: §2 · p. 2

2.1 Definition and Basics

  • Normalizing flows construct flexible distributions over continuous variables by representing x as an invertible transformation T(u) of a sample u from a base distribution.The base distribution and transformation may each have parameters, inducing a parameterized family over x.Source: §2.1 · p. 3
  • Diffeomorphic transformations make the model density well-defined through the change-of-variables formula, using the inverse map and the absolute Jacobian determinant.Specifically, px(x) = pu(u) |det JT(u)|^-1 where u = T^-1(x).Source: §2.1 · p. 3
  • In practice, implementations often parameterize T or T^-1 with a neural network and choose a simple base density such as a multivariate normal.The inverse flow maps observed samples back to the prescribed base density, explaining the terms “flow” and “normalizing.”Source: §2.1 · pp. 3–4
  • The Jacobian determinant measures local volume change: expansion lowers transformed density, whereas contraction raises it while preserving probability mass.The determinant approximately equals Vol(dx)/Vol(du) for corresponding infinitesimal neighborhoods.Source: §2.1 · p. 3
  • Because invertible differentiable transformations compose, flows can chain simple maps into complex transformations while retaining invertibility, differentiability, and tractable density calculation.A flow typically uses T = T_K ◦ · · · ◦ T_1, progressively transforming z_0 = u into z_K = x.Source: §2.1 · pp. 3–4
  • A flow-based model supports sampling and density evaluation, but sampling uses the base sampler and forward map, whereas density evaluation uses the inverse map, Jacobian determinant, and base density.The application determines which operation must be implemented and optimized.Source: §2.1 · p. 4

2.2 Expressive Power of Flow-Based Models

Under reasonable smoothness and positivity conditions, flow-based models can represent any target distribution from any similarly well-behaved base distribution through a diffeomorphic transformation. The constructive proof maps both distributions through conditional cumulative transforms to a uniform distribution, then composes the resulting bijections.

Universal representation

  • Flow-based models universally represent any well-behaved target distribution from any well-behaved base distribution via a diffeomorphism.The construction first maps the base to a uniform distribution and then maps that uniform distribution to the target.Source: §2.2 · pp. 4–5

Constructive proof

  • The target-to-uniform transformation uses conditional cumulative distribution functions, yielding a differentiable invertible map under positive conditional densities.The map is inverted element-by-element because each conditional CDF is strictly increasing; its Jacobian is lower triangular with a nonzero determinant.Source: §2.2 · p. 5
  • The Jacobian structure establishes that the conditional-CDF map is a diffeomorphism and transforms the target distribution into the uniform distribution on (0, 1)^D.Lower triangularity makes the determinant the product of diagonal elements, which is nonzero everywhere under the positivity assumption.Source: §2.2 · p. 5

Universal representation

  • Composing the base-to-uniform map G with the inverse target-to-uniform map F^-1 gives T = F^-1 ◦ G, which transforms the base distribution into the target.The same diffeomorphic construction applies to any base distribution satisfying the stated conditions.Source: §2.2 · p. 5

2.3 Using Flows for Modeling and Inference

Flow-based models can be fitted by minimizing divergences, with forward KL supporting maximum-likelihood learning from samples and reverse KL supporting density-based fitting without target samples. These objectives are equivalent under a change-of-variables view, while alternative f-divergences and integral probability metrics broaden training options.

2.3 Using Flows for Modeling and Inference

  • Flow models are fitted by optimizing parameters θ={φ, ψ} to minimize a divergence or discrepancy between the model and target distributions, especially KL divergence.The parameters φ describe the transformation T, while ψ describes the base density p_u(u; ψ).Source: §2.3 · p. 6

2.3.1 Forward KL Divergence and Maximum Likelihood Estimation

  • Forward KL is suitable when target samples are available even if the target density cannot be evaluated, and its Monte Carlo objective is equivalent to maximum-likelihood estimation.The objective uses log p_u(T^-1(x_n; φ); ψ) plus the inverse-Jacobian determinant term, and optimization requires T^-1, its Jacobian determinant, and p_u, but not T or base sampling until model sampling.Source: §2.3.1 · p. 6

2.3.2 Reverse KL Divergence

  • Reverse KL is suitable when the target density can be evaluated, even up to a multiplicative constant, without requiring target samples.It requires sampling from and differentiating through the base distribution and transformation, but not evaluating the base density or computing T^-1; it is used in variational inference and model distillation.Source: §2.3.2 · pp. 7–8

2.3.3 Relationship Between Forward and Reverse KL Divergence

  • Under a change of variables, forward-KL maximum likelihood on the target is equivalent to reverse-KL fitting of the induced inverse-flow distribution to the base, and vice versa.The target and base uniquely determine each other given the transformation, so fitting either side yields the corresponding fit on the other.Source: §2.3.3 · p. 8

2.3.4 Alternative Divergences

  • Alternative objectives include f-divergences and integral probability metrics, whose applicability depends on available sampling, density evaluation, and transformation operations.Choosing a normalizing flow as a generative-adversarial-network generator enables adversarial, Wasserstein, maximum-mean-discrepancy, and other training approaches (Grover et al., 2018; Danihelka et al., 2017; Arjovsky et al., 2017; Bi´nkowski et al., 2018).Source: §2.3.4 · pp. 8–9

2.4 Brief Historical Overview

Normalizing flows grew from whitening and Gaussianization into compositions of simple maps, establishing expressive yet tractable transformations. Their development also connects to change-of-measure methods, copulas, and optimal transport.

Early predecessors

  • Whitening transformations preceded Gaussianization by Chen and Gopinath (2000), while Tabak and Vanden-Eijnden (2010) connected it to diffusion processes and statistical mechanics.Tabak and Turner (2013) introduced the term normalizing flow and defined it as a composition of K simple maps.Source: §2.4 · p. 9

Flow formulation

  • The composition principle enabled expressive normalizing flows while preserving computational and analytical tractability.Figure 2 illustrates a flow composed of K transformations.Source: §2.4 · p. 9, Figure 2 · p. 10

Machine-learning development

Broader probability transformations

  • Normalizing flows also relate to change-of-measure methods, including Liouville’s theorem and copulas, where dimensions are transformed independently through empirical marginal cumulative distribution functions.These connections place modern flows within a broader pre-history of probability transformations.Source: §2.4 · p. 9

Optimal transport

  • Optimal transport formulates measure transformations through the Monge problem, with triangular maps related to autoregressive flows and arising as limiting solutions to certain Monge–Kantorovich problems.Rosenblatt (1952) studied triangular maps for transforming multivariate distributions uniformly over the hypercube, while the review largely sidesteps this framework.Source: §2.4 · p. 10

3. Constructing Flows Part I: Finite Compositions

Finite compositions construct expressive normalizing flows by chaining simple invertible transformations with tractable Jacobians. Their computational cost grows linearly with composition depth, while inverse efficiency and Jacobian tractability determine practical sampling and density evaluation.

  • Flows compose simple transformations with tractable inverses and Jacobian determinants to obtain more expressive transformations while localizing computation to sub-flows.Forward evaluation applies T_k sequentially, while inverse evaluation applies the component inverses in reverse order.Source: §3 · p. 10
  • Increasing composition depth K increases computational complexity by only O(K), making additional expressivity practically affordable.Source: §3 · p. 11
  • Each component f_φ must be invertible with a tractable Jacobian determinant, but guaranteed invertibility does not ensure an efficiently computable inverse.An inefficient inverse can make sampling or density evaluation inefficient or intractable, depending on whether the component implements T_k or T_k^-1.Source: §3 · p. 11
  • Whether to design an efficient inverse and whether f_φ implements T_k or T_k^-1 should depend on the intended use of the flow.Forward transformations are used for sampling, whereas inverse transformations are used for density evaluation.Source: §3 · p. 11
  • For D-dimensional inputs, flow components should enable Jacobian-determinant computation in at most O(D) time rather than explicitly forming costly determinants.Source: §3 · p. 12

3.1 Autoregressive Flows

This section presents autoregressive flows as triangular, invertible transformations with tractable Jacobians, covering their approximation properties and the trade-offs among conditioner architectures, expressivity, efficiency, and inversion speed. It also examines coupling-layer implementations and situates continuous-variable autoregressive models as special cases within the broader compositional framework of autoregressive flows.

3.1 Autoregressive Flows

  • Autoregressive flows use monotonic transformers whose parameters are produced by conditioners from preceding variables, making the transformation invertible and its Jacobian triangular.Forward outputs can be computed independently or in parallel, whereas inversion requires sequentially recovering variables; the log-absolute-determinant is computable in O(D) time.Source: §3.1 · pp. 12–13
  • With sufficiently flexible transformers and conditioners, autoregressive flows are universal approximators, although this representational result does not guarantee practical behavior.This follows because the universal conditional-CDF transformation is itself an autoregressive flow.Source: §3.1 · p. 13

3.1.1 Implementing the Transformer

  • Affine transformers are simple and analytically tractable but limited because one layer preserves Gaussian conditional structure, while stacked layers are expressive but not known to be universal.Affine transformers appear in NICE (Dinh et al., 2015), Real NVP (Dinh et al., 2017), IAF, MAF (Papamakarios et al., 2017), and Glow.Source: §3.1.1 · p. 14
  • Combination-based transformers can approximate any monotonic function, but generally require iterative inversion such as bisection search.They construct monotonic functions through positive combinations and compositions, and variants include NAF (Huang et al., 2018), block-NAF (De Cao et al., 2019), and Flow++ (Ho et al., 2019).Source: §3.1.1 · pp. 14–15
  • Integration-based transformers obtain flexible monotonic maps by integrating positive functions, while positive-polynomial integrands provide analytically tractable sum-of-squares variants.General integrals may require numerical approximation; spline-based transformers can instead become arbitrarily flexible as the number of segments K increases.Source: §3.1.1 · pp. 15, 17

3.1.2 Implementing the Conditioner

  • Naïve independent conditioners scale poorly with dimensionality, but shared-parameter implementations such as RNNs and masked networks make autoregressive conditioning practical.RNN conditioners require O(D) sequential steps, whereas masked conditioners produce all parameters in one pass while preserving the autoregressive dependency structure.Source: §3.1.2 · pp. 17–18
  • Masked autoregressive flows evaluate efficiently in one conditioner pass, compute transformed dimensions in parallel, and are universal approximators with sufficiently flexible components.Their inversion is sequential because each conditioner output depends on previously recovered variables.Source: §3.1.2 · p. 18
  • Exact inversion of masked autoregressive flows requires D conditioner calls and is about D times more expensive than forward evaluation, while Newton-style inversion is approximate and only locally guaranteed to converge.The approximate procedure can be faster when substantially fewer than D iterations suffice, but it may diverge.Source: §3.1.2 · p. 19
  • Intermediate coupling designs trade inversion cost against expressivity by splitting the input into K parts, with inversion O(K) times more expensive than evaluation.K = 2 gives a coupling layer, whereas K = D gives a fully autoregressive flow.Source: §3.1.2 · p. 21
  • Composing coupling layers increases expressivity, and D layers with progressively shifted partitions are theoretically universal approximators, though they require D sequential computations.A single coupling layer is not a universal approximator; permutations between layers let all dimensions interact and be transformed.Source: §3.1.2 · p. 21
  • Coupling layers enable fast density evaluation and sampling, making them popular in tractable maximum-likelihood flow models for high-dimensional images, audio, and video.Examples include NICE, Real NVP, Glow, WaveGlow, FloWaveNet, and Flow++.Source: §3.1.2 · p. 21

3.1.3 Relationship with Autoregressive Models

  • Every autoregressive model of continuous variables is an autoregressive flow whose transformers are conditional CDFs and whose base distribution is uniform on (0, 1)^D.Sampling the flow is exactly sequential conditional sampling via inverse transform sampling.Source: §3.1.3 · pp. 22–23
  • Autoregressive conditionals map to flow transformers: Gaussian models yield affine flows with a standard Gaussian base, while Gaussian mixtures and histograms yield combination-based and linear-spline transformers.Viewing autoregressive models as flows permits flexible base distributions and composition with other autoregressive or non-autoregressive flows.Source: §3.1.3 · p. 23

3.2 Linear Flows

Linear flows generalize permutations with invertible matrix transformations, improving mixing between autoregressive or coupling layers while introducing inversion and determinant-computation challenges. Practical parameterizations therefore restrict matrices structurally, but continuous invertible parameterizations must omit one determinant-sign component.

Motivation

  • Input ordering limits finite-capacity autoregressive and coupling flows, while coupling layers require permutations to ensure every variable is eventually transformed.Permutations are volume-preserving and have unit absolute Jacobian determinant, so they compose seamlessly with other invertible differentiable transformations.Source: §3.2 · pp. 23–24

Linear flows

  • A linear flow applies an invertible D×D matrix W, with Jacobian determinant det W; permutations are the special case where W is a permutation matrix.Alternating invertible linear transformations with autoregressive or coupling layers is used in practice (Durkan et al., 2019b).Source: §3.2 · p. 24

Computational challenges

  • Directly learning W risks non-invertibility and makes inversion and determinant computation prohibitively expensive in high dimensions.Inversion requires solving Wz = z′, while determinant computation is also costly in general.Source: §3.2 · p. 24

Structured parameterizations

  • Structured matrix parameterizations address these costs: triangular W can guarantee invertibility with positive diagonals, while inversion and determinant evaluation become efficient.For triangular matrices, inversion costs about the same as matrix multiplication, and determinant computation costs O(D).Source: §3.2 · p. 24

Parameterization limits

  • No continuous parameterization can cover all invertible D×D matrices, because positive- and negative-determinant matrices form disconnected components separated by singular matrices.Consequently, any continuous invertibility-guaranteeing parameterization must fix the determinant sign and omit one component.Source: §3.2 · p. 24

3.3 Residual Flows

Residual flows use transformations of the form z′ = z + gφ(z), resembling residual networks, and achieve invertibility through either contractive maps or matrix-determinant-lemma constructions. These approaches trade expressive, dense transformations against inversion and Jacobian-determinant computation costs.

3.3.1 Contractive Residual Flows

  • Contractive residual flows are invertible when gφ is contractive, with inversion obtained by fixed-point iteration that converges exponentially but becomes slower as flexibility increases.The Lipschitz constant L controls the trade-off: smaller L accelerates convergence but constrains the transformation, with L = 0 reducing it to adding a constant.Source: §3.3.1 · pp. 25–26
  • Contractive residual flows generally have dense Jacobians and can be flexible, but exact density evaluation is costly and sampling is iterative because no general efficient Jacobian-determinant procedure is known.Unbiased log-determinant estimates can support stochastic-gradient training, using power-series methods with Hutchinson and Russian-roulette estimators.Source: §3.3.1 · pp. 26–27

3.3.2 Residual Flows Based on the Matrix Determinant Lemma

  • Matrix-determinant-lemma residual flows design Jacobians as diagonal matrices plus low-rank updates, enabling efficient determinant computation while requiring parameter restrictions for invertibility.Planar flows use rank-1 updates, while Sylvester flows extend them to M hidden units; radial flows perform radial contraction or expansion around z0.Source: §3.3.2 · pp. 27–29
  • Planar, Sylvester, and radial flows have O(D) Jacobian-determinant computation, but no analytical inverse, limiting their use primarily to variational-autoencoder posterior approximation.Their individual transformations are relatively simple, so greater flexibility generally requires increasing the number of transformations.Source: §3.3.2 · pp. 28–29

3.4 Practical Considerations when Combining Transformations

Deep flows require practical techniques that stabilize optimization and reduce computational demands. Normalization addresses gradient stability, while multi-scale architectures reduce cost and organize representations across semantic levels.

Normalization

  • Batch normalization stabilizes gradients and can be inserted between sub-transformations because its elementwise Jacobian determinant is easy to compute.It can be treated as another composition member, Tk ◦ BN ◦ Tk−1.Source: §3.4 · p. 29
  • During training, batch normalization is not inherently invertible because batch statistics depend on the whole batch, although its determinant formula can approximate training gradients when batches are sufficiently large.Cached statistics restore invertibility for a forward pass.Source: §3.4 · p. 30
  • Act norm avoids noisy batch statistics by data-dependent initialization followed by learned scale and translation parameters, making it preferable for small mini-batches.A batch initializes the parameters so the transformed data has zero mean and unit variance.Source: §3.4 · p. 30

Multi-scale architectures

  • Multi-scale architectures reduce computation by clamping sub-dimensions at intervals, while skip-connections may distribute the objective across the flow’s depth.All K steps operate on only a subset of dimensions rather than the full representation.Source: §3.4 · p. 30
  • Multi-scale architectures encode more global semantic information in dimensions transformed throughout the flow, while earlier-factored dimensions represent lower-level information.This structure is especially natural for granular data such as pixels and waveforms.Source: §3.4 · p. 30

4. Constructing Flows Part II: Continuous-Time Transformations

Continuous-time flows define transformations by integrating an ODE parameterized through infinitesimal dynamics, rather than composing discrete steps. Their implementation relies on numerical ODE solvers, with Euler discretization connecting them to residual flows and adjoint sensitivity enabling memory-efficient optimization.

4.1 Definition

  • Continuous-time flows parameterize infinitesimal dynamics with an ODE and integrate from z_t0 = u to z_t1 = x, replacing discrete transformation compositions.The dynamics function g_φ must be uniformly Lipschitz in z_t and continuous in t, ensuring a unique solution without special invertibility or Jacobian-tractability architecture constraints.Source: §4 · pp. 30–31, §4.1 · p. 31
  • Forward and inverse continuous-time transformations have the same computational complexity because inversion reverses integration limits, unlike many discrete flows.Log-density changes are expressed through the Jacobian trace, but exact computation requires O(D) backpropagation passes; Hutchinson’s estimator reduces this to one pass approximately.Source: §4.1 · pp. 31–32

4.2 Solving and Optimizing Continuous-Time Flows

  • Numerical integration is required because general continuous-time flow integrals are not analytically feasible, and solver backpropagation can optimize the dynamics parameters.Euler and Runge–Kutta methods discretize the ODE, with gradients obtained by backpropagating through the resulting solver computation.Source: §4.1 · p. 32, §4.2 · p. 32, §4.2.1 · p. 32

4.2.1 Euler’s Method and Equivalence to Residual Flows

  • Euler discretization realizes a continuous-time flow as a contractive residual flow when the step size satisfies ϵ < 1/L.This equivalence follows from uniform Lipschitz continuity of g_φ and connects the continuous-time log-density evolution to residual-flow Jacobian expansions.Source: §4.2.1 · p. 33

4.2.2 The Adjoint Method

  • The adjoint sensitivity method treats gradient computation as a separate ODE, allowing black-box solvers without backpropagating through the solver’s computational graph.This reduces computation and memory costs while allowing instance-dependent solver effort controlled by tolerances that can be tuned at test time.Source: §4.2.2 · pp. 33–34

5. Generalizations

This section generalizes normalizing flows through a conservation-of-probability framework to non-Euclidean, discrete, many-to-one, and symmetry-constrained settings. It also develops equivariant transformations under orthogonal-representation symmetries while exposing expressivity, topological, and computational limitations.

5. Generalizations

  • The general framework extends flows beyond standard Euclidean invertible transformations and identifies broader probability transformations as a frontier for normalizing-flow research.The section explicitly surveys specific implementations that extend the standard definition and explores further transformation spaces, measures, and applications.Source: §5 · p. 34, §5.1 · p. 35

5.1 General Probability-Transformation Formula

  • The conservation-of-probability relation unifies general transformations T between spaces U and X, with standard flows recovered when T is a Euclidean diffeomorphism.The framework equates probability mass on any subset with the mass on its image, using the appropriate integration measures.Source: §5.1 · pp. 34–35

5.2 Piecewise-Invertible Transformations and Mixtures of Flows

  • Piecewise-invertible many-to-one transformations represent mixtures of flows, while one-to-many RAD transformations become mixtures with non-overlapping components after augmenting the input with an index.The RAD formulation permits selection probabilities p(i | u), producing component-specific base distributions and mixture weights.Source: §5.2 · pp. 35–36

5.3 Flows for Discrete Random Variables

  • Discrete flows use bijections over counting-measure spaces and therefore omit Jacobian terms, but can only permute base probabilities and may require autoregressive dependencies.A fully factorized base cannot represent every discrete target; extended spaces can restore factorization by rearranging the probability table into rank 1.Source: §5.3 · pp. 37–38

5.4 Flows on Riemannian Manifolds

  • Flows can operate on Riemannian manifolds through embedding-induced metrics and volume factors, but coordinate singularities arise when manifolds are not homeomorphic to Euclidean space.For Lie groups, exponential-map parameterization is analytic in some cases but generally has cubic cost in dimensionality D.Source: §5.4 · pp. 39–40

5.5 Bypassing Topological Constraints

  • Topological constraints require flow input, output, and intermediate spaces to be homeomorphic, motivating lifted or latent-variable mixtures that broaden mappings but lose analytic density evaluation.Auxiliary variables require numerical integration, while latent-variable flows generally rely on variational inference for a lower bound on log p_x(x).Source: §5.5 · pp. 40–41

5.6 Symmetric Densities and Equivariant Flows

  • Equivariant flows preserve a prescribed symmetry group when the transformation is equivariant and the base density is invariant, yielding an invariant target density.For orthogonal group representations, gradients of invariant functions provide equivariant transformations, supporting practical construction of symmetry-respecting flows.Source: §5.6 · p. 42
  • 5.6 Symmetric Densities and Equivariant Flows: Lemma 2 constructs equivariant transformations for orthogonal-representation symmetries, leveraging the easier construction of invariant functions such as functions of the ℓ2 norm.Because ∥Ru∥=∥u∥ for any orthogonal matrix R, any function of the ℓ2 norm is invariant under these symmetries.Source: §5.6 · p. 43

6. Applications

Normalizing flows support applications requiring tractable density calculation or sampling, spanning probabilistic modeling, inference, supervised learning, and reinforcement learning. Their flexibility enables density estimation, generation, posterior approximation, likelihood-free inference, classification, and policy modeling.

6. Applications

  • Flows provide tractable density calculation and sampling, making them applicable to probabilistic modeling, inference, supervised learning, and reinforcement learning.These are the two primitive operations underlying the applications summarized in this section.Source: §6 · p. 43

6.1 Probabilistic Modeling

  • In probabilistic modeling, flows fit unknown data distributions through likelihood-based or other divergence objectives, supporting density estimation and generation.Discrete data typically requires dequantization with fixed or learned noise before applying continuous flows.Source: §6.1 · pp. 43–44

6.1.1 Density Estimation

  • Flow-based density estimation has represented skewed and multimodal densities, detected image anomalies, and improved over MADE and Real NVP in systematic comparisons.Generation applications include images, video, audio, text, graphs, molecules, point clouds, and motion-related part models.Source: §6.1.1 · p. 44, §6.1.2 · pp. 44–45

6.2.1 Importance and Rejection Sampling

  • Flows make importance and rejection sampling attractive by providing both proposal-density evaluation and sampling, with proposals optimized using divergence or estimator-variance objectives.Real NVP coupling layers were used for fast proposal density evaluation and sampling in rejection sampling.Source: §6.2.1 · p. 46

6.2.2 Markov Chain Monte Carlo

  • In MCMC, flows construct symmetric volume-preserving proposals or reparameterize difficult targets into simpler base distributions to improve exploration.The reparameterization approach can substantially ease exploration when the inverse-transformed target follows a simpler base distribution, though Hamiltonian proposals may remain useful.Source: §6.2.2 · p. 47

6.2.3 Variational Inference

  • For variational inference, flows provide flexible, readily reparameterizable approximate posteriors for local or global variables, including conditional models that amortize inference across observations.Training maximizes the ELBO using the base distribution, transformed joint density, and Jacobian determinant.Source: §6.2.3 · pp. 47–48

6.2.4 Likelihood-Free Inference

  • For simulator-based models with intractable likelihoods, conditional flows approximate either the posterior p(η | x) or likelihood p(x | η) from simulated parameter-data pairs.The resulting models support downstream tasks requiring density evaluation or sampling.Source: §6.2.4 · pp. 48–49

6.3 Using Flows for Representation Learning

  • Beyond density modeling, flows serve as invertible feature extractors for classification and hybrid generative-predictive models, while conditional flows model continuous policies and intermediate states in reinforcement learning.Invertible ResNets achieved nearly the same ImageNet accuracy as non-invertible ResNets, and flow policies were used in maximum-entropy, soft actor-critic, and imitation-learning settings.Source: §6.3.1 · p. 49, §6.3.2 · p. 50

7. Conclusions

This review unifies normalizing flows for probabilistic modeling and inference, covering their expressive power, construction principles, generalized probability transformations, and applications. It also distills implementation trade-offs and identifies flexible, tractable transformations and theoretical approximation guarantees as continuing challenges.

Scope

  • The review unifies flow design around expressive power, discrete and continuous-time construction, generalized probability transformations, and applications from density estimation to likelihood-free inference and classification.The generalized perspective extends flows to discrete domains, mixtures of flows, and Riemannian manifolds.Source: §7 · p. 50

Foundations

  • Foundational principles are expected to remain useful despite changing implementations, including the chain rule with triangular Jacobians for autoregressive flows and the Banach fixed-point theorem for contractive residual flows.Contractive residual flows must retain their underlying Lipschitz constraints, even if translation parameterizations or normalization strategies change.Source: §7 · pp. 50–51

Implementation guidance

  • Choosing a flow requires balancing transformation direction, computational constraints, sampling, density evaluation, interactions between dimensions, and expressive power.Autoregressive flows suit one-sided sampling or density-estimation objectives; coupling-based conditioners improve both operations at an expressive-power cost, while residual flows are expressive but typically require iterative sampling and evaluation.Source: §7 · p. 51

Future directions

  • The central open challenge is constructing increasingly flexible transformations while preserving exact density evaluation and sampling with tractable computation.Further theory is needed on approximation capabilities at finite sample sizes and finite depths to guide flow-class selection.Source: §7 · p. 51

Appendix A. Proof of KL Dualities

This appendix derives KL dualities for distributions induced by a flow and its inverse, using flow-density formulas and change of variables.

  • The proof defines p_x(x) from a flow transformation T with base distribution p_u(u).Source: Appendix A · p. 52
  • It also defines p*_u(u) from the inverse transformation T^-1 with base distribution p*_x(x).Source: Appendix A · p. 52
  • The appendix applies the density formula for flow-based models and change of variables to derive the stated identities, then gives an analogous derivation.Source: Appendix A · p. 52

Appendix B. Constructing Linear Flows

The appendix constructs efficient linear flows by decomposing invertible matrices into structured factors with tractable determinants and inverses. It covers PLU and QR flows, then discusses orthogonal flows and their parameterizations and limitations.

Constructing Linear Flows

  • Matrix decompositions parameterize invertible linear flows through structured factors whose invertibility, determinants, and inverses are tractable.The appendix introduces matrix decompositions as an efficient approach to parameterizing invertible matrices.Source: Appendix B · p. 52

PLU flows

  • PLU flows constrain lower- and upper-triangular factors to have positive diagonals, enabling invertibility and O(D)-time determinant computation.Inversion proceeds by undoing the permutation and solving lower- and upper-triangular systems; typically, P is fixed while L and U are learned.Source: Appendix B · p. 52

QR flows

  • QR flows represent any invertible matrix with an orthogonal factor and positive-diagonal upper-triangular factor, with tractable determinant and inversion.Inversion multiplies by Q⊤ and then solves a triangular system; QR flows can be viewed as autoregressive followed by orthogonal transformations.Source: Appendix B · p. 53

Orthogonal flows

  • Orthogonal flows are volume-preserving and trivially invertible, generalize permutations, and can compose with other structured matrices.Their effective parameterization is challenging because continuously parameterizing both determinant components is impossible within a single continuous parameterization.Source: Appendix B · p. 53
  • The exponential map represents every orthogonal matrix with determinant 1, the Cayley map yields determinant-1 orthogonal matrices but excludes those with −1 eigenvalues, and Householder transformations use at most D reflections.Source: Appendix B · pp. 53–54
Loading 1912.02762v2…