PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Autoregressive Flows

Chin-Wei Huang, David Krueger, Alexandre Lacoste, Aaron Courville

arXiv:1804.00779v1cs.LGstat.ML

TL;DR

Existing affine autoregressive flows can struggle with multimodal distributions despite their success in density estimation and variational inference. NAF replaces their affine univariate transformations with invertible monotonic neural networks. The resulting flows are universal approximators, model multimodality, and outperform related methods in density estimation and variational inference.

  • Problem

    Affine autoregressive transformations can fail to capture multimodal target densities, motivating more expressive normalizing-flow transformations.

  • Method

    NAF replaces conditional affine transformations with invertible univariate monotonic neural networks whose parameters are produced by an autoregressive model.

  • Results

    NAF models multimodal distributions, is a universal approximator for proper distributions in real space, and outperforms related methods including IAF in density estimation and variational inference.

  • Takeaways & Limitations

    More expressive neural transformers improve the ability of autoregressive flows to represent multimodal target distributions.

  • Takeaways & Limitations

    NAF architectures used in the paper lack analytic inverse expressions, so inverse computations are approximated numerically; alternative transformer architectures remain unexplored.

Abstract

from arXiv · show

Normalizing flows and autoregressive models have been successfully combined to produce state-of-the-art results in density estimation, via Masked Autoregressive Flows (MAF), and to accelerate state-of-the-art WaveNet-based speech synthesis to 20x faster than real-time, via Inverse Autoregressive Flows (IAF). We unify and generalize these approaches, replacing the (conditionally) affine univariate transformations of MAF/IAF with a more general class of invertible univariate transformations expressed as monotonic neural networks. We demonstrate that the proposed neural autoregressive flows (NAF) are universal approximators for continuous probability distributions, and their greater expressivity allows them to better capture multimodal target distributions. Experimentally, NAF yields state-of-the-art performance on a suite of density estimation tasks and outperforms IAF in variational autoencoders trained on binarized MNIST.

1. Introduction

Normalizing flows support tractable density evaluation and richer variational approximations, but affine autoregressive transformations can miss multimodal targets. NAF replaces those affine transformations with more expressive neural transformers and shows stronger empirical and theoretical expressivity.

  • Motivation: Normalizing flows enable direct density optimization and can improve variational posterior approximations.Poor posterior approximations can misrepresent uncertainty and produce inaccurate predictions.
  • Motivation: Affine autoregressive flows can become trapped in bad local minima and fail to capture multimodal target densities.The limitation is illustrated through IAF and MAF comparisons in Figures 1 and 2.
  • Contributions: NAF replaces conditional affine transformations with richer monotonic neural transformations whose weights and biases are produced by an autoregressive model.Each transformer operates on a single random variable, while the conditioner outputs its pseudo-parameters.
  • Contributions: NAF works better than affine autoregressive flows as both a multimodal sample generator and a density model for evaluating data likelihoods.The comparison includes the affine flows of Kingma et al. (2016) and Papamakarios et al. (2017).
  • Contributions: NAF is a universal approximator for proper distributions in real space, supporting the expressiveness of its parameterization.

2. Background

Normalizing flows use invertible transformations and change-of-variables densities, with affine autoregressive designs making Jacobian computation efficient. The background motivates more powerful transformers while retaining invertibility and cheap derivative computation.

  • Normalizing flows: A normalizing flow is an invertible function that transforms one random variable into another and relates their densities through change of variables.The Jacobian determinant accounts for local expansion or contraction under the transformation.
  • Normalizing flows: Flow notation uses x and y for function inputs and outputs, not supervised-learning inputs and targets.
  • Normalizing flows: Complex normalizing flows are often built by stacking simpler invertible flows through function composition.
  • Applications: In maximum likelihood, flows map complex data toward a simple latent prior, whereas variational inference maps simple noise toward a complex approximate posterior.These settings correspond respectively to structured-to-unstructured and unstructured-to-structured transformations.
  • Practical considerations: For the NAF architectures used, inverse computation lacks an analytic expression and is approximated numerically.
  • Affine autoregressive flows: Affine autoregressive flows use triangular Jacobians, allowing log-determinants to be computed in linear time from diagonal entries.An autoregressive conditioner computes each output component from preceding inputs, and MADE can produce all conditioner outputs in one forward pass.
  • Affine autoregressive flows: The transformer must be invertible and have a cheaply computable derivative, motivating more powerful alternatives to affine transformations.Affine transformers are easy to invert but place most expressivity in the conditioner and stacked layers.

3. Neural Autoregressive Flows

NAF replaces affine autoregressive transformers with invertible monotonic neural networks, increasing the flow’s ability to represent multimodal distributions while retaining efficient computation.

  • NAF replaces previous affine transformers with neural networks, producing richer distributions with only a minor increase in computation and memory.
  • The autoregressive conditioner outputs transformer-network parameters for each scalar input, while the transformer maps that scalar to the output.
  • Strictly positive weights and strictly monotonic activations are sufficient to make each neural transformer, and thus the flow, strictly monotonic and invertible.
  • NAF can induce multimodality through inflection points in the transformer, unlike affine transformers that require multimodal information to pass through preceding variables.
  • Transformer Architectures: DSF and DDSF use sigmoid-based monotonic architectures; small networks with 1 or 2 hidden layers and 8 or 16 sigmoid units perform well across experiments.
  • Transformer Architectures: DSF uses an inverse sigmoid output to avoid bounded outputs, while its sigmoid activations form a convex combination before mapping back to the original space.
  • Transformer Architectures: DDSF generalizes stacked sigmoidal transformations to a fully connected MLP, with positive parameters and normalized weights enforcing its constraints.
  • Efficient Parametrization of Larger Transformers: Larger multilayer transformers can reduce conditioner outputs from O(d^2) to O(d) pseudo-parameters through per-unit modulation techniques such as conditional weight normalization.

4. NAFs are Universal Density Approximators

NAFs, specifically DSF, are proved capable of approximating arbitrary continuous probability distributions under stated positivity and continuity conditions. The construction supports both transforming uniform variables into target distributions and transforming structured variables into desired distributions.

  • NAFs can approximate any probability distribution over real vectors arbitrarily well when the transformer has sufficient hidden units and autoregressive conditioning.The result is stated for DSF architectures parameterized by generic autoregressive neural networks.
  • Combining the two directions, DSF can transform structured noise into a random variable with any desired distribution.The IAF direction resembles inverse transform sampling, while the MAF direction addresses disentangling into uniform independent variables.
  • DSF universally transforms uniform random variables into any desired random variables with positive and continuous density.The transformed samples converge in distribution to the target random variable.
  • DSF universally transforms random variables with positive and continuous density into uniformly distributed random variables.The transformed variables converge in distribution to a uniform distribution on the unit cube.
  • The theorem assumes positive and continuous probability densities for both the input and target random vectors.Under these assumptions, a sequence of autoregressive neural-network transformations converges in distribution to the target.

5. Related work

Related work establishes normalizing flows as flexible density and variational-inference tools, while prior autoregressive flows motivate NAFs and provide comparison baselines. The supplied figures and table captions identify evaluations of multimodal fitting, learning curves, and variational inference.

  • NAFs generalize affine autoregressive flows developed through IAF, AF, and MAF formulations.The related approaches reinterpret autoregressive models as invertible transformations and exploit parallel inverse computation.
  • Normalizing flows provide richer variational posteriors and tractable density models for generative modeling and inference.Their applications include density estimation, variational posteriors, and generative models.
  • The related-work landscape includes accelerated IAF speech synthesis, MAF density estimation, and other normalizing-flow applications and construction techniques.The cited applications span generative modeling, approximate posteriors, maximum entropy, and MCMC proposals.
  • Table 1 evaluates DSF for variational inference against affine IAF and reports five-trial averages and standard deviations.The caption also notes a log likelihood of 78.88 reported by Kingma et al. (2016).
  • Figure 6 compares the true Gaussian grid with affine autoregressive flow and neural autoregressive flow fits under maximum likelihood.The three panels are ordered true distribution, AAF, and NAF.

6. Experiments

Experiments show that NAFs capture multimodal distributions more faithfully than affine flows and achieve strong results in variational inference and density estimation.

  • 6. Experiments: The experiments also evaluate NAFs on multimodal posterior inference and compare them with IAF and MAF baselines.The study includes a limited-data Bayesian posterior task and larger-scale density-estimation experiments.
  • 6.1.1. EXPRESSIVENESS: Affine flows produce a single mode, whereas neural flows fit Gaussian mixtures with up to 100 modes on a 10x10 grid.The experiment uses marginally independent mixture-of-Gaussian targets.
  • 6.4. Density Estimation with Masked Autoregressive Flows: NAFs produce state-of-the-art test log-likelihood results on all five density-estimation datasets.Table 2 reports test log-likelihoods with error bars of two standard deviations across five trials.
  • 6.3. Amortized Approximate Posterior: DSF outperforms standard IAF and independent Gaussian posteriors by a statistically significant margin on binarized MNIST.The evaluation uses a variational autoencoder benchmark.
  • 6.4. Density Estimation with Masked Autoregressive Flows: Replacing affine transformers with DDSF yields substantial gains over prior MAF architectures while also outperforming TAN.The comparisons retain the other settings of the best-performing architecture for each task.

7. Conclusion

The paper introduces neural autoregressive flows (NAF), using monotonic neural networks to model rich and multimodal distributions. It establishes monotonicity through positive weights and increasing activations, and uses log-scale computations for stable Jacobian evaluation.

  • Conclusion: NAF is a flexible method that models multimodal distributions and outperforms inverse autoregressive flow in density estimation and variational inference.The paper emphasizes multimodality because previous methods fail even on simple toy tasks.
  • Conclusion: Change-of-variable training transforms the empirical distribution toward a tractable base density by minimizing a metric equivalent to exclusive KL divergence.The model uses an invertible transformation with tractable base-density evaluation.
  • Conclusion: Strictly positive weights and strictly increasing activations ensure that each monotonic neural-network unit, and ultimately the full NAF transformation, is strictly monotonic.The argument proceeds by showing layerwise monotonicity and applying mathematical induction across layers.
  • Conclusion: The DDSF Jacobian is derived through chained dot products, enabling log-determinant computation as part of the objective.The chain rule expresses each transformation Jacobian as a sequence of dot products with intermediate tensor operations.
  • Conclusion: Jacobian products are computed in log-scale, while softmax and softplus enforce positivity and summing-to-one constraints on transformation parameters.Logarithmic matrix dot products use logsumexp to represent products without unstable multiplicative operations.

D. Scalability and Parameter Sharing

The paper reduces DDSF’s conditioner-output burden by sharing learned statistical parameters while retaining conditional scaling through the autoregressive conditioner.

  • D. Scalability and Parameter Sharing: A multi-layer DDSF conditioner would otherwise output O(Ld^2) pseudo-parameters.L is the transformer depth and d is the average number of hidden units per layer.
  • D. Scalability and Parameter Sharing: Parameter sharing decomposes transformer weights so learned statistical parameters remain fixed while the conditioner outputs only conditional pseudo-parameters.The shared parameters are represented by v^(l+1), while η is produced by the conditioner.
  • D. Scalability and Parameter Sharing: Conditional normalized weight exponentiation reduces the conditioner’s output count from O(Ld^2) to O(Ld).Weights are softmax-normalized to be positive and sum to one before conditional rescaling.

E. Identity Flow Initialization

The flow is initialized near the identity so training begins from a minimally transformed distribution, using small conditioner weights and suitable output biases.

  • E. Identity Flow Initialization: Identity-flow initialization preserves the quality of the input approximate posterior before transformation.This is motivated by variational inference, where the input is usually a fully factorized Gaussian.
  • E. Identity Flow Initialization: For DSF and DDSF, pseudo-weights a are initialized near 1 and pseudo-biases b near 0.These settings make the initial transformation have a minimal effect.
  • E. Identity Flow Initialization: The conditioner uses zero output biases, adds softplus^-1(1) ≈ 0.5413 before softplus for a, and samples weights from Unif(-0.001, 0.001).These choices implement the near-identity initialization of the transformer parameters.
  • E. Identity Flow Initialization: Sigmoidal superpositions approximate monotonic functions by matching an intermediate step-function representation with selected transition locations.The construction is illustrated for functions on [0,1].

F. Lemmas: Uniform Convergence of DSF

The section establishes uniform approximation of strictly monotonic functions by DSF, then extends this result to autoregressively conditioned multivariate functions using neural-network parameterizations.

  • Uniform approximation: DSF can universally approximate any strictly monotonic function on a bounded interval.The proof first constructs step-function approximations and then replaces steps with superimposed sigmoids.
  • Step-function construction: Step-function approximation places biases at inverse target quantiles and solves a lower-triangular linear system for positive weights.The resulting weights satisfy simplex constraints and reproduce the target’s interpolated step values.
  • Sigmoid approximation: Superimposed sigmoids approximate the step construction arbitrarily closely when their positive, bounded temperature parameters are chosen appropriately.The proof uses a common temperature selected from the minimum separation between bias values.
  • Neural parameterization: The pre-logit DSF transformation inherits universal approximation by matching sigmoid-mixture parameters and approximating their conditioning functions with neural networks.Cybenko’s result supplies the neural-network approximation of continuous parameter functions.
  • Multivariate extension: For multivariate targets, each autoregressive component is approximated uniformly by choosing a common sufficiently large width across all dimensions.The construction yields ||S_n − S||∞ < ϵ over the full bounded domain.

G. Proof of Universal Approximation of DSF

The proof represents target variables through conditional CDFs and inverse CDFs, then uses DSF uniform approximation and convergence in distribution to establish universal approximation of distributions.

  • Distributional convergence: Pointwise convergence of transformations implies convergence in distribution of the induced random variables.The proof uses bounded continuous test functions, dominated convergence, and the Portmanteau lemma.
  • CDF construction: Conditional CDFs transform the target into a uniform distribution, while inverse conditional CDFs map uniform variables back to the target.The construction relies on positive conditional densities and the resulting triangular Jacobian structure.
  • DSF approximation: DSF approximates the inverse-CDF transformation uniformly, and continuity of the inverse sigmoid converts this into pointwise convergence of the corresponding transformations.The same argument is applied to the relevant CDF or inverse-CDF maps.

H. Experimental Details

The experiments use specified neural architectures for variational inference and density estimation, including convolutional encoders, flow layers, and MADE configurations.

  • Amortized variational inference: The variational autoencoder uses an encoder with [16, 32, 32] feature maps, stride-2 3 × 3 convolutions, and a fully connected context layer of size 450.The context feeds the flow layers that transform the latent representation.
  • Density estimation: The MAF density-estimation experiment evaluates models with 5 or 10 flow layers followed by one linear flow layer.MADE hidden-layer and hidden-unit settings are specified separately for each dataset.
  • Density estimation: Table 3 specifies the number of MADE hidden layers and hidden units used in the MAF experiment.The table provides the architecture specification for the density-estimation models.
Loading 1804.00779v1…