PaperlayerView exact revision on arXiv

Source-linked AI summary

Masked Autoregressive Flow for Density Estimation

George Papamakarios, Theo Pavlakou, Iain Murray

arXiv:1705.07057v4stat.MLcs.LG

TL;DR

Density estimation requires models that represent complex distributions while retaining tractable evaluation and learning, but autoregressive models can be sensitive to variable ordering. MAF stacks autoregressive models by modeling their internal random numbers, yielding a flexible normalizing flow built with MADE. Across general-purpose tasks, MAF outperforms Real NVP and achieves state-of-the-art performance, while its broader approximation properties remain open and density accuracy does not guarantee high-quality generation.

  • Problem

    Density estimation needs flexible, tractable models, while autoregressive models are sensitive to variable ordering and applications may require accurate densities rather than only synthetic samples.

  • Method

    MAF stacks autoregressive models so each models the next model’s internal random numbers, using MADE as a building block for a tractable normalizing flow.

  • Results

    MAF outperforms Real NVP on all evaluated datasets and achieves state-of-the-art performance across a variety of general-purpose density-estimation tasks.

  • Takeaways & Limitations

    MAF improves density-estimation performance while remaining focused on accurate densities, not necessarily latent-space inference or high-quality data generation.

  • Takeaways & Limitations

    MAF’s Gaussian-base approximation properties remain an open question, and its general-purpose Real NVP comparison differs from the original image-specific version.

Abstract

from arXiv · show

Autoregressive models are among the best performing neural density estimators. We describe an approach for increasing the flexibility of an autoregressive model, based on modelling the random numbers that the model uses internally when generating data. By constructing a stack of autoregressive models, each modelling the random numbers of the next model in the stack, we obtain a type of normalizing flow suitable for density estimation, which we call Masked Autoregressive Flow. This type of flow is closely related to Inverse Autoregressive Flow and is a generalization of Real NVP. Masked Autoregressive Flow achieves state-of-the-art performance in a range of general-purpose density estimation tasks.

1 Introduction

Density estimation seeks flexible models with tractable densities and learning, and neural methods have become successful across several applications. The paper introduces Masked Autoregressive Flow by stacking autoregressive models to increase flexibility while retaining tractability.

  • Estimating the joint density p(x) from examples supports inference, prediction, data completion, and data generation.
  • Neural density estimators combine neural-network flexibility with structural prior knowledge and have achieved strong results across images, audio, inference, and variational applications.
  • Exact density evaluations make neural density estimators suitable when applications prioritize evaluating densities over generating synthetic data.
  • The central challenge is constructing models flexible enough for complex densities while retaining tractable density functions and learning algorithms.
  • Autoregressive generation can be viewed as a differentiable transformation of external randomness with a tractable Jacobian, enabling stacked normalizing flows.
  • MAF uses MADE as a building block, supports parallel density evaluation and training, connects theoretically to IAF and Real NVP, and achieves state-of-the-art density-estimation performance.

2 Background

Autoregressive models factorize densities into conditionals but face ordering and sequential-computation challenges. Normalizing flows address tractability through invertible transformations whose compositions can form deeper models.

  • Autoregressive density estimation: Autoregressive density estimators decompose a joint density into one-dimensional conditionals modeled using parameters generated from hidden states.
  • Autoregressive density estimation: Variable ordering can determine whether an autoregressive model represents a target density, while factorially many possible orders make the best choice difficult to identify.
  • Autoregressive density estimation: Masked fully connected networks remove connections so output i depends only on inputs 1 through i−1, enabling parallel computation while preserving autoregressive structure.
  • Normalizing flows: Normalizing flows transform an easily evaluated base density into a target density through an invertible differentiable transformation.
  • Autoregressive models as normalizing flows: A five-layer MAF can learn multimodal conditionals even when each constituent autoregressive model has unimodal conditionals.
  • Normalizing flows: Composing transformations that are easy to invert and whose Jacobian determinants are tractable produces a deeper valid normalizing flow.
  • Normalizing flows: Some flow approaches have Jacobian-determinant costs that scale cubically with data dimensionality, motivating tractable alternatives such as autoregressive flows.

3 Masked Autoregressive Flow

MAF interprets autoregressive models as normalizing flows and increases their flexibility by stacking models that transform successive sources of randomness. Its MADE-based implementation supports efficient density evaluation, while its relationships with IAF and Real NVP clarify distinct computational trade-offs and generality.

  • 3.1 Autoregressive models as normalizing flows: An autoregressive model can be viewed as an invertible transformation from Gaussian random numbers u to data x with a tractable Jacobian.The inverse transformation recovers u from x, and the triangular Jacobian makes the determinant easy to compute, yielding a normalizing-flow interpretation.
  • 3.1 Autoregressive models as normalizing flows: Transforming training data into random numbers and checking for independent standard normals provides a diagnostic for autoregressive-model fit.Poorly fitting models can produce substantially non-Gaussian scatter plots of the transformed random numbers.
  • 3.1 Autoregressive models as normalizing flows: Stacking autoregressive models, each modelling the next model’s random numbers, produces a more flexible but still tractable flow.A stack of five models can learn multimodal conditionals even when each component has unimodal conditionals.
  • 3.1 Autoregressive models as normalizing flows: MAF uses MADE with Gaussian conditionals, enabling density evaluations without the sequential loop typical of autoregressive models.Masking enforces the autoregressive property while allowing a single forward pass, making MAF fast to evaluate and train on GPUs.
  • 3.2 Relationship with Inverse Autoregressive Flow: MAF and IAF use the same MADE-based components but condition their scale and shift functions on previous data variables x1:i−1 and random numbers u1:i−1, respectively.MAF evaluates an external datapoint’s density in one pass but samples sequentially; IAF has the opposite computational trade-off.
  • 3.2 Relationship with Inverse Autoregressive Flow: Maximum-likelihood training of MAF is theoretically equivalent to stochastic variational inference with an implicit IAF.The equivalence follows by relating the KL objectives over data space and random-number space.
  • 3.3 Relationship with Real NVP: Real NVP coupling layers are special cases of both MAF and IAF, making the autoregressive flows more flexible but giving them different one-pass capabilities.Real NVP can generate data and estimate densities in one pass, whereas MAF and IAF each require D sequential passes for one of those operations.
  • Conditional density estimation: Conditional MAF is formed by supplying side-information y as an additional input to every MADE-based layer.The conditional construction models p(x | y) while requiring y to precede x in the autoregressive ordering.

4 Experiments

The experiments compare MADE, Real NVP, and MAF on unconditional and conditional density estimation tasks using standardized implementations and likelihood-based evaluation. MAF consistently outperforms Real NVP and is often the strongest model across datasets.

  • Experimental setup: The study evaluates MADE, Real NVP, and MAF on unconditional density estimation using four UCI datasets and natural image patches.The evaluation reports average test log likelihoods for unconditional density estimation.
  • Experimental setup: The general-purpose Real NVP implementation uses coupling layers with separate scaling and shifting networks and is designed for fair comparison with autoregressive layers.This implementation differs from the original image-specific Real NVP.
  • Experimental setup: MAF is tested with 5 or 10 autoregressive layers, using either a standard Gaussian or a 10-component Gaussian-mixture MADE base density.The mixture-base variant is trained jointly with a MADE MoG.
  • Experimental setup: MADE, MADE MoG, and MAF use masked feedforward networks to enforce the autoregressive property, with reversed input order across successive MAF layers.Batch normalization is added after autoregressive layers and improves training time, stability, and performance.
  • Unconditional density estimation: On 3 of 5 unconditional datasets, MAF is best; MADE MoG leads on the other 2, and MAF outperforms Real NVP on every dataset.MAF MoG (5) achieves 156.36 nats on BSDS300, the best reported result there for a single model in the cited comparison.
  • Conditional density estimation: For conditional density estimation, MADE MoG performs best on MNIST, MAF performs best on CIFAR-10, and MAF outperforms Real NVP in all cases.On CIFAR-10, MADE and MADE MoG perform significantly worse than the Gaussian baseline.

5 Discussion

The discussion interprets MAF as an alternative way to increase MADE’s flexibility and compares it with MADE MoG and Real NVP. It also identifies unresolved expressivity questions and a sampling trade-off.

  • Flexibility and expressivity: MAF improves MADE by modelling the density of its internal random numbers, while increasing conditional mixture components offers an alternative flexibility enhancement.The comparison treats these as two distinct ways to improve MADE.
  • Flexibility and expressivity: MAF outperforms MADE MoG in 5 of 9 cases, indicating that the better flexibility strategy is dataset specific in these experiments.MADE MoG is stated to be a universal density approximator, whereas the corresponding property for Gaussian-base MAF remains open.
  • Relation to Real NVP: The Real NVP coupling layer is a special case of the MAF autoregressive layer, and MAF outperforms Real NVP in all reported experiments.The discussion suggests replacing coupling layers with autoregressive layers in image-oriented Real NVP as a promising direction.
  • Relation to Real NVP: Real NVP retains an advantage over MAF because its samples can be generated efficiently in parallel.This advantage concerns sampling, whereas the reported MAF advantage concerns density-estimation performance.
  • Scope of density estimation: Accurate density estimates do not necessarily imply strong performance on other generative-modelling tasks such as data generation.The discussion distinguishes density estimation from alternative generative-modelling objectives.

A Equivalence between MAF and IAF

The appendix establishes an equivalence between maximum-likelihood training of MAF and stochastic variational inference with an implicit IAF. It also details batch normalization as an invertible flow layer.

  • MAF–IAF equivalence: A MAF maps base variables u to data x through an invertible transformation, while its inverse can be interpreted as an implicit IAF.The MAF base density is treated as the target density for the implicit IAF interpretation.
  • MAF–IAF equivalence: Minimizing D_KL(π_x(x) ∥ p_x(x)) is equivalent to fitting the induced density p_u(u) to the base density π_u(u) by minimizing D_KL(p_u(u) ∥ π_u(u)).The equivalence follows from the change-of-variables formulation of the MAF density.
  • MAF–IAF equivalence: The inverse transformation implements the reparameterization trick in the implicit IAF view.The variational objective is evaluated using expectations rewritten with respect to the base distribution.
  • MAF–IAF equivalence: Training MAF by maximizing total log likelihood is equivalent to variationally training an implicit IAF with MAF’s base distribution as its target.The training data are samples from the data density used in the variational interpretation.
  • Batch normalization: Batch normalization is inserted between flow layers and implemented as an invertible elementwise scaling and shifting transformation with a tractable Jacobian.The implementation exponentiates γ to ensure positivity and simplify the log absolute determinant.

C Number of parameters

The parameter analysis compares model capacity under connection-weight counts and finds that stacking MAF layers has a cost comparable to adding MADE MoG components, while Real NVP uses more parameters for similar size.

  • Parameter-count setup: The analysis counts connection weights while ignoring biases and batch-normalization parameters, assuming masking removes approximately half the connections.For MAF MoG, C denotes mixture components; for Real NVP and MAF, K denotes coupling or autoregressive layers.
  • Stacking versus conditional flexibility: Adding an autoregressive layer to MAF costs about as many parameters as adding a component to MADE MoG when L is small and D is comparable to H.The paper states this comparison for one or two hidden layers.
  • Real NVP comparison: Real NVP has about 1.3 to 2 times more parameters than a comparable MAF.The comparison counts connection weights and excludes biases and batch-normalization parameters.
  • Real NVP comparison: Despite its larger parameter count, Real NVP is reported as less flexible than a comparable MAF in the experiments.The authors conclude that MAF makes better use of its available capacity; tying Real NVP weights could reduce its parameter count.

D Additional experimental details

The experiments compare feedforward masked autoregressive networks with Real NVP coupling layers across selected architectural options and nonlinearities, using parameter-count tables to document model sizes.

  • Model architectures: MADE, MADE MoG, and each MAF autoregressive layer use masked feedforward networks with L hidden layers of H units.Real NVP coupling layers instead contain separate scaling and shifting feedforward networks.
  • Model architectures: Real NVP coupling layers use one scaling network and one shifting network, each with L hidden layers of H units.The supplied table caption defines its parameter values as approximate connection-weight counts.
  • Hidden-unit nonlinearities: MADE, MADE MoG, and MAF use rectified linear hidden units except on GAS, where hyperbolic tangent units are used.Real NVP uses hyperbolic tangent units for scaling and rectified linear units for shifting.
  • Architecture selection: Table 4 lists the L × H architecture combinations offered for each dataset.The same options were given to all models, and validation performance selected the option for each model.

D.2 Datasets

The study evaluates seven datasets with documented preprocessing and dataset-specific splits, including four UCI datasets, BSDS300, MNIST, and CIFAR-10.

  • UCI datasets: POWER and GAS are time series treated as i.i.d. samples from their marginal distributions.POWER uses household electric-consumption measurements, while GAS contains chemical-sensor readings reduced to eight dimensions.
  • UCI datasets: HEPMASS uses positive examples from the particle-collision dataset with particle mass 1000, after removing five problematic features.The removed features had too many recurring values and could create density spikes.
  • UCI datasets: MINIBOONE uses positive electron-neutrino examples after removing 11 outliers and seven features with excessive counts at particular values.The removed outliers had values of exactly −1000 in every column.
  • Data splits: For the four UCI datasets, 10% of the data was held out for testing and 10% of the remainder for validation.BSDS300 used 1M training, 50K validation, and 250K testing patches; MNIST and CIFAR-10 held out 10% of training data for validation.

E Additional results

On MINIBOONE, pairwise testing finds MAF MoG (5) significantly better than every other model except MAF (5), despite overlapping error bars in some comparisons.

  • Pairwise comparison: MAF MoG (5) is significantly better than all other models except MAF (5) on MINIBOONE.The test compares each model's per-example log-probability difference against MAF MoG (5).
  • Pairwise comparison: The pairwise comparison was used because several models had overlapping error bars with MAF MoG (5).The authors describe the pairwise comparison as a more powerful statistical test for assessing significance.

E.2 Bits per pixel

The section reports conditional density-estimation results in bits per pixel, converting model densities from logit space back to image space before computing the metric.

  • The reported image results use different colour components as separate pixels for CIFAR-10, giving images 32×32×3 pixels.
  • The conversion first maps image-space values in [0, 256] to logit space using dataset-specific λ values.The specified λ values are 10−6 for MNIST and 0.05 for CIFAR-10.
  • Bits per pixel are reported for conditional density estimation, with lower values indicating better performance.
  • The image-space density is obtained with the change-of-variables transformation using the logistic sigmoid, then converted into bits per pixel.
  • The appendix equation converts the average log likelihoods reported in the main text into the Table 7 results.

E.3 Generated images

The paper compares generated and real images across BSDS300, MNIST, and CIFAR-10. BSDS300 samples are visually indistinguishable from real images, while MNIST and CIFAR-10 samples have lower fidelity than modern image-focused approaches.

  • Generated and real examples are shown for BSDS300, MNIST, and CIFAR-10.The examples use MAF MoG (5) for BSDS300, conditional MAF (5) for MNIST, and conditional MAF (10) for CIFAR-10.
  • BSDS300 generated images are visually indistinguishable from the real images.
  • MNIST and CIFAR-10 generated images lack the fidelity of modern image-based generative approaches such as RealNVP and PixelCNN++.
  • The lower image fidelity reflects that this MAF version has no knowledge of image structure and targets general-purpose density estimation rather than realistic-looking image synthesis.
Loading 1705.07057v4…