Source-linked AI summary
Density estimation using Real NVP
Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio
TL;DR
Unsupervised modeling of high-dimensional structured data requires models that are both expressive and tractable. Real NVP uses invertible transformations to enable exact inference, sampling, and log-likelihood evaluation, while achieving competitive sample quality and likelihood.
Problem
Modeling high-dimensional, structured data requires expressive models that remain tractable to train and evaluate.
Method
Real NVP stacks affine coupling layers into flexible bijections with tractable Jacobian determinants for exact probabilistic modeling.
Results
The model achieves competitive sample quality and log-likelihood while supporting exact and tractable inference, sampling, and evaluation.
Takeaways & Limitations
Real NVP learns a semantically meaningful latent space as high-dimensional as the input, supporting latent representation and manipulation.
Takeaways & Limitations
The experiments use an isotropic unit-norm Gaussian prior, although alternative or learned priors could also be used.
Abstract
from arXiv · showhide
Unsupervised learning of probabilistic models is a central yet challenging problem in machine learning. Specifically, designing models with tractable learning, sampling, inference and evaluation is crucial in solving this task. We extend the space of such models using real-valued non-volume preserving (real NVP) transformations, a set of powerful invertible and learnable transformations, resulting in an unsupervised learning algorithm with exact log-likelihood computation, exact sampling, exact inference of latent variables, and an interpretable latent space. We demonstrate its ability to model natural images on four datasets through sampling, log-likelihood evaluation and latent variable manipulations.
1 Introduction
The introduction motivates unsupervised generative modeling for leveraging unlabeled data and presents real NVP transformations as a tractable, expressive approach to high-dimensional data. The model supports exact, efficient inference, sampling, log-density estimation, and image reconstruction from hierarchical features.
- Unsupervised learning can exploit large pools of unlabeled data and extend advances to otherwise impractical or impossible modalities.
- Generative probabilistic models provide a principled unsupervised-learning approach that can create novel content and support reconstruction applications.
- Real-valued non-volume preserving transformations address high-dimensional modeling by combining tractability with expressiveness.
- The model enables efficient and exact inference, sampling, and log-density estimation, plus exact and efficient reconstruction from hierarchical features.
2 Related work
Prior generative-modeling approaches trade off tractable likelihoods, inference, and sampling in different ways. Real NVP builds on bijective maximum-likelihood models using the change-of-variable formula to target exact inference and likelihood computation.
- Undirected and directed models: Maximum-likelihood undirected models such as Restricted Boltzmann Machines and Deep Boltzmann Machines exploit bipartite conditional independence for exact or approximate latent-posterior inference.Their approximations can produce highly correlated samples and hinder performance.
- Undirected and directed models: Directed graphical models simplify ancestral sampling but lack undirected models’ conditional independence structure, making exact and approximate latent-posterior inference cumbersome.Stochastic variational inference and amortized inference enable efficient approximate inference and learning for deep directed models.
- Autoregressive models: Autoregressive models avoid latent-variable approximations by factorizing the joint distribution into conditionals under a fixed dimension ordering, enabling tractable likelihood evaluation and sampling.They typically retain substantial flexibility.
- Generative adversarial models: GANs avoid maximum likelihood and intractable log-likelihoods by training a discriminator to distinguish generated samples from real data.The discriminator supplies the generative network’s adversarial training signal.
- Bijective density models: A bijective generative network can instead be trained by maximum likelihood through the change-of-variable formula, avoiding GAN discriminators and variational-autoencoder approximate inference.This formulation connects Real NVP to maximum-likelihood independent components analysis, gaussianization, and deep density models.
3 Model definition
The model defines flexible bijections between data and latent spaces, enabling exact density evaluation, sampling, and inference through the change-of-variables formula. Affine coupling layers make these transformations tractable and invertible, while alternating masks and a multi-scale architecture improve coverage and efficiency.
- Change-of-variables model: A bijection f maps data x to latent z under a simple prior, defining the data density through the change-of-variables formula and its Jacobian determinant.This provides exact and tractable density evaluation for continuous data.
- Change-of-variables model: Exact samples are generated by drawing z from the prior and applying the inverse transform x = f^-1(z), while density evaluation uses f(x) and the Jacobian determinant.Exact and efficient inference enables accurate and fast model evaluation.
- Affine coupling layers: Affine coupling layers update part of a D-dimensional input using scale and translation functions conditioned on the remaining components, preserving simple inversion.Their triangular Jacobian makes the determinant efficient to compute, while s and t can be arbitrarily complex, including deep convolutional networks.
- Architecture: Alternating coupling layers update components left unchanged previously, and the model uses spatial checkerboard and channel-wise masks to exploit image correlations.The two masking schemes are arranged so that channel-wise partitioning is not redundant with checkerboard masking.
- Architecture: The multi-scale architecture squeezes spatial dimensions into channels, applies coupling layers at each scale, and factors out half the dimensions at regular intervals.Earlier factoring creates finer-grained intermediary representations and distributes the loss while reducing computation and memory, enabling larger models.
4 Experiments
Experiments evaluate real NVP on four natural-image datasets using a multi-scale convolutional architecture and report competitive likelihoods, sharp and coherent samples, and semantically organized latent spaces.
- Experimental setup: The model is trained on CIFAR-10, 32×32 and 64×64 downsampled ImageNet, LSUN categories, and CelebA.LSUN uses bedroom, tower, and church outdoor categories with random 64×64 crops.
- Experimental setup: The experiments use a recursively repeated multi-scale architecture with residual-network coupling layers and learned scaling and translation functions.The recursion continues until the final input is a 4 × 4 × c tensor, with tanh-based scaling and affine translation outputs.
- Experimental setup: The isotropic unit-norm Gaussian is used as the prior, although the framework permits learned or alternative prior distributions.Examples include autoregressive priors and, with slight objective modifications, variational-autoencoder priors.
- Likelihood results: Bits per dimension are competitive with other generative methods but do not improve over the Pixel RNN baseline, while larger models are likely to improve performance.CelebA and LSUN validation bits per dimension decreased throughout training, suggesting little overfitting.
- Sample quality: Generated samples are globally coherent and sharp, but limited capacity can produce highly improbable outputs, especially on CelebA.The paper attributes this tradeoff to maximum likelihood valuing diversity over sample quality in a limited-capacity setting and contrasts it with fixed-form L2 reconstruction costs in variational autoencoders.
- Latent representations: Latent-space manifolds show smooth, semantically consistent organization extending beyond pixel-space interpolation.A class-conditional CelebA model further showed that the learned representation had a consistent semantic meaning.
5 Discussion and conclusion
Real NVP defines invertible transformations with tractable Jacobians, enabling exact likelihood evaluation, inference, and sampling while achieving competitive sample quality and likelihood. Its flexible, high-dimensional latent representation supports links to other generative models and potential applications beyond unsupervised learning.
- Contributions: Real NVP provides exact and tractable log-likelihood evaluation, inference, and sampling through invertible functions with tractable Jacobian determinants.The resulting generative models achieve competitive performance in both sample quality and log-likelihood.
- Connections to other generative models: The technique bridges auto-regressive models, variational autoencoders, and GANs by combining exact likelihood evaluation, flexible transformations, and fast exact sampling.It retains properties associated with each model family while using a more flexible functional form than auto-regressive models.
- Latent representations: Unlike variational autoencoders and GANs, Real NVP learns a semantically meaningful latent space as high dimensional as the input space.This property may make it particularly suitable for semi-supervised learning, identified as a direction for future work.
- Extensions: Real NVP can be conditioned on variables such as class labels and used modularly to improve probabilistic models, including auto-regressive models and variational autoencoders.For variational autoencoders, the transformations could enable more flexible reconstruction.
- Applications beyond generative modeling: Trainable invertible functions may also support reinforcement learning by expanding tractable argmax operations for continuous Q-learning or improving local linear Gaussian representations.These examples illustrate applications beyond generative unsupervised learning.
A Samples
The section presents samples generated by models trained on Imagenet, CelebA, and three LSUN scene categories. Imagenet samples are shown at 64 × 64 resolution.
- Samples are shown from a model trained on Imagenet at 64 × 64 resolution.
- Samples are shown from a model trained on CelebA.
- Samples are shown from a model trained on LSUN’s bedroom category.
- Samples are shown from a model trained on LSUN’s church outdoor category.
- Samples are shown from a model trained on LSUN’s tower category.
B Manifold
The paper visualizes learned image manifolds for ImageNet, CelebA, and several LSUN categories by varying angular coordinates φ and φ′ over eight values. Red-bordered training or validation images define each manifold, which is computed using Equation 19.
- ImageNet: ImageNet’s 64 × 64 manifold uses validation images with red borders as reference points and varies φ and φ′ across eight angular values.The axes correspond to φ and φ′, with each taking values in {0, π/4, · · ·, 7π/4}.
- CelebA: CelebA’s manifold is defined by red-bordered training images and constructed by varying φ and φ′ according to Equation 19.Both angular coordinates range over {0, π/4, · · ·, 7π/4}.
- LSUN categories: LSUN bedroom manifolds use red-bordered validation images and vary φ and φ′ across the same eight angular values.The manifold computation follows Equation 19.
- LSUN categories: The LSUN church outdoor manifold is computed from red-bordered validation images using φ and φ′ as axes with values from 0 to 7π/4 in π/4 increments.The construction follows Equation 19.
- LSUN categories: The LSUN tower manifold follows the same Equation 19 construction, with φ and φ′ varying over the stated angular grid and red-bordered validation images defining the manifold.The axes correspond to φ and φ′.
C Extrapolation
The model is evaluated on extrapolating images to twice or ten times the dataset size. It appears to capture dataset texture while preserving spatial smoothness across the generated images.
- Extrapolation: The model generates images twice or ten times larger than those in the datasets to evaluate its captured statistics.This extrapolation test is inspired by prior texture-generation work and DCGAN experiments.
- Extrapolation: The generated samples appear to preserve a texture representation of each dataset while maintaining spatial smoothness through the image.Examples are shown for ImageNet, CelebA, and LSUN bedroom, church outdoor, and tower categories.
- Extrapolation: Figures 17–21 show samples generated beyond the training image size for ImageNet, CelebA, and three LSUN categories.The LSUN categories are bedroom, church outdoor, and tower.
D Latent variables semantic
Ablation tests probe the semantics of Real NVP’s learned latent layers by resampling progressively higher levels from a standard Gaussian. The latent space appears to capture graphic-level structure rather than higher-level concepts, despite convolution improving learning through image priors.
- Latent-variable ablations: Ablation tests infer latent variables and resample progressively higher latent levels from a standard Gaussian to examine learned-layer semantics.The highest affected level is increased across tests.
- Latent-variable ablations: The learned latent space appears more semantic at a graphic level than at the level of higher-level concepts.The passage attributes this observation to the ablation tests.
- Conceptual compression: Conceptual compression experiments retain 100%, 50%, 25%, 12.5%, or 6.25% of latent variables while resampling the rest on ImageNet 64 × 64.Higher-level latent variables are stored while lower-level variables are resampled progressively.
- Conceptual compression: The same conceptual compression procedure is shown for CelebA and LSUN bedroom, church outdoor, and tower models.Each figure compares 100%, 50%, 25%, 12.5%, and 6.25% of retained latent variables.
E Batch normalization
The model uses batch normalization with moving-average layer statistics while propagating gradients only through current batch statistics. This lag helps training with very small minibatches and was used for CIFAR-10 results.
- E Batch normalization: Batch normalization uses a weighted average of moving-average layer statistics and propagates gradients only through current batch statistics.The moving-average statistics are denoted ˜µ_t and ˜σ^2_t, while current batch statistics are denoted ˆµ_t and ˆσ^2_t.
- E Batch normalization: The lag between moving-average and current batch statistics helps the model train with very small minibatches.The authors specifically observed this training benefit when using the lagged statistics.
- E Batch normalization: Batch normalization with a moving average was used for the CIFAR-10 results.This describes the batch-normalization configuration used in the reported CIFAR-10 experiments.
F Attribute change
The conditional Real NVP model uses CelebA attribute labels to control image decoding. Changing attributes alters corresponding facial properties while preserving other characteristics such as position and background.
- Conditional attribute manipulation: The conditional model uses CelebA attributes y to define the invertible function f’s parameters.Images are encoded with original attributes and decoded using shuffled attributes y′ from the same batch.
- Conditional attribute manipulation: Shuffling attributes changes faces to respect the new attributes while preserving properties such as position and background.The decoded images are x′ = g(z, y′), using the latent representation from the original encoding.
- Conditional attribute manipulation: Decoded images often share characteristics with the originals, including position and background.This behavior is illustrated for a model trained on CelebA image–attribute pairs.