PaperlayerView exact revision on arXiv

Source-linked AI summary

NICE: Non-linear Independent Components Estimation

Laurent Dinh, David Krueger, Yoshua Bengio

arXiv:1410.8516v6cs.LG

TL;DR

Modeling complex high-dimensional densities requires representations with distributions that are easier to model. NICE learns an invertible nonlinear transform to independent latent variables, using tractable exact likelihood and easy inversion; it achieves competitive log-likelihood results and supports inpainting. The method's likelihood can be inflated by invertible preprocessing that contracts the data, a limitation addressed through the Jacobian term.

  • Problem

    Unsupervised learning seeks ways to capture complex data distributions with unknown structure by finding representations whose distributions are easier to model.

  • Method

    NICE learns an invertible nonlinear transformation to a factorial latent distribution, with tractable Jacobian determinant and inverse, and trains it by exact maximum likelihood.

  • Results

    NICE achieves competitive log-likelihood results, with test log-likelihoods of 1980.50 on MNIST, 5514.71 on TFD, 11496.55 on SVHN, and 5371.78 on CIFAR-10.

  • Takeaways & Limitations

    The framework provides efficient unbiased ancestral sampling and can also produce reasonable qualitative inpainting despite not being trained for that task.

  • Takeaways & Limitations

    Invertible preprocessing can increase likelihood arbitrarily by contracting the data, so the change-of-variables determinant is needed to counteract this effect.

Abstract

from arXiv · show

We propose a deep learning framework for modeling complex high-dimensional densities called Non-linear Independent Component Estimation (NICE). It is based on the idea that a good representation is one in which the data has a distribution that is easy to model. For this purpose, a non-linear deterministic transformation of the data is learned that maps it to a latent space so as to make the transformed data conform to a factorized distribution, i.e., resulting in independent latent variables. We parametrize this transformation so that computing the Jacobian determinant and inverse transform is trivial, yet we maintain the ability to learn complex non-linear transformations, via a composition of simple building blocks, each based on a deep neural network. The training criterion is simply the exact log-likelihood, which is tractable. Unbiased ancestral sampling is also easy. We show that this approach yields good generative models on four image datasets and can be used for inpainting.

1 INTRODUCTION

NICE learns a nonlinear transformation that makes latent components independent while preserving tractable exact likelihood, inversion, and sampling. Its construction combines simple neural building blocks to retain capacity for complex transformations.

  • 1 INTRODUCTION: NICE learns h = f(x) so the transformed components are independent, making the latent distribution easier to model.
  • 1 INTRODUCTION: The training criterion follows directly from exact log-likelihood under an invertible, dimension-preserving change of variables.
  • 1 INTRODUCTION: The transformation is designed so its Jacobian determinant and inverse are both trivial to compute, enabling easy sampling.
  • 1 INTRODUCTION: A key novelty is combining easy determinant computation and inversion with enough capacity to learn complex nonlinear transformations.
  • 1 INTRODUCTION: The model builds the transformation by splitting x into two blocks and applying coupling-based neural transformations with unit Jacobian determinant and trivial inversion.

2 LEARNING BIJECTIVE TRANSFORMATIONS OF CONTINUOUS

NICE models data by transforming it through an invertible nonlinear map into a simpler factorial prior and fitting the resulting density by maximum likelihood. The formulation addresses preprocessing-induced likelihood inflation through the change-of-variables determinant and supports ancestral sampling through the inverse map.

  • 2 LEARNING BIJECTIVE TRANSFORMATIONS OF CONTINUOUS: NICE learns a continuous, differentiable almost-everywhere nonlinear transformation from the data distribution to a simpler distribution using maximum likelihood.
  • 2 LEARNING BIJECTIVE TRANSFORMATIONS OF CONTINUOUS: With a factorial prior pH, the model maximizes likelihood under a deterministic transform whose latent dimensions are independent.
  • 2 LEARNING BIJECTIVE TRANSFORMATIONS OF CONTINUOUS: The change-of-variables determinant counteracts arbitrary likelihood increases from contracting the data and encourages expansion in high-density regions.
  • 2 LEARNING BIJECTIVE TRANSFORMATIONS OF CONTINUOUS: The inverse transform acts as a decoder, allowing easy ancestral sampling from the directed latent-to-data model.

3 ARCHITECTURE

NICE builds complex bijections by composing transformations with tractable Jacobian determinants and straightforward inverses. Its additive coupling layers provide invertibility and unit Jacobian determinants, while a final diagonal scaling stage restores flexibility.

  • Triangular structure: Composed transformations make forward and inverse computation layerwise, while their Jacobian determinants multiply across layers.This architecture constructs complex bijections from elementary components whose computations remain tractable.
  • Triangular structure: Triangular-Jacobian transformations keep determinants tractable by making diagonal Jacobian elements easy to compute.The design avoids heavily constraining neural-network weights and activations.
  • Coupling layers: A coupling layer partitions the input and transforms one subset through an invertible coupling law conditioned on the other subset.The mapping can be inverted with respect to the transformed subset while leaving the conditioning subset available.
  • Coupling layers: Additive coupling uses g(a; b) = a + b, allowing a neural network coupling function while keeping inversion as inexpensive as the forward transformation.The coupling function may be an unconstrained neural network with the appropriate dimensions.
  • Coupling layers: Additive coupling layers have unit Jacobian determinants and trivial inverses; NICE chooses them for numerical stability with rectified neural networks.The resulting transformation becomes piece-wise linear when the coupling function is a rectified neural network.
  • Combining coupling layers: Alternating partitions across at least three coupling layers let all dimensions influence one another, and NICE generally uses four layers.Because each layer leaves part of its input unchanged, later layers exchange the roles of the subsets.
  • Scaling and prior: A final diagonal scaling matrix addresses the volume-preserving limitation by modeling more variation in some dimensions and less in others.The scaling factors enter the NICE criterion through log(|Sii|), while the prior term and determinant term constrain their behavior.
  • Scaling and prior: NICE uses a factorial prior, selecting standard Gaussian or logistic distributions; the logistic prior is favored for better-behaved gradients.The prior distribution is chosen independently across latent dimensions.

4 RELATED METHODS

NICE differs from earlier generative approaches by using deterministic, invertible encoding with tractable exact likelihood rather than stochastic approximate inference or sampling-based training. It retains efficient ancestral sampling while avoiding variational lower-bound training.

  • Compared with undirected models: Deep Boltzmann machines require MCMC for training and sampling, while their log-likelihood is intractable.Their chains can mix slowly when the target distribution has sharp modes.
  • Compared with variational auto-encoders: Variational auto-encoders use a stochastic encoder and imperfect decoder, so training relies on a reconstruction term and a variational lower bound.The latent code is sampled from q(h | x), which approximates the true posterior p(h | x).
  • Compared with variational auto-encoders: NICE treats its transformation and inverse as a perfect auto-encoder pair, making the reconstruction term constant and leaving the prior and entropy terms.The prior term evaluates the transformed code, while the Jacobian term represents local volume expansion.
  • Connections to related methods: Combining the variational criterion with reparameterization is described as maximizing the joint log-likelihood of (x, ϵ) in a NICE model with two affine coupling layers.The stated construction uses an auxiliary noise variable and a Gaussian prior.
  • Connections to related methods: Unlike maximum-likelihood ICA, NICE learns a richer nonlinear transformation without requiring costly orthogonalization between parameter updates.The related nonlinear approaches cited use proxies such as regularized auto-encoding or variational lower bounds when bijectivity is unavailable.
  • Connections to related methods: Like neural autoregressive density models, NICE exploits triangular structure for tractable density modeling, but its encoding is deterministic and training requires no sampling.The paper contrasts this with stochastic encoding in variational auto-encoders.

5 EXPERIMENTS

NICE is evaluated as a generative density model on four image datasets and additionally applied to MNIST inpainting. The experiments report test log-likelihoods, generated samples, and qualitative inpainting results from a model not trained specifically for that task.

  • Experimental setup: NICE is trained on MNIST, TFD, SVHN, and CIFAR-10 using dequantized and rescaled data.The preprocessing adds uniform noise and maps the data into dataset-specific intervals.
  • Experimental setup: The architecture uses four coupling layers, a final exponential diagonal scaling stage, and dataset-dependent whitening.Approximate whitening is used for TFD, while exact ZCA is used for SVHN and CIFAR-10.
  • Experimental setup: The coupling functions are deep rectified networks with linear outputs, using dataset-dependent hidden-layer widths.MNIST uses five hidden layers of 1000 units; TFD uses four of 5000; SVHN and CIFAR-10 use four of 2000.
  • Experimental setup: The models use logistic priors for MNIST, SVHN, and CIFAR-10, and a standard normal prior for TFD.Training maximizes log-likelihood with Adam and selects the model by validation log-likelihood after 1500 epochs.
  • Log-likelihood and generation: 1980.50, 5514.71, 11496.55, and 5371.78 are the reported test log-likelihoods for MNIST, TFD, SVHN, and CIFAR-10, respectively.The TFD and CIFAR-10 values are compared with reported Deep MFA results, whose value is a variational lower bound.
  • Log-likelihood and generation: The trained models produce samples by sampling from the latent prior and transforming back to data space.The samples are presented as unbiased samples from a trained NICE model.
  • Inpainting: For inpainting, observed pixels are clamped and hidden pixels are optimized by projected gradient ascent on likelihood with Gaussian noise.The procedure keeps inputs within their original value interval and uses an iteration-dependent step size.
  • Inpainting: Although not trained for inpainting, NICE produces qualitatively reasonable MNIST completions, with occasional spurious modes.The results are shown on test examples under multiple masking patterns.

6 CONCLUSION

NICE learns highly non-linear bijective transformations that factorize data distributions while optimizing log-likelihood directly. The model supports efficient unbiased ancestral sampling and achieves competitive log-likelihood results, with potential extensions to other inductive principles and variational models.

  • NICE learns a flexible, highly non-linear bijective transformation that maps training data to a space with a factorized distribution.
  • The framework directly maximizes log-likelihood for the generative model.
  • NICE provides efficient unbiased ancestral sampling and competitive log-likelihood results.
  • The architecture could also be trained with other inductive principles, including toroidal subspace analysis.
  • NICE may support more powerful approximate inference in variational auto-encoders through richer approximate posteriors or priors.

A.1 MANIFOLD VISUALIZATION

The learned manifold is visualized by rotating a 3D sphere in latent space and mapping the rotated sphere back into data space.

  • A random rotation of a 3D sphere in latent space is transformed into data space to illustrate the learned manifold.The resulting manifold is f^-1(R(S)).

A.2 SPECTRUM

The diagonal scaling layer is analyzed through component-wise scale parameters, which indicate the importance assigned to independent components and the success of manifold learning.

  • The diagonal scaling coefficients define component-wise scale parameters when considered jointly with the prior distribution.These parameters are sorted and plotted to examine their spectrum.
  • The scale parameters indicate the importance assigned to each independent component and ultimately how successfully the model learned manifolds.

B APPROXIMATE WHITENING

Approximate whitening is learned within the NICE framework using an affine transformation and a standard Gaussian prior. The resulting model is equivalent to learning a Gaussian distribution with the same optimization procedure as NICE.

  • Approximate whitening uses the NICE framework with an affine function and a standard Gaussian prior.
  • The affine transformation has a lower-triangular matrix L and a bias vector b.
  • This procedure is equivalent to learning a Gaussian distribution.
  • Optimization uses RMSProp with early stopping and momentum, as in NICE.

C VARIATIONAL AUTO-ENCODER AS NICE

The variational auto-encoder formulation uses SGVB with a Gaussian latent prior and examines learned manifold structure through component-wise scale parameters. The resulting spectrum is interpreted as a nonlinear analogue of PCA's eigenspectrum.

  • Variational auto-encoder formulation: SGVB is presented as maximizing log-likelihood for the data and noise pair, using a recognition network.The text identifies the associated Monte Carlo objective as the SGVB cost function.
  • Variational auto-encoder formulation: A standard Gaussian prior p(z) and conditional p(x | z) define the variational model.The formulation also introduces a Gaussian prior on h = (z, ξ) and an equivalent cost expression.
  • Experimental figures: Models trained on TFD and CIFAR-10 are identified in the associated figure labels.The supplied passages provide dataset labels but no comparative performance values.
  • Learned manifold structure: The sorted scale spectrum is described as the nonlinear equivalent of PCA's eigenspectrum, with components d on the x axis and σd on the y axis.This spectrum represents variation across latent dimensions.
Loading 1410.8516v6…