Source-linked AI summary
Neural Autoregressive Distribution Estimation
Benigno Uria, Marc-Alexandre Côté, Karol Gregor, Iain Murray, Hugo Larochelle
TL;DR
Distribution estimation requires flexible models whose probabilities remain tractable, whereas common graphical-model approaches can become intractable or require difficult normalization. The paper introduces NADE, a weight-shared autoregressive neural architecture, and shows competitive modeling of binary and real-valued data, including order-agnostic and convolutional variants.
Problem
Directed and undirected graphical models can make exact probability computation intractable, limiting tractable distribution estimation.
Method
NADE factors distributions into ordered one-dimensional conditionals and parameterizes them with feed-forward neural networks using shared parameters.
Results
NADE achieves competitive results across binary and real-valued observations, supports arbitrary orderings, and extends to convolutional image models.
Takeaways & Limitations
NADE provides a tractable, flexible, and competitive alternative to directed and undirected graphical models for unsupervised distribution estimation.
Takeaways & Limitations
With a fixed variable ordering, exact arbitrary conditional probabilities are computationally intractable and only a limited subset remains tractable.
Abstract
from arXiv · showhide
We present Neural Autoregressive Distribution Estimation (NADE) models, which are neural network architectures applied to the problem of unsupervised distribution and density estimation. They leverage the probability product rule and a weight sharing scheme inspired from restricted Boltzmann machines, to yield an estimator that is both tractable and has good generalization performance. We discuss how they achieve competitive performance in modeling both binary and real-valued observations. We also present how deep NADE models can be trained to be agnostic to the ordering of input dimensions used by the autoregressive product rule decomposition. Finally, we also show how to exploit the topological structure of pixels in images using a deep convolutional architecture for NADE.
1. Introduction
Distribution estimation supports many inference tasks, but common directed and undirected graphical-model approaches can make exact probability computation intractable. NADE provides a tractable autoregressive neural-network alternative that models binary and real-valued data and achieves competitive results.
- Distribution estimation can support classification, regression, missing-value imputation, and other predictive tasks.
- Directed graphical models become intractable as latent-state complexity increases because computing p(x) requires marginalizing over h.
- Undirected graphical models often cannot compute p(x) exactly because of the normalizing constant Z.
- NADE uses autoregressive models and feed-forward neural networks to compute p(x) tractably and efficiently for arbitrary dimension orderings.
- NADE models binary and real-valued observations, supports order-agnostic training and convolutional image architectures, and achieves competitive results against directed and undirected alternatives.
2. NADE
NADE factors a distribution into ordered one-dimensional conditionals and parameterizes them with shared feed-forward neural-network components. Weight sharing reduces parameter and computation costs while retaining a tractable O(HD) evaluation, and the forward pass connects to one-pass RBM mean-field inference.
- The model initially assumes binary observations, while its framework extends to other observation types.
- NADE factors any D-dimensional distribution into D one-dimensional conditionals under an arbitrary ordering of input dimensions.
- Each conditional is parameterized by a feed-forward neural network using the preceding dimensions in the chosen ordering.
- Sharing hidden-layer weights and biases gives NADE O(HD) parameters instead of O(HD2) for separate conditional networks.The shared parameters also allow all hidden layers to be computed in O(HD) time and can reduce over-fitting risk.
- NADE computes p(x) and negative-log-likelihood gradients in O(HD), using maximum likelihood training typically implemented with minibatch gradient descent.
- Relationship with the RBM: NADE’s forward pass is equivalent to one mean-field inference pass for approximating all RBM conditionals, with separate hidden-to-input parameters.
3. NADE for non-binary observations
NADE extends naturally beyond binary observations, including real-valued data modeled with autoregressive conditional distributions. RNADE-MoG uses shared-parameter mixture-density networks and can provide competitive performance relative to mixture models.
- NADE’s framework naturally extends from binary observations to other data types, including real-valued observations.
- A fixed-variance Gaussian conditional is not competitive with mixture models on perceptual datasets.
- RNADE-MoG models each autoregressive conditional with a one-dimensional Gaussian mixture produced by a shared-parameter neural network.
- Parameter sharing in RNADE provides the same computational and statistical advantages as in binary NADE.
- Different one-dimensional conditional families may be appropriate given limited dataset size or domain knowledge about conditional distributions.
- Stochastic gradient descent trains RNADE by minimizing the negative log-density of the training set.
4. Orderless and Deep NADE
DeepNADE makes deep autoregressive density estimation order-agnostic by training one masked network across variable orderings, while retaining tractable computation. Its shared computations support flexible conditional prediction, and ensembles exploit ordering variability for improved estimation.
- Order-agnostic DeepNADE: DeepNADE assigns conditional distributions for any variable given any subset of the others, supporting every ordering with shared parameters.The ordering is sampled uniformly during training, and the expected likelihood over orderings is optimized.
- Deep architecture: Masked inputs restrict each hidden-layer computation to variables currently being conditioned on, while the final layer predicts all variables.The network uses a mask before subsequent nonlinear layers, and its final layer has D units for predictions of any element.
- Training: Training uses masked observations and backpropagates loss only from outputs corresponding to variables not supplied as inputs.A single forward pass computes predictions for all masked variables; outputs for provided variables are ignored.
- Training: The mask must be included alongside masked inputs because otherwise missing variables and zero-valued observations are indistinguishable.The authors report this modification as important for obtaining competitive statistical performance.
- Ensembles of NADE models: Different orderings can assign different probabilities, but averaging uniformly sampled ordering-specific models forms an ensemble that improves estimation without increasing training cost.Ensemble density-estimation cost grows linearly with the number of orderings, while sampling cost remains unchanged.
5. ConvNADE: Convolutional NADE
ConvNADE extends DeepNADE with convolutional layers to exploit image topology while preserving autoregressive conditional modeling. Hybrid convolutional and fully connected architectures perform better than convolution-only models in the reported experiments.
- 5. ConvNADE: Convolutional NADE: ConvNADE replaces DeepNADE’s fully connected hidden layers with convolutional layers to exploit spatial topology while retaining autoregressive modeling.Convolutions preserve input structure and reuse filters across image sub-regions.
- 5. ConvNADE: Convolutional NADE: ConvNADE uses a binary image mask marking pixels preceding each conditional in the chosen ordering.The mask is supplied as an additional convolutional input channel and concatenated with fully connected inputs.
- 5. ConvNADE: Convolutional NADE: The convolutional network preserves the input shape by combining valid and full convolutions, with stride 1 and symmetric filter shapes.Valid convolutions shrink feature maps, whereas full convolutions expand them through zero-padding.
- 5. ConvNADE: Convolutional NADE: Pooling and upsampling can be included in ConvNADE, but the authors observed little improvement from using them.This is reported as an architectural limitation of the explored variants.
- 5. ConvNADE: Convolutional NADE: Hybrid ConvNADE models fuse separate convolutional and fully connected networks at their final layer.The fused output layer aggregates the last-layer pre-activations of both subnetworks.
- 5. ConvNADE: Convolutional NADE: The hybrid architecture works better than using only convolutional layers in the reported experiments.More complex architectures were left for future work.
6. Related Work
NADE is positioned as an autoregressive alternative to directed, undirected, and hybrid generative models. Its defining advantage is tractable likelihood computation without latent stochastic hidden units, enabling exact sampling and unbiased maximum-likelihood gradients.
- 6. Related Work: NADE was motivated by whether a tractable estimator could match the modeling power of an intractable restricted Boltzmann machine.The paper compares its autoregressive approach with established directed and undirected generative models.
- 6. Related Work: NADE’s autoregressive inspiration includes fully visible sigmoid belief networks and neural-network conditionals proposed for earlier autoregressive models.The paper presents NADE as a more flexible parameterized continuation of this line of work.
- 6. Related Work: The related literature includes classical tractable models such as mixture models and Chow–Liu trees, alongside directed, undirected, and hybrid neural networks.The paper compares NADE with classical tractable estimators and situates it among several generative-network paradigms.
- 6. Related Work: Unlike undirected models, NADE does not rely on latent stochastic hidden units and can compute data likelihood exactly for a given ordering.This distinguishes NADE from RBM-based approaches whose normalizing constants can prevent exact probability computation.
- 6. Related Work: Exact likelihood computation also permits efficient exact sampling and unbiased gradients for maximum-likelihood training.These properties are presented as consequences of NADE’s absence of latent stochastic hidden units.
7. Results
On binary-vector benchmarks, fixed-order NADE is competitive with tractable baselines, while order-agnostic NADE is overall strongest. Fixed-order performance is robust to ordering, yet ensembling helps because different orderings succeed on different examples.
- 7. Results: The binary-vector evaluation compares NADE variants with mixture, RBM, fully visible sigmoid belief network, Chow–Liu, and MADE baselines.The experiments restrict comparisons to estimators whose p(x) can be evaluated exactly on test items.
- 7. Results: NADE with a fixed input ordering achieves very competitive performance against the tractable baselines.The comparison uses average log-likelihood on eight binary-vector datasets.
- 7. Results: The order-agnostic NADE variant is among the top-performing models on 5 of 8 datasets.The authors identify it as the overall best method in this benchmark comparison.
- 7. Results: Fixed-order NADE is robust to input-order changes, with average-log-likelihood standard deviations of 0.045 on Mushrooms, 0.05 on DNA, and 0.15 on NIPS-0-12.These values quantify variation across orderings for the named datasets.
2. Does the use of a convolutional architecture improve the performance of NADE?
Convolutional architectures improve NADE’s performance on binarized MNIST, while masking information and combining convolutional with fully connected layers further help. The resulting ConvNADE+DeepNADE remains slightly below Pixel RNN but supports random orderings.
- Results: Order-agnostic NADE with two hidden layers is competitive with intractable generative models.DeepNADE also supports marginalization and conditional sampling, using a convenient variable ordering for these inference tasks.
- Results: Convolutions improve NADE’s performance on binarized MNIST.The comparison concerns models exploiting versus ignoring the image’s 2D topology.
- Results: Providing information about which inputs are masked is essential for obtaining good results.
- Results: Combining convolutional and fully connected layers helps performance.ConvNADE+DeepNADE aggregates a separate DeepNADE model at the end of the convolutional network.
- Results: ConvNADE+DeepNADE performs slightly worse than Pixel RNN, although it is order-agnostic while Pixel RNN requires a fixed ordering.
- Real-valued observations: For real-valued observations, RNADE with mixture-of-Gaussian conditionals belongs to the statistically significant group of best models across all evaluated datasets.RNADE-MoG can capture hard thresholds and heteroscedasticity.
- Real-valued observations: 3.3 nats per patch higher log-density was obtained on average by the best RNADE model than by a MoG fitted with the same training data.
- Real-valued observations: 15 nats higher likelihood per test example was obtained by the best RNADE than by the best mixture of Gaussians on the TIMIT core-test dataset.Samples from both models looked like blurred spectrograms, while RNADE appeared to capture sharper formant structures.
8. Conclusion
The paper presents NADE as a tractable, flexible, and competitive alternative for unsupervised distribution estimation. Subsequent work extended NADE to additional settings, including documents, images, music, natural images, and convolutional or recurrent architectures.
- Conclusion: NADE is a tractable, flexible alternative to directed and undirected graphical models for unsupervised distribution estimation.
- Conclusion: NADE has been extended to topic modeling of documents and images and to modeling sequential music data.
- Conclusion: Other NADE models have been proposed for natural images using convolutional and LSTM hidden units.