Source-linked AI summary
A Deep and Tractable Density Estimator
Benigno Uria, Iain Murray, Hugo Larochelle
TL;DR
NADE’s fixed variable ordering makes arbitrary conditioning and marginalization difficult, while deep versions lack an efficient formulation. The paper shares parameters to train models for all orderings, adds input masks, and enables on-the-fly ensembles. The resulting models provide tractable inference, with strong density-estimation performance and reported state-of-the-art results on BSDS300.
Problem
NADE handles conditioning and marginalization conveniently only for variables aligned with its fixed ordering, while deep NADE lacks an efficient formulation.
Method
The procedure shares parameters across NADEs for every variable ordering, uses input masks to distinguish conditioning patterns, and forms ensembles without extra training cost.
Results
Order-agnostic NADEs outperform mixture models across investigated datasets, while ensembles improve performance on most analyzed datasets and achieve the best reported BSDS300 performance.
Takeaways & Limitations
The learned shared parameters provide convenient ordering selection, fast exact marginalization and sampling, and on-the-fly ensembles for inference tasks.
Takeaways & Limitations
Deep NADE training previously incurred O(DH2L) complexity, making it impractical for high-dimensional datasets.
Abstract
from arXiv · showhide
The Neural Autoregressive Distribution Estimator (NADE) and its real-valued version RNADE are competitive density models of multidimensional data across a variety of domains. These models use a fixed, arbitrary ordering of the data dimensions. One can easily condition on variables at the beginning of the ordering, and marginalize out variables at the end of the ordering, however other inference tasks require approximate inference. In this work we introduce an efficient procedure to simultaneously train a NADE model for each possible ordering of the variables, by sharing parameters across all these models. We can thus use the most convenient model for each inference task at hand, and ensembles of such models with different orderings are immediately available. Moreover, unlike the original NADE, our training procedure scales to deep models. Empirically, ensembles of Deep NADE models obtain state of the art density estimation performance.
1. Introduction
Probabilistic models support conditioning, prediction, and marginalization, but many useful inference tasks require approximate computation. The paper introduces shared-parameter training for NADE models across all variable orderings, enabling convenient ordering selection and ensembles.
- Probabilistic models represent multidimensional data with joint distributions and support conditioning, prediction, and marginalization over unobserved variables.
- Exact solutions to many inference tasks are unavailable, motivating iterative approximations such as Monte Carlo and variational methods.
- NADE and RNADE are competitive joint-density models that sequentially predict variables using an arbitrary ordering fixed during training.
- The proposed procedure simultaneously trains a NADE for every possible ordering by sharing parameters and optimizing the mean cost across orderings.
- A trained shared-parameter system can provide the ordering most convenient for an inference task and generate ensembles of models with different orderings.
2. Background: NADE and RNADE
NADE factorizes a joint density into sequential one-dimensional conditionals, with efficient recursive hidden-unit computation and direct sampling. Its fixed ordering limits arbitrary inference, while deep extensions incur cubic layer-size complexity.
- Autoregressive models factorize a D-dimensional density into one-dimensional conditional distributions under an ordering of the variables.
- NADE uses tied-parameter neural networks to calculate each conditional, while RNADE extends the approach to real-valued data with mixture density networks.
- NADE supports gradient-based likelihood training, tractable datapoint densities, and direct ancestral sampling without Markov chain Monte Carlo.
- Recursive hidden-unit activations reduce NADE training and evaluation complexity to O(DH), compared with O(D2H) for untied autoregressive networks.
- Inference is exact when conditioned variables precede and marginalized variables follow the ordering, but arbitrary probabilistic queries generally require approximation.
- Deep NADE extensions lack an efficient recursive formulation, causing O(DH2L) complexity and making them impractical for high-dimensional datasets.
3. Training a factorial number of NADEs
The method avoids explicitly training O(D!) ordering-specific NADEs by tying parameters and optimizing an ordering-averaged objective stochastically. Input masks distinguish observed inputs from dimensions to predict, while training remains tractable.
- 3. Training a factorial number of NADEs: Training separate NADEs for all orderings would require O(D!) time and memory, so the paper instead minimizes their mean negative log-likelihood through shared parameters.
- 3. Training a factorial number of NADEs: The method uses shared parameters across ordering-specific models, whose dth conditional depends only on the ordering’s first d variables.
- 3. Training a factorial number of NADEs: The ordering-averaged objective is approximated by sampling datapoints, dimensions, and preceding variable subsets, while summing cheaply over candidate predicted dimensions.
- 3. Training a factorial number of NADEs: A stochastic training update costs O(DH + H2L), matching regular multilayer neural networks, although test-time evaluation costs O(DH2L).
- 3.1. Improved parameter sharing using input masks: Input masks indicate which dimensions are present in the conditioning set, distinguishing zero-valued observed inputs from dimensions that must be predicted.
4. On the fly generation of NADE ensembles
Different variable orderings define distinct NADE hypotheses, enabling ensembles whose size and computational trade-offs can be adjusted after training. The shared training procedure generates these ensemble components without increasing training cost as ensemble size grows.
- Ensemble disagreement can improve estimates relative to a set of consistent models.
- Each input ordering defines a distinct NADE hypothesis space and inductive bias despite shared parameters.
- The procedure generates ensembles by uniformly sampling input orderings and averaging their probabilities.
- Ensemble density-estimation cost grows linearly with ensemble size, while sampling complexity remains unchanged.Sampling selects an ordering at random and uses its corresponding NADE.
- Training cost remains unchanged, and the number of ensemble components can be selected after training or adapted to available computation.
5. Related work
Prior work explored autoregressive density estimation, stochastic orderings, related variational procedures, and masking-noise training. This work differs by combining order-agnostic training with tractable density estimation through NADE models.
- Earlier binary autoregressive models generally selected and retained a single, usually random, input ordering during training.
- Gregor and LeCun trained under stochastic orderings but observed worse performance than with a single ordering, motivating input-mask-based parameter sharing.
- Training deep Boltzmann machines with generalized pseudo-likelihood resembles predicting a subset of dimensions from the remaining values.
- The procedure resembles masking-noise denoising autoencoders but averages reconstruction only over missing inputs.
- Unlike denoising autoencoders, the trained NADE models serve as tractable density estimators.
6. Experimental results
Experiments evaluate order-agnostic NADE and RNADE models across binary, real-valued, and image datasets. Ensembles generally improve density estimation and enable convenient marginalization and sampling, while depth benefits depend on the dataset.
- Binary UCI datasets: Order-agnostic NADEs achieved test log-likelihoods close to fixed-ordering NADEs across eight binary UCI datasets, while ensembles obtained higher values on all datasets.The experiments used average test-set log-likelihood per datapoint, with comparisons against mixture models, RBMs, fully visible Bayesian networks, and fixed-ordering NADEs.
- Binarized MNIST: Ensembles of order-agnostic NADEs outperformed fixed-ordering NADEs and could surpass RBMs while approaching a larger two-hidden-layer deep belief network on MNIST.Single one- and two-hidden-layer order-agnostic NADEs were marginally worse than fixed-ordering NADEs but much better than multivariate Bernoulli mixtures.
- Binarized MNIST: Input masks were necessary for order-agnostic NADEs to obtain competitive results.The masks indicate which dimensions are present in the input and allow the shared model to distinguish ordering and conditioning contexts.
- Binarized MNIST: A two-hidden-layer order-agnostic NADE produced recognizable binarized-MNIST digit samples, and many first-layer receptive fields resembled pen strokes.Associated mask receptive fields tended to activate units when unknown regions were flanked by known regions.
- Inference tasks: With a convenient pixel ordering, the model calculated marginal densities and sampled hollowed image regions in constant time without approximate inference.The figure demonstrates marginalization of random 10 by 10 pixel regions and sampling for the missing regions.
- Real-valued datasets: On real-valued UCI datasets, order-agnostic RNADEs were close to fixed-ordering RNADEs on two datasets but worse on Parkinsons, while ensembles improved performance on all three.On natural-image patches, RNADEs with more than three hidden layers beat the fixed-ordering NADE and Gaussian-mixture baselines; fewer than three layers did not.
7. Conclusions
The procedure jointly trains order-specific NADEs with shared parameters, supports deep models efficiently, and enables ensembles without extra training cost. These models achieve strong empirical density-estimation results, including best reported performance on BSDS300 8×8 image patches.
- Training procedure: The procedure simultaneously fits a NADE for every variable ordering while sharing parameters across models.It optimizes the mean cost over orderings and permits extracting a convenient ordering after training.
- Deep models and ensembles: Deep NADE training requires only a linear increase in computation under the new procedure.The method also constructs ensembles on the fly without additional training computational cost.
- Empirical results: NADEs trained with this procedure outperform mixture models across all investigated datasets.Several hidden layers are usually needed to match or surpass fixed-ordering NADEs.
- Empirical results: Order-agnostic models achieve the best reported statistical performance on the BSDS300 8×8 image-patches dataset.Ensembles improve statistical performance on most analyzed datasets, with only a mild test-time cost.