Source-linked AI summary
RNADE: The real-valued neural autoregressive density-estimator
Benigno Uria, Iain Murray, Hugo Larochelle
TL;DR
RNADE addresses joint density estimation for real-valued vectors by extending NADE with shared-parameter mixture density networks in an autoregressive factorization. Its likelihood is tractable for optimization and comparison, and it outperformed mixture models on all but one evaluated dataset, while remaining subject to neural-network hyperparameter choices and output-family limitations.
Problem
Flexible joint density estimation for real-valued vectors remains needed for heterogeneous and perceptual data beyond binary-variable models.
Method
RNADE factorizes a vector density into ordered one-dimensional conditionals modeled by mixture density networks with shared neural parameters.
Results
RNADE outperformed mixture models on all datasets considered except image patches, where performance was close to a reported state-of-the-art mixture model.
Takeaways & Limitations
RNADE provides a black-box density estimator whose likelihood computation time and parameter count scale linearly with dataset dimensionality.
Takeaways & Limitations
RNADE requires choosing several training hyperparameters, and its suitability can depend on the chosen one-dimensional output family.
Abstract
from arXiv · showhide
We introduce RNADE, a new model for joint density estimation of real-valued vectors. Our model calculates the density of a datapoint as the product of one-dimensional conditionals modeled using mixture density networks with shared parameters. RNADE learns a distributed representation of the data, while having a tractable expression for the calculation of densities. A tractable likelihood allows direct comparison with other methods and training by standard gradient-based optimizers. We compare the performance of RNADE on several datasets of heterogeneous and perceptual data, finding it outperforms mixture models in all but one case.
1 Introduction
The paper motivates flexible real-valued density models by extending successful distributed-representation approaches beyond binary data. RNADE combines tractable autoregressive densities with neural-network flexibility for heterogeneous and perceptual data.
- Real-valued probabilistic models must impose structure because general distributions become exponentially expensive to describe as dimensionality grows.
- Image restoration requires modeling multivariate distributions over uncorrupted pixel patches, where Gaussian mixtures are strong practical baselines.
- RBMs can outperform mixture models on binary data by using distributed representations and exponentially many effectively parameter-tied mixture components.
- RNADE extends NADE to real-valued vectors, combining tractable density computation with mixture-based conditionals whose parameters vary nonlinearly with context.
2 Background: Autoregressive models
Autoregressive models factor a joint distribution into ordered one-dimensional conditionals, enabling tractable likelihood computation and gradient-based fitting. NADE adds shared neural parameters, reducing model size and computation while retaining flexible conditional representations.
- The chain rule factorizes any joint distribution into one-dimensional conditionals ordered over the variables.
- Choosing tractable parametric conditionals makes joint densities computable and permits local penalized maximum-likelihood training with gradient-based optimizers.
- NADE models binary conditionals with one-hidden-layer neural networks using parameter sharing inspired by mean-field RBM approximations.
- Tied input-to-hidden weights reduce the parameter count from quadratic to linear in input dimensionality.
- Shared hidden-activation computations reduce datapoint probability evaluation to O(DH) time.
- Untying NADE output weights from input weights improved statistical performance across tasks with negligible extra computational cost.
3 Real-valued neural autoregressive density estimators
RNADE extends NADE to real-valued vectors by modeling each one-dimensional conditional with a shared-parameter mixture density network. The model supports flexible conditional distributions while retaining tractable likelihood computation and efficient parameter sharing.
- Parameter sharing: Tied input-to-hidden weights reduce the parameter count from quadratic to linear in input dimensionality and allow probability computation in O(DH) time.Shared hidden activations are reused across conditionals.
- Model construction: RNADE replaces NADE’s Bernoulli conditionals with mixture-of-Gaussians mixture density networks for real-valued data.The conditional parameters are computed from preceding dimensions x<d using a shared hidden layer.
- Model construction: Its autoregressive density factorizes the joint distribution into one-dimensional conditionals, each modeled as a mixture of Gaussians.The mixture parameters for conditional d are θd = {αd, µd, σd}.
- Hidden units: Preliminary experiments favored rectified-linear or learned-rescaled hidden units over sigmoidal units in RNADE.The authors used rectified-linear units throughout except for one sigmoidal example, and found learned rescaling worked slightly better.
- Optimization: Learning uses minibatch stochastic gradient ascent on likelihood, with gradient scaling for mixture means enabling higher learning rates empirically.Multiplying each mean gradient by its standard deviation makes tight components move more slowly than broad ones.
- Output distributions: RNADE supports alternative output families: RNADE-MoG uses Gaussian mixtures, while RNADE-MoL uses mixtures of Laplacian outputs.The Gaussian choice is described as arbitrary, with alternatives potentially better suited to particular data.
4 Experiments
Experiments compare RNADE with mixture-based baselines on low-dimensional tabular data, natural-image patches, and speech acoustics using held-out log-likelihood. RNADE methods outperform mixture models on the UCI datasets and TIMIT, but RNADE trails a previously trained MoG on natural-image patches.
- Low-dimensional data: The UCI evaluation used held-out log-likelihood with 10-fold splits, cross-validated hyperparameters, and normalized continuous attributes.The baselines included full-covariance Gaussians and mixtures of factor analyzers.
- Low-dimensional data: Autoregressive methods achieved statistically superior performance to mixture models on all five UCI datasets.RNADE-MoG was among the statistically significant group of best models on every dataset.
- Natural image patches: Natural-image experiments modeled 8-by-8 monochrome BSDS300 patches, adding uniform noise before scaling pixel values to [0, 1].Models were trained on random patches and evaluated on one million randomly sampled test patches.
- Natural image patches: RNADE test log-likelihood was on average 0.7 nats per patch lower than Zoran and Weiss’s MoG on natural-image patches.The comparison used an RNADE with 512 rectified-linear hidden units and 20 one-dimensional Gaussian components per output.
- Natural image patches: Randomizing pixel order made little difference to natural-image results, whose performance changes were comparable to variation across runs with one ordering.The reported results used raster-scan order.
- Speech acoustics: RNADE obtained on average 10 nats more per test example than a MoG on the TIMIT core-test dataset.Both models’ log-likelihoods were measured on the complete core-test set.
5 Discussion
RNADE applies one-dimensional mixture density networks within an autoregressive framework, with output-distribution choice affecting predictions across context sizes. The model generalizes across heterogeneous and perceptual datasets, outperforming mixture models except on image patches where performance was close to a state-of-the-art mixture model.
- Model variants: RNADE applies practical one-dimensional mixture density networks to unsupervised density estimation through an autoregressive framework.This avoids the quadratic target-parameter growth of multidimensional mixture density networks.
- Model variants: Natural-image pixel brightness is heavy-tailed and closer to a Laplace distribution than a Gaussian.
- Model variants: RNADE-MoL predicts the first image-patch pixel better than RNADE-MoG, while Gaussian outputs predict later pixels better.The comparison is illustrated in Figure 3b, c, and f.
- Limitations and extensions: The mixture of Laplace model is unsuitable for predicting with large contexts, motivating more flexible one-dimensional output forms.The discussion suggests scale mixtures for image patches but considers them too restrictive for general applications.
- Limitations and extensions: A main drawback is the need to choose several training hyperparameters.The paper notes that automatic learning-rate adjustment and more efficient hyperparameter methods may improve performance.
- Overall performance: RNADE generalizes across arbitrary feature vectors, image patches, and auditory spectrograms, outperforming mixture models on all other considered datasets.Image-patch performance was close to a recently reported state-of-the-art mixture model.
A Implementation details
RNADE computes densities by traversing dimensions autoregressively with shared hidden activations and mixture-of-Gaussian conditionals. Learning uses recursively backpropagated gradients, with manual implementation and automatic differentiation both available.
- Density computation: Algorithm 1 computes p(x) by multiplying one-dimensional mixture-of-Gaussian conditional densities while updating activations recursively across dimensions.
- Gradient computation: Algorithm 2 computes learning gradients by backpropagating errors from d = D down to d = 1.The gradients are calculated recursively due to the autoregressive dependencies.
- Density computation: Shared parameters include the hidden-layer weights and output-layer parameters for mixture weights, means, and scales.The RNADE parameter set contains ρ, W, c, and the α, µ, and σ output parameters.
- Gradient computation: Training maximizes log-likelihood with gradient ascent, using either automatic differentiation libraries or a faster manual implementation.The reported speed advantage is attributed possibly to cache-friendly recomputation of activation terms.