Source-linked AI summary
Deep AutoRegressive Networks
Karol Gregor, Ivo Danihelka, Andriy Mnih, Charles Blundell, Daan Wierstra
TL;DR
The paper addresses efficient generative modeling with tractable inference and non-iterative sampling. It introduces DARNs, which combine autoregressive stochastic hidden layers with MDL-based joint training. DARNs achieve favorable results across UCI data, MNIST, and Atari frames, while exact calculation of the learning objective remains intractable.
Problem
Existing probabilistic generative models often make observation-to-representation or representation-to-observation mappings intractable, while probabilistic autoencoder samplers are often iterative and correlated.
Method
DARNs are deep generative autoencoders with autoregressive stochastic hidden layers, trained by jointly backpropagating an MDL cost through the encoder and decoder.
Results
DARN achieved better test log-likelihood than reported baseline models on four of eight binary UCI data sets and favorable performance on MNIST and Atari frames.
Takeaways & Limitations
DARN combines efficient exact ancestral sampling with deep stochastic representations and scalable generative modeling across several classic data sets.
Takeaways & Limitations
The paper approximates the otherwise intractable objective with a Monte Carlo method.
Abstract
from arXiv · showhide
We introduce a deep, generative autoencoder capable of learning hierarchies of distributed representations from data. Successive deep stochastic hidden layers are equipped with autoregressive connections, which enable the model to be sampled from quickly and exactly via ancestral sampling. We derive an efficient approximate parameter estimation method based on the minimum description length (MDL) principle, which can be seen as maximising a variational lower bound on the log-likelihood, with a feedforward neural network implementing approximate inference. We demonstrate state-of-the-art generative performance on a number of classic data sets: several UCI data sets, MNIST and Atari 2600 games.
1. Introduction
DARNs are deep generative autoencoders designed to combine efficient two-way inference with fast, exact ancestral generation. They use autoregressive stochastic hidden layers and train the joint encoder–decoder by an MDL cost linked to variational free energy.
- Motivation: DARNs extend earlier probabilistic autoencoders by placing autoregressive connections among stochastic hidden units rather than relying only on iterative approximate sampling.The paper positions this design as distinct from prior autoregressive models with deterministic hidden units.
- Contribution: DARNs generate independent, exact samples efficiently through top-down ancestral sampling in the decoder.Sampling starts at the deepest hidden layer and proceeds downward, sampling one unit at a time.
- Learning principle: DARN training minimizes the total information needed to reconstruct inputs by backpropagating an MDL cost through the joint encoder and decoder.The MDL objective coincides with minimizing Helmholtz variational free energy.
- Architecture: Autoregressive hidden-layer connections capture within-layer dependencies at low computational cost during learning and generation.This avoids the potentially prohibitive costs associated with within-layer undirected connections.
2. Model Architecture
DARN is a layered generative autoencoder with a stochastic encoder, autoregressive decoder prior, and decoder conditional. Its architecture supports deeper stochastic and deterministic representations while retaining efficient ancestral sampling.
- Core components: DARN comprises an encoder q(H|X), a decoder prior p(H), and a decoder conditional p(X|H).The encoder infers representations, the prior generates representations, and the conditional produces observations from them.
- Autoregressive prior: The decoder prior factorizes each binary hidden representation autoregressively, conditioning each unit on preceding units.The conditional p(h_j|h_1:j−1) models the activity of the current unit given earlier activities.
- Deeper architectures: DARN increases representational power by adding stochastic hidden layers, deterministic hidden layers, or more flexible autoregressive parameterizations.Additional stochastic units increase representation capacity, while deterministic layers add nonlinear transformations.
- Sampling: DARN sampling proceeds layer by layer from the top-most stochastic layer to the observation, while encoder sampling runs in the opposite direction.Each hidden unit is sampled sequentially from its preceding units and the relevant adjacent layer.
- Computational scaling: With sparse stochastic hidden units, fDARN has expected sampling complexity O(ns(nx + nh)).Without observation autoregression, a single stochastic hidden-layer DARN has complexity O(nh(nx + nh)).
3. Minimum Description Length and Autoencoders
The paper frames autoencoder training as minimizing expected communication cost under MDL. This objective recovers Helmholtz variational free energy and jointly optimizes the generative model and encoder.
- MDL formulation: MDL trains the autoencoder to compress data by encoding a representation and then communicating the residual relative to that representation.The representation is intended to provide a more concise code than the original datum.
- Coding objective: Description length is determined by the negative logarithm of a variable’s probability, so training minimizes expected coding cost.The encoder probability q(h|x) contributes to the representation’s coding cost under bits-back coding.
- Variational interpretation: Substituting the coding terms recovers Helmholtz variational free energy.The resulting objective is the quantity used to assess the expected description length of data and representations.
- Optimization: The encoder and generative model are optimized simultaneously, unlike variational procedures that use coordinate descent.The minimizing encoder corresponds to the posterior p(H|X) when the variational free-energy objective is minimized.
4. Learning
DARN learning jointly trains encoder and decoder parameters by minimizing an MDL objective, using sampled representations and approximate gradients. The procedure updates the autoencoder through backpropagation, with implementation choices that support efficient computation.
- Encoder and decoder weights and biases θ are jointly trained to minimise the expected description-length objective in Eq. 12.
- Exact calculation of Eq. 15 is intractable, so learning uses a Monte Carlo approximation.
- For each observation x, the procedure samples a representation h from q(H|x), evaluates q(h|x), p(x|h), and p(h), then calculates the gradient.
- The autoencoder parameters are updated by following the gradient ∇θL(x).
- Backpropagation through stochastic binary units can produce highly biased gradients, motivating approximations to the gradient calculation.
- When the encoder is not autoregressive, standard matrix operations and point-wise nonlinearities make the system suitable for GPU implementation.
5. Results
DARN was evaluated on binary UCI data, binarised MNIST, and Atari frames using likelihood, sampling, and representation-quality analyses. It achieved competitive or improved results on several benchmarks, with speed and depth benefits alongside limitations on irregular game frames.
- Binary UCI data sets: DARN achieved better test log-likelihood than reported baseline models on four of eight binary UCI data sets.
- Binarised MNIST data set: On binarised MNIST, DARN with 9 stochastic hidden units nearly matched a 500-component mixture of Bernoullis, while the 500-unit model compared favourably with DBMs and DBNs.Networks with 16 or fewer stochastic hidden units permitted exact log-likelihood evaluation; larger models used importance sampling with repeated estimates and confidence intervals.
- Binarised MNIST data set: Figure 2 compares generated MNIST samples with nearest training examples and shows sample probabilities for models trained directly on pixel intensities.The caption states that the generated samples were not simple memorisations of training examples.
- Binarised MNIST data set: 96.1 was the upper-bound negative log-likelihood for a 400-unit fDARN with 5% sparsity, while generation required 2.4×10^4 multiplications per sample versus NADE’s 2.2 × 10^6.This was described as a nearly 100-fold speedup, although the likelihood was worse and samples appeared reasonable by ocular inspection.
- Binarised MNIST data set: A 12-stochastic-layer DARN with 80 stochastic units per layer achieved the best upper bound among DARN models, although only that upper bound was evaluated.The architecture used 400 tanh units in each deterministic layer and skip connections from previous stochastic layers.
- Atari 2600 game frames: On Atari frames, DARN captured scores, objects, and object correlations, but failed to generate reasonable frames for less regular games such as River Raid and Sea Quest.The games were ordered from left to right by decreasing log probability.
6. Conclusion
DARN is a deep generative autoencoder with autoregressive stochastic hidden units, enabling high-quality samples and efficient, scalable training. Its MDL-based stochastic training procedure supports state-of-the-art performance across multiple data sets.
- DARN combines a stochastic encoder with autoregressive stochastic hidden units in a deep generative autoencoder.The architecture captures high-level structure while retaining efficient and tractable inference.
- Training backpropagates an MDL cost through the joint model, approximately minimizing Helmholtz variational free energy with a Monte Carlo procedure.
- DARN samples efficiently, trains efficiently, and scales to locally connected and convolutional architectures.
- The experiments report state-of-the-art performance on multiple data sets.
A. Derivation of Gradients
The gradient method estimates derivatives through stochastic binary hidden units using Monte Carlo sampling and a variance-reducing baseline. The baseline is chosen through a Taylor approximation, but the resulting estimator is biased for non-quadratic functions.
- Monte Carlo sampling estimates the gradient of the expected network contribution when stochastic binary units are sampled from q(h_i).The estimator is unbiased in principle but can have high variance.
- A baseline b based on control variates is introduced to reduce Monte Carlo gradient variance while preserving unbiasedness under the required condition.The baseline should correlate with f(h_i), have low variance, and make the expected score-function baseline term zero.
- The chosen baseline is a first-order Taylor approximation of f evaluated at h′, with derivatives obtained by backpropagation.
- For linear f any h′ can be used, whereas higher-order cases impose restrictions on the Taylor-expansion point.
- The estimator is biased for non-quadratic functions.
- Backpropagation with respect to a sampled binary value estimates the gradient with respect to q(H_i=1) after scaling by 1/(2q(h_i)).