Source-linked AI summary
MADE: Masked Autoencoder for Distribution Estimation
Mathieu Germain, Karol Gregor, Iain Murray, Hugo Larochelle
TL;DR
Distribution estimation is broadly useful but difficult in high dimensions, while existing autoregressive estimators can be computationally expensive. MADE masks autoencoder connections to enforce autoregressive conditional predictions, yielding tractable distribution estimators that are competitive and faster than existing alternatives. The approach also extends to deep networks and multiple input orderings.
Problem
High-dimensional distribution estimation is challenging, and previous deep autoregressive models require O(D) more computation than simple neural-network point predictors.
Method
MADE masks autoencoder connections so each output depends only on preceding inputs, making outputs conditional probabilities whose product gives the joint probability.
Results
MADE is competitive with tractable distribution estimators, preserves single-pass autoencoder efficiency, and scales better than existing autoregressive alternatives.
Takeaways & Limitations
The framework supports deep architectures, multiple input orderings, and straightforward vectorized GPU implementations.
Takeaways & Limitations
Unrestricted autoencoders can learn trivial copying representations, producing an implied distribution that is not properly normalized.
Abstract
from arXiv · showhide
There has been a lot of recent interest in designing neural network models to estimate a distribution from a set of examples. We introduce a simple modification for autoencoder neural networks that yields powerful generative models. Our method masks the autoencoder's parameters to respect autoregressive constraints: each input is reconstructed only from previous inputs in a given ordering. Constrained this way, the autoencoder outputs can be interpreted as a set of conditional probabilities, and their product, the full joint probability. We can also train a single network that can decompose the joint probability in multiple different orderings. Our simple framework can be applied to multiple architectures, including deep ones. Vectorized implementations, such as on GPUs, are simple and fast. Experiments demonstrate that this approach is competitive with state-of-the-art tractable distribution estimators. At test time, the method is significantly faster and scales better than other autoregressive estimators.
1. Introduction
Distribution estimation is broadly useful but becomes difficult in high dimensions because the volume requiring accurate probability estimates grows exponentially. MADE adapts autoencoders with autoregressive masks to provide tractable, scalable estimation while preserving a single-pass computation.
- Distribution estimation underlies tasks including classification, denoising, missing-input imputation, and data synthesis.
- The curse of dimensionality makes distribution estimation harder as input dimensions increase because the space requiring accurate p(x) estimates grows exponentially.
- Autoregressive models compute p(x) exactly, but previous deep models require O(D) more computation than a simple neural-network point predictor.
- MADE masks a standard autoencoder’s weighted connections so each input dimension is reconstructed only from preceding dimensions in a chosen ordering.
- MADE preserves a single regular-autoencoder pass, supports straightforward GPU implementation, and is explored in deep and multiple-ordering variants.
2. Autoencoders
The paper introduces autoencoders as flexible reconstruction models trained with binary cross-entropy, while highlighting that unrestricted copying can produce trivial representations and invalid likelihoods.
- An autoencoder learns a hidden representation from which it reconstructs the input, using nonlinear transformations with input-to-hidden and hidden-to-output weights.
- For binary observations, cross-entropy is a natural training loss and can be interpreted as negative log-likelihood when reconstructions represent probabilities.
- Deep autoencoders are obtained by inserting additional hidden layers between the input and output layers.
- If the hidden layer is at least as large as the input, units can copy individual dimensions and achieve perfect reconstruction.
- Perfect reconstruction means the implied distribution need not be properly normalized, so the autoencoder loss is not necessarily a valid log-likelihood.
3. Distribution Estimation as Autoregression
The paper resolves the autoencoder likelihood problem by imposing an autoregressive structure: each output predicts one dimension from only its preceding inputs, making the joint probability a product of valid conditionals.
- The probability product rule decomposes a joint distribution into a product of nested conditional probabilities.
- Interpreting each reconstruction output as p(x_d = 1 | x<d) makes binary cross-entropy a valid negative log-likelihood.
- A proper distribution estimator requires each output p(x_d | x<d) to depend only on preceding inputs and not on x≥d.
- The autoregressive property makes likelihood computation equivalent to sequentially predicting each input dimension.
4. Masked Autoencoders
MADE enforces autoregressive dependencies by masking connections so forbidden computational paths from current or later inputs to each output are removed. Hidden-unit assignments determine which connections remain allowed.
- To enforce autoregression, each output must have no computational path from the current or later input units.
- Binary masks zero selected weights in the input-to-hidden and hidden-to-output matrices.
- Each hidden unit receives an integer m between 1 and D−1 that limits the maximum number of input units it may connect to.
- The output mask connects output d only to hidden units with m(k)<d, ensuring dependence on at most the preceding inputs.
- The product of the connectivity masks represents input-to-output connectivity and supports demonstrating the resulting autoregressive property.
M V,W
MADE enforces autoregressive dependencies by masking connections, and extends this construction to deep networks, multiple orderings, and connectivity-agnostic training. These masks preserve efficient computation while enabling valid probabilistic modeling.
- M V,W: The path-count matrix M V,W must be strictly lower diagonal so each output depends only on preceding inputs.The matrix entry counts network paths between an output unit and an input unit; entries are zero when d′ ≤ d.
- 4.1. Deep MADE: MADE extends autoregressive masking to deep architectures by assigning connectivity constraints to units across all hidden layers.Each layer connects units only when the preceding layer’s maximum-input constraint is no greater than the receiving unit’s constraint.
- 4.1. Deep MADE: The output mask uses the last hidden layer’s connectivity constraints, permitting an output d only when d exceeds mL(k).The same mask definition also applies to the first hidden layer by treating the input layer as layer zero.
- 4.2. Order-agnostic training: Order-agnostic training samples input orderings during updates, allowing one parameter set to represent models for many orderings.The ordering is encoded by m0, obtained by permuting [1, . . . , D].
- 4.3. Connectivity-agnostic training: Connectivity-agnostic training resamples hidden-unit constraints for each example or minibatch because mask construction is easy to parallelize.Algorithm 1 jointly samples orderings and layer connectivity constraints before constructing the masks and computing p(x).
- 4.3. Connectivity-agnostic training: Sampling masks for every example could over-regularize MADE and cause underfitting, so the method also considers cycling through a finite mask list.At test time, probabilities from the masks in that list are averaged.
5. Related Work
Earlier feed-forward generative models established tractable autoregressive estimation but faced costly or intractable probability evaluation. MADE extends this line with deeper and ordering-flexible models while retaining efficient exact evaluation.
- Feed-forward architectures can form state-of-the-art tractable distribution estimators, motivating MADE’s autoregressive design.
- Deep NADE improves distribution estimation but requires D network passes to evaluate p(x) for a D-dimensional test vector.
- DARN permits fast simulation, but exact test probabilities require summing over latent configurations, making computation exponential.
- MADE uses one autoencoder pass for exact probabilities, whereas NADE requires D passes; MADE is polynomial and single-layer MADE is observed to be an order of magnitude faster than one-layer NADE.
- MADE extends the single-hidden-layer architecture of Bengio and Bengio by exploring deep variants and training across multiple orderings and connectivity structures.
- MADE’s autoregressive masks resemble structured dropout or dropconnect, but they are constrained to preserve the autoregressive property.
6. Experiments
Experiments evaluate MADE on UCI binary datasets and binarized MNIST using negative log-likelihood, hyperparameter searches, and comparisons with other estimators. MADE is competitive across datasets, outperforms single-layer NADE on MNIST, and exhibits a mask-dependent regularization trade-off.
- Evaluation setup: The evaluation uses seven UCI binary datasets plus OCR-letters and binarized MNIST, reporting average test negative log-likelihood.The MNIST split contains 50,000 training, 10,000 validation, and 10,000 test examples.
- UCI evaluation suite: UCI results place MADE among the best-performing models on half of the datasets and competitive otherwise.Reported standard deviations were fairly small and consistent across models, with additional values supplied separately.
- UCI evaluation suite: Mask sampling helps substantially when beneficial, but has negligible impact on most datasets when it does not.No clear hyperparameter winner emerged, while conditioning weights mattered mainly on NIPS-0-12.
- Binarized MNIST evaluation: The best MADE model outperforms single-layer NADE, otherwise the strongest model requiring one feed-forward pass for log probabilities.The comparison is reported for binarized MNIST alongside results from the literature.
- Binarized MNIST evaluation: Using too many masks over-regularizes MADE: performance improves up to 64 masks for two layers and around 8 masks for one layer.When more than four orderings were used, the deeper variant consistently performed better; Figure 2 examines the single-layer case.
- Binarized MNIST evaluation: Generated MNIST samples were compared with nearest training-set neighbors to assess whether sampling was simple memorization.Rows use different masks encountered during training.
7. Conclusion
The conclusion presents MADE as a simple autoencoder modification for tractable distribution estimation. It provides direct joint-probability estimates in one pass while retaining competitive statistical performance and favorable scaling.
- 7. Conclusion: MADE modifies autoencoders so they can serve as distribution estimators.The approach is described as a simple modification of standard autoencoders.
- 7. Conclusion: MADE obtains direct, inexpensive estimates of high-dimensional joint probabilities through a single autoencoder pass.The method remains easy to vectorize and implement on GPUs.
- 7. Conclusion: MADE maintains state-of-the-art statistical performance while offering better scaling for high-dimensional probability evaluation.The conclusion emphasizes computational scaling together with statistical performance.
Supplementary Material
The supplementary material provides uncertainty information for the main experimental results. It reports 95% confidence intervals for the UCI and binarized MNIST tables.
- Supplementary Material: Supplementary Table 7 reports negative log-likelihood and 95% confidence intervals for the main-document UCI results.It supplements Table 4.
- Supplementary Material: Supplementary Table 8 reports binarized MNIST negative log-likelihood and 95% confidence intervals for the main-document results.It supplements Table 6.