Source-linked AI summary
High-Dimensional Probability Estimation with Deep Density Models
Oren Rippel, Ryan Prescott Adams
TL;DR
High-dimensional density estimation is difficult when data concentrate near complex lower-dimensional manifolds. The deep density model uses a bijective map to construct tractable, normalized densities; on MNIST, its classifier reduced error from 9.5% to 1.614% after penalizing density assigned to foreign examples.
Problem
Method
The deep density model learns a bijective transformation to a representation space with an approximately factorized distribution and known marginals, enabling fully normalized densities without a partition function.
Source: §1 · p. 2Results
1.614% error versus 9.5% for a raw mixture of Bayesian classifiers on MNIST after penalizing density assigned to foreign examples.
Source: §6 · p. 11Takeaways & Limitations
Normalized density can be leveraged across separately trained models, a capability the authors say is not typically possible for energy-based probabilistic approaches.
Source: §6 · p. 11Bijectivity requires the representation space to have the same dimension as the observed space, foregoing the greater parametric flexibility attributed to overcomplete representations.
Source: §2 · p. 3
Abstract
from arXiv · showhide
One of the fundamental problems in machine learning is the estimation of a probability distribution from data. Many techniques have been proposed to study the structure of data, most often building around the assumption that observations lie on a lower-dimensional manifold of high probability. It has been more difficult, however, to exploit this insight to build explicit, tractable density models for high-dimensional data. In this paper, we introduce the deep density model (DDM), a new approach to density estimation. We exploit insights from deep learning to construct a bijective map to a representation space, under which the transformation of the distribution of the data is approximately factorized and has identical and known marginal densities. The simplicity of the latent distribution under the model allows us to feasibly explore it, and the invertibility of the map to characterize contraction of measure across it. This enables us to compute normalized densities for out-of-sample data. This combination of tractability and flexibility allows us to tackle a variety of probabilistic tasks on high-dimensional datasets, including: rapid computation of normalized densities at test-time without evaluating a partition function; generation of samples without MCMC; and characterization of the joint entropy of the data.
1 Introduction
Existing density and manifold methods trade tractability, scalability, or probabilistic interpretation, while flexible invertible maps with tractable latent densities have been difficult to obtain. The DDM uses deep-learning ideas to learn such maps and applies them to density estimation, generation, classification, and latent-distribution analysis.
1 Introduction
- Graphical and nonparametric density models can require costly inference or provide only unnormalized estimates, and nonparametric methods often scale poorly to high-dimensional data.Undirected models can provide compact latent representations but only unnormalized densities; directed belief networks are normalized but require costly inference.Source: §1 · p. 1
- Manifold-learning methods often prioritize visualization, while autoencoders lack a clear probabilistic interpretation and invertible mappings have been difficult to parameterize flexibly.Existing bijective approaches based on invertible linear transformations have limited expressiveness.Source: §1 · pp. 1–2
- The DDM learns rich bijective transformations that produce approximately factorized latent distributions with known marginals, enabling fully normalized densities without a partition function.The Jacobian determinant accounts for measure change across the invertible transformation.Source: §1 · p. 2
- The generative DDM can sample without MCMC and supports Bayesian classifiers with calibrated class-conditional probabilities and EM-trained mixtures that use unlabeled data.The paper presents proof-of-concept applications on CIFAR and MNIST.Source: §1 · p. 2
- The model connects latent sparsity and independence, using this relationship to find approximately factorized representations and characterize observed-data entropy for model selection.The authors use information-theoretic tools to study these connections.Source: §1 · p. 2
2 Bijections and Normalized Densities
The paper addresses the challenge of fitting tractable distributions to data concentrated near complicated lower-dimensional manifolds. It models observations through a bijection from a simple latent distribution, enabling normalized density evaluation while requiring equal-dimensional observed and representation spaces.
2 Bijections and Normalized Densities
- Learning distributions on high-dimensional data is difficult because the probability mass may concentrate on a complicated lower-dimensional manifold that is hard to parameterize and fit tractably.Directly studying the observed-space distribution also presents theoretical and computational difficulties.Source: §2 · p. 2
- The approach assumes a simple factorized latent distribution with known univariate marginals and a bijection whose inverse is analytically available, allowing observed-data densities to be computed.Source: §2 · pp. 2–3
- The DDM uses rich bijections to map simple latent distributions into complex observed densities while retaining a manageable, normalized density estimator.The model is intended to discover structure in high-dimensional datasets.Source: §2 · p. 3
- Bijectivity enables normalized density estimates but requires the representation space to match the observed space in dimension, trading off overcompleteness for normalization.Overcomplete representations are considered more parametrically flexible.Source: §2 · p. 3
3 The Deep Density Model
The DDM learns an invertible map between observed data and a representation space with simple, approximately factorized marginals. Separate regularizers shape those marginals, preserve invertibility and reconstruction, and connect sparsity to latent independence and entropy.
3 The Deep Density Model
- The model learns an invertible decoder whose representation-space data should follow a simple factorized distribution, using layers of invertible linear maps and bijective sigmoid nonlinearities.A separate encoder maps observed data into that representation space.Source: §3 · p. 3
3.1 Regularizing the Transformations
- Training combines penalties for matching representation marginals to a target, maintaining decoder invertibility, and reconstructing data through the encoder-decoder pair.The objective weights the three terms jointly.Source: §3.1 · p. 4
3.2 Divergence Penalty: Sculpting the Latent Marginals
- A symmetrized KL penalty matches each encoded coordinate's empirical marginal to a chosen target; for the unit-hypercube representation, the target is from the Beta family.The empirical marginals are approximated with Beta distributions using moment matching.Source: §3.2 · pp. 4–5
3.2.1 Sparsity in Distribution
- Distributional sparsity matches empirical activations to a target peaked at zero, unlike L1 activity penalties that can collapse all examples to zero.The target shapes sparsity across examples rather than directly penalizing each activation.Source: §3.2.1 · p. 5
3.3 Invertibility: condition number penalty
- Layer-condition-number regularization protects decoder invertibility, preserving volume accounting at unseen points; the authors report this requirement is easily satisfied in practice.Invertibility makes it possible to compute the decoder Jacobian determinant for density evaluation.Source: §3.3 · p. 6
3.4 Reconstruction and the independence of latent dimensions
- As target marginal entropy falls while reconstruction preserves information, latent dimensions must become more independent; Figure 2 contrasts incoherent dependent samples with diversified generations.The paper calls this reduction in overlap between dimensions diversification.Source: §3.4 · p. 6, Figure 2 · p. 7
3.4.1 Entropy characterization
- Observed-space entropy can be upper-bounded and characterized after mutual information is minimized; the selected representation is maximally sparse while retaining input information.Model entropy is also treated as an effective configuration count governing mass allocation between training examples and out-of-sample data.Source: §3.4 · pp. 6–7, §3.4.1 · pp. 7–8
3.4.2 On why we need gΨ(·)
- A separate gΨ brings observed points close in X to control capacity and marginals, since asking the bijective decoder and its inverse to serve both purposes is contradictory and numerically unstable.The encoder also provides a computationally useful path for back-propagating penalties defined on the observed space.Source: §3.4.2 · p. 8
4 Training
Training recursively stacks single-layer DDMs, then fine-tunes by layer; initialization gradually moves the target distribution from an easier Beta distribution toward the final one. Adaptive optimization and implicit momentum are used to manage training.
4 Training
- Pretraining recursively trains single-layer DDMs, then fine-tuning uses block coordinate descent to optimize one layer at a time.The layer-wise steps avoid mixing distinct gradient magnitudes, sidestepping exponentially decaying gradients during back-propagation.Source: §4 · p. 8
- The optimizer adapts step size with an inexact Armijo line search and adjusts penalty coefficients to maintain gradient-magnitude ratios.Exact overall-objective line search is computationally expensive and cannot use mini-batches.Source: §4 · p. 8
- Masking noise and implicit momentum are used during training; the authors report that implicit momentum stabilizes convergence.Examples recur across several consecutive minibatches while each minibatch adds new training examples.Source: §4 · p. 8
4.1 Distribution sequencing and initialization
- Rather than impose the final Beta target immediately, initialization schedules a sequence from an easier Beta distribution toward the target.The procedure adapts transitions as the empirical representation distribution approaches each target; weights start as scaled random orthonormal matrices.Source: §4.1 · pp. 8–9
- Table 1 reports similar mean log-probabilities for training and test examples, slightly lower values for rotated tests, and significantly lower values for corrupted examples.Both models have three layers.Source: Table 1 · p. 9
5 Empirical results
The empirical section evaluates normalized densities on MNIST and CIFAR-10, including test examples, distortions, and marginal entropy. It also shows that sampling in the representation space produces observed-space generations.
5.1 Density evaluation
- The density evaluation compares probabilities for training examples, test examples, and distortions of test examples.The section frames useful density estimates as assigning high probability to unseen data without overfitting and low density to unrealistic points.Source: §5.1 · p. 9
- A model trained only on 9s assigns a real 9 higher probability than an inverted 6, with log-densities shown across rotation angles.The figure also shows the rotations of a 6 assigned the highest and lowest probabilities by the 9-model.Source: §5.1 · p. 9, Figure 3 · p. 10
- 20.7217 is the computed mean Bernoulli log-probability, close to the expected 21.0181, supporting the stated marginal-entropy estimate.The MNIST representation marginals are approximated as Bernoulli variables by rounding elements to 0 or 1.Source: §5.1 · pp. 9–10
5.2 Generation
- Sampling in the representation space and decoding through the invertible map produces observed-space samples that visualize high-density regions.Figure 4 presents generations from DDMs, including a three-layer MNIST model with diversified Beta marginals.Source: §5.2 · p. 10, Figure 4 · p. 10
6 Discussion
The discussion presents DDM as a way to construct rich, tractable, normalized density estimates and explores class-conditional classification. On MNIST, penalizing density on foreign examples improves the reported classification results; density estimates may also be used with unlabeled data.
6 Discussion
- Fully normalized class-conditional densities support Bayesian classifiers with calibrated probabilities and rejection when no class model is sufficiently confident.A threshold can reject an example when every class likelihood falls below it.Source: §6 · pp. 10–11
- Class models can be exposed to examples from other classes and trained to assign them low probability, mapping them away from their own class manifold.The discussion presents this as a way to distinguish class-manifold examples from foreign examples.Source: §6 · p. 11
- On MNIST, penalizing density assigned to foreign examples yields 1.614% error versus 9.5% for a raw mixture; among roughly 95% confident test examples, error is 0.45%.The authors note these rates are not state of the art, but use them to illustrate classifier flexibility and the use of normalized densities across separately trained models.Source: §6 · p. 11
- Density estimates may extract information from unlabeled data when only a small fraction of observations are labeled, for example through weighted-data EM in a mixture model.Source: §6 · p. 11