Source-linked AI summary
Posterior Collapse and Latent Variable Non-identifiability
Yixin Wang, David M. Blei, John P. Cunningham
TL;DR
Posterior collapse makes VAE latent posteriors equal their priors, preventing meaningful representations, while existing explanations often focus on neural networks or variational inference. This paper proves that collapse is equivalent to latent-variable non-identifiability and proposes flexible identifiable VAEs. The approach mitigates collapse across synthetic and real datasets, though its gradient-based decoder can raise computational cost and challenge optimization.
Problem
Posterior collapse makes VAE latent posteriors equal their priors, undermining meaningful representations, while existing approaches often attribute it to neural networks or variational inference.
Source: §1 · p. 1Method
The paper proposes latent-identifiable VAEs that enforce identifiability with Brenier maps while retaining flexibility and using the standard variational inference objective.
Source: §1 · p. 2Results
Across empirical studies, LIDVAE outperforms existing methods in mitigating posterior collapse, and the paper proves collapse is equivalent to latent-variable non-identifiability.
Source: §1 · p. 2, §5 · p. 10Takeaways & Limitations
Posterior collapse is an intrinsic issue of the model and data, not a phenomenon specific to neural networks or particular inference algorithms.
Source: §5 · pp. 9–10LIDVAE’s gradient-based decoder can increase inference complexity and sometimes challenge optimization.
Source: §3.2 · p. 7
Abstract
from arXiv · showhide
Variational autoencoders model high-dimensional data by positing low-dimensional latent variables that are mapped through a flexible distribution parametrized by a neural network. Unfortunately, variational autoencoders often suffer from posterior collapse: the posterior of the latent variables is equal to its prior, rendering the variational autoencoder useless as a means to produce meaningful representations. Existing approaches to posterior collapse often attribute it to the use of neural networks or optimization issues due to variational approximation. In this paper, we consider posterior collapse as a problem of latent variable non-identifiability. We prove that the posterior collapses if and only if the latent variables are non-identifiable in the generative model. This fact implies that posterior collapse is not a phenomenon specific to the use of flexible distributions or approximate inference. Rather, it can occur in classical probabilistic models even with exact inference, which we also demonstrate. Based on these results, we propose a class of latent-identifiable variational autoencoders, deep generative models which enforce identifiability without sacrificing flexibility. This model class resolves the problem of latent variable non-identifiability by leveraging bijective Brenier maps and parameterizing them with input convex neural networks, without special variational inference objectives or optimization tricks. Across synthetic and real datasets, latent-identifiable variational autoencoders outperform existing methods in mitigating posterior collapse and providing meaningful representations of the data.
1 Introduction
Posterior collapse makes VAE latent posteriors equal their priors, undermining meaningful representations. The paper links this problem to latent-variable non-identifiability and proposes identifiable VAEs as a direct response.
1 Introduction
- Posterior collapse makes per-datapoint latent posteriors identical to the prior, preventing VAE from producing meaningful representations.The paper also calls this phenomenon latent variable collapse, KL vanishing, and over-pruning.Source: §1 · p. 1
- Prior approaches often target variational objectives, optimization schemes, or generative-model capacity, reflecting hypotheses about neural networks or variational inference.The introduction contrasts these approaches with the paper’s non-identifiability account.Source: §1 · pp. 1–2
- Posterior collapse occurs if and only if the latent variable is non-identifiable in the generative model.The paper characterizes non-identifiability as a likelihood that does not depend on the latent variable.Source: §1 · p. 2
- Collapse can also occur in classical probabilistic models with exact inference, so it is not specific to neural networks or variational inference.The paper names Gaussian mixture models and probabilistic PCA as examples.Source: §1 · p. 2
- Latent-identifiable VAEs use Brenier maps to enforce identifiability while retaining VAE flexibility, and mitigate collapse across synthetic and real datasets without special objectives or optimization tricks.The paper reports that they outperform existing methods in mitigating posterior collapse.Source: §1 · p. 2
2 Posterior collapse and latent variable non-identifiability
The paper establishes an exact equivalence between posterior collapse and latent-variable non-identifiability, then illustrates how both the model and the data determine whether collapse occurs. The examples include collapse in exact posteriors, without variational inference.
2 Posterior collapse and latent variable non-identifiability
- A VAE generates each datapoint from a low-dimensional latent variable and fits model parameters by maximizing a variational approximation to the log marginal likelihood.The paper focuses its theoretical argument on exact inference, while noting that a restrictive variational family can also collapse an otherwise non-collapsed exact posterior.Source: §2 · p. 3
2.1 Posterior collapse ⇔Latent variable non-identifiability
- Posterior collapse occurs if and only if the latent variable is non-identifiable, meaning the dataset likelihood does not vary with the latent value.The paper’s definition concerns a particular dataset and parameter value; identifiability can therefore differ across datasets or parameter values.Source: §2.1 · pp. 3–4
- Collapse is mainly a model-and-data issue rather than an inference or optimization issue, and can change as parameter iterates change during VAE fitting.The paper also interprets some existing optimization adjustments as avoiding non-identifiable parameter values, although they may no longer approximate the maximum-likelihood estimate.Source: §2.1 · p. 4
- In practice, a nearly constant latent likelihood corresponds to a nearly collapsed posterior and suggests near-non-identifiability.The cited empirical studies report KL divergence between prior and posterior close to, but not exactly, zero.Source: §2.1 · p. 4
2.2 Examples of latent variable non-identifiability and posterior collapse
- In the GMVAE example, identifiable and non-identifiable likelihood maximizers fit the same well-separated two-cluster data, but only the latter collapses the latent posterior.At the collapsing maximizer, all fitted clusters share the data’s marginal distribution while still capturing the mixture.Source: §2.2 · pp. 4–5
- With data generated from one cluster but a two-cluster GMVAE, both fitted clusters can match the true cluster, making memberships indistinguishable and the posterior collapsed.This contrasts with the well-separated two-cluster data example, where one likelihood maximizer is identifiable.Source: §2.2 · p. 5
- The examples show that exact posteriors can collapse without variational inference, and that identifiability depends on both the model and the data.The paper uses GMVAE, PPCA, and GMM examples to illustrate these points.Source: §2.2 · pp. 4–5
3 Latent-identifiable VAE via Brenier maps
The section constructs identifiable VAEs by enforcing injective likelihoods with Brenier-map compositions, retaining standard inference and extending the strategy to mixture and sequential VAEs.
3.1 The latent-identifiable VAE
- Injective likelihoods guarantee identifiable latents and therefore prevent posterior collapse; LIDVAE constructs them by composing two Brenier maps with an injective matrix multiplication.The construction replaces the classical VAE's arbitrary mapping with an injective mapping while retaining the model's flexibility.Source: §3.1 · pp. 5–6
- Input convex neural networks parameterize Brenier maps as gradients of convex functions and can approximate these maps on compact domains.The ICNN parameterization is a universal approximator of Brenier maps.Source: §3.1 · p. 6
3.2 Inference in LIDVAE
- LIDVAE retains classical VAE capacity and uses standard amortized inference with the ELBO, but its gradient-of-gradient decoder can increase computational cost and challenge optimization.Classical VAE fitting has O(k · p) complexity, while LIDVAE fitting may require O(k · p^2).Source: §3.1 · p. 6, §3.2 · p. 7
3.1 The latent-identifiable VAE
- Figure 1 shows LIDGMVAE avoids pinwheel posterior collapse and produces more informative Fashion-MNIST posteriors with higher test log likelihood than GMVAE.On pinwheel, classical GMVAE latents predict the same value for all datapoints, while LIDGMVAE representations are meaningful.Source: Figure 1 · p. 7
3.3 Extensions of LIDVAE
- The injective-mapping strategy extends to mixture and sequential VAEs, including LIDGMVAE and LIDSVAE, by replacing their nonlinear mappings with injective counterparts.LIDGMVAE is a Gaussian special case of the latent-identifiable mixture VAE, and LIDSVAE models text.Source: §3.3 · pp. 7–8
4 Empirical studies
Experiments find that latent-identifiable VAEs mitigate posterior collapse across image and text datasets. PPCA results using Hamiltonian Monte Carlo show that increasing noise makes the latent less identifiable and the posterior closer to its prior.
4.1 LIDVAE on images and text
- The evaluation measures collapse with posterior–prior KL divergence, active units, and approximate mutual information, alongside held-out test log-likelihood.For mixture VAE, predictive accuracy of categorical latents against ground-truth labels also measures informativeness.Source: §4.1 · p. 8
- Across four image datasets, LIDGMVAE achieves higher AU, KL, MI, and test log-likelihood than other VAE variants; across three text datasets, LIDSVAE generally achieves higher AU, KL, and MI.The image results include pinwheel, MNIST, Fashion MNIST, and Omniglot; the text results include synthetic, Yahoo, and Yelp datasets.Source: §4.1 · p. 9
4.2 Latent variable non-identifiability and posterior collapse in PPCA
- In PPCA, Hamiltonian Monte Carlo shows that increasing noise flattens the likelihood and brings the posterior closer to the prior; at σ = 1.5, the posterior collapses.This demonstrates posterior collapse without variational approximation.Source: §4.2 · p. 9, Figure 2 · p. 9
5 Discussion
The discussion frames posterior collapse as latent-variable non-identifiability and proposes LIDVAE to address it. It distinguishes latent identifiability from joint model identifiability and notes limits in theory and computation.
5 Discussion
- LIDVAE uses Brenier maps to resolve latent-variable non-identifiability and mitigate posterior collapse; empirical studies report better mitigation than existing methods.Its latent variables are guaranteed identifiable, but the parameters and latent variables are not necessarily jointly identifiable.Source: §5 · pp. 9–10
- Ensuring an injective likelihood in z suffices for latent identifiability, whereas identifiability of the general VAE model remains an open problem.The discussion distinguishes identifiable latent variables from joint identifiability of the model.Source: §5 · p. 10
- The theory and algorithm do not address cases where the exact posterior remains uncollapsed but its variational approximation collapses.LIDVAE also has higher computational cost than classical VAE because fitting it requires gradients of gradients of a neural network.Source: §5 · p. 10
Supplementary Materials
The supplementary materials present two additional classical-model examples of posterior collapse.
Supplementary Materials
- The supplementary materials add probabilistic PCA and Gaussian mixture models as examples of posterior collapse.The passage names both examples without describing their results.Source: A Examples of posterior collapse… · p. 14
A.1 Probabilistic principal component analysis
The PPCA example shows that local latent variables can collapse even at maximum-likelihood global optima. Whether collapse occurs depends on the data-generating distribution.
A.1 Probabilistic principal component analysis
- PPCA local latent variables can collapse at maximum-likelihood parameter values that are global maxima of the log marginal likelihood.This extends the cited PPCA example of collapse at local maxima by showing it can occur without that optimization issue.Source: §A.1 · p. 14
- The example fits PPCA to two datasets of 500 samples each and examines maximum-likelihood parameter values.The model has two latent dimensions; z_i are latent variables and θ=(w,σ²) are model parameters.Source: §A.1 · p. 14
- Depending on the dataset’s true distribution, PPCA may or may not suffer from posterior collapse.Source: §A.1 · p. 14
1. Sample the data from a one-dimensional PPCA,
In PPCA, collapse of a latent dimension is associated with a likelihood that is constant along that dimension, while identifiable latents have peaked posteriors. A GMM example also considers collapse of a global latent variable and reports a peaked posterior for identifiable α.
1. Sample the data from a one-dimensional PPCA,
- When the one-dimensional PPCA data are fit with two latent dimensions, one maximum-likelihood solution makes the likelihood constant in z_i1, whose posterior then matches the prior, while z_i2 remains peaked.The latent variables are not fully identifiable under this parameter setting.Source: §1 · p. 14
- For the one-dimensional PPCA example, identifiable latent variables have likelihoods that vary across both dimensions and peaked posteriors for z_i1 and z_i2.Source: §1 · p. 14
A.2 Gaussian mixture model
- The GMM example considers posterior collapse of a global latent variable, α, in a two-cluster model fit to three datasets of 10^5 samples each.Here α is the global latent variable and θ=(μ1,μ2,σ1,σ2) are model parameters.Source: §A.2 · p. 14
1. Sample the data from two non-overlapping clusters,
- For data from two non-overlapping clusters, α is identifiable and has a peaked posterior that differs substantially from its prior.Under the maximum-likelihood parameters, the likelihood varies across α ∈ [0,1].Source: §1 · p. 14
2. Sample the data from two overlapping clusters,
The Gaussian-mixture examples show that identifiability depends on the data even when the model is fixed. Overlapping clusters make the mixture weight nearly non-identifiable, with a posterior close to its prior.
2. Sample the data from two overlapping clusters,
- Overlapping clusters are distinct but similar, so the mixture-weight likelihood varies slowly and its posterior remains close to the prior.The example uses parameters (-0.5, 0.5, 1, 1).Source: §2 · p. 15
- Across three datasets fitted with the same Gaussian mixture model, the latent variable’s identifiability differs with how the data were generated.The examples use one Gaussian cluster, two overlapping clusters, and two non-overlapping clusters.Source: §2 · p. 15
B Proof of Proposition 2
This appendix establishes that LIDVAE is identifiable while retaining the distributions that classical VAEs can generate. It describes the model’s construction, special cases, and proof strategy.
B Proof of Proposition 2
- LIDVAE replaces an arbitrary classical-VAE function with a constrained composition of full-row-rank linear maps and monotone transport maps.The general form uses two continuous monotone transport maps and supports both i.i.d. and sequential data through the history function.Source: B Proof of Proposition 2 · pp. 15–16
- The general construction includes categorical Gaussian-mixture and Bernoulli-mixture variants, as well as LIDSVAE configurations; real-valued latents mimic classical VAE.The Bernoulli-mixture variant is described as common for MNIST.Source: B Proof of Proposition 2 · p. 16
- Proposition 2 states that each latent variable in LIDVAE is identifiable, and the appendix extends the result to the general model.The defining condition equates conditional likelihoods only when their latent inputs are equal.Source: B Proof of Proposition 2 · p. 16
- The appendix argues that any classical-VAE data distribution can also be generated by an LIDVAE, preserving distributional flexibility.It first reduces to an injective-likelihood VAE, then reparameterizes that model as LIDVAE.Source: B Proof of Proposition 2 · pp. 16–17
- The proof establishes identifiability by composing injective transformations, including full-row-rank linear maps, bijective transport maps, and an injective exponential-family likelihood.The text cites for bijectivity of the transport maps.Source: B Proof of Proposition 2 · p. 16
- In a 9-layer generative model, LIDGMVAE does not suffer from posterior collapse and achieves better fit than its classical counterpart, as reported in Table 2.Table 2 reports means and standard deviations over ten random seeds; higher values are better.Source: Table 2 · p. 17
C Experiment details
The experiment details specify separate network and latent-variable configurations for image and text experiments.
C Experiment details
- Image experiments use 512-unit hidden layers, 64 continuous latent variables, and categorical dimensionality equal to the number of ground-truth labels.Two-layer RealNVP approximates inference to isolate the effect of variational inference.Source: C Experiment details · p. 17
- Text experiments use 1024-unit hidden layers, 1024-dimensional embeddings, and a two-layer LSTM inference family.Source: C Experiment details · p. 17
D Additional experimental results
The additional results illustrate posterior collapse in over-dimensioned PPCA and relate mixture-model behavior to latent identifiability. The section also points to further LIDVAE results on Pinwheel and MNIST.
D Additional experimental results
- Table 2 contains additional LIDVAE experimental results on Pinwheel and MNIST.Source: D Additional experimental results · p. 17
- When two-dimensional PPCA is fit to one-dimensional data, one latent dimension has a constant likelihood and its posterior collapses; fitting two-dimensional data does not show collapse.Source: Figure 3 · p. 18
- In Gaussian-mixture examples, a constant or nearly constant likelihood over the mixture weight accompanies a posterior close to or matching the prior; otherwise the likelihood varies and the posterior is peaked.The examples span one Gaussian cluster, overlapping clusters, and non-overlapping clusters.Source: D Additional experimental results · p. 18, Figure 4 · p. 18