PaperlayerView exact revision on arXiv

Source-linked AI summary

Variational Autoencoders and Nonlinear ICA: A Unifying Framework

Ilyes Khemakhem, Diederik P. Kingma, Ricardo Pio Monti, Aapo Hyvärinen

arXiv:1907.04809v4stat.MLcs.LG

TL;DR

VAEs generally lack guarantees that their latent variables and joint distribution are identifiable, limiting principled recovery of latent structure. The paper introduces iVAE, using an additionally observed variable to condition a factorized latent prior, and proves identifiability up to simple transformations, including noisy and undercomplete settings.

  • Problem

    Deep latent-variable models generally lack identifiability guarantees for recovering the true joint distribution and latent structure from observed data.

  • Method

    The paper connects VAEs with identifiable nonlinear ICA by conditioning a factorized latent prior on an additionally observed variable.

  • Results

    The paper proves identifiability for VAEs under relatively mild conditions, recovering the original latents up to linear invertible transformations and point-wise nonlinearities.

  • Takeaways & Limitations

    The framework provides principled disentanglement while supporting likelihood-based estimation, forward modeling, data generation, and noisy settings with fewer components.

  • Takeaways & Limitations

    The paper withdraws its claim of identifiability for discrete observations because an error was found in the proof.

Abstract

from arXiv · show

The framework of variational autoencoders allows us to efficiently learn deep latent-variable models, such that the model's marginal distribution over observed variables fits the data. Often, we're interested in going a step further, and want to approximate the true joint distribution over observed and latent variables, including the true prior and posterior distributions over latent variables. This is known to be generally impossible due to unidentifiability of the model. We address this issue by showing that for a broad family of deep latent-variable models, identification of the true joint distribution over observed and latent variables is actually possible up to very simple transformations, thus achieving a principled and powerful form of disentanglement. Our result requires a factorized prior distribution over the latent variables that is conditioned on an additionally observed variable, such as a class label or almost any other observation. We build on recent developments in nonlinear ICA, which we extend to the case with noisy, undercomplete or discrete observations, integrated in a maximum likelihood framework. The result also trivially contains identifiable flow-based generative models as a special case.

1 INTRODUCTION

VAEs efficiently learn deep latent-variable models whose observed-variable marginals fit data, but their latent joint distributions are generally unidentifiable. The paper shows that mild conditions and a factorized latent prior can make the joint distribution identifiable, connecting VAEs with nonlinear ICA.

  • VAE setting: VAEs provide scalable techniques for learning deep latent-variable and inference models, fitting observed-variable marginals and synthesizing pseudo-data.Their standard theory concerns optimization of the observed-variable marginal rather than recovery of the true latent-variable joint distribution.
  • Identifiability problem: Recovering the true joint distribution over observed and latent variables is difficult because latent variables are never directly observed.The original VAE theory does not establish when the correct joint distribution is learned.
  • Prior work: Earlier disentanglement approaches lack identifiability guarantees, and non-conditional latent priors leave these models non-identifiable.The introduction cites β-VAEs, extensions, and GAN-based independent-component methods as examples of this limitation.
  • Contribution: Under relatively mild conditions, the paper makes the VAE joint distribution identifiable and learnable using a factorized latent prior.This establishes a principled connection between VAEs and identifiable nonlinear ICA models.

2 UNIDENTIFIABILITY OF DEEP LATENT VARIABLE MODELS

Deep latent-variable models can flexibly fit observed data distributions, but unconditional latent distributions generally do not identify the latent variables, prior, posterior, or joint distribution. VAEs therefore guarantee meaningful marginal modeling of x without generally guaranteeing meaningful learned latent distributions.

  • Model structure: Deep latent-variable models define pθ(x, z) through a prior pθ(z) and decoder pθ(x|z), yielding the observed marginal pθ(x).A neural-network decoder can represent a rich class of data distributions.
  • VAE learning: VAEs optimize an approximate maximum marginal likelihood objective, estimating the marginal density of x after optimization.The learned generative model also includes an inference model qφ(z|x) approximating pθ(z|x).
  • Consequences: Consequently, the marginal distribution over x may be meaningful while the learned prior, posterior, and other latent distributions are generally meaningless.The VAE learns a full generative model and an approximate inference model, but provides no general guarantees about what those learned distributions represent.
  • Identifiability goal: Matching the observed marginal would identify the true joint distribution, prior, and posterior only if the model were identifiable.This requires that any two parameter choices producing the same pθ(x) also have matching joint distributions.
  • Unidentifiability: For unconditional latent distributions, deep latent-variable models are generally unidentifiable because transformations can change z while preserving its distribution.For a spherical Gaussian prior, rotations provide an example of such transformations.

3 AN IDENTIFIABLE MODEL BASED ON CONDITIONALLY FACTORIAL PRIORS

The section introduces Identifiable VAE (iVAE), a deep latent-variable model with conditionally factorial exponential-family priors driven by an auxiliary observation and estimated through a VAE. Under these assumptions, the latent variables are identifiable up to component-wise transformations and, initially, a linear transformation that can often be reduced to permutation.

  • Model and estimation: Identifiable VAE (iVAE) defines an identifiable family of deep latent-variable models together with a VAE-based estimation method.The method targets the true generative parameters and variational posterior within the stated model family.
  • Model and assumptions: The model assumes a conditionally factorial prior pθ(z|u), where each latent component has a univariate exponential-family distribution whose parameters depend on an additionally observed variable u.The auxiliary variable may be a time index, previous data points, a noisy class label, or another concurrently observed variable.
  • Model and assumptions: Observations follow x = f(z) + ε, with injective but otherwise arbitrarily nonlinear f; the framework also covers noiseless observations and flow-based generative models.Flexible function approximators such as neural networks can parameterize f, while discrete observations require a bespoke identifiability theory.
  • Model and estimation: The VAE maximizes a lower bound on data log-likelihood while learning the generative parameters and a variational approximation qφ(z|x,u) to the posterior.The reparameterization trick provides low-variance gradient estimates, and latent estimates can be obtained by sampling from the variational posterior.
  • Identifiability result: In the no-noise, k = 1 case, the recovered latents equal component-wise transformations of the true latents followed by an invertible linear transformation, often reducible to permutation.The remaining component-wise transformations are described as potentially inconsequential in many applications.
  • Relation to nonlinear ICA: The framework extends nonlinear ICA by combining identifiable source recovery with VAE evidence-lower-bound estimation and learning both forward and backward models.It also retains a tight link between maximum likelihood estimation and maximizing latent independence.

4 IDENTIFIABILITY THEORY

The theory establishes identifiability of the generative model up to an invertible linear transformation, and under additional conditions reduces this ambiguity to permutation and componentwise nonlinearities. It also shows that, with a sufficiently broad variational family and infinite data, a VAE consistently learns the true parameters up to the relevant equivalence class.

  • Basic identifiability: The main theorem guarantees identifiability up to an invertible linear transformation of the sufficient statistics.With noise, the learned latent posteriors are identified up to an analogous indeterminacy.
  • Reduction to permutation ambiguity: For k ≥2, additional conditions make the parameters identifiable up to a block permutation.Theorem 2 assumes the hypotheses of Theorem 1, k ≥2, and further regularity conditions on the sufficient statistics and mixing function.
  • Reduction to permutation ambiguity: For k = 1, nonmonotonic sufficient statistics and continuous partial derivatives of the mixing function also yield identifiability up to a block permutation.These conditions are given in Theorem 3.
  • Interpretation: Under these sufficient conditions, the learned latents equal a permutation and pointwise nonlinearity of the original latents.Equivalently, the mapping ˜f −1 ◦f is pointwise, with each component depending on only one original latent.
  • Limitations: The linear ambiguity cannot generally be removed when sufficient statistics are linear, including Gaussian components with only changing location.Orthogonal latent transformations can preserve the observational distribution after adjusting parameters.
  • VAE consistency: With a variational family containing the true posterior, infinite-data VAE training learns the true parameters up to the equivalence class defined by the theory.The result is stated for the true parameters θ∗ := (f∗, T∗, λ∗).

5 EXPERIMENTS

Experiments on synthetic nonstationary-source data evaluate iVAE using mean correlation after optimal source–latent assignment. iVAE recovers sources up to trivial indeterminacies, outperforms vanilla and disentanglement-oriented VAEs in comparisons, and remains effective where TCL fails.

  • Dataset: Synthetic datasets use nonstationary Gaussian time-series sources divided into uniformly labeled segments, with conditional priors selected from the specified family.The conditioning variable u is the segment label, and each segment contains L samples.
  • Performance metric: Performance is measured by mean correlation coefficient between original sources and posterior-sampled latents after linear-sum assignment resolves component permutations.The assignment pairs each latent component with the most strongly correlating source component.
  • Results: 2D example: In the 2D visualization, iVAE recovers the original sources up to rotation and sign flip, whereas vanilla VAE fails to separate the latent variables.The comparison covers original sources, observations, and posterior distributions learned by both models.
  • Results: Comparison to VAE variants: iVAE is compared with vanilla VAE, β-VAE, and β-TC-VAE using the same architecture except for iVAE’s conditional prior.The VAE prior is Gaussian or Laplace when the data is centered, while β parameters follow the respective authors’ instructions.
  • Results: Comparison to TCL: Across simulations varying the number of segments from 10 to 50, iVAE slightly outperforms TCL on the same dataset used for the main comparison.TCL is the nonlinear ICA baseline based on a self-supervised classification task.
  • Results: Comparison to TCL: On data where x2 alone enables segment classification, TCL achieves high classification accuracy without unmixing observations and fails to recover latent variables.The failure is induced by significantly modulating x2’s mean across segments, exposing a self-supervised-method failure mode.

6 CONCLUSION

The paper combines VAE-based deep latent-variable modeling with nonlinear ICA to create iVAE, providing the first rigorous VAE identifiability proof and principled recovery of original latents. The framework supports the four stated unsupervised-learning goals while strengthening nonlinear-ICA results through likelihood-based estimation, forward-model learning, and data generation.

  • 6 CONCLUSION: iVAE combines VAE-based deep latent-variable models with nonlinear ICA to address distribution modeling, sampling, feature learning, and latent-code identifiability.These are the four goals listed for unsupervised learning, and the model is described as satisfying all four.
  • 6 CONCLUSION: iVAE provides the first rigorous proof of identifiability in the context of VAEs.The conclusion explicitly characterizes this as a first rigorous VAE identifiability result.
  • 6 CONCLUSION: The framework recovers the original latent variables, thereby providing principled disentanglement beyond typical deep latent-variable VAE models.The stated advantage is recovery of the original latents rather than only fitting observed-data distributions.
  • 6 CONCLUSION: Compared with Hyvärinen et al. (2019), the nonlinear-ICA algorithm strengthens identifiability, obtains a likelihood for MLE, learns a forward model, and generates new data.The passage lists these advantages as the algorithm’s main improvements for solving nonlinear ICA.

Corrigendum

The authors retract the claim that their identifiability result extends to discrete observations because they found an error in the proof after publication. They leave a correct treatment of the discrete case for future work and made minimal revisions to remove the incorrect claims.

  • Corrigendum: The identifiability result can no longer be claimed for discrete observations because the published proof contains an error.The error was discovered after publication.
  • Corrigendum: The authors suspect that the discrete case requires a different type of proof and leave it for future work.
  • Corrigendum: They made minimal changes to the abstract, Section 3.1, and Section 6 to remove the incorrect claims, and rewrote Appendix C.

Supplementary Material for … B.4.1 Main steps of the proof

The supplementary material establishes assumptions and lemmas for exponential-family identifiability, then proves that equal noisy observations imply equivalent latent representations through invertible transformations. The proofs culminate in componentwise, permutation-structured transformations and explain the geometric argument for the nonmonotonic case.

  • Supplementary Material for: The supplementary analysis assumes univariate distributions on R, while the results also hold on any open positive-measure subset of R.These assumptions introduce the domain used throughout the exponential-family lemmas.
  • A.1 Exponential family distributions: Exponential families use a minimal sufficient-statistic dimension k, whose components are linearly independent.The minimality condition prevents rewriting the density with fewer sufficient-statistic components.
  • A.2 Strongly exponential distributions: Strongly exponential distributions retain their exponential component almost surely, and include standard Gaussian, Laplace, Pareto, Chi-squared, Gamma, and Beta families.The lemmas show that differentiable sufficient statistics provide independent derivative vectors and that the constructed matrix has rank 2n almost everywhere.
  • A.2 Strongly exponential distributions: A non-strongly-exponential example arises from T(x) = (min(0, x), −max(0, x)), whose derivative vanishes on separate half-lines.This demonstrates why the strong exponential condition is needed for the rank arguments.
  • B.2 Proof of Theorem 1: The proof of Theorem 1 first removes observation noise by convolution, then subtracts equations around a pivot auxiliary value, and finally proves the resulting linear transformation is invertible.Assumption (i) makes the noise characteristic function nonzero almost everywhere; assumption (iv) supplies the pivot points, while assumption (iii) yields full-rank Jacobians.
  • B.2.2 Proof: Theorem 1’s matrix argument establishes equivalence of parameterizations and implies invertibility of the transformed auxiliary-variable matrix.For k > 1, Lemma 3 makes the constructed Jacobian matrix Q invertible, which forces A and the alternative Q matrix to be invertible.
  • B.2.3 Understanding assumption (iv) in Theorem 1: Assumption (iv) holds almost surely for independently random modulation parameters, and in the Gaussian n = 2, k = 1 example it follows from the independence of u and u^2.For any pivot u0 = 1 and distinct nonzero u1 and u2, the resulting matrix L is invertible.

B.4.2 Proof … D.1 Factorial priors

The proofs establish the remaining identifiability and non-identifiability boundaries: the transformation is restricted to a scaled permutation under the theorem’s assumptions, while unconditional factorial priors leave latent variables unidentifiable. They also connect VAE optimization to maximum likelihood, note the special challenge of discrete observations, and show orthogonal invariance for Gaussian and general factorial priors.

  • B.4.2 Proof: The minimum-set construction identifies Ci with ˜Ci and shows it is a smooth C1 submanifold of co-dimension 1.This follows by restricting to a cube around a unique local minimum and using invertibility of vi.
  • B.4.2 Proof: Under the proof’s assumptions, exactly one transformed component is nonconstant per latent dimension, forcing A to be a scaled permutation matrix and ˜f ◦f to be point-wise nonlinear.The argument uses the codimension of the minimum sets to exclude two nonconstant components in any row of A.
  • B.5 Proof of Proposition 1: For an orthogonal latent transformation, the transformed exponential-family model has the same observed-data distribution, so θ and ˜θ cannot be distinguished from observations.The base measure remains unchanged for rotationally invariant choices such as Qi(zi) = 1 or Qi(zi) = e−z2.
  • B.6 Proof of Theorem 4: When qφ(z|x, u) contains pθ(z|x, u), the KL term reaches zero and VAE optimization becomes maximum likelihood, converging to the true parameter’s equivalence class.Identifiability up to equivalence classes changes the conclusion of MLE consistency from the exact parameter to its equivalence class.
  • C DISCRETE OBSERVATIONS: Discrete observations require a separate identifiability proof because the mapping z →x is non-injective, although supplementary experiments strongly suggest identifiability remains achievable.A logistic model can replace additive Gaussian noise for discrete observations.
  • D UNIDENTIFIABILITY OF GENERATIVE MODELS WITH UNCONDITIONAL PRIOR: The general nonlinear setting offers little hope of recovering original latents, because any random vector can be transformed into independent components, including standardized Gaussian ones.This section presents a general proof of generative-model unidentifiability alongside the simpler factorial-prior proof.
  • D.1 Factorial priors: For factorial Gaussian priors, any orthogonal transformation preserves the latent distribution and can be canceled in p(x|z), yielding identical observed data; the same non-identifiability extends to any factorial prior.Each marginal can be mapped through its CDF to uniform, then standardized Gaussian, transformed orthogonally, and mapped back.

D.2 General priors

For any d-dimensional random vector, a nonlinear transformation can produce independent standardized Gaussian components, revealing severe non-identifiability under general priors. Consequently, observed data alone cannot recover the true latent variables, regardless of the prior distribution.

  • Theorem 5 states that any d-dimensional random vector can be transformed into independent components, each with a standardized Gaussian distribution.The first transformed component is a monotonic transformation of the first original component.
  • An iterative procedure analogous to Gram–Schmidt makes each new variable independent of previously considered variables, leaving z1 essentially unchanged.
  • The theorem implies infinitely many independent-component definitions can generate the same observation, because z can first be transformed arbitrarily before applying the theorem.An arbitrarily selected variable can always become one of the independent components, creating an additional form of indeterminacy.
  • Applying the theorem directly to observed data means any xi could become an independent component under factorial priors, contradicting meaningful latent recovery.
  • For any prior, transforming z to independent Gaussian variables, applying an orthogonal transformation, and inverting it yields a complex nonlinear transformation with the same distribution.Therefore, the true latent variables cannot be recovered from data alone, regardless of the prior.

E ALTERNATIVE FORMULATION OF THEOREM 1 … H SIMULATION DETAILS

The supplementary sections formalize identifiability under differentiability and invertibility assumptions, connect likelihood maximization to latent independence, and position the framework against nonlinear ICA and VAE work. They also describe simulation sources, including segment-modulated distributions.

  • E ALTERNATIVE FORMULATION OF THEOREM 1: Theorem 6 establishes ∼-identifiability of (f, T, λ) when the noise characteristic function is nonzero almost everywhere, f is injective, sufficient statistics are regular, and Jλ(u0) is invertible.Any alternative parameterization producing the same conditional distribution also satisfies the sufficient-statistic and λ regularity assumptions.
  • E ALTERNATIVE FORMULATION OF THEOREM 1: The theorem further shows that any alternative parameters yielding the same p(x|u) preserve assumptions (iii) and (iv).This extends the identifiability statement to the regularity properties required by the alternative formulation.
  • F LINK BETWEEN MAXIMUM LIKELIHOOD AND TOTAL CORRELATION: Maximizing data likelihood in the noiseless case decreases conditional total correlation, which measures latent independence and vanishes exactly when latent components are independent.The derivation uses change-of-variable formulas for the log-likelihood and conditional entropies.
  • G.1 Previous work in nonlinear ICA: Time Contrastive Learning assumes independent, slowly non-stationary sources whose distributions vary across known non-overlapping segments, then recovers the inverse transformation through segment classification.Its model uses a neural feature extractor followed by a classifier, with segment-dependent source parameters.
  • G.1 Previous work in nonlinear ICA: The framework builds on auxiliary-variable nonlinear ICA but strengthens prior identifiability results for exponential-family models, including non-monotonic sufficient statistics when k = 1.Related methods use self-supervised binary discrimination, whereas this work concentrates on exponential-family identifiability.
  • G.2 Previous work on identifiability in VAEs: Unlike semi-supervised VAE settings, the auxiliary variable u may be time, history, or another modality, and unconditional priors yield unidentifiable models.Prior disentanglement work showed rotation invariance for isotropic priors; the paper states a more general non-identifiability result for unconditional priors.
  • H SIMULATION DETAILS: The simulations visualize sources with segment-modulated variance and means, with the latter potentially facilitating TCL’s surrogate classification task.Figure 5 includes a single source with modulated variance and two sources where the second source’s mean varies by segment.

H.1 Implementation detail for VAE experiments · H.2 Description of significant mean modulated data · I FURTHER EXPERIMENTS

The VAE experiments use segment-conditioned latent priors, MLP-based generative and inference networks, and controlled noise and optimization settings. Additional data test significant segment-dependent mean modulation and motivate TCL’s segment-classification surrogate.

  • H.1 Implementation detail for VAE experiments: The conditioning variable u is a uniformly distributed segment label, and each segment uses a conditional prior with independently randomized variances in [.5, 3].The latent variables are sampled from these segment-specific distributions before mixing.
  • H.1 Implementation detail for VAE experiments: Latent sources are mixed by a 4-layer MLP, and observations receive small noise with σ2 = 0.01.This noise step is omitted when comparing against ICA methods designed for noiseless data.
  • H.1 Implementation detail for VAE experiments: The inference model is a diagonal-covariance multivariate Gaussian, while decoder, inference, and conditional-prior parameters are implemented with MLPs.The supplied implementation description specifies qφ(z|x, u) as Gaussian and identifies f, g, σ2, and λ as functional MLP parameters.
  • H.1 Implementation detail for VAE experiments: The networks use leaky ReLU activations, 3–6 layers, hidden widths from 10 to 200, batch size 64, and Adam learning rates of 0.01 or 0.001.A scheduler is also used to decay the learning rate.
  • H.1 Implementation detail for VAE experiments: The VAE implementation follows Kingma and Welling, with hyperparameter ranges chosen to match the complexity of the proposed method’s additional λ network.β-VAE uses β in [1, 45], while β-TC-VAE uses α = γ = 1 and β in [1, 35].
  • H.2 Description of significant mean modulated data: The modified 2D dataset makes the second source mean vary by segment as µ2(u) = αγ(u), while µ1(u) remains zero and segment-specific variances are randomized.The sources are then mixed so that x2 preserves the significant mean modulation.
  • H.2 Description of significant mean modulated data: TCL learns to unmix observations by training a deep network to classify each observation into its corresponding segment.The constructed mapping is intended to highlight this segment-classification-based surrogate task.

I.1 Additional general nonlinear ICA experiments … I.4 Additional visualisations for comparison to VAE variants

The additional experiments test iVAE’s capabilities for discrete observations, dimensionality and hyperparameter selection, causal discovery, real fMRI data, and visual comparison with VAE variants. Across these settings, the passages describe discrete ICA, latent-dimension selection, causal-discovery behavior, and stronger source recovery than vanilla VAE.

  • I.1 Additional general nonlinear ICA experiments: The method supports dimensionality reduction, ELBO-based latent-dimension selection, and discrete ICA, extending capabilities beyond earlier self-supervised nonlinear ICA methods.These claims motivate the additional simulations.
  • I.1 Additional general nonlinear ICA experiments: With d = 100 observed dimensions and n = 10 latent dimensions, iVAE performs discrete ICA and is compared against a vanilla VAE.The comparison appears in Figure 6a.
  • I.1 Additional general nonlinear ICA experiments: Because the objective derives from the observations’ marginal log-likelihood, the ELBO can guide hyperparameter selection across network dimensions, learning rates, and nonlinearities.The experiments vary distinct hyperparameter choices.
  • I.1 Additional general nonlinear ICA experiments: The post-training ELBO shows an elbow around latent dimension 15 when d∗= 80 and n∗= 15, successfully guessing the real latent dimension.The figure reports ELBO as a function of latent dimension.
  • I.2 Additional causality experiments for comparison to TCL: For causal discovery, iVAE and TCL first recover latent disturbances and then use HSIC independence tests, with experiments covering normal and significantly mean-modulated sources.The causal setup uses lower-triangular mixing matrices to impose acyclic structure over observations.
  • I.2 Additional causality experiments for comparison to TCL: Significant mean modulation makes TCL’s surrogate classification easier, so TCL need not accurately represent within-segment source log-densities, harming nonlinear unmixing rather than merely reflecting sub-optimal optimization.This is presented as a negative consequence of TCL’s surrogate-classification reliance.
  • I.3 Real data experiments: The real-data experiments analyze daily resting-state hippocampal fMRI scans from one 45-year-old individual over 84 successive days, treating each day as a segment encoded in u.The data follow earlier causal work using TCL.
  • I.4 Additional visualisations for comparison to VAE variants: Visualization of recovered latents shows iVAE estimates the original sources excellently, whereas VAE performs worse and other models are nearly indistinguishable from vanilla VAE.The comparison uses randomly sampled contiguous source subsets after correcting component permutations.
Loading 1907.04809v4…