PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA

Sébastien Lachapelle, Pau Rodríguez López, Yash Sharma, Katie Everett, Rémi Le Priol, Alexandre Lacoste, Simon Lacoste-Julien

arXiv:2107.10098v3stat.MLcs.LG

TL;DR

General nonlinear mixing makes disentanglement difficult, motivating methods that use auxiliary variables and causal structure. The paper jointly learns latent factors and a sparse causal graph, proves permutation recovery under graph conditions, and validates a binary-mask VAE in simulations.

  • Problem

    General nonlinear mixing is non-identifiable, while reconciling semantically meaningful causal variables with high-dimensional observations remains difficult.

  • Method

    The method jointly learns latent factors and their sparse causal graphical model using a VAE with binary-mask regularization.

  • Results

    The theory shows that sparse mechanism regularization and a suitable graph connectivity criterion can recover latent variables up to permutation, with disentangled representations demonstrated in simulations.

  • Takeaways & Limitations

    Mechanism sparsity provides a formal route to disentanglement and connects nonlinear ICA with unknown-target interventions and sparse mechanism shifts.

  • Takeaways & Limitations

    The current theory uses only a fixed number of previous time steps; extending the mechanism function to all past steps with recurrent networks is left for future work.

Abstract

from arXiv · show

This work introduces a novel principle we call disentanglement via mechanism sparsity regularization, which can be applied when the latent factors of interest depend sparsely on past latent factors and/or observed auxiliary variables. We propose a representation learning method that induces disentanglement by simultaneously learning the latent factors and the sparse causal graphical model that relates them. We develop a rigorous identifiability theory, building on recent nonlinear independent component analysis (ICA) results, that formalizes this principle and shows how the latent variables can be recovered up to permutation if one regularizes the latent mechanisms to be sparse and if some graph connectivity criterion is satisfied by the data generating process. As a special case of our framework, we show how one can leverage unknown-target interventions on the latent factors to disentangle them, thereby drawing further connections between ICA and causality. We propose a VAE-based method in which the latent mechanisms are learned and regularized via binary masks, and validate our theory by showing it learns disentangled representations in simulations.

Pau Rodr´ıguez L´opez

The listed affiliations place the authors at Mila and DIRO, Université de Montréal, Canada CIFAR AI Chair. The editors are Bernhard Schölkopf, Caroline Uhler, and Kun Zhang.

  • The authors are affiliated with Mila and DIRO at Université de Montréal.
  • The affiliation information identifies Canada CIFAR AI Chair involvement.
  • The listed editors are Bernhard Schölkopf, Caroline Uhler, and Kun Zhang.

1. Introduction

The paper addresses disentanglement of meaningful latent variables from high-dimensional observations by jointly learning representations and sparse causal structure. Its theory connects nonlinear ICA with mechanism sparsity and unknown-target interventions.

  • The paper jointly disentangles high-level variables from low-level observations and learns the causal graph relating them.
  • Nonlinear ICA can identify latent factors under sufficiently strong auxiliary-variable effects, but general nonlinear mixing is otherwise non-identifiable.
  • Mechanism sparsity regularization recovers latent variables when temporal or action dependencies are sparse and the graphical criterion holds.
  • The theory formally connects unknown-target latent interventions with the sparse mechanism shift hypothesis.
  • A VAE-based estimator learns sparse latent mechanisms with binary masks and demonstrates the theoretical predictions on synthetic datasets.

2. Disentanglement via Mechanism Sparsity Regularization

The model combines nonlinear ICA with temporal and auxiliary-variable structure, then uses sparse graph regularization to obtain permutation-identifiable latent factors under explicit assumptions. A VAE estimates the representation and graph, with the theory applying to sparse mechanisms and suitable graph connectivity.

  • 2.1. An identifiable latent causal model: The model observes sequences of images and auxiliary action vectors generated from hidden semantic latent variables through a diffeomorphic mixing function.
  • 2.1. An identifiable latent causal model: Latent factors are conditionally independent given past latent factors and past actions, with transition mechanisms drawn from a flexible exponential-family model.
  • 2.1. An identifiable latent causal model: Binary masks select direct latent and action parents, jointly defining the causal graph whose mechanisms govern latent transitions.
  • 2.1. An identifiable latent causal model: In the motivating environment, object interactions and action effects are sparse: the action affects only the robot, while objects have limited dependencies.
  • 2.4. Permutation-identifiability via mechanism sparsity regularization: Sparse mechanism shifts correspond to sparse changes in mechanisms, and Theorem 5 formalizes when they induce disentanglement.

3. Related work

The paper extends nonlinear ICA connections to temporal dependencies, auxiliary variables, and sparse latent causal graphs. It differs from prior methods by formally linking mechanism sparsity to disentanglement while accommodating broader dependency structures.

  • Nonlinear ICA methods use nonstationarity, temporal dependencies, or auxiliary variables to establish identifiability beyond independent latent factors.
  • The theory differs from iVAE by covering k = 1 and monotonic sufficient statistics, including Gaussian fixed-variance settings.
  • Unlike PCL and SlowVAE, the framework allows non-diagonal latent dependency graphs rather than assuming mutually independent latent sequences.
  • Compared with Locatello et al., the theory does not assume that only a random subset of latent components changes between consecutive times.
  • The paper addresses a literature gap by jointly learning latent factors and sparse causal graphs without requiring the graph structure to be known.

4. Experiments

Experiments evaluate a regularized VAE on synthetic temporal- and action-sparsity datasets using disentanglement, identifiability, and graph-recovery metrics. Appropriate mechanism sparsity regularization improves disentanglement and graph estimation across the tested datasets, while assumption violations reduce the gains.

  • Effect of regularization: The experiments vary temporal- and action-graph sparsity coefficients αz and αa, with R2 and MCC higher-is-better and SHD lower-is-better.
  • Synthetic datasets: The method learns latent transition models with Gaussian noise and MLP-predicted means on Markovian synthetic datasets with one-dimensional monotonic sufficient statistics.
  • Performance metrics: MCC measures permutation-identifiability, while normalized SHD measures the proportion of incorrectly estimated graph edges.
  • Effect of regularization: Regularization selected by filtered UDR substantially outperforms baselines, whereas the unregularized method performs similarly to them across all four datasets.
  • Effect of regularization: High R2 but low MCC for most baselines indicates linear identifiability without disentanglement in these experiments.
  • Violating assumptions: When sufficient variability, the graphical criterion, or the theorem’s k = 1 assumption is violated, regularization usually still improves MCC but by a smaller margin.

5. Conclusion

The conclusion presents mechanism sparsity regularization as a principle for disentanglement, supported by identifiability theory and a VAE-based estimator. Synthetic experiments demonstrate improved disentanglement and motivate extensions toward causal and interactive settings.

  • Mechanism sparsity regularization assumes that high-level dynamics are governed by sparse mechanisms, such as sparse object interactions or action effects.
  • The theory gives conditions on the dependency graph under which sparse mechanism regularization yields disentanglement.
  • Unknown-target interventions on latent factors provide a special case connecting disentanglement with sparse mechanism shifts.
  • The proposed estimator is VAE-based and learns causal mechanisms whose sparsity is regularized through binary masks.
  • Controlled synthetic experiments show that the approach can improve disentanglement, while broader realistic scenarios remain a future direction.

A.2. Proof of linear identifiability (Thm. 4)

The proof establishes linear identifiability by showing that observationally equivalent models have equivalent latent representations under stated regularity and variability assumptions. It combines denoising, support, density, and matrix-invertibility arguments to derive the result.

  • Theorem 4: The proof extends prior work by allowing non-differentiable sufficient statistics, λ depending on past latents, and no required relationship between λ and ˆλ.
  • Invertibility of L: Minimal sufficient statistics and sufficient variability provide the assumptions used to construct an invertible matrix linking transformed latent representations.
  • Theorem 4: Theorem 4 states that equality of conditional observation distributions for every auxiliary value implies linear equivalence of the two models.
  • Equality of denoised distributions: Equality of observation distributions is first converted into equality of denoised distributions using convolution and Fourier-transform arguments.
  • Equality of data manifolds: The equality of data manifolds follows from the shared support of the two observation models and their corresponding mixing functions.
  • Equality of densities: The remaining proof derives conditional-density equality through a change of variables involving v = ˆf−1 ◦ f and its Jacobian.
  • Proof structure: The proof organization separates denoising, density, linear-relation, and invertibility steps, with the overall proof structure summarized in Figure 4.

A.3.1. CENTRAL LEMMAS FOR THM. 5, 21 & 22

The central lemmas show that an invertible transformation preserving or reducing mechanism sparsity must be permutation-scaling under sufficient variability and a graphical criterion. They establish this for both square transformations and rectangular mechanism matrices, then combine the two cases.

  • Definitions: A permutation-scaling matrix has exactly one nonzero element in every row or column and can be written as a permutation matrix times a full-rank diagonal matrix.This characterizes the remaining ambiguity in the identifiability lemmas.
  • Lemma 17: Lemma 17 concludes that L is permutation-scaling when transformed matrix sparsity does not exceed the original under its three assumptions.The assumptions are sufficient variability, sparsity, and a graphical criterion.
  • Combined lemma: Lemma 19 combines the two preceding results and concludes that the shared transformation L is permutation-scaling when both mechanism sparsity patterns satisfy the required conditions.The combined setting includes a square and a rectangular mechanism family.

A.3.2. PROOF OF THE SPECIALIZED TIME-SPARSITY THEOREM (THM. 21)

The time-sparsity theorem maps latent transition Jacobian sparsity to the dependency graph and applies the central lemma to prove permutation-identifiability. Under sufficient variability, graph sparsity, and connectivity, equivalent models differ only by a permutation-scaling transformation.

  • Theorem statement: Theorem 21 states that two models representing the same distribution satisfy PGzP⊤ ⊂ Ĝz under the theorem’s sufficient-statistic and variability assumptions.The sufficient statistic is dz-dimensional and a diffeomorphism from Z to T(Z).
  • Assumptions: The sparsity assumption ||Ĝz||0 ≤ ||Gz||0 supplies the support-size condition required by the central lemma.The proof identifies the sparsity patterns of the relevant Jacobians with the dependency graphs.
  • Proof correspondence: The proof applies Lemma 17 by matching the abstract function Λ with the latent transition Jacobian and matching its sparsity pattern with Gz.The argument proceeds through the lemma’s three statements.
  • Conclusion: The theorem’s proof therefore establishes time-sparsity-based identifiability by converting graph structure into the matrix-sparsity conditions of Lemma 17.This is the final step from the specialized model assumptions to the abstract identifiability result.
  • Conclusion: The graphical criterion transfers to the dependency graph and yields a permutation-scaling transformation between equivalent models.The proof concludes this after establishing the lemma’s variability, sparsity, and graphical assumptions.

A.3.3. PROOF OF THE SPECIALIZED ACTION-SPARSITY THEOREM (THM. 22)

The action-sparsity theorem adapts the central lemma to finite-difference mechanism matrices. It proves permutation-identifiability when action-conditioned mechanisms have sufficient variability, no greater learned graph sparsity, and the required graphical criterion.

  • Theorem statement: Theorem 22 considers equivalent models with action dependency graphs Ga and Ĝa and applies the rectangular sparsity lemma to their mechanism differences.Its proof is explicitly described as analogous to Theorem 21 but uses Lemma 18.
  • Assumptions: The action-sparsity assumptions include one-dimensional sufficient statistics, sufficient variability for every action coordinate, and ||Ĝa||0 ≤ ||Ga||0.A graphical criterion is also required for every latent coordinate.
  • Proof correspondence: The proof identifies finite-difference sparsity with the action dependency graph, allowing Lemma 18 to transfer graph support constraints through the learned transformation.The argument uses the finite-difference matrices Δτλ and Δτλ̂.
  • Identifiability steps: Sufficient variability yields a permutation-supported inclusion between the original and learned action graphs.The proof then uses the sparsity bound to establish equality of the relevant supports.
  • Conclusion: The graphical criterion completes the argument by forcing L to be permutation-scaling.This is the third and final statement of the theorem’s proof.

A.3.4. PROOF OF THE COMBINED THEOREM (THM. 5)

The proof combines prior identifiability results by mapping their assumptions to the present model, then uses sparsity and the graphical criterion to obtain permutation-scaling identifiability. The appendix also explains extensions and practical boundaries of the theorem.

  • Theorem 5 assumes a dz-dimensional diffeomorphic sufficient statistic, sufficient time- and action-variability, model sparsity, and a graphical criterion.
  • The proof maps the theorem’s variability, sparsity, and graph assumptions to the corresponding assumptions of Lemma 19.
  • The graphical criterion is preserved under the correspondence between dependency graphs and sparsity patterns.
  • The combined argument concludes that the equivalence matrix L is a permutation-scaling matrix.
  • The identifiability argument extends to learned observation-noise variance when dz < dx and to learned parameters independent of the past.
  • The theory does not cover non-invertible mixing with occlusions, while combining multiple identifiability results remains future work.

A.8. Derivation of the ELBO

The ELBO derivation rewrites the observation and KL terms using the variational posterior and conditional latent model, then combines them into the desired lower bound.

  • The variational posterior is introduced before rewriting the observation term and decomposing the conditional KL term.
  • Combining the rewritten terms yields the ELBO used in the paper.
  • The derivation begins from the observation log-likelihood and bounds it using a KL divergence between variational and model distributions.

B.1. Synthetic datasets

The synthetic datasets vary temporal or action sparsity, graph structure, variability, and sufficient-statistic dimension to test the theorem’s conditions and violations.

  • The synthetic observations have dx = 20, and the ground-truth mixing function is an injective random neural network.
  • The experiments use Gaussian latent transitions with dz = da = 10, fixed covariance in most settings, and means designed around Theorem 5 assumptions.
  • Temporal-sparsity datasets include diagonal and triangular dependencies, insufficient variability, graphical-criterion violations, and k = 2.
  • Additional datasets alter the transition variance to create k = 2 settings or modify the dependency graph and variability assumptions.
  • Action-sparsity datasets include diagonal and double-diagonal dependencies, insufficient variability, graphical-criterion violations, and k = 2.

B.2. Implementation details of our regularized VAE approach

The method learns Gaussian latent mechanisms, graph masks, and VAE encoders and decoders, with experiments covering random graphs and noise levels.

  • Each latent coordinate has a Gaussian learned mechanism with neural-network mean and a learned variance independent of past steps.
  • Each graph edge is modeled as a Bernoulli variable with sigmoid probability, optimized using a Gumbel-Softmax gradient estimator.
  • Temporal-only and action-only experiments freeze the graph absent from the corresponding dependency type, while zero regularization keeps all edges active.
  • The encoder and decoder use six-layer fully connected neural networks, and the observation model has learned isotropic covariance.
  • Figure 5 varies randomly sampled graph edge probabilities across rows, with the first row representing the sparsest graphs.
  • Regularization improves MCC except for edge probability 0.75, where denser graphs are less likely to satisfy the graphical criterion.
  • Additional experiments vary latent noise and observation noise alongside randomly sampled graph structures.

B.4. Experiments that violate assumptions

Experiments probe the method beyond its theoretical assumptions and under varying noise levels. Regularization generally improves disentanglement, but performance can weaken when assumptions fail or observations become noisier.

  • Violating assumptions: Regularization improved MCC on datasets violating assumptions, except time-sparsity data with insufficient variability, where no improvement occurred.The gains were smaller than when the theoretical assumptions were satisfied.
  • Violating assumptions: When sufficient variability held, the graph could still be learned despite violation of the graphical criterion.The graphical-criterion violation used block-diagonal graphs.
  • Noise robustness: The theory applies across latent noise levels, while learning was evaluated at standard deviations 0.01, 0.1, and 0.5.The experiments used 0.01 as the standard noise level and regarded 0.5 as very high.
  • Noise robustness: At latent standard deviation 0.5, the approach performed well except on the time-sparsity dataset.The authors characterize this as a very high noise level relative to the Normal(0, I) initial latent.
  • Noise robustness: Higher observation noise harmed time-sparsity performance but not the action-sparsity dataset.The authors suggest noisier data may worsen sample complexity or make the approximate posterior inadequate.
  • Learned graphs: Figures 10 and 11 show regularization improving disentanglement while learned adjacency matrices remain reasonably close to, but not exactly equal to, ground truth.The visualizations pair learned graphs with Pearson correlation matrices for learned and ground-truth representations.
Loading 2107.10098v3…