Source-linked AI summary
The Incomplete Rosetta Stone Problem: Identifiability Results for Multi-View Nonlinear ICA
Luigi Gresele, Paul K. Rubenstein, Arash Mehrjou, Francesco Locatello, Bernhard Schölkopf
TL;DR
Recovering shared independent sources from nonlinear views is difficult because a single-view nonlinear ICA problem is unidentifiable. The paper uses contrastive learning to prove identifiability for multiple views under stated conditions, with simultaneous demixing and source recovery indicated in the limit of infinitely many views.
Problem
A single nonlinear view does not identify shared independent sources, while multiview measurements need a common representation without observed parent variables.
Source: §1 · pp. 1–2, §5 · p. 9Method
The paper uses contrastive learning on paired observations of the same sources and analyzes component-wise noisy views under a sufficiently distinct views assumption.
Source: §3 · p. 3, §3.1 · p. 4Results
The proofs establish identifiability in multiple-view settings, and Theorem 8 points to simultaneous demixing and source recovery in the limit of infinitely many views.
Source: §3.2 · p. 6, §3.3 · p. 7, §5 · p. 9Takeaways & Limitations
Additional views can constrain nonlinear ICA ambiguities when the views are sufficiently different; identical views add nothing.
Source: §3.1 · p. 4, §5 · p. 9The analysis assumes component-wise independent corruption and requires stochasticity in the relationship between the sources and at least one observation.
Source: §3 · p. 3
Abstract
from arXiv · showhide
We consider the problem of recovering a common latent source with independent components from multiple views. This applies to settings in which a variable is measured with multiple experimental modalities, and where the goal is to synthesize the disparate measurements into a single unified representation. We consider the case that the observed views are a nonlinear mixing of component-wise corruptions of the sources. When the views are considered separately, this reduces to nonlinear Independent Component Analysis (ICA) for which it is provably impossible to undo the mixing. We present novel identifiability proofs that this is possible when the multiple views are considered jointly, showing that the mixing can theoretically be undone using function approximators such as deep neural networks. In contrast to known identifiability results for nonlinear ICA, we prove that independent latent sources with arbitrary mixing can be recovered as long as multiple, sufficiently different noisy views are available.
1 INTRODUCTION
The paper asks whether multiple nonlinear observations of shared independent sources can jointly resolve ambiguities that make nonlinear ICA unidentifiable from a single view. It uses contrastive learning to establish identifiability under assumptions on how the views relate, rather than restrictions on the sources or mixing functions.
1 INTRODUCTION
- The paper studies recovering shared independent sources from two observations produced by arbitrary smooth, invertible transformations of those sources.The motivating setting includes recordings of the same process made with different instruments or modalities.Source: §1 · pp. 1–2
- Nonlinear ICA is unidentifiable without additional assumptions, unlike linear ICA, whose identifiability holds when at most one source component is Gaussian.Existing nonlinear ICA approaches have therefore relied on assumptions about source properties or mixing-function form.Source: §1 · p. 1
- The paper employs contrastive learning for this setting without requiring observations of parent variables of the sources, unlike.This targets common representations from paired measurements in multiview biomedical and neuroimaging datasets.Source: §1 · p. 2
- Jointly addressing multiple demixing problems yields identifiability under assumptions on the conditional distribution of one observation given the other, rather than directly on sources or mixing functions.The views must be sufficiently diverse.Source: §1 · p. 2
2 NONLINEAR ICA WITH CONTRASTIVE LEARNING
Contrastive learning recasts unsupervised learning as supervised function approximation and supports constructive identifiability proofs. The paper reviews its use with auxiliary variables in nonlinear ICA and motivates adapting the technique to paired views of shared sources.
2 NONLINEAR ICA WITH CONTRASTIVE LEARNING
- Nonlinear ICA with an auxiliary variable u replaces unconditional source independence with conditional independence given u and requires variability in u’s influence on x.This setup interprets u as a parent of the sources.Source: §2 · p. 2
- Contrastive learning recasts unsupervised learning as supervised function approximation, enabling constructive identifiability proofs and practical algorithms.The cited auxiliary-variable setup distinguishes joint samples (x, u) from samples (x, u*) with u* drawn independently from its marginal.Source: §2 · p. 2
- In the auxiliary-variable method, reconstruction of s is possible only up to an invertible scalar gauge transformation, a fundamental ambiguity of nonlinear ICA.The passage characterizes this ambiguity as trivial rather than a limitation of those results.Source: §2 · pp. 2–3
3 NONLINEAR ICA WITH MULTIPLE VIEWS
This section studies how multiple corrupted views can resolve ambiguities in nonlinear ICA and enable reconstruction of their common source. It covers recovery across different corruption settings, including sufficient conditions for reconstruction up to a shared affine ambiguity using infinitely many corrupted views.
3.1 ONE NOISELESS VIEW
- With one noiseless view, joint contrastive learning recovers the independent source components up to component-wise invertible transformations, given SDV and infinite data with universal approximation capacity.The SDV condition requires the corruption distribution to vary sufficiently with the source.Source: §3.1 · p. 4
- When both views satisfy SDV, the learned representations of the source and its corruption are aligned, under stricter assumptions than for recovering the source alone.Theorem 4 requires SDV for both views; Theorem 1 requires it for only one.Source: §3.1 · p. 5
3.1.1 Equivalence with Permutation Contrastive Learning for Time Dependent Sources
- The analysis also covers temporally dependent stationary sources considered in, while allowing asynchronous measurements and different mixing functions across views.The paper notes that its SDV assumption does not necessarily imply the uniform dependency assumed in.Source: §3.1.1 · p. 5
3.2 TWO NOISY VIEWS
- With two noisy views, the method recovers aligned representations of each view’s corrupted source, and exact source recovery is approached as one view’s noise vanishes.Theorem 5 assumes SDV for both views; exact recovery in the small-noise limit also requires the stated conditions on the corrupters.Source: §3.2 · p. 6
3.3 MULTIPLE NOISY VIEWS
- With many noisy views, pairwise recovery can be aligned across views by independence testing, matching components that are deterministically related and permuting the others.Pairwise methods recover components up to different component-wise invertible functions, so alignment across pairs is not automatic.Source: §3.3 · pp. 6–7
- For additive zero-mean noise, averaging recovered s + n_i approaches s almost surely as the number of views grows, under mild conditions on noise-variance growth.With unknown component-wise invertible functions, the text further states that suitable choices can make dependence on most or all noises disappear in the limit.Source: §3.3 · p. 7
- When noise variances are bounded by C and K ≥ Var(s) + C, the specified set G_K is non-empty for sufficiently large K.The stated bound is Var(n_i) ≤ C for all i.Source: §3.3 · p. 7
- Theorem 8 allows recovery of s and each n_i up to shared affine parameters, with Ω_e(s) = αs + β and R_e,i(n_i) = αn_i.The theorem provides sufficient conditions under which each e_i inverts k_i up to an affine ambiguity shared across views.Source: §3.3 · pp. 7–8
- Conditions 17–21 are verifiable from known information; the authors conjecture that condition 22 could be relaxed using independence and additional regularity assumptions.The proposed alternative requires independence between Ω_e(s) and R_e,i(s, n_i) for every i, with regularity such as smoothness.Source: §3.3 · p. 7
4 RELATED WORK
The related work reviews methods that use multiple views and distinguishes their objectives and identifiability guarantees from this paper’s goal of recovering continuous independent sources.
4.1 CANONICAL CORRELATION ANALYSIS
- CCA finds cross-correlated linear subspaces, whereas this method targets independent components with nonlinear mappings; identifiability results for nonlinear CCA are lacking.The paper relates its Section 3.2 model to CCA as nonlinear ICA relates to PCA.Source: §4.1 · p. 8
4.2 MULTI-VIEW LATENT VARIABLE MODELS
- Diffusion maps find the common source of variability; learns mixture parameters rather than exact latent realizations, whereas this work inverts each view’s mixing and recovers the common latent variables.Source: §4.2 · p. 8
4.3 HALF-SIBLING REGRESSION
- Half-sibling regression estimates shared noise by regressing one corrupted observation on other noise-affected variables, then subtracts it to recover the source up to an offset.This requires additive corruption and sufficient information about the noise effect in the observation.Source: §4.3 · p. 8
5 DISCUSSION AND CONCLUSION
The discussion argues that multiple sufficiently different views can support nonlinear ICA identifiability, while noise and the number of views constrain the results. The appendix reviews why a single-view nonlinear ICA solution exists but is non-unique, including a component-wise invertible ambiguity.
5 DISCUSSION AND CONCLUSION
- Theorem 8 points toward simultaneously undoing the mixing and recovering sources in the limit of infinitely many views.The discussion distinguishes these tasks because each view is a corrupted version of the common sources.Source: §5 · p. 9
- A second view helps resolve nonlinear ICA ambiguities only when it is sufficiently different from the first.Identical views add no information; the discussion links sufficient diversity to the complexity of the joint distribution of corrupted sources.Source: §5 · p. 9
- The setting treats noise as crucial for contrastive learning and as view-specific variability in practical applications.Without noise variables, the sufficiently distinct views assumption could not hold; the authors note that stochasticity is prevalent in real-world systems.Source: §5 · p. 9
- The authors identify future work on using more than pairwise information and on finite-view guarantees; Theorem 8 is asymptotic.They conjecture that algorithms using N > 2 views could relax the additivity assumption on corruptions.Source: §5 · p. 9
A.1 EXISTENCE
- For single-view nonlinear ICA, a mapping to independent uniform components exists, but the construction is not unique.The appendix describes a recursive construction and states that the resulting components are independent and uniformly distributed in the unit cube.Source: §A.1 · p. 12, §A.2 · p. 12
A.2 NON-UNIQUENESS
- The appendix describes multiple sources of non-uniqueness, including measure-preserving transformations that generate infinitely many solutions.It also describes linear transformations before demixing and element-wise transformations of sources as further ambiguities.Source: §A.2 · pp. 12–13
A.3 THE SCALAR INVERTIBLE FUNCTION GAUGE
- Even non-Gaussianity cannot remove the element-wise invertible-function ambiguity in nonlinear ICA.The appendix relates this indeterminacy to the scalar multiplicative ambiguity in linear ICA.Source: §A.3 · p. 13
B WHY DOES CLASSIFICATION RESULT IN THE LOG RATIO?
The appendix explains why contrastive classification estimates a density ratio and outlines conditions for its use. It also illustrates the sufficiently distinct views assumption with cases where it fails or holds.
B WHY DOES CLASSIFICATION RESULT IN THE LOG RATIO?
- At the global optimum of the cross-entropy loss, the classifier estimates p0(x)/(p0(x)+p1(x)), yielding the density ratio through its logit.In the paper's setting, the classifier distinguishes paired observations from samples drawn from the product of their marginals.Source: B WHY DOES CLASSIFICATION RESULT IN… · p. 13
- If x1 and x2 are deterministically related, their joint and product distributions are not absolutely continuous with respect to each other, so the log-ratio is ill-defined.The appendix notes that in this case the log-ratio is everywhere either 0 or infinite and learned features are not useful.Source: B WHY DOES CLASSIFICATION RESULT IN… · p. 13
C THE SUFFICIENTLY DISTINCT VIEWS ASSUMPTION
- The sufficiently distinct views assumption fails for the illustrated Gaussian conditional distribution, although the latent sources themselves may be Gaussian.The appendix states that the source distribution does not enter the expression used for the assumption.Source: C THE SUFFICIENTLY DISTINCT VIEWS… · pp. 13–14
- For the other illustrated conditional distribution, the appendix argues that the SDV condition holds by constructing 2D linearly independent vectors through different choices of z.Its argument factors w(s, z) into terms depending only on s and z, respectively.Source: C THE SUFFICIENTLY DISTINCT VIEWS… · p. 14
D.1 PROOF OF THEOREM 1
The proof uses the sufficiently distinct views assumption to constrain the form of a recovered representation. It concludes that source components are recovered up to component-wise invertible transformations and permutation.
D.1 PROOF OF THEOREM 1
- The proof uses the SDV assumption to obtain 2n linearly independent vectors, forcing the matrix M(y) to be zero.The matrix is described as n(n −1)/2 × 2n, and the vectors as 2n-dimensional.Source: §D.1 · p. 14
- Since M(y) is zero, each recovered component depends on only one coordinate of y.The proof argues that dependence on multiple coordinates would make a matrix entry nonzero.Source: §D.1 · p. 14
D.2 PROOF OF COROLLARY 3
- The recovered source components are determined up to component-wise invertible ambiguity and permutation; Corollary 3 follows by exchanging x1 and x2.The proof explicitly states both remaining ambiguities and says the corollary repeats the argument with the view roles exchanged.Source: §D.1 · p. 14, §D.2 · p. 15
E PROOF OF THEOREMS 4 AND 5
The proofs establish component-wise recovery from each view and then address whether the two recovered representations are aligned. Theorem 4 is treated as a special case of Theorem 5.
E PROOF OF THEOREMS 4 AND 5
- Theorem 4 is a special case of Theorem 5 when g1(s, n1) = s.Source: E PROOF OF THEOREMS 4 AND 5 · p. 15
- Theorem 5's proof shows that each recovered representation depends on only one coordinate, yielding component-wise recovery up to invertible functions and permutation.The proof invokes the SDV assumption on η to obtain linearly independent vectors and force the matrix M(y) to be zero.Source: E PROOF OF THEOREMS 4 AND 5 · p. 15
- The proof states that the two views estimate their respective corrupted sources up to two different gauges, then addresses possible misalignment.The two permutations need not initially agree.Source: E PROOF OF THEOREMS 4 AND 5 · p. 16
- The alignment argument uses independence relations to conclude that the corresponding transformed variables from the two views are independent.The proof relies on invertibility of the component transformations for the final implication.Source: E PROOF OF THEOREMS 4 AND 5 · p. 17
F PROOF OF COROLLARY 6
The proof considers recovery up to a component-wise invertible ambiguity. It uses a lower bound valid for every candidate in the class, including the inverse of g1.
F PROOF OF COROLLARY 6
- Recovery is described up to a component-wise invertible ambiguity in the relevant transformed source expression.The passage fragments refer to g(s, n(k)) and its component-wise invertible ambiguity.Source: F PROOF OF COROLLARY 6 · p. 17
- The lower bound applies to every e* in E by the definition of the infimum.Source: F PROOF OF COROLLARY 6 · p. 17
- In particular, the proof uses e* = g1^-1, whose existence follows from the assumptions on g1, and takes a Taylor expansion.Source: F PROOF OF COROLLARY 6 · p. 17
G PROOF OF LEMMA 7
The proof applies Kolmogorov’s strong law to show that, for almost all s, the relevant random quantity converges almost surely to a function of s. Independence and variance bounds supply the conditions needed for that application.
G PROOF OF LEMMA 7
- Kolmogorov’s strong law is applied to independent, not necessarily identically distributed variables, with the proof checking its conditions for almost all s.The argument uses bounded variance and independence across noise variables to establish the required conditions.Source: G PROOF OF LEMMA 7 · p. 17
- For almost all s, e(s,n) converges n-almost surely to a constant, so its average over n converges almost surely to a function of s.Source: G PROOF OF LEMMA 7 · p. 17
- The proof uses the law of total expectation and the assumed variance bound to establish finiteness conditions needed for the almost-sure convergence argument.Source: G PROOF OF LEMMA 7 · p. 17
H PROOF OF THEOREM 8
Under the stated variance threshold, the proof shows that admissible coordinate-wise maps must share a common affine form. It first establishes affinity and then rules out differences in their linear and constant terms.
H PROOF OF THEOREM 8
- The argument considers the class under the threshold K ≥ Var(s) + C and verifies the required conditions using the noise variance bound and independence.Source: H PROOF OF THEOREM 8 · p. 18
- For sufficiently large K, every set of maps in the class has the form e_i = αk_i^-1 + β, with diagonal α and shared β.The proof establishes a common linear and constant term across views.Source: H PROOF OF THEOREM 8 · p. 18
- Each composition e_i ◦ k_i is affine, initially with potentially view-specific coefficients.Source: H PROOF OF THEOREM 8 · p. 18
- The proof shows that the coefficients must agree across views: differing linear terms or offsets contradict the stated constancy conditions.Source: H PROOF OF THEOREM 8 · p. 18