PaperlayerView exact revision on arXiv

Source-linked AI summary

Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning

Aapo Hyvarinen, Hiroaki Sasaki, Richard E. Turner

arXiv:1805.08651v3stat.MLcs.LG

TL;DR

Nonlinear ICA lacks identifiability under general nonlinear mixing and i.i.d. observations. The paper introduces an auxiliary-variable framework and estimates it by contrastive logistic regression, proving exact identifiability conditions and consistency while unifying temporal and nonstationary approaches. The method’s scope is bounded by assumptions including invertibility and sufficient variability.

  • Problem

    General nonlinear ICA is seriously unidentifiable from i.i.d. observations, so recovering the underlying latent components requires additional structure.

  • Method

    The framework conditions source independence on an observed auxiliary variable and learns by discriminating real augmented data from data with a randomized auxiliary variable.

  • Results

    The paper proves exact identifiability conditions and shows that contrastive estimation consistently recovers independent components up to component-wise invertible transformations.

  • Takeaways & Limitations

    The framework unifies temporal-dependence and nonstationarity principles and extends nonlinear ICA to auxiliary information such as time, spatial indices, labels, and stimuli.

  • Takeaways & Limitations

    The theory requires assumptions including invertibility of the learned representation and sufficient source variability, with overly simple Gaussian scale-location families excluded.

Abstract

from arXiv · show

Nonlinear ICA is a fundamental problem for unsupervised representation learning, emphasizing the capacity to recover the underlying latent variables generating the data (i.e., identifiability). Recently, the very first identifiability proofs for nonlinear ICA have been proposed, leveraging the temporal structure of the independent components. Here, we propose a general framework for nonlinear ICA, which, as a special case, can make use of temporal structure. It is based on augmenting the data by an auxiliary variable, such as the time index, the history of the time series, or any other available information. We propose to learn nonlinear ICA by discriminating between true augmented data, or data in which the auxiliary variable has been randomized. This enables the framework to be implemented algorithmically through logistic regression, possibly in a neural network. We provide a comprehensive proof of the identifiability of the model as well as the consistency of our estimation method. The approach not only provides a general theoretical framework combining and generalizing previously proposed nonlinear ICA models and algorithms, but also brings practical advantages.

1 INTRODUCTION

Nonlinear ICA seeks identifiable latent variables through nonlinear feature extraction, but arbitrary smooth invertible mixing is seriously unidentifiable from i.i.d. observations. The paper proposes a general auxiliary-variable framework that unifies temporal approaches and supports identifiability proofs and practical estimation.

  • Motivation: Nonlinear ICA aims to recover latent variables through a generative model that is identifiable by design.The approach differs from many unsupervised representation-learning methods by explicitly defining a recoverable generative model.
  • The identifiability problem: Arbitrary smooth invertible nonlinear mixing is seriously unidentifiable when observations are i.i.d.Independence alone can permit alternative constructions in which observed variables are treated as independent components.
  • Existing solutions: Temporal structure has enabled recent nonlinear ICA identifiability conditions, including autocorrelations, temporal dependencies, and nonstationarity.These results extend earlier linear ICA principles to nonlinear settings.
  • Contribution: The proposed framework makes independent components conditionally mutually independent given an auxiliary variable and proves exact identifiability conditions.The auxiliary variable generalizes temporal structure rather than requiring a specifically temporal or spatial dataset.

2 BACKGROUND

The background explains why nonlinear ICA cannot generally be identified from i.i.d. data alone and reviews temporal strategies that restore identifiability. Earlier methods use autocorrelation, temporal dependence, or segmentation-based nonstationarity, often under restrictive source conditions.

  • Non-identifiability: With a general nonlinear mixing function and i.i.d. observations, independence does not determine the underlying components.A construction can transform variables so that alternative observed-variable choices appear independent.
  • Non-identifiability: Marginal non-Gaussianity does not resolve nonlinear ICA because point-wise transformations can produce arbitrary marginal distributions.This removes the key distributional cue used in linear ICA.
  • Temporal approaches: Temporal structure offers an alternative route to identifiability through source autocorrelations, general temporal dependencies, or nonstationarity.These approaches relax the i.i.d. sampling assumption.
  • Temporal approaches: Distinct autocorrelation functions can enable nonlinear source separation, but requiring different source statistics is restrictive.Sprekeler et al. provided an early identifiability proof for general nonlinearities under this condition.
  • Contrastive approaches: Permutation-contrastive learning reframes temporal dependence as classification between real time windows and randomized concatenations.The randomized dataset is formed by permuting the temporal pairing, and a neural network discriminates the two classes.
  • Nonstationary approaches: Nonstationary approaches divide time series into segments and train multinomial regression to predict the segment index.The segment label is artificially assigned from each data point’s time-segment membership.

3 NONLINEAR ICA USING AUXILIARY VARIABLES

The proposed nonlinear ICA model introduces an observed auxiliary variable that modulates conditionally independent sources, unifying temporal dependence and nonstationarity while allowing broader application-specific choices. Estimation uses contrastive discrimination between real and randomized data, implemented with logistic regression or neural networks.

  • Generative model: The model assumes each source depends on an observed auxiliary variable while remaining conditionally independent of the other sources.The mixing function remains broadly specified as smooth and invertible rather than restricted to a particular functional form.
  • Generative model: The auxiliary variable can be a time-series history, time index, segment index, pixel index, related audio or text, class label, or stimulus-related quantity.Multiple auxiliary variables may also be combined depending on the application.
  • Generative model: The framework unifies temporal-dependence and nonstationarity separation principles.These principles had been treated as distinct in linear ICA.
  • Generative model: Conditional independence does not imply marginal independence because the auxiliary variable can induce dependence among sources after marginalization.The formulation therefore generalizes ordinary ICA independence rather than simply reproducing it.
  • Learning algorithm: The learning algorithm discriminates between real augmented data and versions in which the auxiliary variable has been randomized.Randomization is implemented by independently permuting empirical auxiliary-variable samples.
  • Learning algorithm: A nonlinear logistic regression model, possibly a neural network, provides the contrastive estimator.Hidden units compute scalar features, and universal approximation capacity is assumed for the relevant model functions.

4 THEORETICAL ANALYSIS

The theoretical analysis establishes consistency and identifiability conditions for nonlinear ICA with auxiliary variables. Recovery is component-wise up to invertible nonlinearities in the general case, while conditionally exponential sources can retain a linear indeterminacy.

  • General theory: The framework separates general sources from conditionally exponential sources of low order because their consistency conditions differ.The analysis treats the conditionally exponential case separately from the general case.
  • General theory: Under smoothness, independence, variability, and invertible-regressor assumptions, nonlinear logistic regression consistently estimates the independent components with infinite data.The hidden units recover the components up to scalar, component-wise invertible transformations.
  • General theory: The Assumption of Variability requires the auxiliary variable to affect source distributions sufficiently strongly and diversely.For conditionally exponential sources, this excludes order k = 1 and overly simple source modulation; for k > 1, additional independence and support conditions can ensure variability.
  • General theory: Invertibility of the learned demixing function h is a non-trivial theoretical assumption, although simulations and subsequent theory suggest it may not be necessary in practice.The paper explicitly identifies h's invertibility as a constraint of the rigorous theory.
  • Conditionally exponential case: The general theorem differs from TCL because it removes the linear indeterminacy, whereas the conditionally exponential theory recovers the TCL-like ambiguity.The paper presents this distinction as a fundamental difference between the theories.
  • Conditionally exponential case: For conditionally exponential sources, recovery is only up to a linear transformation of point-wise sufficient statistics.The unknown linear transformation is represented by a matrix A and vector c, reflecting linearity of the log-pdf in exponential families.

5 DIFFERENT DEFINITIONS OF AUXILIARY VARIABLES

The framework supports multiple auxiliary variables, including time, history, combined temporal information, and class labels. These choices connect the method to prior nonlinear ICA approaches while extending them beyond specifically temporal data.

  • Auxiliary-variable choices: The auxiliary variable can be defined flexibly, but it must preserve conditional independence of the components.A deterministic function of x can violate this central assumption.
  • Time index: Using the time index as u turns nonstationary source distributions into a discrimination problem between true and randomized augmentations.Logistic regression is used to distinguish the augmented datasets, with component-wise nonstationarity modeled without interactions.
  • Time index: Theorem 1 and Theorem 3 provide consistency conditions for estimating the nonstationary nonlinear ICA model through this logistic-regression procedure.This gives an alternative estimator for the nonstationary model associated with TCL.
  • History: Using the history of each variable as u recovers the temporal-dependence setting of PCL as a special case of the general discrimination framework.The past of x and the past of h(x) are equivalent here because they are invertibly related.
  • Combined temporal information: Setting u = (x(t −1), t) combines temporal dependencies with nonstationarity in one nonlinear ICA method.This avoids choosing between the two types of temporal structure because both are used as auxiliary information.
  • Class labels: Class labels can serve as auxiliary variables for representation learning or structural analysis rather than only for the original classification task.The randomized-label discrimination setup can have no linear indeterminacy except in the low-rank conditionally exponential case.

6 SIMULATIONS

The simulations test whether the general framework includes methods analogous to TCL across nonstationary source settings. They examine both non-conditionally-exponential and conditionally exponential cases.

  • Experimental scope: The experiments target nonstationary sources, the paper’s main simulation setting, rather than claiming superiority over TCL.For temporally correlated sources, the framework yields PCL; the stated goal is inclusion of a TCL-like method.
  • General case: The first experiment tests the general Theorem 1 case using scale-mixture sources on a 2D grid.The sources use standardized Laplacian variables whose scale components vary across the grid.
  • Conditionally exponential case: The second experiment tests the conditionally exponential Theorem 3 case with time-series sources divided into equally spaced segments.A random neural network generates separate variances within each segment, and the segment index is used as the auxiliary variable.

7 CONCLUSION

The framework addresses nonlinear ICA’s non-identifiability by conditioning on an auxiliary variable and learning through generalized contrastive discrimination. It unifies temporal and class-label settings while providing identifiability conditions and a consistent learning algorithm.

  • The framework uses an auxiliary variable whose conditioning changes independent-component distributions to address nonlinear ICA non-identifiability.
  • Temporal history or time indices serve as special cases, unifying earlier nonlinear ICA frameworks in theory and practice.
  • The theory gives exact identifiability conditions and divides the problem according to conditional exponentiality.
  • The method uses contrastive learning by logistic regression and includes a consistency proof.
  • The auxiliary variable can be defined in multiple ways depending on the application.

Nonlinear ICA using auxiliary variables and

The section identifies generalized contrastive learning as the organizing concept.

  • Generalized contrastive learning is identified as the section’s central concept.
  • The section frames generalized contrastive learning as a methodological theme.
  • The section introduces generalized contrastive learning without further elaboration in the supplied passage.

A Proof of Theorem 1

The proof uses logistic-regression density-ratio estimation, a change of variables, and variability assumptions to show that the learned representation separates componentwise. It concludes convergence and a nonlinear ICA identifiability result.

  • After convergence, logistic regression with infinite data and universal approximation recovers the difference between the two class log-densities.
  • The proof rewrites the density relation after changing variables to y = h(x), defining v(y) = g(h^-1(y)).
  • To establish identifiability, the proof requires that the functions vi depend on only one coordinate yi for all y and u.
  • Differentiating with respect to distinct coordinates produces equations involving second-order cross-derivatives, which are collected into vector form.
  • The resulting equations form a linear system whose matrix is forced to vanish by the invertibility of a 2n × 2n matrix under the variability assumption.
  • Consequently, each vi depends on only one yi, each scalar function is invertible, and the method is both convergent and identifiable.

B Proof of Theorem 2

The proof examines the variability condition through basis vectors generated by conditional exponentiality. It shows that multiple components can span the required 2n-dimensional space, whereas k = 1 cannot.

  • The proof simplifies to n = 2 and k = 3, while noting that the argument generalizes to arbitrary dimensions and Qi ≡ 1.
  • For fixed s, w(s, u) is a linear combination of nk fixed basis vectors whose coefficients are the λ values.
  • When k = 1, 2n linearly independent vectors cannot be obtained, but k > 1 allows the basis vectors to span a 2n-dimensional subspace.
  • The construction uses independently and randomly chosen nonzero-measure λij values as a sufficient, though not necessary, condition.
  • Subtracting w(s, u0) preserves linear independence almost surely because it only redefines the origin.

C Proof of Theorem 3

The proof establishes the regression-function form and shows that the alternative variability assumption also suffices for Theorem 1. The resulting representation recovers the sufficient statistics up to the theorem’s stated indeterminacies.

  • Proof strategy: The regression function can represent the required log-density differences because those differences are linear in the same form as the regression function.The construction defines q0(s) = log ps(s), making the approximation possible.
  • Proof strategy: The proof then establishes that this representation is the only solution up to the indeterminacies specified by the theorem.
  • Matrix formulation: Collecting the equations across auxiliary-variable values yields matrix relations involving sufficient statistics, variability coefficients, regression parameters, and normalization terms.The matrices use the auxiliary-variable index as the column index, while the vector collects sufficient statistics for all independent components.
  • Matrix formulation: Subtracting the first matrix row from the others removes the final normalization term from the resulting equation.
  • Theorem consequence: The hidden units recover the sufficient statistics up to a linear transformation, as in TCL.The theorem’s matrix structure identifies the sufficient-statistics representation under the stated assumptions.
  • Alternative variability assumption: Theorem 1 also holds under the alternative variability assumption requiring 2n auxiliary-variable values whose corresponding vectors are linearly independent.The proof differentiates with respect to the auxiliary-variable values, reducing the argument to the same logic used for the original theorem.

E Using a function of x as auxiliary variable

Using a deterministic function of the observations as the auxiliary variable is likely to violate conditional independence. The auxiliary variable therefore needs to contribute information beyond the observed data itself.

  • Deterministic auxiliary variables: Defining the auxiliary variable directly from x can create dependence among latent components after conditioning.For x1 = s1 + s2 and u = x1, conditioning gives s1 = u − s2, violating conditional independence.
  • Deterministic auxiliary variables: The auxiliary variable must provide new information in addition to x, in a sufficiently independent relation to the information in x.
  • Relation to prior work: Earlier temporal nonlinear ICA work used identifiability assumptions involving non-factorizable conditional exponential structure and strong temporal dependency.The present discussion contrasts those assumptions with the framework’s use of an invertible h, which removes the need for the latter condition in the cited comparison.
  • Randomization choices: Randomizing two auxiliary inputs with separate indices is not directly covered by the stated theorems, although the results might be reformulated for that setting.The choice of randomization should depend on how strongly the conditional source distributions are modulated.
Loading 1805.08651v3…