Source-linked AI summary
Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA
Aapo Hyvarinen, Hiroshi Morioka
TL;DR
Nonlinear ICA is appealing for unsupervised learning but lacks identifiability, especially without temporal structure. The paper introduces time-contrastive learning and a nonstationary nonlinear ICA model, showing that TCL plus linear ICA identifies the model up to source-wise transformations, with stronger identifiability for modulated Gaussian sources.
Problem
Existing nonlinear ICA models are not identifiable, while unsupervised feature-learning methods also lack scalability, theoretical justification, or both.
Method
The paper learns features by discriminating nonstationary time segments with TCL and models independent sources whose distributions vary across segments.
Results
TCL combined with linear ICA estimates the nonlinear ICA model up to component-wise nonlinearities, with modulated Gaussian sources reducing the indeterminacy to monotonic component-wise transformations.
Takeaways & Limitations
The results provide a constructive identifiability proof for a high-dimensional nonlinear ICA mixing model together with a practical estimation method.
Takeaways & Limitations
For general sources, TCL followed by linear ICA leaves component-wise nonlinear indeterminacies such as squaring or absolute values.
Abstract
from arXiv · showhide
Nonlinear independent component analysis (ICA) provides an appealing framework for unsupervised feature learning, but the models proposed so far are not identifiable. Here, we first propose a new intuitive principle of unsupervised deep learning from time series which uses the nonstationary structure of the data. Our learning principle, time-contrastive learning (TCL), finds a representation which allows optimal discrimination of time segments (windows). Surprisingly, we show how TCL can be related to a nonlinear ICA model, when ICA is redefined to include temporal nonstationarities. In particular, we show that TCL combined with linear ICA estimates the nonlinear ICA model up to point-wise transformations of the sources, and this solution is unique --- thus providing the first identifiability result for nonlinear ICA which is rigorous, constructive, as well as very general.
1 Introduction
Existing unsupervised feature-learning methods lack scalability, theoretical justification, or both, while nonlinear ICA remains unidentifiable without temporal structure. The paper addresses these gaps by combining time-contrastive learning with a nonstationary nonlinear ICA model to obtain constructive identifiability results.
- Existing unsupervised feature-learning approaches are widely used but generally lack scalability, theoretical justification, or both.
- Temporal coherence and slow feature analysis use time-series structure, but their temporal stability criteria are not clearly optimal.
- Without temporal structure, nonlinear ICA is seriously unidentifiable, and no generally applicable identifiability conditions had been found.
- The paper proposes time-contrastive learning, which extracts features that maximally discriminate among time windows using temporal nonstationarity.
- The proposed nonlinear ICA model gives independent components segment-dependent distributions and observes them through nonlinear mixtures.
- TCL combined with linear ICA estimates the nonlinear ICA model up to component-wise nonlinearities, while modulated Gaussian sources permit full identifiability.
2 Time-contrastive learning
Time-contrastive learning trains a feature extractor to make time segments discriminable through multinomial logistic regression. The approach converts temporal distributional differences into an unsupervised representation-learning objective implementable with standard neural-network training.
- TCL trains a feature extractor with multinomial logistic regression to discriminate all time segments using their segment indices as labels.
- The time series is divided into indexed windows, and every data point receives the label of its containing segment.
- The feature extractor and logistic classifier are trained jointly, with the extractor producing features used to classify segment labels.
- Because segment discrimination requires capturing distributional differences across windows, TCL uses classification to perform unsupervised learning.
- A multilayer perceptron can implement the feature extractor, with its last hidden layer serving as the learned representation.
3 TCL as approximator of log-pdf ratios
Under universal approximation and infinite data, the optimal TCL classifier recovers segment log-density ratios relative to a chosen pivot, with a prior correction for unequal segment probabilities. This gives TCL a direct probabilistic interpretation.
- TCL’s multinomial logistic regression models the posterior probability of each segment label for an observed data point.
- The classifier uses a feature extractor parameterized by θ together with segment-specific weights and biases in W and b.
- The true segment-label posterior is expressed through the segment prior and the conditional density pτ(xt) for each window.
- With universal feature capacity and infinite data, the learned classifier posterior equals the true posterior for every segment.
- At optimum, TCL features encode each segment’s log-density relative to the first segment, plus a segment-prior term that vanishes for equal-length windows.
4 Nonlinear nonstationary ICA model
The paper defines nonlinear nonstationary ICA by combining smooth invertible nonlinear mixing with mutually independent, temporally nonstationary source components. Source densities use an exponential-family form whose segment-dependent parameters capture changing distributions.
- The observed time series is modeled as a smooth, invertible nonlinear mixture of mutually independent source signals.
- Source nonstationarity is assumed slower than sampling, allowing the time series to be divided into segments with approximately constant but differing distributions.
- Each source density belongs to an exponential family whose parameters depend on the segment index, creating temporal nonstationarity.The stationary baseline and nonlinear functions define the source family, while the segment-dependent parameters control distribution changes.
- The model generalizes variance-modulated linear ICA to nonlinear, non-Gaussian densities using non-quadratic functions and multiple functions per source.Examples include modulated Gaussian, Laplacian, and ReLU-like source models.
5 Solving nonlinear ICA by TCL
TCL is applied to data from the nonlinear nonstationary ICA model, where its learned features span the source nonlinearities up to an invertible linear transformation. Linear ICA then resolves that transformation, with stronger identifiability for suitable source nonlinearities.
- Under the simplified single-nonlinearity model, TCL is applied to nonlinear ICA data and learns features related to the source functions q(si).The analysis assumes qi,0 = 0, V = 1, and a common modulated function q across sources.
- The theorem assumes nonlinear ICA data, feature dimension m = n, and a full-column-rank modulation matrix whose source modulations vary sufficiently independently.
- After TCL, the learned nonlinear features equal q(s) up to an invertible linear transformation and a constant bias.The proof identifies a shared linear subspace between the TCL features and the source nonlinearities.
- Applying linear ICA to the TCL hidden representation estimates the transformed source nonlinearities q(si).The result follows because the transformed quantities remain independent and non-Gaussian.
- For q(s) strictly monotonic in |s|, the original sources are identifiable up to strictly monotonic component-wise transformations; generally, q may be noninvertible.
- The framework can incorporate dimension reduction and multiple nonlinearities, but the latter requires expanded feature dimension and an independent-subspace treatment.
6 Simulation on artificial data
Artificial-data simulations test TCL on nonlinear mixtures generated from nonstationary Laplacian sources. TCL learns the data’s nonstationarity and reconstructs source nonlinearities in nonlinear mixtures, while more data helps and stronger mixing nonlinearity makes learning harder.
- The simulation generated n = 20 nonstationary Laplacian sources in segments of length 512, mixed through an invertible MLP.Source modulation parameters were randomly drawn from a uniform distribution in [0, 1].
- TCL used an MLP feature extractor with matching segmentation and layer count, followed by FastICA to estimate q(si).The feature dimension was set to m = n.
- TCL-trained features enabled MLR classification accuracy above chance, indicating that they captured data nonstationarity.Classification became harder with more mixing layers and more segments.
- TCL reasonably reconstructed q(si) for nonlinear mixtures, whereas the comparison methods failed; NSVICA performed well in the linear case.Performance increased with more segments and decreased as the number of mixing layers increased.
- The simulation supports TCL’s ability to estimate the nonlinear ICA model from nonstationarity, with more data improving performance and stronger nonlinearity reducing it.
7 Experiments on real brain imaging data
The study evaluates TCL on MEG by training features on unlabeled resting data and testing stimulation classification on labeled task data. TCL consistently outperformed baseline methods, with multilayer models performing best and yielding interpretable brain-activity patterns.
- TCL settings: The experiment used equal-sized 12.5-second segments and networks with L ∈ {1, 2, 3, 4} layers.A shared feature-MLP and separate session- and subject-specific classifiers were used to reduce learning of artefactual session differences.
- Evaluation methods: TCL features were trained on resting-session MEG data and evaluated by linear SVM classification of stimulation modalities in task sessions.The dataset comprised nine healthy volunteers measured with 306-channel MEG; evaluation used session-wise cross-validation accuracies.
- Results: TCL classification accuracies were consistently higher than those of the baseline methods across network depths.Models with L ≥3 outperformed the linear L = 1 case, indicating the importance of nonlinear demixing for this task.
- Spatial patterns: The learned three-layer features progressively pooled first-layer units into temporal and occipito-parietal patterns, producing bilateral temporal representations.Most third-layer patterns resembled networks previously reported with fMRI and MEG, while hidden units did not appear to represent artefacts.
8 Conclusion
The conclusion presents TCL as an easily implemented approach for learning features from temporal nonstationarity and estimating nonlinear independent components. Under suitable source nonstationarity, the method has strong identifiability guarantees and produces scientifically interesting MEG networks.
- Conclusion: TCL uses an MLP with logistic regression to discriminate time segments and estimate independent components in nonlinear mixtures.The method leaves linear mixing and component-wise nonlinearities as indeterminacies, with linear ICA resolving the remaining linear mixing.
- Practical implications: TCL avoids computing the gradient of the mixing-function Jacobian, a major difficulty of maximum-likelihood estimation.
- Identifiability: For modulated Gaussian sources, the remaining indeterminacies reduce to inevitable monotonic component-wise transformations.The authors describe this as their strongest identifiability result for nonlinear ICA.
- Applications: Experiments on real MEG data found neuroscientifically interesting networks, with video, econometric, EMG, and ECG data suggested as future application domains.
Proof of Theorem
The proof derives the optimal segment-discrimination relation from the nonlinear ICA density and converts it into a linear system involving the learned features. Full-column-rank assumptions then allow inversion of that system, while Jacobian terms cancel.
- Density derivation: The proof begins by computing the log probability density of a data point under the nonlinear ICA model using the probability transformation formula.The transformation uses the inverse mixing function g and its Jacobian.
- Discrimination relation: The optimal discrimination relation expresses the segment-dependent log density through the inverse mixing function and learned features.The relation is asserted for every segment under the stated assumptions.
- Cancellation: After substitution, the log-determinants of the Jacobians cancel when the two expressions are equated across segments.This cancellation removes the Jacobian term from the resulting relation.
- Linear system: Collecting all segment equations yields a linear system whose tall matrix has full column rank, so its pseudoinverse can recover the feature-related quantities.The proof uses L+L = I to multiply the system from the left.
- Identifiability relation: The learned features are related linearly to the source functions q(s), with a bias term and an invertible coefficient matrix.Invertibility follows because otherwise functions of distinct individual sources would become linearly dependent.