ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA

Ilyes Khemakhem; Ricardo Pio Monti; Diederik P. Kingma; Aapo Hyvärinen

Source-linked AI summary

ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA

Ilyes Khemakhem, Ricardo Pio Monti, Diederik P. Kingma, Aapo Hyvärinen

arXiv:2002.11537v4Feb 26, 2020stat.MLcs.LG

TL;DR

The paper asks when representations learned by flexible conditional energy-based models are identifiable despite unobserved latent variables. It develops ICE-BeeM and connects it to IMCA, then shows identifiable image representations and improved transfer and semi-supervised learning performance. The paper’s scope includes theoretical assumptions on feature extractors and the broader-impact boundary that guarantees abstract away from data and implementation choices.

Problem
Unsupervised learning needs identifiable nonlinear representations because latent variables are unobserved and multiple representation functions may fit the same data.
Method
ICE-BeeM uses a dot-product conditional energy between two neural feature extractors and establishes functional identifiability conditions, including a connection to IMCA.
Results
On real-world image datasets, ICE-BeeM learns representations that do not change arbitrarily across runs and improves transfer learning and semi-supervised learning applications.
Takeaways & Limitations
The results extend identifiability theory to conditional energy-based models and support their use for principled representation learning and generalized nonlinear ICA.
Takeaways & Limitations
The theoretical guarantees abstract away from the nature of the data and practical implementation, while IMCA results rely on stated component assumptions such as independence on subsets and dimensionality or non-monotonicity conditions.

Abstract

from arXiv · show

We consider the identifiability theory of probabilistic models and establish sufficient conditions under which the representations learned by a very broad family of conditional energy-based models are unique in function space, up to a simple transformation. In our model family, the energy function is the dot-product between two feature extractors, one for the dependent variable, and one for the conditioning variable. We show that under mild conditions, the features are unique up to scaling and permutation. Our results extend recent developments in nonlinear ICA, and in fact, they lead to an important generalization of ICA models. In particular, we show that our model can be used for the estimation of the components in the framework of Independently Modulated Component Analysis (IMCA), a new generalization of nonlinear ICA that relaxes the independence assumption. A thorough empirical study shows that representations learned by our model from real-world image datasets are identifiable, and improve performance in transfer learning and semi-supervised learning tasks.

1 Introduction

The paper addresses identifiability of nonlinear representations and extends identifiability theory to flexible conditional energy-based models. It also introduces IMCA, connects the model to generalized nonlinear ICA, and reports identifiable image representations that improve transfer and semi-supervised learning.

Motivation: Identifiability is necessary for learning latent representations that are meaningful, interpretable, and useful for downstream applications.Without identifiability, multiple representation functions can fit the observed data even with infinite data.
Model family: The paper extends nonlinear ICA insights to a broad family of unnormalized conditional energy-based models with universal approximation capabilities.The energy is constructed from feature extractors for the dependent and conditioning variables.
Contributions: ICE-BeeM provides mild functional conditions under which learned features are identifiable up to linear transformations, or up to scaling and permutation after modifying the energy.The conditions abstract away from network architecture and do not assume distributions for the learned representations.
Contributions: IMCA generalizes nonlinear ICA to non-independent latents with arbitrary global dependency structure while retaining estimation through ICE-BeeM.The framework relaxes the independence requirement by allowing components to be independently modulated by another variable.
Empirical findings: Experiments show that ICE-BeeM learns identifiable representations from real-world image datasets and supports transfer learning and semi-supervised learning applications.The empirical study evaluates identifiability and downstream usefulness rather than only theoretical guarantees.

2 Identifiable conditional energy-based deep models

ICE-BeeM defines conditional densities through a dot-product energy between neural feature extractors and studies when those features are identifiable. The theory establishes weak and strong identifiability under functional conditions, while also providing an architecture example and universal approximation guarantees.

Model definition: ICE-BeeM models conditional densities with an energy formed by the dot product of feature extractors for the dependent and conditioning variables.The conditional density is proportional to the exponential of the negative energy, subject to a finite normalizing constant.
Model properties: The model family is flexible because it supports multiple estimation methods, abstracts away from latent-feature distributions, and can use overcomplete representations.The paper discusses Flow Contrastive Estimation and Denoising Score Matching for conditional estimation.
Weak identifiability: Weak identifiability makes equivalent feature extractors differ only by linear transformations, with the result applying separately to both feature extractors.The theorem assumes differentiability and full-rank Jacobians together with an invertibility condition over selected conditioning points.
Weak identifiability: When the theorem’s assumptions hold for both extractors, the associated transformation matrices have full row rank equal to the feature dimension.This result also covers overcomplete nonlinear representations and requires few assumptions on the feature extractors.
Strong identifiability: Strong identifiability is obtained by restricting features to be non-negative or augmenting them with their squares, yielding scaled-permutation equivalence.Non-negativity rules out nontrivial rotations, while square augmentation avoids requiring positive feature extractors but doubles the effective dimension.
Architecture and approximation: The proposed MLP architecture can satisfy the identifiability assumptions, and increasing feature dimension gives universal approximation of positive conditional densities on compact supports.For every ε > 0, the model can approximate the target conditional density within ε under the stated compactness and positivity conditions.

3 Independently modulated component analysis

IMCA generalizes nonlinear ICA by allowing arbitrary dependencies among latent components, while ICE-BeeM provides conditions under which the recovered representations are identifiable up to structured transformations.

3 Independently modulated component analysis: The framework connects ICE-BeeM to a generative latent-variable model and uses the energy model to estimate non-independent latent components.Output nonlinearities serve as sufficient statistics and support the strong-identifiability construction.
3 Independently modulated component analysis: ICE-BeeM models conditional densities with sufficient statistics that factorize by latent component, while the base measure can encode global dependencies.The conditional exponential-family form separates component-wise modulation from dependence represented by µ(z).
3 Independently modulated component analysis: IMCA allows an arbitrary base measure, so latent components need not be independent while retaining strong identifiability guarantees.The nonlinear ICA special case uses a factorized base measure; IMCA removes that factorization requirement.
3 Independently modulated component analysis: Under functional assumptions on the mixing function, sufficient statistics, modulation points, and estimator, recovered features identify latent variables up to block permutation and invertible linear transformations.The result holds in the infinite-data limit and allows point-wise nonlinearities with dimensions matched to the sufficient statistics.

4 Relation to previous work on nonlinear ICA

The paper extends nonlinear ICA identifiability beyond conditional independence and distributional assumptions, while supporting overcomplete representations and links to practical neural architectures.

4 Relation to previous work on nonlinear ICA: The framework identifies conditional independence as unnecessary and extends identifiability theory to non-independent latent variables through IMCA.Its proofs generalize prior nonlinear ICA results to the non-independent case.
4 Relation to previous work on nonlinear ICA: Unlike earlier latent-variable results, the conditional EBM drops distributional assumptions on the learned representations.This broadens the model class beyond assumptions commonly used in latent-variable identifiability theory.
4 Relation to previous work on nonlinear ICA: Most theoretical results hold for overcomplete representations, giving the model universal approximation capabilities.This distinguishes the model from earlier cited approaches that did not provide the same overcomplete guarantee.
4 Relation to previous work on nonlinear ICA: The theory is stated through functional conditions and is complemented by neural-network architecture results intended to connect identifiability guarantees with training practice.The architecture results are described as a first step toward bridging theory and practice.

5 Experiments

Experiments show that ICE-BeeM learns more consistent representations on image datasets, supports transfer and semi-supervised learning, and outperforms competing nonlinear ICA methods on dependent-latent simulations.

5 Experiments: ICE-BeeM representations are more consistent across random initializations in both weak and strong identifiability evaluations.The experiments use MCC to compare conditional and unconditional EBMs on image datasets.
5 Experiments: Transfer learning fixes ICE-BeeM features learned on classes 0–7 and estimates coefficients for unseen classes 8–9.The method relies on weak identifiability to recover the learned basis up to a linear transformation.
5 Experiments: ICE-BeeM achieves significantly higher classification accuracy than an unconditional EBM when classifying unseen classes with logistic regression.Both models are trained on classes 0–7 and evaluated on classes 8–9 using learned features.
5 Experiments: For IMCA simulations, ICE-BeeM outperforms alternative nonlinear ICA methods, particularly at mixing-network depth L = 4.The alternatives implicitly assume conditional independence, whereas ICE-BeeM makes no distributional assumptions on the latent space.
5 Experiments: For nonlinear ICA simulations, ICE-BeeM performs competitively with iVAE and TCL, while increasing mixing-network depth reduces performance for all methods.Performance is evaluated with MCC between true and recovered latent variables.

6 Conclusion

ICE-BeeM provides identifiable conditional energy-based representations with universal approximation capabilities and a connection to nonlinear ICA and IMCA. Experiments on real-world image datasets show stable representations across runs and improved transfer and semi-supervised learning performance.

ICE-BeeM combines rigorous identifiability results with the flexibility and universal approximation capability of energy-based models.The model is presented as an identifiable conditional energy-based deep model for unsupervised representation learning.
ICE-BeeM estimates nonlinear ICA as a special case and extends it to non-independent components that are independently modulated by auxiliary variables.The generalized setting permits modulation by a time index, history, or noisy labels.
On real-world image datasets, learned representations remained stable across runs and improved transfer learning and semi-supervised learning performance.
The results extend identifiability theory to energy-based models, supporting principled disentanglement, interpretation, causal discovery, and transfer learning.

Broader Impact

The paper frames identifiability as important for reproducible and interpretable systems while noting that its theoretical guarantees abstract away from data and implementation. It also provides supporting theoretical and appendix analyses rather than decision-making functionality.

Identifiability is presented as key to reproducible science, interpretable results, reduced system arbitrariness, and greater operator control.
The system does not make decisions, so the paper states that it cannot exhibit bias in the decision-making sense.
The theoretical guarantees abstract away from the nature of the data and the practical implementation, leaving their scope independent of those details.
The appendices cover experimental details, estimation algorithms, identifiability and approximation proofs, IMCA estimation, and theoretical analysis of IMCA.

A.1 Model architecture details

The experiments use MLP, ConvMLP, and Unet-style feature-extractor configurations on MNIST, FashionMNIST, CIFAR10, and CIFAR100. Variants alter feature dimensionality and feature constraints, with training performed using conditional denoising score matching.

A.1 Model architecture details: The configurations are evaluated on MNIST, FashionMNIST, CIFAR10, and CIFAR100, with detailed architectures summarized in Tables 2 and 3.
A.1 Model architecture details: Experiments use three architecture families: fully connected MLPs, convolutional-plus-fully-connected ConvMLPs, and RefineNet-based Unets with skip connections.The architectures test identifiability from basic fully connected networks through modern deep image architectures.
A.1 Model architecture details: Feature variants reduce latent dimensionality, impose positivity, or use augmented features, producing names such as ConvMLP-50, ConvMLP-p, and ConvMLP-a.
A.1 Model architecture details: All experiments use Adam with learning rate 0.001, mini-batches of 63, and 5000 training iterations; CIFAR experiments additionally use random horizontal flips.
A.1 Model architecture details: The energy models are trained with conditional denoising score matching using noise parameter σ = 0.01.

A.2 The MCC metric

MCC measures representation agreement by maximizing average componentwise correlation over permutations. The appendix uses it to compare recovered latents with ground truth or representations from different random initializations, alongside experiments on multiple architectures and transfer learning.

A.2 The MCC metric: MCC computes maximum linear correlations between representation components after optimizing over permutations.This accounts for arbitrary component ordering when comparing two vectors.
A.2 The MCC metric: With known latent ground truth, a high MCC indicates that the recovered components match the true latents.
A.2 The MCC metric: With unknown ground truth, consistently high MCC across random initializations indicates that learned representations do not change drastically with the model’s random state.
A.3 Quality of representations: The identifiability experiments use Unet plots and additional ConvMLP and Unet configurations across MNIST, FashionMNIST, CIFAR10, and CIFAR100.Figures 2 and 3 provide further representation-quality plots for these configurations.
A.4.1 Intuition: The transfer-learning formulation represents each dataset’s unnormalized log-density as a linear combination of learned basis functions with dataset-specific scalar coefficients.This is interpreted as PCA in probability-distribution space, allowing models for different datasets to share information.
A.4.1 Intuition: Using sparsity in transfer learning requires strong identifiability without linear mixing, whereas estimating new-task coefficients alone requires only weak identifiability.

A.4.2 Further experiments

The further experiments evaluate transfer learning and related simulation setups across image datasets, with additional figures and tables documenting configurations and outcomes.

Transfer learning: 99 out of 100 returned MNIST digits are class 8 with transfer learning, compared with 58 in the baseline.Figure 4 reports this transfer-learning comparison for MNIST.
Transfer learning: ICE-BeeM assigns high confidence to unseen MNIST samples as the new class 8, whereas the no-transfer baseline assigns high confidence to other digits.The comparison indicates that transfer learning produces a reasonable distribution over unseen classes.
Semi-supervised learning: Semi-supervised experiments compare identifiable ICE-BeeM with an unconditional non-identifiable EBM using classification accuracy on unseen classes.Table 5 reports classification accuracy, with higher values preferred.
ICA and IMCA simulations: The simulations use five-dimensional synthetic datasets with segment-index conditioning and compare ICE-BeeM with TCL and iVAE under nonlinear ICA and IMCA settings.The generated data include conditionally independent and dependent latent-variable regimes.
Estimation: ICE-BeeM is trained with flow contrastive estimation in the simulations, while the broader conditional EBM framework also supports denoising score matching and other estimation methods.The paper emphasizes that the identifiability results apply to conditional EBMs generally.

C.2 Proof of Theorem 1

The proof establishes that equal conditional energy functions force the feature extractors to be related by a linear transformation under rank and richness assumptions, with invertibility yielding stronger equivalence.

Theorem 1: Under differentiability, rank, and point-richness assumptions, equality of the conditional energies implies fθ = A fθ′ + c for a matrix A and vector c.The proof evaluates the equality at multiple conditioning points, forms a matrix equation, and differentiates it.
Theorem 1: If the additional assumptions hold, evaluating the relation at sufficiently many x-points makes A invertible and completes the identifiability proof.The matrix Q is invertible, so A must also be invertible.
Assumptions: The conditioning extractor’s sufficiently rich image restricts the search space for the main extractor, enabling the identifiability argument.The richness condition can alternatively be expressed through a full-rank Jacobian at one point when the conditioning variable is continuous.
Weaker assumptions: The full-rank Jacobian assumption can be weakened to invertibility at one point without changing the theorem’s conclusion.The proof only uses the rank condition at the step establishing the rank of A.
Weaker assumptions: Removing rank restrictions from the equivalence definition is possible, but produces a relatively weak and potentially meaningless equivalence class.This is presented as a trade-off rather than as the preferred formulation.

C.3 Proof of Proposition 1

The proposition proves that suitably structured LeakyReLU MLPs have full-rank Jacobians almost everywhere, using full-rank layers and monotone layer dimensions.

Proposition 1: With LeakyReLU activations, full-rank weight matrices, and monotonically ordered layer widths, the MLP has a full-rank Jacobian almost everywhere.The proof combines full-rank activation matrices with a product-of-full-rank-matrices argument.
Jacobian construction: The piecewise-linear representation expresses each layer’s Jacobian through activation-dependent matrices, which remain full rank away from a measure-zero set.The exceptional set is contained in a finite union of measure-zero sets.
Surjectivity: When the output dimension does not exceed the input dimension, the MLP is also surjective under the proposition’s conditions.Surjectivity follows from surjectivity of the activation and layer maps.
Matrix argument: Products of full-rank matrices retain full rank when their dimensions are monotonically ordered.The proof uses a lower bound on the smallest singular value and induction over the matrix sequence.

C.4 Proof of Proposition 2

The proof constructs enough independent activation regions and points to establish the conditioning extractor’s richness, while positive-feature assumptions later force the identifying matrix to be a scaled permutation.

Linear and nonlinear extractors: In the overcomplete case, a linear conditioning extractor cannot satisfy the richness assumption, motivating deep nonlinear networks for identifiability.The limitation arises when the feature dimension exceeds the conditioning-variable dimension.
Proposition 2: Under full-rank and submatrix-invertibility assumptions, LeakyReLU features generate enough regions to construct an invertible matrix from dL + 1 conditioning points.The proof proceeds by induction over piecewise-affine sign regions.
Region construction: The induction maintains invertibility when a new affine hyperplane splits an existing region into two regions with different activation signs.Lemma 3 supplies the matrix-invertibility step used in the induction.
Feature independence: The resulting LeakyReLU functions and their generalized slopes are linearly independent under the stated conditions.This independence supports the construction required for the proposition.
Positive features: With nonnegative features and attained lower bounds, invertibility and nonnegativity force each identifying row to contain only one nonzero entry, yielding a scaled permutation.The conclusion is that features are related by permutation, scaling, and additive constants.

C.6 Proof of Theorem 3

The proof establishes universal approximation for conditional energy-based models by showing that neural-network feature products can approximate continuous functions on X × Y. Positivity and compactness then yield uniform approximation of any conditional density.

Theorem 3 guarantees uniform approximation of every strictly positive conditional density on compact Hausdorff spaces.For every ε > 0, suitable feature dimension and parameters satisfy supx,y |pθ(x|y) − p(x|y)| < ε.
The energy function is a finite sum of products f_i(x)g_i(y), placing it in the tensor-product function class FX ⊗ FY.The two feature extractors are parameterized as neural networks on the dependent and conditioning variables.
Approximation of log p(x|y), followed by the Lipschitz behavior of exp on compact sets, transfers function approximation to density approximation.For discrete conditioning variables, g(y) can instead be represented by a lookup table while f retains universal approximation capability.
Compact Hausdorff assumptions support bounded continuous functions and the Stone–Weierstrass argument used to establish density.The product space is compact Hausdorff, and the tensor-product algebra is unital and separates points.
Neural-network feature classes are dense in their respective continuous-function spaces, so FX ⊗ FY is dense in C(X × Y).The proof first approximates products of continuous functions and then applies Stone–Weierstrass to the product space.

D.3 Proofs

The proofs derive identifiability for ICE-BeeM and IMCA by converting conditional-density equalities into matrix and Jacobian constraints. Under stronger assumptions, the ambiguity narrows from invertible transformations to permutation or block-permutation structure.

After convergence, Theorem 4a recovers latent variables up to an invertible linear transformation and point-wise nonlinearities.The proof establishes invertibility by combining independent Jacobian columns with rank arguments.
IMCA identifiability: Differences across auxiliary-variable values cancel the non-factorial base-measure terms, enabling nonlinear-ICA identifiability arguments for IMCA.This cancellation permits dependent latent components while retaining independently modulated structure.
Under stronger assumptions, Theorem 4b reduces the ambiguity to component-level transformations indexed by a permutation and invertible square matrices.The resulting correspondence matches feature dimensions to sufficient-statistic dimensions.
Dimensionality reduction: With dimensionality reduction, the learned feature extractor recovers only modulated latent components and can identify them through a block-permutation transformation.The reduced representation uses an invertible square matrix on the retained components.

E.5.3 Proof of Theorem 9

The proof adapts a nonlinear logistic-regression construction to IMCA by contrasting observed data with data randomized through an independent auxiliary variable. With universal approximation and infinite data, the learned feature components recover latent components up to pointwise nonlinearities.

The method constructs randomized and non-randomized datasets, then trains multinomial logistic regression to distinguish their origins.The last hidden layer supplies the feature extractor used for representation learning.
Under Theorem 9’s assumptions, infinite-data logistic regression yields feature components equal to latent components up to pointwise nonlinearities.The result assumes a nonlinear logistic regressor with universal approximation capability and feature dimension d.
The proof uses the difference between class log-densities and auxiliary-variable differences to eliminate the base-measure term.The remaining argument parallels the identifiability proofs for Theorems 4a and 4b, with a full-rank rather than square matrix.