PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Learning Inverts the Data Generating Process

Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, Wieland Brendel

arXiv:2102.08850v4cs.LGcs.CV

TL;DR

The paper addresses why contrastive learning produces representations that generalize across downstream tasks. It proves that InfoNCE-family objectives can implicitly invert the data-generating process, connects this behavior to nonlinear ICA and generative modeling, and finds empirically that factor recovery persists under severe assumption violations.

  • Problem

    It is largely unclear why contrastive learning learns representations that generalize effectively to many downstream tasks.

  • Method

    The paper analyzes feedforward models trained with InfoNCE-family objectives under assumptions about latent conditional and marginal distributions, and derives alternative objectives when those assumptions are violated.

  • Results

    InfoNCE-family objectives can recover the true generative factors up to rotations, while empirical results show factor identification even when theoretical assumptions are severely violated.

  • Takeaways & Limitations

    The results suggest that contrastive representations implicitly and approximately invert the data-generating process, helping explain their usefulness across downstream tasks and motivating more effective contrastive losses.

  • Takeaways & Limitations

    The theory assumes statistical properties of the latent variables and requires the objective's model distribution to match the ground-truth distribution's metric form.

Abstract

from arXiv · show

Contrastive learning has recently seen tremendous success in self-supervised learning. So far, however, it is largely unclear why the learned representations generalize so effectively to a large variety of downstream tasks. We here prove that feedforward models trained with objectives belonging to the commonly used InfoNCE family learn to implicitly invert the underlying generative model of the observed data. While the proofs make certain statistical assumptions about the generative model, we observe empirically that our findings hold even if these assumptions are severely violated. Our theory highlights a fundamental connection between contrastive learning, generative modeling, and nonlinear independent component analysis, thereby furthering our understanding of the learned representations as well as providing a theoretical foundation to derive more effective contrastive losses.

1. Introduction

The paper addresses conflicting explanations for contrastive learning by showing that InfoNCE objectives can recover generative factors through implicit inversion of the data-generating process. Theory and experiments connect contrastive learning with nonlinear ICA and show factor recovery even when assumptions are partially violated.

  • Motivation: Contrastive learning has achieved strong results, but existing theoretical predictions and empirical observations about its representations appear inconsistent.The paper motivates a new analysis of why contrastive representations generalize to downstream tasks.
  • Theory: InfoNCE objectives recover true generative factors up to rotations when the data-generating process satisfies weak statistical assumptions.The result establishes a theoretical connection between contrastive learning and nonlinear ICA.
  • Interpretation: The theory interprets contrastive learning as implicitly and approximately inverting the data-generating process, helping explain the usefulness of learned representations across downstream tasks.The paper presents this as a suggested explanation supported by theoretical and empirical results.
  • Experiments: Controlled experiments find successful inversion even when the theoretical assumptions on the ground-truth generative model are partially violated.The empirical verification tests both settings where the theoretical conditions hold and settings with partial violations.
  • Experiments: 3DIdent extends the CLEVR rendering pipeline into a more visually complex disentanglement benchmark, where a theory-derived contrastive loss identifies ground-truth factors in high-resolution images.The benchmark includes shadows, varied lighting, and a 3D object.

2. Related Work

Related work explains contrastive learning through mutual information, latent classes, alignment and uniformity, and nonlinear ICA, but reports tensions between theory and practice. This paper instead links the learned latent representation directly to the ground-truth generative factors and characterizes when inversion occurs.

  • Contrastive Learning: InfoMax-based explanations motivate contrastive learning through maximizing mutual information, but tighter mutual-information bounds can produce worse representations.This leaves the accuracy of the InfoMax motivation for contrastive learning unclear.
  • Contrastive Learning: Latent-class theory predicts that too many negative samples reduce performance, a prediction that initially conflicted with empirical results.Later work provided some empirical evidence involving negative-sample sampling strategies.
  • Contrastive Learning: Alignment and uniformity properties correlate with downstream performance and motivate a connection to cross-entropy minimization yielding identifiability results.The paper builds on this perspective rather than treating it as a complete account of factor recovery.
  • Nonlinear ICA: Nonlinear ICA seeks the underlying sources of observed data x = g(z) by finding a demixing model f equal to the inverse generative model g^-1.The nonlinear generative model is assumed to be invertible, or injective.
  • Nonlinear ICA: Prior nonlinear ICA results use auxiliary variables and practical estimation methods related to noise contrastive estimation, with extensions to VAEs and invertible energy-based models.These works establish broader context for connecting contrastive objectives with source recovery.
  • Contribution: Unlike distribution-to-parameter matching, this paper associates the learned latent representation with ground-truth generative factors and specifies conditions under which the data-generating process is inverted.This is the paper's stated distinction from closely related theoretical work.

3. Theory

The theory shows that InfoNCE-trained encoders can invert an injective data-generating process and recover latent factors under distributional and geometric assumptions. Extensions to convex latent bodies and alternative metrics yield affine or generalized-permutation recovery, while empirical evidence suggests robustness to model mismatch.

  • Proof strategy: The proof interprets the contrastive loss as cross-entropy between the ground-truth conditional latent distribution and a model distribution induced by the encoder.It then establishes distance preservation and uses this isometry to derive inversion up to orthogonal transformations.
  • Proof strategy: Distance preservation implies that the encoder-composed generative map is an orthogonal linear transformation, scaled according to the representation-space radius.The minimizer preserves latent dot products, which yields the stated linear form.
  • Hyperspherical latent spaces: InfoNCE minimizers recover latent sources up to orthogonal linear transformations when the latent space is a hypersphere and the assumed distributions match.The result assumes a uniform latent marginal, a von Mises–Fisher conditional, an injective generative process, differentiability, and infinitely many negative samples.
  • Scope and robustness: The analysis does not require knowledge of the generative model, and empirical evidence indicates approximate affine recovery despite severe mismatch between assumed and true conditional distributions.The theoretical model distribution is assumed to match the ground truth for the formal guarantees, but experiments examine violations of this condition.
  • Convex latent bodies: For convex latent bodies, adjusted contrastive objectives recover latent sources up to invertible affine transformations when the model and ground-truth conditional distributions use the same metric.This extends the hyperspherical result beyond the unit sphere.
  • Alternative metrics: With Lp-based similarities for p ≥ 1 and p ≠ 2, recovery is restricted to generalized permutations, including coordinate permutations, sign flips, and rescaling.The objective must match the ground-truth conditional distribution's metric, up to constant scaling.

4. Experiments

Experiments test identifiability under matched and violated generative assumptions, evaluate naturalistic benchmarks, and examine how conditional or marginal mismatches affect contrastive learning.

  • Validation of theoretical claim: The experiments compare identity, supervised, and unsupervised encoders using R2 for affine identifiability and MCC for generalized-permutation identifiability.The unsupervised model is trained with a contrastive loss.
  • Validation of theoretical claim: Contrastive learning recovers factors close to the empirical upper bound when the assumed and ground-truth distributions match.Mismatches reduce permutation identifiability more than affine identifiability.
  • Validation of theoretical claim: Identifiability remains robust to many mismatches in marginal and conditional distributions, but can collapse when the marginal is more concentrated than positive-pair conditionals.In that regime, positive pairs become indistinguishable from negative pairs.
  • KITTI Masks: On KITTI Masks, Laplace conditionals identify underlying factors better than normal conditionals because nearby-frame latent transitions are sparse.SlowVAE degrades when positive frames are extremely similar at ∆t = 0.05s.
  • 3DIdent: On 3DIdent, contrastive learning reaches scores close to supervised topline performance, while restrictive encoders and standard image augmentations cause performance drops.The authors attribute the augmentation drop to greater mismatch between assumed and induced positive-pair conditionals.

5. Conclusion

InfoNCE objectives can uncover underlying generative factors despite severe violations of their theoretical assumptions, suggesting that contrastive representations approximately invert the data-generating process. The framework also motivates improved augmentations and contrastive losses.

  • InfoNCE objectives can uncover the true generative factors underlying observational data despite severely violated theoretical assumptions.
  • Contrastive representations implicitly and approximately invert the data-generating process, potentially explaining their usefulness across downstream tasks.
  • The framework may clarify how augmentations affect learned representations and support improved augmentation schemes.
  • The theory opens avenues for contrastive losses informed by latent-space characteristics and for extending the framework with richer marginal distributions or visual inductive biases.The paper identifies non-uniform marginals, compositionality, hierarchy, and objectness as directions for future work.
  • Together, the results provide a theoretical foundation for understanding and extending state-of-the-art self-supervised learning techniques.

A.1.1. ASSUMPTIONS

The theoretical setup models latent variables on a unit hypersphere with a uniform marginal and vMF positive-pair conditionals, observed through an injective differentiable generator. The contrastive model maps observations through an encoder into a conditional distribution with a scale and normalization term.

  • The generator maps latent variables from a unit hypersphere into observations, is injective on that latent space, and is differentiable nearby.The latent dimension is at most the observation dimension.
  • The latent marginal distribution is assumed uniform, while positive pairs follow a von Mises-Fisher conditional distribution.
  • Training observations are samples from the positive-pair and marginal distributions transformed by the generator.
  • The model associates a conditional distribution with the encoder composed with the generator, using a hyperspherical representation.
  • The model conditional includes a partition function and a positive scale parameter τ.

A.1.2. PROOFS FOR SEC. 3

The proof sequence reduces asymptotic InfoNCE optimization to cross-entropy matching, then shows that matching preserves latent dot products and forces the encoder-generator composition to be a scaled orthogonal transformation.

  • As the number of negative samples grows, normalized contrastive loss converges to cross-entropy between the ground-truth and model conditional latent distributions.The reduction assumes a uniform ground-truth marginal and fixed τ > 0.
  • Cross-entropy minimization matches the model conditional to the ground-truth conditional when the model specification is correct.
  • For vMF conditionals, a minimizer preserves latent dot products through h = f ◦ g, with the representation radius set by √(τκ).
  • A differentiable map between hyperspheres that preserves dot products up to a constant factor must be an orthogonal linear transformation with corresponding scaling.
  • Under the stated assumptions, minimizing contrastive loss recovers latent sources up to an orthogonal linear transformation and a constant scaling factor.The proof combines asymptotic loss reduction, conditional matching, dot-product preservation, and the hypersphere result.

A.2. Extension of theory to subspaces of RN

The theory extends beyond the unit hypersphere to latent spaces that are convex bodies in R^N, where mild distributional assumptions imply linear cross-entropy minimizers.

  • For Z ⊆ R^N that is a convex body, mild assumptions imply that all cross-entropy minimizers between the ground-truth and model conditionals are linear functions.A hyperrectangle is given as an example of a convex body.

A.2.1. ASSUMPTIONS

The analysis assumes a convex latent space with a uniform marginal and exponentially distributed positive pairs, while allowing a semi-metric related to a metric by an invertible transformation. The model uses a conditional distribution with the same distance-based form and assumes access to samples from both data distributions.

  • The latent space Z is a convex body, the mixing function is injective, and the latent marginal is uniform.
  • Positive pairs follow an exponential-family conditional distribution controlled by λ and a semi-metric δ.The semi-metric may be expressed as a continuously invertible transformation of a metric.
  • Training assumes access to samples from both the marginal and positive-pair conditional distributions.
  • Unlike the hyperspherical setting, the two members of a positive pair need not have identical marginal distributions.The notation p(·) is nevertheless used for the same marginal density in expectations.
  • The model f maps observations into another convex subset of R^N and induces a conditional distribution qh using the same distance structure.Its normalization is represented by the partition function Cq(z).

A.2.2. MINIMIZING THE CROSS-ENTROPY

As the number of negative samples grows, the contrastive objective converges to a form that decomposes into alignment and uniformity terms. The proof rewrites the model cross-entropy using the conditional density, its normalization, and the uniform latent marginal.

  • As M →∞, the adjusted contrastive loss converges to a limiting expression for fixed temperature τ.This establishes the large-negative-sample regime used in the subsequent analysis.
  • The limiting loss decomposes into alignment and uniformity terms, Lδ-align(f; τ) + Lδ-uni(f; τ).
  • The modified objective is introduced as a practical route to minimizing cross-entropy between the ground-truth conditional distribution p and the model conditional qh.
  • The derivation substitutes the conditional model qh and separates the partition-function contribution from the expectation over positive pairs.The normalization C_h(z) is independent of the paired latent and can be moved outside the inner expectation.
  • Using the uniform marginal p(z) = |Z|^-1 simplifies the normalization term in the limiting cross-entropy.

A.2.3. CROSS-ENTROPY MINIMIZERS ARE ISOMETRIES

Cross-entropy minimizers preserve latent distances and therefore become affine transformations under the stated assumptions. With additional Lα-metric structure, the ambiguity narrows to permutations, sign flips, and rescalings, while regular density matching ensures bijectivity.

  • A.2.3. CROSS-ENTROPY MINIMIZERS ARE ISOMETRIES: Cross-entropy minimizers are isometries: they preserve latent distances up to the λ/τ scaling factor.When λ = τ, the latent and encoded distances are equal directly.
  • A.2.3. CROSS-ENTROPY MINIMIZERS ARE ISOMETRIES: For convex bodies and differentiable invertible mixing, the composed map h = f ◦g is affine, recovering latent sources up to affine transformations.This result assumes the model and ground-truth conditional distributions match in form.
  • A.2.3. CROSS-ENTROPY MINIMIZERS ARE ISOMETRIES: The theory currently leaves extension beyond uniform latent marginals as future work.
  • A.2.3. CROSS-ENTROPY MINIMIZERS ARE ISOMETRIES: For Lα metrics with α ≥ 1 and α ≠ 2, the remaining ambiguity is a composition of input-independent permutations, sign flips, and rescalings.
  • A.2.3. CROSS-ENTROPY MINIMIZERS ARE ISOMETRIES: The uniformity term is sufficient for bijectivity under its assumptions, but violations such as nondifferentiable ReLU points can require the full contrastive loss.
  • A.2.3. CROSS-ENTROPY MINIMIZERS ARE ISOMETRIES: A regular encoded density with the same support as the latent density makes h bijective and prevents information loss.The proposition establishes this for differentiable maps between simply connected, oriented C1 manifolds without boundaries.
Loading 2102.08850v4…