PaperlayerView exact revision on arXiv

Source-linked AI summary

InfoNCE Induces Gaussian Distribution

Roy Betser, Eyal Gofer, Meir Yossef Levi, Guy Gilboa

arXiv:2602.24012v2cs.LGeess.SP

TL;DR

InfoNCE representations have an incompletely characterized distribution, despite Gaussian modeling's relevance to analysis and applications. This paper uses two population-level analytical routes and experiments to show that normalized and unnormalized representations exhibit asymptotically Gaussian behavior, with consistent empirical support across datasets and architectures.

  • Problem

    The distribution of representations trained with InfoNCE remains unclear, limiting theoretical characterization despite the practical relevance of Gaussian modeling.

  • Method

    The paper analyzes InfoNCE through alignment-plateau and vanishing-regularization routes, supported by experiments across synthetic data, CIFAR-10, architectures, and pretrained models.

  • Results

    InfoNCE yields asymptotically Gaussian behavior for normalized and unnormalized representations, with empirical evidence showing consistent Gaussian statistics across multiple settings and encoder architectures.

  • Takeaways & Limitations

    The Gaussian view provides a principled basis for analyzing and modeling contrastive representations, including likelihood scoring and out-of-distribution detection.

  • Takeaways & Limitations

    The theoretical characterization assumes a standard Borel space with a nonatomic augmentation marginal distribution.

Abstract

from arXiv · show

Contrastive learning has become a cornerstone of modern representation learning, allowing training with massive unlabeled data for both task-specific and general (foundation) models. A prototypical loss in contrastive training is InfoNCE and its variants. In this work, we show that the InfoNCE objective induces Gaussian structure in representations that emerge from contrastive training. We establish this result in two complementary regimes. First, we show that under certain alignment and concentration assumptions, projections of the high-dimensional representation asymptotically approach a multivariate Gaussian distribution. Next, under less strict assumptions, we show that adding a small asymptotically vanishing regularization term that promotes low feature norm and high feature entropy leads to similar asymptotic results. We support our analysis with experiments on synthetic and CIFAR-10 datasets across multiple encoder architectures and sizes, demonstrating consistent Gaussian behavior. This perspective provides a principled explanation for commonly observed Gaussianity in contrastive representations. The resulting Gaussian model enables principled analytical treatment of learned representations and is expected to support a wide range of applications in contrastive learning.

1 INTRODUCTION

The paper explains how InfoNCE can induce asymptotically Gaussian representations through two complementary population-level analyses. It combines theoretical results with experiments across datasets and encoder architectures, while motivating Gaussian modeling for downstream applications.

  • 1 INTRODUCTION: InfoNCE balances positive-pair alignment against batch repulsion, encouraging uniformity by spreading representations across the hypersphere.The objective’s two pressures are alignment and uniformity.
  • 1 INTRODUCTION: Gaussian representations are motivated by links to downstream performance and by applications including classification, uncertainty estimation, and test-time adaptation.The passage connects more Gaussian representations with improved downstream performance and Gaussian modeling with practical tasks.
  • 1 INTRODUCTION: The analysis establishes two routes to asymptotically Gaussian representations: an empirical-idealization route and a regularized population-level route.The first uses an HGR maximal-correlation alignment bound and alignment plateau; the second adds a vanishing convex regularizer that prioritizes the isotropic solution.
  • 1 INTRODUCTION: Experiments on synthetic data and CIFAR-10 compare contrastive and supervised training using linear layers, nonlinear MLPs, and ResNet-18 encoders.The comparison isolates the role of the training objective beyond data or architecture effects.
  • 1 INTRODUCTION: In the large-batch limit, InfoNCE alignment is bounded by augmentation strength, while normalized representations converge toward the uniform distribution on the unit sphere.The stated framework gives both normalized and unnormalized representations asymptotically Gaussian behavior.

2 RELATED WORK

Related work connects InfoNCE to alignment and uniformity, studies explicit routes to isotropic or Gaussian-like representations, and develops geometric and theoretical perspectives on representation structure.

  • Contrastive learning and InfoNCE: InfoNCE is a standard self-supervised representation-learning objective underlying SimCLR, MoCo, and CLIP, balancing positive-pair alignment with batch-wise repulsion.Prior work examines these effects from geometric, optimization, and distribution-matching perspectives.
  • Isotropy and Gaussian structure: Prior approaches promote isotropic or Gaussian-like representations through whitening objectives, variance-covariance control, architectural design, and neural collapse.Related JEPA methods also produce highly regular representations and have been shown to encode density-related structure.
  • Hyperspherical geometry and Gaussianity: Classical high-dimensional geometry studies links between uniform measures on the sphere and Gaussian distributions, informing hyperspherical variational families and radial Bayesian priors.These methods leverage approximately uniform distributions over the hypersphere.
  • Additional theoretical perspectives: Complementary theoretical work analyzes contrastive representations through identifiability, characterizing when latent variables or semantic factors can be uniquely recovered under structural data-generating assumptions.These results concern conditional or component-level structure.

3 SETUP

The setup formalizes contrastive learning through augmented pairs, normalized encoder representations, and the finite- and infinite-batch InfoNCE objectives. It also introduces augmentation mildness via squared HGR maximal correlation and derives an alignment bound linking positive-pair similarity to augmentation dependence and representation mean.

  • Data and augmentations: Augmentation channel A generates two conditionally independent views X and Y from one base sample X0, defining the marginal pX and joint pair distribution pXY.The marginal pX is assumed nonatomic.
  • InfoNCE objective: InfoNCE maps inputs through an encoder f into ℓ2-normalized representations, rewarding positive-pair similarity while contrasting each anchor with all other views as negatives.The temperature is fixed at τ > 0, and cosine similarity is used because representations are unit-normalized.
  • Population InfoNCE: As N →∞, empirical InfoNCE converges up to an additive log N term to L(µ, π) = −α E(u,v)∼π[u·v] + Φ(µ), with α = 1/τ.The alignment term depends on positive pairs, whereas Φ(µ) is a uniformity potential depending only on the marginal representation law.
  • Augmentation mildness: The augmentation mildness parameter η² equals the squared HGR maximal correlation ρ_m²(X, X0), ranging from 0 for effectively independent views to 1 for views fully determined by the base.For the jointly Gaussian model, η² = A² when Pearson correlation is A.
  • Alignment bound: Positive-pair alignment satisfies E(u,v)∼π[u · v] ≤ η² + (1 − η²)∥m(µ)∥², linking similarity to augmentation dependence and the representation mean.Here m(µ) = E[u] = E[v], and u, v are normalized representations produced from conditionally independent augmentations.

4 GAUSSIANITY FROM INFONCE

The section explains that minimizing population InfoNCE induces asymptotically Gaussian low-dimensional projections through two complementary analyses. One uses alignment-plateau and thin-shell assumptions, while the other uses a vanishing regularizer promoting low norm and high entropy.

  • Overview: Two complementary routes derive Gaussian structure from population InfoNCE: an idealized alignment-and-concentration analysis and a regularized analysis requiring weaker training assumptions.The first route assumes infinite data, d →∞, sufficient optimization, alignment plateau, and thin-shell concentration; the second introduces a vanishing convex regularizer.
  • Normalized representations: For normalized representations on the sphere, every fixed k-dimensional projection of the global minimizer converges asymptotically to a multivariate Gaussian under the alignment-plateau condition.The proof combines uniformity of the global minimum with the central limit theorem on the sphere.
  • Unnormalized representations: For unnormalized outputs z = ru, the same Gaussian projection result follows when directions are uniform on the sphere and representation radii satisfy thin-shell concentration.The radius is the norm of the representation, while u is its normalized direction.
  • Regularized route: The regularized objective penalizes squared norms and encourages high entropy, with the optimal radial distribution matching a truncated Gaussian and the angular distribution approaching uniformity.For B = Rd, a Gaussian distribution approaches optimality, and the threshold β0 vanishes as d →∞.
  • Regularized route: When B = Rd, β ≥β0, and uniform alignment attains the augmentation ceiling, N(0, (2λ)−1Id) is an optimal choice for the unnormalized representation distribution.The theorem also states that the uniform angular law is asymptotically optimal under the regularized assumptions.

5 EXPERIMENTS

Experiments across synthetic data, CIFAR-10, and pretrained foundation-scale models test whether InfoNCE representations develop thin-shell norms and Gaussian low-dimensional projections. Across settings, the results consistently show norm concentration and coordinate-wise Gaussianity, with stronger effects under contrastive than supervised training.

  • Experimental design: The experiments test thin-shell norm concentration, Gaussian low-dimensional projections, and dependence on contrastive learning across three increasingly complex settings.These include synthetic linear encoders, CIFAR-10 with contrastive and supervised training, and pretrained foundation-scale models, using normalized and unnormalized representations.
  • Metrics: CV measures global radial concentration, while Anderson-Darling and D’Agostino-Pearson tests assess coordinate-wise Gaussianity.AD < 0.752 and DP p > 0.05 correspond to failure to reject normality; lower CV indicates stronger norm concentration.
  • Synthetic data experiments: Across Laplace, Gaussian-mixture, and discrete binary inputs, InfoNCE representations show low norm variation and strong coordinate-wise Gaussianity, indicating marginal Gaussian structure largely independent of input distribution.For Laplace inputs, CV decreases with increasing batch size and dimension, while normality diagnostics show perfect per-coordinate compliance.
  • CIFAR-10 experiments: On CIFAR-10, training progressively decreases CV and the AD statistic while monotonically increasing the fraction of coordinates passing the DP test.A two-layer MLP trained with InfoNCE therefore develops thin-shell concentration and coordinate-wise Gaussianity over optimization.
  • Contrastive vs. supervised training: With matched ResNet-18 architecture, initialization, and capacity on CIFAR-10, supervised training produces higher norm variability and stronger deviations from Gaussianity than contrastive training.The comparison changes only the objective: cross-entropy supervision versus InfoNCE.
  • Pretrained models: Pretrained self-supervised CLIP and DINO representations are compared with supervised ImageNet-pretrained ResNet34 and DenseNet representations using coordinate-wise normality diagnostics.The comparison is conducted on the MS-COCO validation set with normalized and unnormalized embeddings.

6 DISCUSSION AND CONCLUSION

The paper establishes an asymptotic Gaussian law for InfoNCE-trained representations through two complementary analytical routes. Experiments across synthetic, image, and pretrained-model settings support the assumptions and Gaussian hypothesis through norm concentration, alignment saturation, and near-Gaussian projections.

  • 6 DISCUSSION AND CONCLUSION: InfoNCE-trained representations admit an asymptotic Gaussian law through two complementary routes.The routes are an alignment-plateau analysis with thin-shell concentration and a regularized surrogate under milder assumptions.
  • 6 DISCUSSION AND CONCLUSION: Experiments on synthetic data, CIFAR-10, MS-COCO, and ImageNet-R are consistent with the analytical assumptions and Gaussian hypothesis.The pretrained-model evidence comes from models evaluated on MS-COCO and ImageNet-R.
  • 6 DISCUSSION AND CONCLUSION: The experiments reveal norm concentration, alignment saturation, and near-Gaussian projections in the learned representations.These empirical patterns support the proposed Gaussian structure across the reported experimental settings.

ETHICS STATEMENT

The work is theoretical and empirical, studying the statistical behavior of contrastively trained representations without foreseen direct negative societal impacts. However, Gaussian-modeling applications in safety-critical domains require attention to robustness and fairness.

  • ETHICS STATEMENT: The study examines the statistical behavior of representations trained with contrastive learning through theoretical and empirical analysis.The authors do not foresee direct negative societal impacts from the work itself.
  • ETHICS STATEMENT: Downstream uses of Gaussian modeling, including density estimation and OOD detection, may influence safety-critical decisions and therefore require robustness and fairness.The ethical caution concerns potential applications rather than direct impacts of the study.

LLM USAGE

A large language model assisted with several manuscript tasks, while the authors implemented, validated, and independently verified the paper’s results, experiments, mathematical claims, and citations.

  • LLM USAGE: The authors used a large language model for editing, reference search, ideation, derivations, summarization, exposition refinement, formulation suggestions, and notation checks.The model was used interactively, and no content was included without author review.
  • LLM USAGE: All results, proofs, experiments, mathematical claims, experimental details, and citations were implemented, validated, or independently verified by the authors.The passage states that the authors retained responsibility for the manuscript’s substantive content.

OVERVIEW … B REGULARIZED SURROGATE PROOFS

The appendix supplies proofs, supplementary derivations, experiments, and implementation details, including HGR foundations, a Gaussian example, and the alignment-bound proof. It also identifies regularization-surrogate proofs and related alignment-plateau analyses as dedicated appendix sections.

  • OVERVIEW: The appendices contain complete proofs, additional derivations, supplementary experiments, and implementation details.The overview states that these materials supplement the main text.
  • OVERVIEW: The overview assigns Section B to proofs of regularization-surrogate claims and Section C to the alignment-plateau approach.Some general Section C claims also support the regularization-surrogate proof.
  • A.1 HGR DEFINITION AND BASIC PROPERTIES: HGR’s explained-variance characterization measures the fraction of g(X)’s variance predictable from Y, yielding a generalized coefficient of determination in [0, 1].The numerator is explained variance from the optimal predictor E[g(X) | Y], while the denominator is total variance.
  • A.1 HGR DEFINITION AND BASIC PROPERTIES: HGR obeys a multiplicative data-processing inequality: for X − Y − Z, ρm(X, Z) ≤ ρm(X, Y )ρm(Y, Z).The normalized representations lie on Sd−1, so they are bounded and each coordinate belongs to L2.
  • A.2 GAUSSIAN EXAMPLE: For jointly Gaussian variables, HGR maximal correlation equals the absolute Pearson correlation, and nonlinear transformations cannot improve on linear correlation.The supremum is attained by standardized versions of the variables.
  • A.3 PROOF OF THE ALIGNMENT BOUND: The alignment-bound proof decomposes normalized representations into their common mean m(µ) and zero-mean residuals ũ and ṽ.The residual definitions are ũ := u − m(µ) and ṽ := v − m(µ).
  • A.3 PROOF OF THE ALIGNMENT BOUND: The proof bounds residual coordinate correlations using HGR and Cauchy–Schwarz, exploiting the Markov structure X ← X0 → Y and conditional independence.The augmentation views are identically drawn from the same channel A(· | X0).
  • A.3 PROOF OF THE ALIGNMENT BOUND: The alignment proof then computes marginal variances from unit norms and combines the resulting inequalities to conclude the bound.The final steps invoke the coordinate-summed inequality, Cauchy–Schwarz for sequences, and equations (36)–(38).

B.1 PROOF OF PROPOSITION 3 · B.2 PROOF OF LEMMA 1 · C ALIGNMENT-PLATEAU PROOFS

The supplied proof excerpts establish Proposition 3 through KL chain-rule and polar-coordinate arguments, and Lemma 1 through variational and spherical-concentration bounds. No passage text is provided for the listed C alignment-plateau proofs.

  • B.1 PROOF OF PROPOSITION 3: The Proposition 3 proof invokes the KL chain rule for an encoder with angular law µ.The argument cites Dupuis & Ellis (2011), Theorem B.2.1.
  • B.1 PROOF OF PROPOSITION 3: At fixed angular law µ, polar decomposition shows the KL term is minimized when κ(· | u) = ξ(· | u) µ-a.s.The decomposition is λ(dz) = σ(du)ξ(dr | u) in polar coordinates z = ru.
  • B.2 PROOF OF LEMMA 1: The Lemma 1 proof reduces to µ ≪ σ and m(µ) ≠ 0, since the alternative cases are trivial.It states that KL(µ∥σ) = +∞ when µ is not absolutely continuous with respect to σ, and separately handles m(µ) = 0.
  • B.2 PROOF OF LEMMA 1: Using the Donsker–Varadhan variational formula, the proof tests bounded measurable functions φ(u) = tw · u for unit w and real t.The variational formula is cited to Dupuis & Ellis (2011), Lemma 1.4.3.
  • B.2 PROOF OF LEMMA 1: A bound of the form ≤ t2/a for every t and w yields the desired conclusion after choosing t = a.This is the role assigned to Eq. (43) in the proof.
  • B.2 PROOF OF LEMMA 1: The proof establishes Eq. (43) with a = 4C(d −1) using concentration for the 1-Lipschitz spherical function g(u) = w · u.It applies a corollary of Lévy’s isoperimetric inequality and uses that g has median 0.
  • B.2 PROOF OF LEMMA 1: The spherical concentration argument combines Eg = 0 with a universal constant C′ and concludes by taking C = 1/(8C′2).The supplied excerpt states that this choice completes the proof.

C.1 NORMALIZED REPRESENTATIONS … E.1 IMPLEMENTATION DETAILS

At the alignment plateau, normalized InfoNCE representations are governed by a uniformity potential uniquely minimized by the uniform spherical law, whose fixed-dimensional projections become Gaussian. The analysis extends to unnormalized representations and characterizes nonzero-mean stationary points as von Mises–Fisher tilts, while experiments specify synthetic, CIFAR-10, and pretrained-model settings and diagnostics.

  • C.1 NORMALIZED REPRESENTATIONS: At the alignment plateau, minimizing the population InfoNCE objective is equivalent to minimizing the uniformity potential Φ(µ), uniquely minimized by the uniform law σ on S^{d−1}.The alignment term is constant on the plateau, so uniformity is the remaining objective; feasibility of the plateau value at µ = σ must be verified.
  • C.1 NORMALIZED REPRESENTATIONS: Fixed-dimensional coordinate blocks and orthonormal projections of a uniform point on S^{d−1} converge to a standard multivariate Gaussian as d →∞.The limiting distribution is Z ∼N(0, I_k), with convergence quantified by total variation distance.
  • C.2 UNNORMALIZED REPRESENTATIONS: For unnormalized representations z = r u, global plateau minimizers have uniform angular components and thin-shell-concentrated radii, yielding Gaussian fixed-dimensional projections.The proof writes z in polar form, applies the normalized result to u, and uses Assumption 2 for radius concentration.
  • D EXACT ALIGNMENT BOUND IN PLATEAU DISCUSSION: Under a generalized plateau assumption, the population loss reduces to a trade-off between the uniformity potential and the squared mean-vector length.Expected alignment is determined by augmentation mildness η2 and ∥m(µ)∥2 up to the residual rplat.
  • D EXACT ALIGNMENT BOUND IN PLATEAU DISCUSSION: In high dimensions, stationary points are approximated by the uniform distribution when m(µ) = 0 or a von Mises–Fisher tilt aligned with an axis w when m(µ)̸ = 0.The tilt concentration parameter κ measures angular concentration, with larger κ corresponding to a narrower cone; nearly uniform representations and occasional vMF-like bias are discussed for contrastive encoders.
  • E.1 IMPLEMENTATION DETAILS: Synthetic experiments use 1024-dimensional Laplace vectors with 20k training and 5k testing samples, representation dimensions d ∈ {32, 64, 128, 256}, and InfoNCE temperatures τ ∈{0.1, 0.2}.The reported default is τ = 0.1; augmentations use A = 0.6, Adam with learning rate = 10−3, and typically 150 epochs.
  • E.1 IMPLEMENTATION DETAILS: Additional synthetic settings include a 25-component Gaussian mixture and sparse binary vectors, using linear encoders that map inputs to 256-dimensional representations and train with InfoNCE for 100 epochs.The Gaussian mixture has 10k samples in 1024 dimensions; binary augmentations flip 0.1% of zero entries to ones.

E.2 ADDITIONAL EXPERIMENTS

Additional experiments show that pretrained representations exhibit thin-shell concentration and that contrastive models concentrate more strongly than supervised models. They also confirm improved uniformity with larger batch size, dimensionality, and whitening, while synthetic experiments demonstrate stable pushforward from Laplace inputs to Gaussian representations.

  • Thin-shell concentration in pretrained models: All evaluated pretrained models exhibit thin-shell concentration, with contrastive models showing lower coefficient of variation than supervised models.The models include supervised DenseNet and ResNet34, plus self-supervised CLIP image/text and DINO on MS-COCO.
  • Alignment and uniformity: As batch size or dimensionality increases, negative-pair cosine similarities concentrate near zero while positive-pair alignment remains high.At very low dimensionality or very small batch sizes, the corresponding uniformity gains are limited.
  • Whitening and uniformity: Whitening consistently improves uniformity across CLIP image, CLIP text, and DINO representations, including normalized and unnormalized features.Normalized representations are already close to uniform, so whitening yields a modest but consistent improvement; unnormalized features benefit more visibly.
  • Encoder pushforward: Laplace-distributed synthetic inputs are mapped to approximately Gaussian representations, with likelihood scores showing strong correlation across augmentation strengths.The experiment compares tractable likelihoods for input and output sets to assess the encoder’s pushforward behavior.
Loading 2602.24012v2…