PaperlayerView exact revision on arXiv

Source-linked AI summary

Internal Discrepancy for Contrastive Learning: Generalization Bounds and Downstream Behaviors

Jialiang Wang, Qing Ma, Xiong Zhou, Gangfeng Hu, Haoliang Li, Xianming Liu

arXiv:2610.32729v1cs.LGcs.AI

TL;DR

Contrastive learning's success on downstream tasks remains theoretically incompletely understood, particularly when data assumptions and function-class biases are considered. This paper introduces internal discrepancy, proves a downstream generalization bound, and studies supervised losses incurred by contrastive learning. It reports results on class imbalance and optimality for InfoNCE- and SpeCL-incurred losses.

  • Problem

    How contrastive learning yields meaningful downstream representations remains incompletely understood, and prior work uses data assumptions or function-class inductive biases to study the issue.

    Source: §1 · p. 2

  • Method

    The paper defines internal discrepancy, proves an assumption-free downstream generalization bound using it, and analyzes downstream behaviors through contrastive learning-incurred supervised losses.

    Source: §1 · p. 3

  • Results

    Under class imbalance, InfoNCE-incurred loss approximates balanced softmax and SpeCL-incurred cosine similarities remain consistent across class distributions; the losses also exhibit distinct optimality properties.

    Source: §1 · pp. 3–4

  • Takeaways & Limitations

    The bound unifies connections among assumptions about representations and motivates relaxed k-means loss as an approximation for reducing internal discrepancy.

    Source: §1 · p. 3

    The setup assumes deterministic labels and at least two positive-probability classes, while the optimality result additionally requires a class-constant simplex ETF and bounded positive-pair label mismatch.

    Source: §2 · p. 4, §4.2 · p. 17

Abstract

from arXiv · show

Contrastive learning has emerged as a popular paradigm for training representation models. While prior studies pursue more practical and less restrictive assumptions to enhance theoretical understanding of contrastive learning, they essentially describe the interconnections among representations within the data or induced by the function class. To encapsulate these assumptions, we propose the measure of internal discrepancy that quantifies the difference between the mean representation of positive samples and latent cluster centers. We then prove an internal discrepancy-aware generalization bound for downstream supervised tasks following contrastive learning, without requiring additional assumptions. This generalization bound not only serves as a unifying framework connecting various assumptions but also motivates us to further reduce the internal discrepancy, thereby introducing the relaxed k-means loss as an approximation. Moreover, we investigate the downstream behaviors of contrastive learning from the perspective of contrastive learning-incurred supervised losses. In the presence of class imbalance, we shed light on why contrastive learning can facilitate imbalanced learning. Finally, we investigate the optimality of these incurred supervised losses, proving several intriguing properties.

1 Introduction

The paper addresses limited theoretical understanding of how contrastive learning produces representations useful for downstream tasks, including under class imbalance. It introduces internal discrepancy and incurred supervised losses to study downstream generalization and behavior.

1 Introduction

  • Contrastive learning has succeeded empirically, but its ability to learn meaningful downstream representations remains incompletely understood, especially under class imbalance.The paper notes that relying only on contrastive losses does not establish relevant downstream associations without further assumptions about data or inductive biases.Source: §1 · p. 2
  • Prior work studies data assumptions and function-class inductive biases, yet distinct assumptions may describe shared relationships among representations and latent clusters.The paper asks whether these assumptions can be connected and whether downstream generalization can be bounded without additional assumptions.Source: §1 · p. 2
  • Internal discrepancy measures the difference between mean positive-sample representations and latent cluster centers, and supports a downstream generalization bound without additional assumptions.The bound relates supervised loss to contrastive loss, internal discrepancy, and generalization error.Source: §1 · p. 3
  • The bound motivates relaxed k-means loss, an approximation intended to reduce internal discrepancy by reducing representation distances within a truncated range.The paper also associates the measure with various data assumptions.Source: §1 · p. 3
  • Under class imbalance, the InfoNCE-incurred loss approximates balanced softmax, while SpeCL-incurred representations retain cosine similarities regardless of class distribution.The paper presents these results as helping explain contrastive learning's ability to facilitate imbalanced learning.Source: §1 · p. 3
  • The incurred losses have optimality properties: InfoNCE yields variability collapse and convergence to a simplex ETF, while SpeCL yields optimal features approximating the identity matrix.These properties are proved for the respective contrastive learning-incurred supervised losses.Source: §1 · pp. 3–4

2 Preliminaries

The preliminaries establish the data, label, and representation-function notation, then define the contrastive and downstream supervised-learning setup. They specify linear probing with a mean classifier as the supervised-loss surrogate of interest.

2 Preliminaries

  • The setup assumes at least two latent classes with positive class probabilities, deterministic labels, and norm-bounded representation functions.The paper denotes the sample distribution by D, the latent class space by T, and its class distribution by ρ_T.Source: §2 · p. 4
  • For linear probing, the paper fixes the representation and minimizes supervised loss over linear classifiers, then uses a mean classifier built from class-conditional representation means.The resulting supervised loss is defined as an infimum over the classifier matrix; the mean classifier is used for the surrogate bound.Source: §2 · p. 4
  • Contrastive learning brings positive-pair representations closer and pushes negative-sample representations apart, with positive-pair sampling depending on whether learning is supervised or unsupervised.The preliminaries introduce InfoNCE and spectral contrastive loss; the latter is described as a theoretical surrogate for InfoNCE.Source: §2 · pp. 4–5
  • The analysis focuses on downstream supervised loss after contrastive training, distinguishing the population risk minimizer from the empirical contrastive-loss minimizer.For both InfoNCE and spectral contrastive loss, the preliminaries state convexity and Lipschitz continuity under bounded-norm constraints.Source: §2 · p. 5

3 Internal Discrepancy for Contrastive Learning

The paper introduces internal discrepancy to connect prior assumptions about representations and latent classes, then uses it to bound downstream supervised loss and motivate a relaxed k-means loss. It also relates contrastive-learning-incurred supervised losses to downstream classification behavior.

3.1 Internal Discrepancy

  • Internal discrepancy measures the gap between the mean representation of positive samples and the latent class mean, linking assumptions that constrain representations.It is governed by positive-pair alignment and intra-class discrepancy; prior assumptions can constrain the measure or its upper-bound components.Source: §3.1 · pp. 6, 8

3.2 Internal Discrepancy-Aware Generalization Bound

  • InfoNCE-incurred supervised loss can perform comparably to cross entropy, and can be on par with or surpass it with suitable k and t.The reported comparisons include validation results on CIFAR-10 and CIFAR-100.Source: §3.2 · p. 9, Table 1 · p. 10
  • The downstream supervised loss is bounded by contrastive loss, internal discrepancy, and a generalization term, without additional data assumptions.The theorem applies to Lipschitz convex losses; a construction shows the discrepancy term cannot be dropped, even for a near-minimizer of contrastive loss.Source: §3.2 · pp. 8–10

3.3 The Relaxed K-Means Loss on Representations

  • The relaxed k-means loss approximates intra-class discrepancy minimization without labels, and its addition improves linear probing, especially on ImageNet-100.The proposed surrogate reduces distances between nearby representations; Figure 3 compares it with InfoNCE at τ = 0.5.Source: §3.3 · pp. 12–13, Figure 3 · p. 13

4 On the Supervised Losses Incurred by Contrastive Learning

The section analyzes the supervised losses induced by InfoNCE and SpeCL, showing how they behave under class imbalance and what structures their optimal representations exhibit.

4 On the Supervised Losses Incurred by Contrastive Learning

  • The section examines InfoNCE- and SpeCL-incurred supervised losses as a way to study contrastive learning’s downstream behavior.The losses differ in form from typical classification losses.Source: §4 · pp. 13–14

4.1 Implications for Mitigating Class Imbalance

  • InfoNCE-incurred loss approximates balanced softmax with error O(k^-1), and its performance usually improves as k increases at imbalance ratios 50 and 100.The reported validation comparison also finds InfoNCE-incurred loss outperforms cross entropy, while normalized SpeCL-incurred loss outperforms its unnormalized counterpart.Source: §4.1 · p. 15, Table 2 · p. 15
  • For SpeCL, class imbalance scales optimal feature and classifier-weight norms while preserving their cosine geometry; normalized features and weights yield a rebalanced feature space.The optimal SpeCL-incurred loss also exhibits variability collapse, and its optimal class means minimize a constrained approximation error to the identity matrix.Source: §4.1 · p. 16, §4.2 · pp. 17–18

4.2 The Optimality of Contrastive Learning-Incurred Losses

  • For InfoNCE, optimal representations exhibit variability collapse and a simplex ETF; near-optimal representations approximately retain these structures when the stated error terms are small.The cited near-optimality result controls excess risk through internal discrepancy, generalization error, and a term accounting for positive pairs from different classes.Source: §4.2 · p. 17

5 Conclusion

The conclusion presents internal discrepancy as a unifying measure and connects it to a downstream generalization bound and strategies for understanding contrastive learning. The accompanying appendix material establishes loss properties and proof steps supporting the bound.

5 Conclusion

  • Internal discrepancy unifies assumptions about representation relationships, supports a downstream generalization bound without additional assumptions, and motivates relaxed k-means loss.The conclusion also summarizes analysis of incurred supervised losses, class imbalance, and optimality properties.Source: §5 · p. 18

A.1 The Convexity and Lipschitzness of the InfoNCE Loss and Spectral Contrastive Loss

  • InfoNCE and SpeCL are separately convex in each representation, with positive-representation Lipschitz constants R and 2R, respectively.These constants follow by differentiating each loss with respect to the positive representation.Source: §A.1 · p. 19

A.2.1 Proof of Theorem 2

  • Theorem 2’s proof uses loss Lipschitzness and Jensen’s inequality to compare sampled positive representations with class means; under the class-constant condition, Disc(f)=0 and the inequalities are equalities.The proof treats the unsupervised loss as using a random class sample as classifier and the class mean as a lower-loss alternative.Source: §A.2.1 · p. 20

A.2.2 Proof of Proposition 3

The proof uses Jensen’s inequality and a constructed representation to establish the bound in Proposition 3 and illustrate the role of internal discrepancy.

A.2.2 Proof of Proposition 3

  • When x+=x and representations have norm at most R, Jensen’s inequality gives an InfoNCE lower bound of log(1 + k e^-R2).The proof applies Jensen’s inequality first to negative samples and then to the anchor.Source: §A.2.2 · pp. 20–21
  • A constructed representation has zero class means and Disc(f)=R, yet its contrastive loss is within ε of the infimum.The construction assigns opposite radius-R orthogonal vectors to paired samples within each class.Source: §A.2.2 · p. 21
  • For that construction, Lcon(f) + η·Disc(f) is at most log(e^R2 + k) + ε, yielding the stated inequality after rearrangement.Source: §A.2.2 · p. 21

A.2.3 Proof of Lemma 4

These appendix proofs establish optimality characterizations, generalization and classification bounds, and a comparison between relaxed k-means loss and intra-class discrepancy.

A.2.3 Proof of Lemma 4

  • For InfoNCE, the characterized minimizers are class-constant simplex ETFs; the proof also shows the corresponding cross-entropy minimizers are minimizers of the same objective.The ETF class means have equal radius and pairwise inner products -R2/(C-1).Source: §A.2.3 · pp. 21–22

A.2.4 Proof of Theorem 5

  • Theorem 5’s bound follows by applying a Rademacher-complexity argument to Lipschitz, bounded losses on representation tuples, then using empirical optimality.Its proof gives the resulting generalization term in terms of the loss Lipschitz constant, range, and representation-class complexity.Source: §A.2.4 · p. 23

A.2.5 Proof of Corollary 6

  • The Corollary 6 proof lower-bounds classification error using the incurred loss’s margin term ωk(y), with a uniform minimum over classes yielding the stated bounds.It derives the error control by comparing the predicted and target class logits, then applies Theorem 5 for the final claim.Source: §A.2.5 · p. 24

A.2.6 Proof of Theorem 7

  • Under class-conditional independence of positive pairs, the proof sets Disc(f̂)=0 and compares the contrastive objective with its incurred supervised loss.It then applies the Theorem 5 generalization argument to the empirical minimizer.Source: §A.2.6 · p. 24

A.2.7 Proof of Proposition 8

  • The Proposition 8 proof bounds total discrepancy by intra-class and alignment components, then controls estimated intra-class discrepancy and contrastive loss uniformly over F.The displayed result gives Lsup(DT, W f̂, h) ≤ Lcon(f̂) + η Disci(f̂) + η Disca(f̂).Source: §A.2.7 · p. 25

A.2.8 Proof of Proposition 9

  • The proof bounds the relaxed k-means loss’s within-class component between 2R d Disci(f) and 2NmaxN d Disci(f).The between-class component is bounded separately by τ times the number of cross-class pairs.Source: §A.2.8 · pp. 25–26

A.3.1 Proof of Theorem 10

The proof relates the contrastive objective to a biased Monte Carlo estimate of a log-expectation and bounds their discrepancy. The resulting downstream bound includes an O(k^-1) term, whose order is sharp under the stated assumptions.

A.3.1 Proof of Theorem 10

  • Under the norm constraints, h = ˜h + log k + O(k^-1), with the Jensen gap bounded using a second-order logarithm bound.The proof defines h(x, y) using an expectation over latent classes and compares it with its biased Monte Carlo estimate.Source: §A.3.1 · pp. 26–27
  • Lsup(DT, Wbf, ˜h) + log k is bounded by Lcon(f) + η · Disc(bf) + GenM + VR.This is the internal-discrepancy-aware downstream supervised-loss bound.Source: §A.3.1 · p. 26
  • The O(k^-1) order cannot generally be improved: when all logits are zero, the discrepancy is log(1 + 1/k) ∼ k^-1.This example establishes sharpness under the stated assumptions.Source: §A.3.1 · p. 27

A.3.2 Proof of Theorem 11

These proofs characterize optimal risks and representations for several supervised losses, then derive a stability bound. In particular, the SpeCL minimum is distribution-independent, while its minimizers under different class distributions are related by transformations.

A.3.2 Proof of Theorem 11

  • For SpeCL, the minimum risk is −r for every strictly positive class distribution, and the stated transformations carry uniform-distribution global minimizers to global minimizers under any positive scaling.Here r = min{d, C}; the proof also gives the converse transformation.Source: §A.3.2 · pp. 27–28

A.3.3 Proof of Proposition 12

  • The Proposition 12 equality case requires orthogonal normalized class weights and representations equal to their target-class weights almost surely.Such minimizers exist when d ≥ |T|, and the same characterization holds under normalization.Source: §A.3.3 · p. 28

A.4.1 Proof of Theorem 13

  • For the mean-classifier loss, minimizers use class means of norm R with pairwise inner products −R^2/(|T| − 1).The proof states that the representation equals its target-class mean almost surely.Source: §A.4.1 · p. 28

A.4.2 Proof of Corollary 14

  • The Corollary 14 stability argument lower-bounds the excess risk by κD, with κ ≥ e^-2R^2/2 for the first bound.For both choices of F, the proof also gives κ ≥ ke^-aR^2/(k + 1).Source: §A.4.2 · p. 29
  • The coefficient is at most 8e^4R^2 for all k ≥ 1 and both choices of F.The bound follows from the stated lower bounds on γk and κ.Source: §A.4.2 · p. 30

A.4.3 Proof of Theorem 15

The proofs characterize optimal class-mean geometry for the supervised losses and distinguish projected residual control from full variability collapse. For the supervised spectral loss, every minimizer is class-constant and its means minimize a Gram-matrix objective.

A.4.3 Proof of Theorem 15

  • For the supervised spectral objective, global minima are attained by class-constant representations whose means minimize the Gram objective subject to ∥μc∥ ≤ R.Equality in the lower bound only requires H⊤(f(x) − μyx) = 0 almost surely, so it does not generally imply variability collapse for every minimizer.Source: §A.4.3 · p. 31
  • The Gram objective reaches its lower bound exactly when G is a rank-r orthogonal projection with feasible diagonal entries qc satisfying Σc qc = r.When C ≤ d, G = I; when C > d, the projection gives the smallest rank-d approximation error to I.Source: §A.4.3 · p. 31

A.4.4 Optimality of Supervised Contrastive Cases

  • The supervised contrastive equality conditions require class means with pairwise inner products −R^2/(C − 1), attained by the feasible simplex Gram matrix.The proof obtains this value by requiring the off-diagonal inner products in each row to be constant and sum to −R^2.Source: §A.4.4 · p. 32
  • For the supervised spectral loss, f minimizes risk if and only if it equals its target-class mean almost surely and those means minimize Q.The proof establishes that every minimizer is class-constant.Source: §A.4.4 · p. 32

Appendix B. One Negative Sample Case

The one-negative-sample analysis gives a supervised-loss bound under a constant-loss condition and characterizes the incurred InfoNCE loss as weighted binary cross-entropy. The accompanying experimental details specify datasets, architectures, and a common training setup.

Appendix B. One Negative Sample Case

  • Under k = 1 and the condition ℓ(v, u, u) = ℓ(0), a supervised loss and linear classifier exist with Lsup(DT, Wf, h) ≤ Lcon(f) + η · Disc(f) − π · ℓ(0).The condition applies for ∥v∥, ∥u∥ ≤ R and the bound holds for all f ∈ F.Source: Appendix B · p. 33
  • The k = 1 InfoNCE-incurred loss is a multi-class binary cross-entropy using weighted logistic losses for classifiers {wy − wc}c≠y.Each sample x is drawn from Dy.Source: Appendix B · p. 33
  • With ℓ(0) = log 2, the bound becomes Lsup(DT, Wf, h) ≤ Lcon(f) + ηDisc(f) − π log 2.The appendix also gives a direct classification guarantee involving pmin, but the supplied passage does not include its full expression.Source: Appendix B · p. 33

C.1 Experimental Details

  • Experiments cover CIFAR-10, CIFAR-100, and ImageNet-100, using ResNet-18 for CIFAR and ResNet-50 for ImageNet-100.All models train for 200 epochs with SGD, batch size 512, initial learning rate 0.2, cosine decay, and weight decay 5e-4.Source: §C.1 · p. 34

C.2 More Experimental Results

Experiments with the relaxed k-means loss show improvement, particularly on ImageNet-100, but performance deteriorates when λ is too large. The relaxation can bring semantically inconsistent samples closer, especially early in training.

C.2 More Experimental Results

  • Linear-probing performance deteriorates when λ is too large, indicating limitations of the relaxed k-means loss.The experiments tested λ values from 0 to 1.Source: §C.2 · p. 34
  • The relaxed k-means loss improves performance, particularly on ImageNet-100.The experiments report three trials per λ value with random seeds 0, 1, and 2.Source: §C.2 · pp. 34–35
  • Because the relaxation reduces distances within a range rather than rigorously shrinking them to cluster centers, semantically inconsistent samples can become close, particularly early in training.This is presented as a possible explanation for the loss's limitations.Source: §C.2 · p. 35

C.3 Validation of Theorem 11

The validation shows that scaling between the two distributions preserves the shared minimum and cosine similarities of corresponding global minimizers. On long-tailed CIFAR-10, representation-model logits roughly align with the reported class-frequency quantity across classes.

C.3 Validation of Theorem 11

  • Both distributions attain the same minimum, and scaling maps corresponding global minimizers while preserving cosine similarities between nonzero features and classifier weights.The proof considers class means µ_c = E_{x~D_c}f(x) and a class-balanced case.Source: §C.3 · p. 35
  • On long-tailed CIFAR-10 with ResNet-18, 512-dimensional features, and |T| = 10, representation-model logits roughly align with the reported class-frequency quantity across classes.The passage identifies these logits as µ_c^T w_c.Source: §C.3 · p. 35

Related papers

Loading 2610.32729v1…