Source-linked AI summary
Understanding Contrastive Learning Requires Incorporating Inductive Biases
Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, Akshay Krishnamurthy
TL;DR
Prior contrastive-learning analyses focus on augmentation properties and contrastive loss without accounting for function-class or training-algorithm inductive biases. This paper combines experiments and theory to examine those effects, finding that they matter for downstream performance and that appropriate function classes can succeed where agnostic guarantees are vacuous.
Problem
Method
The paper studies contrastive learning experimentally and theoretically while incorporating function-class inductive bias, including transfer analysis for linear representations.
Source: §1 · p. 2, §6 · p. 15Results
Experiments and theory indicate that contrastive loss alone does not predict downstream performance, overlap-based analyses can be vacuous, and appropriate function classes can succeed with little or no overlap.
Source: §1 · pp. 2–3, §4.2 · p. 9Takeaways & Limitations
Understanding contrastive-learning transfer requires accounting for the inductive bias of the representation class, rather than relying only on augmentation properties and contrastive loss.
Source: §2 · p. 5, §6 · p. 15The theoretical results incorporate function-class bias for linear representations, while extending them to more complex settings remains nontrivial.
Source: §4.3 · p. 10, §6 · p. 15
Abstract
from arXiv · showhide
Contrastive learning is a popular form of self-supervised learning that encourages augmentations (views) of the same input to have more similar representations compared to augmentations of different inputs. Recent attempts to theoretically explain the success of contrastive learning on downstream classification tasks prove guarantees depending on properties of {\em augmentations} and the value of {\em contrastive loss} of representations. We demonstrate that such analyses, that ignore {\em inductive biases} of the function class and training algorithm, cannot adequately explain the success of contrastive learning, even {\em provably} leading to vacuous guarantees in some settings. Extensive experiments on image and text domains highlight the ubiquity of this problem -- different function classes and algorithms behave very differently on downstream tasks, despite having the same augmentations and contrastive losses. Theoretical analysis is presented for the class of linear representations, where incorporating inductive biases of the function class allows contrastive learning to work with less stringent conditions compared to prior analyses.
1 Introduction
Contrastive learning learns representations from unlabeled views, but existing theory leaves open how architecture and training choices affect downstream performance. The paper uses experiments and theory to show that contrastive loss and augmentation properties alone can miss important behavior.
1 Introduction
- Contrastive learning encourages views of one input to cluster and views of different inputs to separate, producing representations used for downstream classification.The introduction notes that these representations can support competitive performance even with a linear classifier.Source: §1 · p. 1
- Prior analyses emphasize augmentation overlap and contrastive loss, while omitting properties of the representation class and training algorithm.Arora et al. [2019] assume identical augmentation distributions within each class; HaoChen et al. [2021] allow overlap patterns whose within-class graph is dense.Source: §1 · p. 2
- Downstream performance depends on the function class and training procedure, so matching contrastive losses and augmentations need not yield matching representations or task performance.The paper identifies function-class sensitivity, brittleness under loss minimization, and failures of function-class-agnostic guarantees with disjoint augmentations.Source: §1 · pp. 2–3
- The paper argues that contrastive learning can succeed with little or no augmentation overlap, although function-class-agnostic analyses can then give vacuous guarantees.Appropriate function classes can still succeed in the disjoint-augmentation regime.Source: §1 · pp. 2–3
1.1 Related work
- Related work also studies mutual information, alignment and uniformity, latent-variable and causal perspectives, optimization, and non-contrastive objectives.The paper distinguishes its focus on function-class inductive bias from work on shortcut solutions and objective or training degeneracies.Source: §1.1 · p. 3
2 Preliminaries
The preliminaries define contrastive learning and downstream evaluation, then describe transfer bounds that relate contrastive loss to classification performance. The paper distinguishes prior bounds that ignore the representation class from bounds that incorporate it.
2 Preliminaries
- Contrastive learning maps augmentations to vectors, encouraging views of the same input to be closer than randomly paired views.The generic loss notation Lcont covers SimCLR, spectral contrastive loss, or another contrastive objective.Source: §2 · p. 4
- Downstream quality is measured by linear classification on original inputs using the average representation over their augmentations.The text reports that this average experimentally performs better than the standard unaveraged approach.Source: §2 · pp. 4–5
- Prior transfer bounds relate classification loss to contrastive loss using problem quantities and representation dimension, while assuming overlap between same-class augmentation distributions.Arora et al. [2019] use identical within-class distributions; HaoChen et al. [2021] use an overlap-based spectral quantity.Source: §2 · p. 5
- A function-class-aware transfer bound can reflect that downstream performance at a given contrastive loss depends on the representation class.The paper notes that these bounds concern upstream and downstream population losses, not generalization guarantees.Source: §2 · p. 5
3 Warm-up: contrastive learning on hyper s
The Boolean hypercube example shows that downstream transfer can depend sharply on the function class and training algorithm, despite the same augmentations. A spurious contrastive minimizer can perform at chance, while an appropriate class succeeds.
3 Warm-up: contrastive learning on hyper s
- The example uses inputs in {±1}^D, with labels linear in the first k coordinates and augmentations that preserve those coordinates while scaling down the remaining coordinates.The scaling factor is sampled uniformly from (0, 1].Source: §3 · pp. 5–6
- Linear representations achieve significantly better downstream performance than MLPs despite having higher contrastive loss, and MLP accuracy varies substantially with training configuration.The reported configurations include Adam versus SGD and weight decay versus no weight decay.Source: Figure 2 · p. 6, §3 · p. 6
- A spurious representation can have much lower contrastive loss yet yield random-guessing downstream performance, because the original input is recoverable from an augmentation by sign(x).This gives a case where an appropriate function class can succeed but a function-class-agnostic analysis cannot explain that success.Source: §3 · p. 6
4 Lower bounds and improved analysis
This section shows that bounds based only on augmentations and contrastive loss can be vacuous, then develops linear-representation guarantees that account for the function class and work under weaker augmentation assumptions.
4.1 Lower bound for disjoint augmentations
- With disjoint augmentations and low representation dimension, a global contrastive-loss minimizer can have near-trivial downstream performance, making loss-only transfer bounds vacuous.The result holds in the stated settings for unconstrained representations and for constrained representations with a common source of randomness; the assumptions also include balanced labels and a uniform input distribution.Source: §4.1 · pp. 7–8
4.2 Prior theoretical results and failure modes
- For small representation dimension, prior spectral guarantees are vacuous under disjoint augmentations and require contrastive loss within O(τ^2) of optimal under approximately disjoint augmentations.The paper also reports nearly 100% accuracy when predicting the original input from standard image augmentations, although contrastive learning still succeeds.Source: §4.2 · p. 9, Approximately disjoint augmentations · p. 9
4.3 Function class dependent transfer guarantees
- Linear-representation guarantees incorporate features of the function class and can be non-vacuous under weaker assumptions than function-class-independent analyses.The bound depends on feature-defined eigenvalues and inconsistency and regression metrics; the paper states that it recovers the prior result when the feature map is full rank.Source: §4.3 · pp. 10–11
- In the hypercube example, function-class-agnostic guarantees are vacuous, while minimizing contrastive loss within the linear class suffices for good downstream performance.Table 1 describes contrastive-loss minimizers ranging from perfect downstream classifiers to no better than random guessing across representation classes and training procedures.Source: Table 1 · p. 7, §4.3 · p. 11
5 Experiments
The experiments test whether contrastive-loss transfer depends on the representation class and training procedure, and whether contrastive learning can work with little augmentation overlap. Across CIFAR-10 and AG News, transfer behavior varies by architecture and augmentation, including cases where low contrastive loss accompanies poor downstream performance.
5.1 CIFAR-10 + SimCLR experiments
- On CIFAR-10, removing random cropping or horizontal flipping can accentuate transfer differences across architectures, and adversarial training can produce low-loss representations with poor downstream performance.The latter result uses the same augmentations but modifies the training algorithm and ResNet architecture.Source: §5.1 · p. 12, • Label-orthogonal training · p. 12
- With hash-pixel augmentations, ViT and MLP-Mixer achieve much lower contrastive loss but near-random downstream accuracy, while ResNet performs well despite not minimizing the loss.The setting forces augmentation distributions into the disjoint regime, demonstrating success for one function class despite the disjointness.Source: • Label-orthogonal training · pp. 12–13
5.2 Are we in the disjoint augmentation regime?
- CIFAR-10 classifiers identify the source image from its augmentation with accuracy above 99.5%, suggesting these standard settings approach the disjoint-augmentation regime.Figure 4 reports approximately 100% average accuracy in the 5000-way task across four settings.Source: §5.2 · p. 13, Figure 4 · p. 13
5.3 Experiments on a Text Domain
- Across AG News augmentations, BoW performs best downstream despite somewhat worse contrastive performance, while GRU and Transformer show augmentation-dependent brittle transfer.For split augmentation, adding the original text as an augmentation lets both GRU and Transformer perform well downstream.Source: Figure 5 · p. 14, §5.3 · p. 15
- For AG News, dropping and permuting tokens helps Transformer transfer without closing its gap, while split augmentation yields brittle transfer for both GRU and Transformer.Including the original text alongside split views makes both models perform well; the passage says this is not explained by distribution shift.Source: §5.3 · p. 15
- The text experiments suggest that less expressive BoW representations can succeed with weaker augmentations, whereas GRU and Transformer may need stronger augmentations for downstream transfer.The passage says this architecture-dependent pattern is not well understood by current theory.Source: §5.3 · p. 15
6 Conclusion
The paper diagnoses gaps in current understanding of contrastive learning and argues that progress requires accounting for inductive biases. Its theoretical results cover linear representations, while extending the analysis to more complex classes and training procedures remains future work.
6 Conclusion
- The paper's study is diagnostic: it identifies gaps in understanding rather than converting its insights into algorithmic approaches.The authors describe algorithmic approaches as a promising direction.Source: §6 · p. 15
- The paper incorporates function-class bias into transfer bounds for linear representations, while extending results to more complex classes and training procedures remains open.The conclusion notes that inductive bias has primarily been studied in simplistic architectures and names depth-2 or depth-3 examples.Source: §6 · p. 15
- The authors suggest that visualizing contrastive loss against downstream performance may aid selection of more robust augmentations.Source: §6 · p. 15
A.1 Proof of Proposition 4.2
The proof sketch uses disjoint augmentations and a covariance calculation to establish eigenvalues, then applies Theorem 4.5.
A.1 Proof of Proposition 4.2
- The proof of part (a) follows from Lemma 4.1 because its conditions hold and the augmentations are disjoint.Source: §A.1 · p. 19
- The covariance calculation uses τ sampled uniformly from (0, 1].Source: §A.1 · p. 19
- The resulting eigenvalues are λ_i = 0 for i ≤ k and λ_i = 1/4 for i > k; substituting them into Theorem 4.5 with d′ = k completes the proof.Source: §A.1 · p. 19
B Low-rank Linear Representation Proof
The proof recasts contrastive learning objectives and representations as matrix problems, then uses the feature-defined linear function class to derive downstream guarantees. The guarantee relies on augmentation overlap and label-related assumptions, as well as sufficient expressive capacity within that function class.
B Low-rank Linear Representation Proof
- The proof strategy seeks transfer bounds that depend on both contrastive loss and membership in a feature-defined linear representation class.It adapts HaoChen et al.'s strategy, with key changes to incorporate the function class.Source: B Low-rank Linear Representation Proof · p. 19
- For unrestricted representations, the spectral objective becomes matrix factorization, whose optimum corresponds to learning the top d eigen-directions of A◦.For the feature-defined class, the factorized matrix instead depends on both A◦ and the feature map.Source: B Low-rank Linear Representation Proof · p. 21
- The downstream connection assumes sufficient overall augmentation overlap, approximate label invariance across augmentations, and a function class expressive enough to solve classification on augmentations.The analysis uses less stringent overlap conditions than prior analyses, according to the passage.Source: B Low-rank Linear Representation Proof · p. 21
B.1.1 Distributions to matrices
- The proof represents input and augmentation distributions as matrices, including a normalized similarity matrix A◦ whose entries derive from similar-pair probabilities and augmentation marginals.The matrix notation also defines normalized augmentation distributions and their relation to the similarity matrix.Source: §B.1.1 · p. 22
B.1.2 Representations to matrices
- Feature maps and representations are stacked as matrices, with distributional normalization; the linear class consists of representations formed by multiplying the feature matrix by W.Source: §B.1.2 · p. 23
B.3 Proof of main result
The proof links contrastive-loss minimization within a fixed feature class to downstream classification through the signal captured by the learned representation. The discussion identifies feature expressivity, in-class loss optimality, and feature-dependent eigenvalues as conditions for good transfer.
B.3 Proof of main result
- The proof relates eigenvalues of the feature-covariance expression to those of the augmentation-graph Laplacian when φ is full rank.This supplies the spectral link between the projected matrix used in the proof and the theorem’s feature-dependent eigenvalues.Source: §B.3 · pp. 28–29
- Theorem 4.5 bounds downstream performance for representations in Fφ using eigenvalues of the feature-dependent covariance expression.The discussion says good performance also requires small in-class contrastive-loss suboptimality and a small Lreg(φ; g).Source: §B.3 · p. 29, §B.4 · p. 31
- Contrastive learning in Fφ reduces to a rank-d matrix factorization of an augmentation adjacency matrix projected onto the feature span.The minimizer spans the top d singular directions of the projected matrix.Source: §B.3 · pp. 29–30
- An ε-optimal representation captures most of the top d′ singular directions when d′ ≤ d, with the singular-value gap controlling the suboptimality effect.The proof connects those directions to downstream error by bounding how much ground-truth label signal they retain.Source: §B.3 · p. 30
B.4 Discussion of upper bound
- Good transfer requires a predictor on augmentations that is label-consistent and expressible by φ, together with favorable feature-dependent eigenvalues and eigen-gaps.The discussion notes that cross-class augmentation overlap prevents ΔA(g, y⋆) from being small.Source: §B.4 · pp. 31–32
C Lower bounds for (approximately) disjoint augmentations
With disjoint augmentation distributions, a global contrastive-loss minimizer can have trivial downstream performance. The section gives conditions for this lower bound and relates approximate disjointness to augmentation-identification Bayes error.
C Lower bounds for (approximately) disjoint augmentations
- A global minimizer of the contrastive objective can have trivial downstream performance when augmentation distributions do not overlap.The theorem’s setting includes balanced labels and either unconstrained representations or constrained representations with a common invertible source of randomness.Source: C Lower bounds for (approximately)… · p. 32
- The lower-bound construction preserves contrastive loss while permuting instance embeddings, allowing a permutation whose labels are far from every linear classifier on those embeddings.For unnormalized representations, convexity permits replacing each instance’s augmentations by their mean embedding without increasing contrastive loss.Source: C Lower bounds for (approximately)… · p. 33
C.1 Approximately disjoint augmentations
- Bayes-error(A) is the minimum error in identifying which input generated an augmentation.The section connects this augmentation-classification error to the normalized augmentation matrix and its Laplacian eigenvalues.Source: §C.1 · p. 35
- Bayes-error(A) ≥ 1/2 D̄min(|X̄| − d)λd+1, relating approximate disjointness to the Laplacian spectrum.Here D̄min is the minimum input probability, and λd+1 is an eigenvalue of the normalized Laplacian.Source: §C.1 · pp. 35–36
D.1 Synthetic experiments: hypercube example
The synthetic experiments use a hypercube data setup with augmentations that scale down selected coordinates. The passage specifies the downstream task and training and evaluation sizes.
D.1 Synthetic experiments: hypercube example
- Data are sampled uniformly from a D = 50 hypercube, and augmentations randomly scale down the last 40 coordinates.The scaling factors are sampled independently from Unif([0, 1]).Source: §D.1 · p. 37
- A random linear classifier defines downstream labels using 10 nonzero coefficients drawn from N(0, 1).The other coefficients are zero.Source: §D.1 · p. 37
- Contrastive loss is minimized on a training set of 50,000 examples, and downstream accuracies are evaluated under a linear classifier.The two-layer MLP representation dimension is 20; means and 95% confidence intervals are computed from 10 random seeds.Source: §D.1 · p. 37
D.2 CIFAR-10 + SimCLR experiments
The CIFAR-10 experiments describe the SimCLR training and linear-evaluation setup, then specify hash-pixel and label-orthogonal variants. The variants alter the information or representation passed into the contrastive loss.
D.2 CIFAR-10 + SimCLR experiments
- CIFAR-10 SimCLR experiments use ResNet-18, ViT, and MLP-Mixer configurations with an added projection MLP, and evaluate learned representations by linear classification accuracy.Training runs for 1,000 epochs; representations are normalized to unit norm for the SimCLR loss.Source: §D.2 · pp. 37–39
D.2.1 Hash experiment.
- The hash experiment replaces 16 image pixels with a 128-bit MD5 hash, making the generating image’s identity recoverable from each augmentation.The modified augmentation pipeline is used to enforce the disjoint-augmentation regime.Source: Figure 7 · p. 38, §D.2.1 · p. 39
D.2.2 Label-orthogonal training.
- Label-orthogonal training subtracts the class mean representation from each augmentation before applying the SimCLR loss.The class mean is computed at each step from a memory bank of 10,240 image-label pairs; the passage says this subtraction makes the representation orthogonal to labels and reduces linear classification ability.Source: §D.2.2 · p. 39
D.3 Experiments on Text Domain
The text-domain experiments evaluate contrastive representations on AG News classification using BoW, GRU, and Transformer models. They train with sampled augmentation pairs and assess downstream performance with a linear classifier on fixed representations.
D.3 Experiments on Text Domain
- AG News has four topic classes, and text sequences longer than 60 tokens are trimmed, yielding a vocabulary of 11,970 tokens.Source: §D.3 · p. 39
- Each training epoch samples augmentation pairs for 50,000 texts and minimizes SimCLR loss; downstream evaluation predicts the text's class.The contrastive loss uses temperature τ = 1.Source: §D.3 · p. 40
- The experiments compare BoW, a four-layer unidirectional GRU, and a four-layer unidirectional Transformer, each producing a d-dimensional text representation.A linear classifier is trained on fixed representations, with validation-based model selection and early stopping; hyperparameters are listed in Table 4.Source: §D.3 · p. 40, Table 4 · p. 40
D.3.1 Robust evaluation.
Robust evaluation with augmentation-averaged representations produces transfer plots almost identical to standard evaluation. In two-dimensional experiments, downstream transfer and representation geometry vary across architectures and augmentations.
D.3.1 Robust evaluation.
- Augmentation-averaged evaluation produces AG News contrastive-loss–accuracy plots almost identical to standard evaluation, so the distribution shift does not appear to explain brittle transfer.Figure 8 uses fA for downstream evaluation; its plots are almost identical to Figure 5.Source: §D.3.1 · p. 40, Figure 8 · p. 41
D.3.2 Visualizing 2-dimensional representations.
- For drop augmentation, BoW representations are roughly linearly separable, whereas GRU and Transformer representations are augmentation-invariant but not linearly separable.GRU points are spread out; Transformer class distributions are nearly identical, consistent with its poor downstream performance.Source: §D.3.2 · p. 41, Figure 10 · p. 43
- At d = 2, BoW performs around 80% downstream, while Transformer or both recurrent models and Transformer show brittleness depending on the augmentation.For split augmentation, GRU performs almost as well as BoW at d = 2 despite not doing so at d = 128, a non-monotonic dimensionality pattern.Source: §D.3.2 · p. 41, Figure 9 · p. 42
- With drop-and-permute augmentation, BoW representations are roughly linearly classifiable, GRU representations somewhat classifiable, and Transformer representations augmentation-invariant but not linearly separable.Source: Figure 11 · p. 44