Debiased Contrastive Learning

Ching-Yao Chuang; Joshua Robinson; Lin Yen-Chen; Antonio Torralba; Stefanie Jegelka

Source-linked AI summary

Debiased Contrastive Learning

Ching-Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, Stefanie Jegelka

arXiv:2007.00224v3Jul 1, 2020cs.LGstat.ML

TL;DR

Contrastive learning often treats randomly sampled unlabeled points as negatives even when they share the anchor’s label, creating sampling bias. The paper corrects this bias with a debiased contrastive objective using unlabeled data and positive examples, and reports improvements across vision, language, and reinforcement learning benchmarks with downstream classification generalization guarantees.

Problem
Contrastive learning typically samples negatives without labels, so some negatives may actually share the anchor’s label; the ideal unbiased objective requires label knowledge.
Method
The paper develops a debiased contrastive objective that corrects negative-sampling bias using only unlabeled data and positive examples.
Results
The objective consistently improves state-of-the-art baselines across vision, language, and reinforcement learning benchmarks and provides downstream classification generalization guarantees.
Takeaways & Limitations
Debiased contrastive learning offers an unsupervised framework for reducing negative-sampling bias while retaining theoretical guarantees for downstream classification.
Takeaways & Limitations
The method may suffer from biases in learned representations similar to standard contrastive learning, including possible worse performance for smaller classes or groups.

Abstract

from arXiv · show

A prominent technique for self-supervised representation learning has been to contrast semantically similar and dissimilar pairs of samples. Without access to labels, dissimilar (negative) points are typically taken to be randomly sampled datapoints, implicitly accepting that these points may, in reality, actually have the same label. Perhaps unsurprisingly, we observe that sampling negative examples from truly different labels improves performance, in a synthetic setting where labels are available. Motivated by this observation, we develop a debiased contrastive objective that corrects for the sampling of same-label datapoints, even without knowledge of the true labels. Empirically, the proposed objective consistently outperforms the state-of-the-art for representation learning in vision, language, and reinforcement learning benchmarks. Theoretically, we establish generalization bounds for the downstream classification task.

1 Introduction

Contrastive learning commonly samples negatives from unlabeled data, allowing semantically similar points to be treated as negatives and creating sampling bias. The paper proposes a debiased objective to correct this bias without labels, evaluates it broadly, and provides supervised-task generalization guarantees.

Contrastive learning brings semantically similar pairs together and pushes dissimilar pairs apart in representation space.
Because labels are unavailable, negative examples are commonly sampled uniformly from training data and may actually be similar to the anchor.The paper calls this phenomenon sampling bias.
The ideal unbiased objective requires label knowledge, motivating correction of the gap using only unlabeled data and positive examples.
The debiased contrastive objective corrects negative-sampling bias and is compatible with algorithms that optimize the standard contrastive loss.
The theoretical analysis relates debiased contrastive optimization to an upper bound on supervised loss and yields a downstream classification generalization bound.
The approach is evaluated in vision, language, and reinforcement learning benchmarks and improves over state-of-the-art methods.

2 Related Work

The paper builds on contrastive representation learning and positive-unlabeled learning while addressing sampling bias in contrastive loss, which existing related analyses identify without resolving.

Contrastive representation learning has achieved state-of-the-art results, with approaches differing mainly in how they obtain positive pairs.
Prior theoretical work analyzes downstream classification and identifies sampling bias as a problem but does not propose models to address it.
The paper relates to positive-unlabeled learning because it uses unlabeled data and positive examples, but addresses contrastive loss rather than zero-one losses.

3 Setup and Sampling Bias in Contrastive Learning

Contrastive learning ideally uses negatives from different latent classes, but unlabeled sampling can include same-class examples. This sampling bias separates the standard objective from the ideal unbiased objective.

Setup: Contrastive learning maps observations and semantically similar pairs into embeddings, with temperature controlling the hypersphere radius.The setup assumes positive pairs share a latent class.
Ideal objective: The ideal unbiased objective uses negative examples drawn from classes different from the anchor’s latent class.Its negative distribution is defined by conditioning on a different latent class.
Sampling bias: Unlabeled training instead samples negatives from p(x), so a negative can come from the same class as the anchor with probability τ+.The standard approach therefore optimizes a biased loss rather than the ideal loss.
Sampling bias: The ideal unbiased objective is unavailable in practice because obtaining truly different-label negatives requires label knowledge.The paper frames reducing this gap as the central setup question.
Consequences: For any embedding and finite N, the standard biased loss upper-bounds the ideal unbiased loss.The bound uses a ∧ operator for the minimum of two real numbers.
Consequences: With large N, the bound’s last term can become negligible, but minimizing the upper bound can still widen the gap as the unbiased loss decreases.The paper also reports that biased and unbiased losses can produce very different representations.

4 Debiased Contrastive Loss

The debiased objective estimates the ideal loss using only unlabeled and positive samples, correcting sampled negatives with additional positive samples. Its finite-sample error decreases with N and M, and larger values consistently improve performance.

Derivation: The paper decomposes the data distribution to derive an objective estimable using samples only from p and the positive distribution p+.This removes the need to directly sample from the inaccessible negative distribution.
Practical objective: The straightforward objective is computationally expensive for large N and requires at least N positive samples.These constraints motivate a more practical asymptotic form.
Practical objective: The debiased objective continues sampling negatives from p but uses additional positive samples v to reweight positive and negative denominator terms.This provides the correction without direct access to true negative examples.
Estimation: The empirical estimate is easier to compute than the straightforward objective and uses an estimator constrained above its theoretical minimum.The constraint prevents taking the logarithm of a negative number.
Theory: Theorem 3 bounds finite-N and finite-M estimation error at rate O(N^-1/2 + M^-1/2).The class prior τ+ can be estimated from data or treated as a hyperparameter.
Empirical behavior: Larger N and M consistently improve empirical performance, while the resulting embeddings are closer to those learned with the unbiased loss.The implementation averages the debiased loss over T data points.

5 Experiments

Experiments evaluate the debiased objective across vision, language, and reinforcement learning, finding improvements over biased contrastive baselines and representations closer to the ideal unbiased objective. Increasing positive samples further improves performance, while the method remains effective under nonuniform classes and surrogate positive distributions.

The debiased objective outperforms state-of-the-art contrastive learning on vision, language, and reinforcement learning benchmarks.
CIFAR10 and STL10: Increasing the number of negative or positive examples improves performance, with even one additional positive example producing a noticeable gain.
CIFAR10 and STL10: The debiased objective with M > 1 outperforms a biased baseline trained for 50% more epochs.
CIFAR10 and STL10: On CIFAR10, debiased representations show better class separation than biased representations and are closer to those from the ideal unbiased objective.
ImageNet-100: On ImageNet-100, the debiased objective improves over the biased baseline despite the dataset’s smaller class probabilities.
Sentence Embeddings: The experiments use six sentence-classification tasks and compare sentence representations using 10-fold cross-validation for four binary tasks.
Reinforcement Learning: In reinforcement learning, the debiased objective outperforms CURL in all environments and often has smaller variance.
Discussion: The objective still improves over baselines when class distributions are not well balanced or true positive distributions are approximated by surrogate distributions.

6 Theoretical Analysis: Generalization Implications for Classification Tasks

The analysis connects debiased contrastive learning to supervised classification by showing that its objective bounds supervised loss and yields downstream generalization guarantees. The empirical objective uses positive and negative samples, while the bound depends on representation complexity and sample sizes.

Theoretical connection: The analysis relates debiased contrastive learning to a downstream supervised classification task through a generalization bound.The bound is derived for representations learned by minimizing an empirical debiased contrastive objective.
Downstream task: The supervised analysis fixes representations and trains a linear classifier using multiclass softmax cross-entropy.The classifier is defined as q(x) = Wf(x), and the analysis also considers a mean classifier whose class weights are representation means.
Theoretical connection: The asymptotic unbiased contrastive loss upper-bounds the supervised loss of the mean classifier when N ≥ K −1.This relationship is the key bridge from the contrastive objective to supervised classification guarantees.
Objective and complexity: The empirical debiased objective averages over T data points with M positive and N negative samples per point, and the learned representation is an empirical risk minimizer.The resulting generalization analysis uses the empirical Rademacher complexity R_S(F) of the function class.
Generalization implications: If F contains an embedding with small debiased loss, a sufficiently data-rich training procedure produces a representation that performs well on downstream classification.The bound also reflects the roles of M, N, T, and the function-class complexity; its dependence on N and T is roughly equivalent to prior work but not directly comparable.

7 Conclusion

The paper proposes debiased contrastive learning to correct sampling bias from drawing negative examples from the overall data distribution. It reports consistent improvements over state-of-the-art baselines across vision, language, and reinforcement learning, with downstream classification guarantees.

Conclusion: The proposed framework corrects bias from sampling negative examples from the overall data distribution while using unlabeled data.The authors identify semi-supervised and few-shot learning, along with positive-example sampling, as future directions.
Conclusion: The debiased objective consistently improves state-of-the-art baselines across vision, language, and reinforcement learning benchmarks.The framework is also accompanied by generalization guarantees for downstream classification.

Broader Impact

The work is motivated by the potential value of improved unsupervised representations when labeled data are scarce. Its broader-impact boundary is that data biases may still propagate into the learned representations, similarly to standard contrastive learning.

Potential benefits: Unsupervised representation learning can help when only small amounts of labeled data are available in applications such as medicine, science, and drug discovery.The passage frames improved representation learning as potentially beneficial across these application areas.
Limitations: Data biases can produce biased learned representations and worse performance for smaller classes or groups.The passage gives more frequent sampling of majority groups than minority groups as an example.
Limitations: The proposed method may suffer from biases similar to standard contrastive learning, motivating further testing across groups and settings.The authors identify thorough evaluation of this issue as future research.

A.3 Proof of Theorem 3

The proof of Theorem 3 bounds the difference between finite-sample and asymptotic debiased objectives using tail-probability arguments. It applies concentration tools to obtain the theorem’s finite-N and finite-M approximation guarantee.

A.3 Proof of Theorem 3: The proof controls the discrepancy between asymptotic and non-asymptotic objectives by bounding the tail probability of their integrand difference.The difference is analyzed for fixed x and x+ before integrating the tail probability.
A.3 Proof of Theorem 3: Theorem 3 follows by combining the tail bound with Jensen’s inequality, which moves the absolute value inside the expectation.The stated result applies to any embedding f and finite N and M.
A.3 Proof of Theorem 3: The proof decomposes the discrepancy into terms whose tails can be bounded separately, including a max-based term.It uses the inequality | max(a, b) − c| ≤ |a − c| when c ≥ b.
A.3 Proof of Theorem 3: Hoeffding’s inequality supplies concentration because the critic satisfies e^-1 ≤ h(x, x̄) ≤ e.The bounded critic yields tail bounds for both terms in the decomposition.
A.3 Proof of Theorem 3: The broader theoretical setup considers uniform K-way classification tasks and uses Jensen-based inequalities to relate unbiased contrastive loss to supervised loss.The proof notes that N = K −1 gives the smallest loss and introduces a task-specific class distribution over possible K-way tasks.

A.5 Proof of Theorem 5

The proof derives a downstream generalization bound by applying concentration results to the debiased contrastive objective and controlling its loss and Lipschitz properties. It combines these intermediate bounds with prior lemmas to establish Theorem 5.

Proof strategy: The proof applies a concentration-of-measure result to bound the loss of an empirical risk minimizer.The cited result assumes a Lipschitz, bounded loss over sampled training data.
Proof strategy: The representation class is bounded by R, and the proof specializes the analysis to k = N + M and R = e.Normalized embeddings further restrict the analysis to e−1 ≤ a_i, b_i ≤ e.
Technical bounds: The loss is bounded and shown to have controlled Lipschitz behavior through derivative and Jacobian estimates.The argument establishes φ as e-Lipschitz and bounds the Lipschitz constant of g using the Frobenius norm of its Jacobian.
Conclusion: Theorem 5 follows by combining the earlier results with Lemma A.3 and Lemma A.4, yielding a high-probability bound for all f ∈ F and N ≥ K − 1.The proof explicitly combines Lemma 4, Theorem 3, and concentration arguments.
Objective derivation: The unbiased objective is decomposed into N + 1 expectations using a binomial expansion and inclusion–exclusion structure.The derivation groups terms according to how many negative samples are drawn from p.
Limitation: The exact objective becomes numerically extremely small for large N, and the authors report that attempted optimization approaches failed.This is identified as a practical limitation of the objective formulation.

B Experimental Details

The experiments implement the debiased objective across vision, language, and reinforcement-learning setups using established codebases and specified training procedures. Evaluation uses fixed learned embeddings with downstream linear classification where described.

Vision: CIFAR10 and STL10 use SimCLR with ResNet-50, Adam, temperature 0.5, 128-dimensional latents, and 400 training epochs.The implementation uses learning rate 0.001 and weight decay 1e−6.
Evaluation: CIFAR10 and STL10 representations are evaluated by training a linear classifier with cross-entropy loss after fixing the learned embedding.The classifier is optimized with Adam at learning rate 0.001 and weight decay 1e−6.
Vision: ImageNet-100 experiments modify only the NCE/NCECriterion.py file in the official CMC codebase.The CMC temperature is set to 0.07.
Stability: When the estimator falls below e−1/t, the implementation switches to the biased loss to improve convergence and stability.This safeguard is described for the CMC implementation.
Language: Sentence-embedding experiments use quick-thought vectors with an unofficial BookCorpus version and constrain the estimator above zero because features are not normalized.The debiased objective is implemented by modifying only src/s2v-model.py.
Reinforcement learning: Reinforcement-learning experiments use CURL and similarly constrain the estimator above zero because CURL feature vectors are not normalized.Only curl-sac.py is modified while the remaining official code is retained.