PaperlayerView exact revision on arXiv

Source-linked AI summary

Debiased Contrastive Learning of Unsupervised Sentence Representations

Kun Zhou, Beichen Zhang, Wayne Xin Zhao, Ji-Rong Wen

arXiv:2205.00656v1cs.CL

TL;DR

Random negative sampling in contrastive learning may introduce sampling bias, including false negatives that are semantically close. DCLR combines instance weighting with noise-based negatives, and outperforms competitive baselines on seven STS tasks.

  • Problem

    Randomly sampled negatives may introduce sampling bias, including false negatives that are semantically close to the sentence being represented.

    Source: §1 · p. 1

  • Method

    DCLR weights negatives to punish false negatives and generates noise-based negatives to address anisotropy in sentence representations.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    DCLR outperforms several competitive baselines on seven semantic textual similarity tasks.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The reported results support DCLR as an effective framework for reducing the influence of negative-sampling bias in unsupervised sentence representation learning.

    Source: §7 · p. 9

    Because the approach uses pre-trained language models, it may inherit biases captured from their training data.

    Source: Ethical Consideration · p. 9

Abstract

from arXiv · show

Recently, contrastive learning has been shown to be effective in improving pre-trained language models (PLM) to derive high-quality sentence representations. It aims to pull close positive examples to enhance the alignment while push apart irrelevant negatives for the uniformity of the whole representation space. However, previous works mostly adopt in-batch negatives or sample from training data at random. Such a way may cause the sampling bias that improper negatives (e.g. false negatives and anisotropy representations) are used to learn sentence representations, which will hurt the uniformity of the representation space. To address it, we present a new framework \textbf{DCLR} (\underline{D}ebiased \underline{C}ontrastive \underline{L}earning of unsupervised sentence \underline{R}epresentations) to alleviate the influence of these improper negatives. In DCLR, we design an instance weighting method to punish false negatives and generate noise-based negatives to guarantee the uniformity of the representation space. Experiments on seven semantic textual similarity tasks show that our approach is more effective than competitive baselines. Our code and data are publicly available at the link: \textcolor{blue}{\url{https://github.com/RUCAIBox/DCLR}}.

1 Introduction

Randomly sampled negatives can include semantically close examples and representations confined to a narrow cone, creating sampling bias for contrastive sentence learning. DCLR combines instance weighting with noise-based negatives and outperforms competitive baselines on seven STS tasks.

1 Introduction

  • Contrastive learning pulls semantically close examples together for alignment and pushes non-neighbors apart for representation-space uniformity.The introduction frames this as a way to refine PLM-derived sentence representations.Source: §1 · p. 1
  • Random negatives may be false negatives that are semantically close, while anisotropic PLM representations leave sampled negatives unrepresentative of the whole space.The introduction reports that about half of in-batch negatives have cosine similarity above 0.7 to the input sentence.Source: §1 · pp. 1–2
  • DCLR assigns lower weights to negatives with higher similarity scores, using a complementary model to identify semantically close false negatives.The framework is presented as a way to reduce the influence of those negatives during training.Source: §1 · p. 2
  • DCLR initializes noise-based negatives from Gaussian noise and optimizes them toward nonuniform points to extend the occupied representation space.The authors say contrasting with these negatives can improve representation-space uniformity.Source: §1 · p. 2
  • DCLR outperforms competitive baselines on seven STS tasks using BERT and RoBERTa.The authors describe their approach as the first attempt to reduce sampling bias in unsupervised sentence-representation contrastive learning.Source: §1 · p. 2

2 Related Work

Related work covers supervised and unsupervised sentence representations, contrastive learning and its augmentation strategies, and virtual adversarial training. The authors identify the quality of randomly sampled negatives as relatively understudied.

Sentence Representation Learning

  • Sentence-representation work includes supervised approaches using annotated data and unsupervised approaches without labels; PLM methods also address anisotropy through flow-based or whitening strategies.Contrastive learning has also been used to refine PLM representations.Source: §2 · p. 2

Contrastive Learning

  • Sentence contrastive learning uses augmentations such as back translation, token shuffling, or dropout, while the quality of randomly sampled negatives is seldom studied.Related work describes contrastive learning as balancing alignment and uniformity in semantic representation space.Source: §2 · p. 3

Virtual Adversarial Training

  • Virtual adversarial training perturbs inputs with learnable noise to maximize prediction divergence, and related methods have been applied across several NLP tasks.The cited applications include text classification, machine translation, and natural language understanding.Source: §2 · p. 3

3 Preliminary

The preliminary setup learns sentence representations from unlabeled inputs for downstream use. It fine-tunes BERT-based PLMs on unlabeled corpora and uses the final-layer [CLS] representation for each sentence.

3 Preliminary

  • The task is to learn a representation for each input sentence from an unlabeled corpus for downstream use.The paper gives semantic textual similarity as an example downstream task.Source: §3 · p. 3
  • The paper denotes each sentence representation as h_i = f(x_i), with f a parameterized function.Source: §3 · p. 3
  • The setup fine-tunes BERT-based PLMs on unlabeled text and takes each sentence’s final-layer [CLS] representation as its representation.Source: §3 · p. 3

4 Approach

DCLR reduces negative-sampling bias by combining noise-based negatives with instance weighting. Its pipeline generates and adjusts negatives, weights them using a complementary model, and uses them in contrastive learning with augmented positives.

4 Approach

  • DCLR pairs noise-based negatives to address anisotropy with instance weighting to reduce the influence of false negatives.The framework combines these interventions with augmented positives in contrastive learning.Source: §4 · p. 3, §4.3.2 · p. 5

4.1 Generating Noise-based Negatives

  • Gaussian-initialized noise negatives are updated by gradient ascent on a non-uniformity objective to target non-uniform points in the semantic space.The paper states that contrasting with these negatives can further improve representation-space uniformity.Source: §4.1 · pp. 3–4

4.2 Contrastive Learning with Instance Weighting

  • A complementary model scores negatives, and those above the similarity threshold receive weight 0 in the debiased contrastive loss.The framework uses SimCSE as the complementary model and combines weighted negatives with dropout-augmented positives.Source: §4.2 · p. 4

4.3.1 Overview of DCLR

  • DCLR generates and adjusts Gaussian noise negatives, mixes them with in-batch negatives, weights the set with SimCSE, then applies contrastive learning to weighted negatives and dropout-augmented positives.The overview describes this as a three-step pipeline.Source: §4.3.1 · p. 5

4.3.2 Discussion

  • The authors state that noise negatives broaden the representation space beyond PLM-derived anisotropic representations, while complementary-model weighting detects and punishes semantically similar false negatives.The approach is described as agnostic to positive-data augmentation methods such as token cutoff and dropout.Source: §4.3.2 · p. 5

5 Experiment - Main Results

On seven standard STS tasks, DCLR outperforms baselines in most settings. The experiments compare unsupervised sentence representation methods using Spearman correlation.

5.1 Experiment Setup

  • Experiments evaluate sentence embeddings on seven standard STS tasks with SentEval, cosine similarity, and Spearman correlation.The tasks include STS 2012–2016, STS Benchmark, and SICK-Relatedness.Source: §5.1 · p. 5

5.2 Main Results

  • DCLR outperforms all baselines in most settings, including contrastive-learning approaches.Table 1 reports sentence embedding performance on STS tasks using Spearman correlation.Source: Table 1 · p. 6, §5.2 · p. 7
  • BERT-base and RoBERTa-base are selected to test the framework’s effectiveness on pre-trained language models.The methods are compared on seven STS tasks.Source: §5.2 · p. 7
  • GloVe and USE mostly outperform native PLM representations, while USE also outperforms GloVe.The paper attributes native PLM results to anisotropy and gives a potential explanation for USE’s advantage over GloVe.Source: §5.2 · p. 7
  • Contrastive approaches outperform other baselines in most cases, and SimCSE is the strongest baseline.The paper attributes SimCSE’s result to dropout rarely hurting positive-pair semantics.Source: §5.2 · p. 7

6 Experiment - Analysis and Extension

The analysis tests DCLR across positive-augmentation strategies, component ablations, uniformity, data scarcity, and hyperparameter settings. The findings support its use across several augmentations and show distinct effects from its two negative-sampling components and settings.

6.1 Debiased Contrastive Learning on Other Methods

  • DCLR improves performance with token shuffling, cutoff, and dropout augmentations; dropout performs best among these variants.Only the negative-sampling strategy was revised across the variants.Source: §6.1 · pp. 7–8

6.3 Uniformity Analysis

  • DCLR’s uniformity loss is lower than SimCSE’s through almost all training and decreases faster, while SimCSE shows no significant downward trend.The authors suggest noise-based negatives that extend beyond the representation space may explain the pattern.Source: Figure 4 · p. 8, §6.3 · p. 8

6.2 Ablation Study

  • Removing either instance weighting or noise-based negatives degrades average Spearman correlation across seven STS tasks, with a larger drop when weighting is removed.Random Noise, Knowledge Distillation, and Self Instance Weighting also perform worse than the original DCLR.Source: §6.2 · p. 8

6.4 Performance under Few-shot Settings

  • With 0.3% of training data, DCLR’s performance drops by 9% on STS-B and 4% on SICK-R, and results remain stable across data proportions.The few-shot experiments use BERT-base and evaluate on STS-B and SICK-R.Source: Figure 5 · p. 8, §6.4 · p. 8

6.5 Hyper-parameters Analysis

  • On STS-B, very high or low weighting thresholds φ can reduce performance, whereas SICK-R is insensitive to φ; performance improves when noise-based negatives approach batch size.The authors associate the STS-B threshold sensitivity with ineffective punishment at high φ and possible misjudgment of true negatives at low φ.Source: §6.5 · p. 9, Figure 6 · p. 9

7 Conclusion

The conclusion presents DCLR as a framework for reducing sampling bias in unsupervised sentence representation learning, using instance weighting and noise-based negatives. Experiments on seven STS tasks outperform several competitive baselines; multilingual and multimodal applications are left for future work.

7 Conclusion

  • DCLR addresses bias from random negative sampling with instance weighting to punish false negatives and noise-based negatives to mitigate anisotropy in PLM-derived representations.The framework targets unsupervised sentence representation learning.Source: §7 · p. 9
  • On seven STS tasks, DCLR outperforms several competitive baselines.Source: §7 · p. 9
  • Future work will explore other ways to reduce contrastive-learning bias and apply the method to multilingual or multimodal representation learning.Source: §7 · p. 9

Ethical Consideration

The authors flag potential bias inherited from pre-trained language models as an ethical concern.

Ethical Consideration

  • Pre-trained language models may capture biases from their pre-training data, creating a potential bias concern for this approach.Source: Ethical Consideration · p. 9

Related papers

Loading 2205.00656v1…