SimCSE: Simple Contrastive Learning of Sentence Embeddings

Tianyu Gao; Xingcheng Yao; Danqi Chen

Source-linked AI summary

SimCSE: Simple Contrastive Learning of Sentence Embeddings

Tianyu Gao, Xingcheng Yao, Danqi Chen

arXiv:2104.08821v4Apr 18, 2021cs.CLcs.LG

TL;DR

Learning strong sentence embeddings from unlabeled or labeled data remains challenging. SimCSE applies contrastive learning with dropout noise or NLI supervision, substantially improving STS performance to 76.25% unsupervised and 81.57% supervised averaged Spearman’s correlation.

Problem
The paper addresses how to learn superior sentence embeddings from either unlabeled or labeled data.
Method
SimCSE uses contrastive learning with dropout-generated views for unlabeled sentences and NLI entailment positives plus contradiction hard negatives for labeled data.
Results
Across seven STS tasks, unsupervised SimCSE-BERTbase reaches 76.25% averaged Spearman’s correlation and supervised SimCSE-BERTbase reaches 81.57%.
Takeaways & Limitations
Contrastive learning improves sentence embeddings by promoting uniform representations, while NLI supervision further improves alignment between positive pairs.
Takeaways & Limitations
The evaluation does not compare SimCSE with CLEAR because the authors use different pretrained models and report lower numbers.

Abstract

from arXiv · show

This paper presents SimCSE, a simple contrastive learning framework that greatly advances state-of-the-art sentence embeddings. We first describe an unsupervised approach, which takes an input sentence and predicts itself in a contrastive objective, with only standard dropout used as noise. This simple method works surprisingly well, performing on par with previous supervised counterparts. We find that dropout acts as minimal data augmentation, and removing it leads to a representation collapse. Then, we propose a supervised approach, which incorporates annotated pairs from natural language inference datasets into our contrastive learning framework by using "entailment" pairs as positives and "contradiction" pairs as hard negatives. We evaluate SimCSE on standard semantic textual similarity (STS) tasks, and our unsupervised and supervised models using BERT base achieve an average of 76.3% and 81.6% Spearman's correlation respectively, a 4.2% and 2.2% improvement compared to the previous best results. We also show -- both theoretically and empirically -- that the contrastive learning objective regularizes pre-trained embeddings' anisotropic space to be more uniform, and it better aligns positive pairs when supervised signals are available.

1 Introduction

SimCSE is a contrastive sentence-embedding framework that works with unlabeled or labeled data and improves representations through simple dropout-based learning and NLI supervision. Across seven STS tasks, its BERTbase models achieve substantial gains over previous best results.

Framework: SimCSE applies contrastive learning with pre-trained language models such as BERT or RoBERTa to produce sentence embeddings from unlabeled or labeled data.The framework is presented as simple and superior to existing sentence-embedding approaches.
Unsupervised SimCSE: Unsupervised SimCSE creates positive pairs by encoding the same sentence twice with independent standard-dropout masks, while other minibatch sentences serve as negatives.The model predicts the positive embedding among in-batch negatives.
Unsupervised SimCSE: Dropout acts as minimal hidden-representation augmentation, enabling unsupervised SimCSE to outperform next-sentence objectives and discrete augmentation while matching prior supervised methods.The passage attributes the method’s effectiveness to dropout-based noise and contrasts it with word deletion or replacement.
Supervised SimCSE: Supervised SimCSE uses NLI entailment pairs as positives and contradiction pairs with other in-batch instances as negatives in contrastive learning.The approach incorporates annotated sentence pairs rather than treating NLI solely as a three-way classification task.
Representation Analysis: Contrastive learning improves embedding quality by increasing uniformity while avoiding degenerated alignment through dropout noise in the unsupervised setting.The analysis evaluates alignment of semantically related pairs and uniformity of the overall representation space.
Evaluation: 76.3% and 81.6% averaged Spearman’s correlation are achieved by the unsupervised and supervised BERTbase models, respectively, on seven STS tasks.These results improve previous best results by 4.2% and 2.2%, respectively.

2 Background: Contrastive Learning

Contrastive learning learns representations by bringing semantically related examples together and separating unrelated ones through an in-batch-negative objective. Its representation quality can be understood through alignment of positive pairs and uniformity of embeddings over the hypersphere.

Contrastive objective: Contrastive learning pulls semantically close neighbors together while pushing non-neighbors apart.The framework assumes paired examples and learns representations from their semantic relationships.
Contrastive objective: The training objective uses cross-entropy with in-batch negatives, temperature scaling, and cosine similarity between encoded representations.Input sentences are encoded with a pretrained language model, then all parameters are fine-tuned using the contrastive objective.
Positive instances: Constructing positive pairs is difficult in NLP because discrete data augmentation can alter sentence meaning.Prior augmentations include word deletion, reordering, and substitution, whereas standard dropout on intermediate representations performs better.
Alignment and uniformity: Alignment measures the distance between positive-pair embeddings, while uniformity measures how evenly embeddings spread across the hypersphere.These metrics correspond to keeping positive instances close and scattering random instances, and are used to analyze the proposed approaches.

3 Unsupervised SimCSE

Unsupervised SimCSE trains on identical sentence pairs whose embeddings differ only through independently sampled standard dropout masks. This minimal augmentation outperforms discrete alternatives and next-sentence objectives, while removing or synchronizing dropout causes severe degradation.

Method: SimCSE feeds each sentence to the encoder twice and uses independently sampled dropout masks to create positive pairs.It adds no dropout beyond the standard Transformer dropout mask.
Dropout noise as data augmentation: Deleting even one word hurts performance, and none of the discrete augmentations outperforms dropout noise.The comparison includes crop, word deletion, synonym replacement, and masked-language-model replacement.
Objective comparison: 82.5 vs 67.4 on STS-B: SimCSE substantially outperforms next-sentence objectives, with one shared encoder outperforming two independent encoders.The result is reported on the STS-B development set using Spearman’s correlation.
Alignment and uniformity: Training from pre-trained checkpoints greatly improves uniformity across models, while the alignment and uniformity metrics provide the analysis framework for comparing dropout noise and discrete augmentation.Lower values are better for both metrics.

4 Supervised SimCSE

Supervised SimCSE directly uses sentence pairs from labeled datasets in its contrastive objective, selecting NLI entailment pairs as the strongest positives and contradiction pairs as hard negatives. Adding hard negatives further improves performance to 86.2, defining the final supervised model.

Choices of labeled data: Supervised SimCSE directly takes sentence pairs from labeled datasets and optimizes the contrastive objective.Candidate datasets include QQP, Flickr30k, ParaNMT, SNLI, and MNLI.
Choices of labeled data: Among the evaluated datasets, SNLI + MNLI entailment pairs perform best when training-pair counts are matched.Their advantage is attributed to high-quality crowd-sourced pairs and manually written hypotheses.
Choices of labeled data: 39% lexical overlap characterizes SNLI + MNLI entailment pairs, versus 60% for QQP and 55% for ParaNMT.The overlap is measured by F1 between two bags of words.
Contradiction as hard negatives: NLI contradiction hypotheses provide hard negatives accompanying each premise–entailment pair.Annotators write entailment, neutral, and contradiction hypotheses for the same premise.
Contradiction as hard negatives: 84.9 →86.2 after adding hard negatives, while adding ANLI or combining with unsupervised SimCSE yields no meaningful improvement.A dual encoder instead hurts performance, changing 86.2 →84.2.

5 Connection to Anisotropy

This section explains that sentence embeddings suffer from anisotropy, concentrating in a narrow cone, and shows that contrastive learning alleviates this problem by flattening their singular spectrum. Unlike post-processing methods, SimCSE also aligns positive pairs, which contributes to its effectiveness.

Motivation: Anisotropy confines learned embeddings to a narrow cone, limiting their expressiveness; singular values also decay sharply beyond a few dominant components.These findings motivate methods that make representations more isotropic.
Contrastive objective: Contrastive learning alleviates anisotropy by pushing negative instances apart, improving uniformity and making sentence representations more isotropic.The paper demonstrates this effect both theoretically and empirically.
Theoretical explanation: The contrastive objective flattens the singular spectrum by reducing the top eigenvalue of the sentence-embedding Gram matrix while its trace remains constant.With normalized embeddings, minimizing the negative-pair term reduces an upper bound on the sum of pairwise similarities, thereby lowering the largest eigenvalue.
Relation to prior methods: Unlike post-processing methods that target isotropy alone, contrastive learning also aligns positive pairs through the objective’s first term.The paper identifies this joint alignment and uniformity optimization as key to SimCSE’s success.

6 Experiment

SimCSE is evaluated primarily on seven STS tasks under a unified Spearman-correlation setting, where both unsupervised and supervised variants substantially outperform prior sentence-embedding methods. Ablations further show that α = 1 is the best hard-negative weighting, while neutral hypotheses provide no additional gains.

Evaluation setup: Seven STS tasks are the main evaluation because sentence embeddings aim to cluster semantically similar sentences.The STS experiments are fully unsupervised, including supervised SimCSE evaluation without using STS training sets.
Evaluation setup: The evaluation follows Reimers and Gurevych’s setting: no additional regressor, Spearman’s correlation, and “all” aggregation.The paper identifies comparison differences involving regressors, correlation types, and aggregation, and calls for a unified setting.
Main results: 76.25% average Spearman’s correlation raises the previous best from 72.05% for unsupervised SimCSE-BERTbase, while supervised SimCSE-BERTbase reaches 81.57%.SimCSE improves results across all seven STS datasets and remains competitive with supervised baselines even without NLI supervision.
Main results: 83.76% is achieved by supervised SimCSE with RoBERTalarge, exceeding the corresponding BERTbase result and showing larger gains with RoBERTa encoders.The paper also reports on-par or better transfer-task performance and additional gains from an auxiliary MLM objective in Appendix E.
Ablations: α = 1 performs best among hard-negative weighting policies, while using neutral hypotheses as hard negatives provides no further gains.These ablations are evaluated on the STS-B development set.

7 Analysis

The analysis shows that SimCSE improves sentence-embedding geometry by balancing uniformity and alignment, while its retrieval results qualitatively surpass SBERT. Supervised signals further improve alignment, whereas unsupervised SimCSE mainly improves uniformity while preserving alignment.

Uniformity and alignment: Models with better alignment and uniformity generally achieve better STS performance.This confirms the relationship between embedding geometry and sentence-representation quality.
Uniformity and alignment: Pre-trained embeddings have good alignment but poor uniformity, reflecting a highly anisotropic embedding space.BERT-flow and BERT-whitening improve uniformity but cause alignment degeneration.
Uniformity and alignment: Unsupervised SimCSE improves pre-trained embeddings’ uniformity while preserving good alignment, and supervised data further improves alignment.The analysis also reports that SimCSE can effectively flatten singular values.
Qualitative comparison: In a Flickr30k retrieval experiment, SimCSE-BERTbase produced higher-quality retrieved sentences than SBERTbase.The experiment used 150k captions and cosine similarity to retrieve sentences for random queries.

8 Related Work

Related work spans distributional and n-gram-based sentence embeddings, contrastive objectives using alternative views of sentences, supervised learning from semantic-pair data, and regularization against representation degeneration.

Early sentence-embedding methods predict surrounding sentences, following the distributional hypothesis.
Augmenting word2vec with n-gram embeddings was shown to produce strong sentence-embedding results.
Recent approaches use contrastive objectives with different views, including data augmentations or separate model copies, of the same sentence.
Supervised sentence embeddings use NLI, bilingual, or back-translation corpora to learn semantic similarity.
Another research direction regularizes embeddings to alleviate representation degeneration and improve over pre-trained language models.

9 Conclusion

SimCSE is presented as a simple contrastive learning framework that improves sentence embeddings on semantic textual similarity tasks. The paper includes unsupervised and supervised approaches and analyzes alignment and uniformity to explain its behavior.

SimCSE is proposed as a simple contrastive learning framework that improves state-of-the-art sentence embeddings on semantic textual similarity tasks.
The framework includes an unsupervised approach that predicts the input sentence itself with dropout noise and a supervised approach that uses NLI datasets.
The paper analyzes SimCSE and baseline models through the lenses of alignment and uniformity to justify the approach’s inner workings.

A Training Details · B Different Settings for STS Evaluation · C Baseline Models

The authors specify SimCSE’s training choices, clarify inconsistent STS evaluation protocols, and standardize baseline reproduction. They report that learning-rate tuning reduces sensitivity to batch size and that NLI-based postprocessing outperforms target-based variants.

A Training Details: Supervised SimCSE trains for 3 epochs, unsupervised SimCSE for 1 epoch, and both select checkpoints using STS-B development performance.They evaluate every 250 training steps and grid-search batch sizes {64, 128, 256, 512} with learning rates {1e-5, 3e-5, 5e-5}.
A Training Details: SimCSE is not sensitive to batch size when learning rates are tuned accordingly, contrary to claims that contrastive learning requires large batches.The authors attribute this partly to initialization from pretrained checkpoints.
A Training Details: An optional MLM auxiliary loss helps prevent catastrophic forgetting and improves transfer-task performance, but not sentence-level STS performance.The loss is ℓ + λ · ℓmlm, where λ is a hyperparameter.
B Different Settings for STS Evaluation: Evaluation protocols across prior sentence-embedding papers are very incoherent, motivating calls for unified settings and released evaluation code.Differences include additional regressors, reported metrics, and aggregation methods.
B Different Settings for STS Evaluation: The authors use raw sentence embeddings with cosine similarity, report Spearman’s rank correlation, and generally default to the “all” aggregation setting.The “all” setting concatenates topics, whereas “mean” and “wmean” average subset results.
C Baseline Models: Baseline results combine reported numbers, downloaded checkpoints, reevaluated models, retrained implementations, and reproduced whitening results depending on each method.This procedure covers GloVe, InferSent, Universal Sentence Encoder, BERT, RoBERTa, SBERT, SRoBERTa, DeCLUTR, contrastive tension, BERT-flow, and BERT-whitening.
C Baseline Models: “Target” postprocessing is generally worse than “NLI” for both BERT-flow and BERT-whitening, so the main results report only the NLI variant.The two variants respectively use NLI data or learn embedding distributions on target sets.

D Ablation Studies

The ablations examine similarity normalization, temperature, and the MLM auxiliary objective. Cosine similarity performs better than dot product at τ = 0.05, while MLM modestly improves transfer performance but consistently harms semantic textual similarity.

Normalization and temperature: At the tuned temperature τ = 0.05, cosine similarity outperforms dot product on the STS-B development set.The comparison evaluates both similarity functions across different temperatures.
Normalization and temperature: The ablation evaluates SimCSE with both dot product and cosine similarity under different temperature settings.Performance is measured on the STS-B development set.
MLM auxiliary task: The token-level MLM objective modestly improves averaged transfer-task performance but consistently decreases performance on semantic textual similarity tasks.The study varies the MLM objective’s λ parameter using BERTbase development sets.

E Transfer Tasks · F Distribution of Singular Values · G Cosine-similarity Distribution

Transfer evaluation finds supervised SimCSE competitive, while auxiliary MLM improves transfer performance and post-processing hurts it. Singular-value and cosine-similarity analyses show that SimCSE improves representation uniformity and distinguishes semantic similarity levels effectively.

E Transfer Tasks: Transfer tasks evaluate frozen sentence embeddings with logistic regression across MR, CR, SUBJ, MPQA, SST-2, TREC, and MRPC under SentEval defaults.The transfer results are measured as accuracy, with MLM defined as an auxiliary task using λ = 0.1.
E Transfer Tasks: Supervised SimCSE performs on par with or better than previous approaches, whereas unsupervised-model trends remain unclear.
E Transfer Tasks: Adding the MLM term consistently improves transfer-task performance, suggesting sentence-level objectives may not directly benefit transfer tasks.
E Transfer Tasks: BERT-flow and BERT-whitening hurt transfer performance relative to their base models, despite producing more uniform representations.
F Distribution of Singular Values: SimCSE flattens the singular-value spectrum for both unsupervised and supervised embeddings, unlike vanilla BERT or SBERT, whose singular values drop fastest.BERT-flow and BERT-whitening flatten the spectrum further because they directly target isotropic embeddings.
G Cosine-similarity Distribution: STS-B cosine-similarity analyses compare sentence-pair groups divided by five ground-truth rating levels, with higher ratings indicating greater similarity.
G Cosine-similarity Distribution: Both SimCSE variants better distinguish sentence pairs across similarity levels than baseline models, improving STS performance.SimCSE is more scattered than BERT or SBERT while retaining lower variance for semantically similar pairs than whitened representations.