Source-linked AI summary
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljačić, Shang-Wen Li, Wen-tau Yih, Yoon Kim, James Glass
TL;DR
Sentence embeddings should capture rich semantics across tasks, yet direct augmentations can alter sentence meaning and conflict with invariance-based contrastive learning. DiffCSE combines dropout contrastive learning with conditional prediction of differences from MLM-based edits. It improves over unsupervised SimCSE on STS and transfer-task evaluations, with reported gains across BERT and RoBERTa settings.
Problem
Learning universal sentence representations that capture rich semantics and perform across downstream NLP tasks without task-specific finetuning remains an open issue, while direct augmentations can change sentence meaning.
Method
DiffCSE combines dropout-based contrastive learning with a conditional difference-prediction objective for MLM-based word replacement.
Results
DiffCSE improves over SimCSE on STS and transfer tasks, raising averaged BERTbase STS Spearman correlation from 76.25% to 78.49%.
Takeaways & Limitations
DiffCSE shows that sentence representations can be trained to remain insensitive to dropout while being aware of MLM-based word replacement.
Abstract
from arXiv · showhide
We propose DiffCSE, an unsupervised contrastive learning framework for learning sentence embeddings. DiffCSE learns sentence embeddings that are sensitive to the difference between the original sentence and an edited sentence, where the edited sentence is obtained by stochastically masking out the original sentence and then sampling from a masked language model. We show that DiffSCE is an instance of equivariant contrastive learning (Dangovski et al., 2021), which generalizes contrastive learning and learns representations that are insensitive to certain types of augmentations and sensitive to other "harmful" types of augmentations. Our experiments show that DiffCSE achieves state-of-the-art results among unsupervised sentence representation learning methods, outperforming unsupervised SimCSE by 2.3 absolute points on semantic textual similarity tasks.
1 Introduction
DiffCSE addresses the challenge of learning universal sentence representations by making embeddings sensitive to meaning-changing surface edits while retaining contrastive learning for insensitive transformations. It reports improvements over unsupervised SimCSE on semantic textual similarity tasks.
- Universal sentence representations must capture rich semantic information and perform across downstream NLP tasks without task-specific finetuning.
- Contrastive learning can learn useful sentence embeddings without labeled data, but direct word deletions or replacements may change sentence meaning.
- DiffCSE combines dropout-based contrastive learning with an additional loss predicting the difference between original and MLM-edited sentences.The edited sentence is produced through stochastic masking and masked-language-model replacement.
- 2.3% absolute improvement on STS datasets over SimCSE, the previous state-of-the-art model, is reported for DiffCSE.
2 Background and Related Work
DiffCSE extends contrastive learning through equivariance: representations remain insensitive to dropout but encode differences introduced by MLM-based word replacement. The framework adapts this principle from computer vision to sentence embeddings and uses it to study useful NLP transformations.
- Related Work: Prior sentence-embedding work includes unsupervised and supervised universal representation methods, followed by contrastive approaches such as SimCSE.
- Contrastive Learning: Contrastive learning encourages encoder representations to be invariant to selected input transformations, but useful transformations differ across vision and NLP.
- Equivariant Contrastive Learning: Equivariance generalizes invariance by allowing a transformation T on inputs to induce a corresponding transformation T′ on output features.
- Equivariant Contrastive Learning: DiffCSE treats dropout as an insensitive transformation and MLM-based replacement as a sensitive transformation, using conditional difference prediction to encode the latter.
- Contribution: The authors present equivariant contrastive learning as applicable beyond computer vision, including sentence difference operations without algebraic structures.
3 Difference-based Contrastive Learning
DiffCSE combines SimCSE-style contrastive learning with a conditional difference-prediction objective, training sentence embeddings to capture distinctions between original and edited sentences.
- DiffCSE combines the standard SimCSE contrastive objective with a difference-prediction objective conditioned on the sentence embedding.
- Given an unlabeled sentence, SimCSE forms a positive pair by applying different dropout masks and represents the sentence as h = f(x).
- A random mask and fixed pretrained masked language model produce an edited sentence, after which a discriminator detects replaced tokens.
- DiffCSE jointly optimizes contrastive and replaced-token-detection losses using weighting coefficient λ, then discards the discriminator and retains the sentence encoder for evaluation.
- The discriminator is conditional because it uses the fixed-dimensional sentence embedding h, allowing gradients to encourage h to encode enough information to distinguish x from x′′.
4 Experiments
The experiments evaluate DiffCSE on unsupervised STS and transfer tasks using BERT and RoBERTa encoders, finding consistent improvements over SimCSE.
- Evaluation: DiffCSE is evaluated on 7 unsupervised semantic textual similarity tasks and 7 SentEval transfer tasks with frozen sentence embeddings.
- Semantic Textual Similarity (STS): 78.49% average Spearman correlation is achieved by DiffCSE-BERTbase on STS, compared with 76.25% for SimCSE-BERTbase.
- Semantic Textual Similarity (STS): 77.80% average Spearman correlation is achieved by DiffCSE-RoBERTabase on STS, compared with 76.57% for SimCSE-RoBERTabase.
- Transfer Tasks: 86.86% average transfer-task performance is achieved by DiffCSE-BERTbase, improving over SimCSE-BERTbase at 85.56%.
- Transfer Tasks: 87.04% average transfer-task performance is achieved by DiffCSE-RoBERTabase, improving over SimCSE-RoBERTabase at 84.84%.
- Transfer Tasks: DiffCSE uses 115MB of Wikipedia data, whereas CMLM reports higher transfer performance using 1TB of Common Crawl data.
5 Ablation Studies
Ablations show that DiffCSE depends on combining contrastive and difference-prediction objectives, while MLM replacement and carefully balanced settings generally outperform alternatives.
- Objectives: Removing contrastive loss drops STS-B performance by 30% and average transfer-task scores by 2%.The contrastive and RTD losses maintain insensitive and sensitive representations, respectively.
- Objectives: Using the next sentence for conditioning significantly decreases STS-B performance, while transfer performance remains similar.Combining same- and next-sentence conditioning does not improve results.
- Objectives: Conditional MLM does not improve STS-B or transfer performance over DiffCSE, while corrective language modeling significantly decreases STS-B performance.Next-sentence conditional MLM performs even worse on STS-B but slightly better on transfer tasks than same-sentence conditional MLM.
- Augmentation Methods: MLM replacement slightly outperforms insertion or deletion on STS-B, while combining all three augmentations provides no improvement.Insertion randomly adds mask tokens, whereas deletion removes 15% of tokens; both use prediction-based auxiliary tasks.
- Pooler Choice: A two-layer BatchNorm pooler is better than BERT’s original one-layer tanh pooler for extracting contrastive features.The ablation compares DiffCSE and SimCSE with and without BatchNorm.
- Generator, Masking Ratio, and λ: Transfer-task performance changes little across generators, but STS-B decreases with smaller generators and is best with DistilBERTbase.Masking ratios between 15% and 40% differ little, with around 30% giving the best performance; λ = 0.005 performs best.
6 Analysis
DiffCSE improves sentence retrieval by distinguishing subtle semantic differences that SimCSE can miss, while producing embedding similarities aligned with human ratings. Its representation-space behavior differs from SimCSE, with the improvement attributed to better alignment.
- 6.1 Qualitative Study: DiffCSE retrieves the correct top-ranked sentence when SimCSE confuses nearly identical wording or opposite meanings, though both fail on double negation.Examples include distinguishing “you can do it, too” from “you can use it, too” and resolving a semantic contrast despite less similar wording.
- 6.2 Quantitative Retrieval Results: DiffCSE’s retrieval evaluation uses 2,758 STS-B test sentences and 97 human-labeled perfect-similarity pairs, measuring recall@1/5/10.For each positive pair, one sentence retrieves the other, and success is recorded at top-1, top-5, and top-10 rankings.
- 6.3 Representation Space Analysis: Both SimCSE and DiffCSE assign cosine similarities consistent with human ratings across five STS-B split groups.Figure 2 plots cosine similarity distributions across groups defined by human ratings.
- 6.3 Representation Space Analysis: For the same human rating, DiffCSE assigns slightly higher cosine similarities than SimCSE, possibly reflecting representation-space squeezing through conditional ELECTRA training.The authors relate this effect to pretrained Transformer models’ tendency to squeeze representations to fit ELECTRA’s input distribution.
- 6.3 Representation Space Analysis: SimCSE and DiffCSE optimize the representation space in different directions, and the authors suggest DiffCSE’s improvement may come from better alignment.Alignment and uniformity are measured on the STS-B test set, with smaller values better and averaged STS scores also reported.
7 Conclusion
The paper concludes that DiffCSE is an effective unsupervised sentence-embedding framework that uses augmentation to model sensitivity rather than invariance. Its evaluations and retrieval analyses support improved embeddings, while supervised training remains unexplored.
- 7 Conclusion: DiffCSE is an unsupervised sentence-embedding framework aware of, but not invariant to, MLM-based word replacement.The framework is presented as a new way to use augmentations for natural-language representation learning.
- 7 Conclusion: Empirical results on semantic textual similarity and transfer tasks show DiffCSE’s effectiveness compared with current state-of-the-art sentence-embedding methods.The paper also reports extensive ablations and qualitative retrieval analyses supporting its modeling choices and embedding space.
- 7 Conclusion: DiffCSE produces a better sentence-retrieval embedding space according to the paper’s qualitative study and retrieval results.The conclusion connects the retrieval evidence with the framework’s broader embedding-quality claims.
- 7 Conclusion: The paper does not explore the supervised setting using human-labeled NLI datasets to further boost performance, leaving it for future work.This is the stated limitation and future-work direction.
A Training Details
Training uses a single 2080Ti GPU, grid-searched hyperparameters, development-set checkpoint selection, and a two-model training setup whose inference size matches SimCSE.
- Hardware and Hyperparameters: DiffCSE experiments use a single NVIDIA 2080Ti GPU, with averaged running times of 3–6 hours.The reported grid search varies batch size, learning rate, masking ratio, and λ.
- Hyperparameter Search: The grid search covers batch sizes {64, 128}, learning rates {2e-6, 3e-6, 5e-6, 7e-6, 1e-5}, masking ratios {0.15, 0.20, 0.30, 0.40}, and λ values {0.1, 0.05, 0.01, 0.005, 0.001}.SimCSE’s temperature τ is fixed at 0.05.
- Model Selection: Training saves the checkpoint with the highest STS-B development-set score, using that development set to select hyperparameters.The supplied training-details passages identify STS-B development performance as the selection criterion.
- Testing: At testing, DiffCSE discards the MLP projector and extracts sentence embeddings from the [CLS] output, following SimCSE.This describes the inference representation rather than the full training architecture.
- Model Size: DiffCSE trains a sentence encoder and discriminator together, but testing uses only the sentence encoder, so its model size matches SimCSE.The two-model requirement applies during training; the deployed inference model has the same size as SimCSE.
- Projector: The appendix includes a projector with BatchNorm as the model’s final layer and provides its PyTorch structure.The supplied code passage begins with a linear layer specification.
B Using Augmentations as Positive/Negative Examples
Using insertion, deletion, or replacement augmentations as additional positives generally hurts performance, while MLM-replaced examples as additional negatives help only slightly and remain below DiffCSE’s proposed approach.
- Positive Examples: Adding insertion, deletion, or replacement augmentations as additional positive examples decreases performance.These augmentations are evaluated alongside the SimCSE training paradigm.
- Negative Examples: MLM 15% replaced examples are the only tested augmentation that slightly improves performance when used as additional negative examples.The improvement is described as small relative to the alternatives.
- Comparison with DiffCSE: None of the tested augmentation configurations performs better than DiffCSE’s proposed equivariance-based method.The comparison covers the replace, insert, and delete augmentation strategies evaluated in the appendix.
C Uniformity and Alignment
Alignment and uniformity are proposed measures of sentence-representation quality: alignment evaluates paired-sentence proximity, while uniformity evaluates distribution across representation space.
- Alignment computes the expected distance between normalized embeddings of paired sentences.
- Uniformity measures how well embeddings are uniformly distributed in the representation space.
- Smaller alignment and uniformity values indicate a higher-quality representation space.
- The model is built using a PyTorch implementation of SimCSE based on HuggingFace Transformers.
- The authors provide code and pretrained models and direct readers to README instructions for reproducing results.
E Potential Risks
The paper identifies a potential risk from relying on pretrained language models: DiffCSE may inherit and propagate their harmful biases.
- DiffCSE may inherit and propagate harmful biases present in the pretrained language models it uses.