Source-linked AI summary
ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding
Xing Wu, Chaochen Gao, Liangjun Zang, Jizhong Han, Zhongyuan Wang, Songlin Hu
TL;DR
Unsupervised SimCSE may be biased because its positive pairs preserve identical sentence-length information, which can affect semantic similarity learning. ESimCSE uses word repetition for length-varied positive pairs and momentum contrast for more negatives, achieving higher STS performance than SimCSE, including a +2.02% average Spearman gain on BERT-base.
Problem
SimCSE positive pairs preserve identical sentence-length information while negative pairs generally differ, potentially biasing semantic similarity learning toward sentence length.
Method
ESimCSE combines word repetition for positive-pair construction with momentum contrast to increase negative pairs.
Results
+2.02% average Spearman correlation is ESimCSE’s gain over SimCSE on BERT-base across seven STS-B test sets.
Takeaways & Limitations
ESimCSE improves semantic textual similarity measurement over SimCSE in different model settings.
Takeaways & Limitations
The authors identify future work on improving discrimination among negative pairs and performance on both semantic textual similarity and transfer tasks.
Abstract
from arXiv · showhide
Contrastive learning has been attracting much attention for learning unsupervised sentence embeddings. The current state-of-the-art unsupervised method is the unsupervised SimCSE (unsup-SimCSE). Unsup-SimCSE takes dropout as a minimal data augmentation method, and passes the same input sentence to a pre-trained Transformer encoder (with dropout turned on) twice to obtain the two corresponding embeddings to build a positive pair. As the length information of a sentence will generally be encoded into the sentence embeddings due to the usage of position embedding in Transformer, each positive pair in unsup-SimCSE actually contains the same length information. And thus unsup-SimCSE trained with these positive pairs is probably biased, which would tend to consider that sentences of the same or similar length are more similar in semantics. Through statistical observations, we find that unsup-SimCSE does have such a problem. To alleviate it, we apply a simple repetition operation to modify the input sentence, and then pass the input sentence and its modified counterpart to the pre-trained Transformer encoder, respectively, to get the positive pair. Additionally, we draw inspiration from the community of computer vision and introduce a momentum contrast, enlarging the number of negative pairs without additional calculations. The proposed two modifications are applied on positive and negative pairs separately, and build a new sentence embedding method, termed Enhanced Unsup-SimCSE (ESimCSE). We evaluate the proposed ESimCSE on several benchmark datasets w.r.t the semantic text similarity (STS) task. Experimental results show that ESimCSE outperforms the state-of-the-art unsup-SimCSE by an average Spearman correlation of 2.02% on BERT-base.
1 Introduction
ESimCSE addresses a length-related bias in unsupervised SimCSE by modifying positive-pair construction with word repetition and expanding negative pairs with momentum contrast. On seven STS-B test sets, it improves over SimCSE, including a +2.02% average Spearman correlation gain on BERT-base.
- Problem: SimCSE positive pairs share sentence length, while negative pairs generally differ, allowing length information to distinguish pairs and potentially bias semantic similarity.Transformer positional embeddings encode sentence length, so the contrastive objective may favor sentences with similar lengths.
- Evidence of bias: Across seven semantic textual similarity datasets, prediction differences are higher for sentence pairs whose length difference is ≤3, supporting the hypothesized length bias.The evaluation partitions each dataset by whether sentence-length difference is ≤3 and compares predictions with normalized ground truths.
- Positive-pair optimization: Word repetition randomly duplicates words to change a sentence’s length while preserving its meaning more reliably than random insertion or deletion.Random insertion may add noise, whereas deleting keywords can substantially alter semantics.
- Negative-pair optimization: Momentum contrast increases the number of negative pairs without changing the batch-size setting, using embeddings enqueued from preceding mini-batches.A momentum-updated encoder generates the enqueued embeddings, with dropout disabled for that encoder.
- Results: +2.02% average Spearman correlation on BERT-base is ESimCSE’s gain over SimCSE across seven STS-B test sets.The method also improves similarity measurement in different model settings.
2 Background: SimCSE
SimCSE constructs positive pairs by encoding paired sentences and uses their embeddings in a contrastive objective based on similarity.
- SimCSE represents paired sentences as positive examples for contrastive sentence-embedding learning.
- The method feeds the same sentence to the encoder twice with different dropout masks to obtain two embeddings.
- The contrastive objective compares each sentence embedding with its paired positive embedding and other batch embeddings.
3 Proposed Enhanced SimCSE
ESimCSE enhances SimCSE by modifying positive pairs through sub-word repetition and enlarging negative-pair comparisons with a momentum-updated queue.
- ESimCSE combines word repetition for positive-pair construction with momentum contrast for expanding negative pairs.
- Word Repetition: The repetition mechanism randomly duplicates selected sub-words, with repetition length sampled to diversify sequence extensions.
- Word Repetition: Unlike SimCSE, ESimCSE encodes the original sequence and its repeated counterpart to form a positive pair.
- Momentum Contrast: Momentum contrast reuses embeddings from preceding mini-batches through a fixed-size queue that progressively replaces older entries.
- Momentum Contrast: A momentum-updated encoder keeps queued embeddings consistent despite their being produced at different training steps.
- Momentum Contrast: The ESimCSE loss incorporates queued momentum-encoder embeddings, whose number is controlled by the queue size M.
4 Experiment
The experiments train and evaluate ESimCSE on English Wikipedia and seven STS datasets, comparing it with SimCSE and other baselines. ESimCSE improves over SimCSE across model settings, including a +2.02% BERT-base gain.
- Experimental Setup: Training uses 1-million English Wikipedia sentences, while evaluation covers seven standard semantic textual similarity datasets.
- Experimental Setup: Spearman correlation is used to report sentence-embedding performance on the STS task.
- Experimental Setup: The implementation starts from BERT or RoBERTa base and large checkpoints and follows SimCSE’s training setup.
- Main Results: Table 3 reports sentence-embedding performance across seven semantic textual similarity test sets and compares ESimCSE with multiple baselines.
- Main Results: +2.02% Spearman correlation is ESimCSE’s improvement over SimCSE on BERTbase.
- Main Results: Table 4 isolates improvements that word repetition or momentum contrast brings to SimCSE on STS-B development sets.
5 Ablation Study
The ablation study evaluates ESimCSE’s pair-construction components, sentence-length-extension alternatives, batching strategies, and hyperparameters. Word repetition and momentum contrast improve SimCSE, while ESimCSE reduces length-related bias without introducing a repetition-overlap bias.
- Component ablations: Word repetition and momentum contrast each substantially improve SimCSE, and combining them yields further improvements.
- Sentence-length extension: Sub-word repetition performs best among sentence-length-extension methods, while word repetition also improves performance.Inserting [MASK] improves slightly, whereas inserting stop words decreases the effect.
- Sentence-length extension: Masked prediction improves performance but requires a pretrained masked language model, creating high additional computational overhead.
- Length-based batching: Length-based training buckets do not significantly improve results and sometimes decrease performance, whereas word repetition performs better.The authors attribute this to insufficient contrastive comparison when shuffling is restricted within buckets.
- Length-bias analysis: ESimCSE reduces the average similarity-difference gap between sentence-pair length groups from 1.84 to 0.71.The groups separate pairs with length difference greater than 3 from those with difference at most 3.
- Hyperparameters: The best repetition rate is dup_rate = 0.32; both larger and smaller rates degrade performance.
- Hyperparameters: The optimal momentum queue size is 2.5 times the batch size; smaller or larger queues reduce the effect.Larger queues may contain embeddings produced by increasingly outdated encoder models.
6 Related Work
Prior unsupervised sentence-representation methods learn from sentence structure, surrounding sentences, or word-vector composition. Recent contrastive methods instead create augmented sentence views and optimize their similarity, with ConSERT and SimCSE most closely related to ESimCSE.
- Unsupervised sentence representation learning: Earlier unsupervised methods learn sentence representations from internal sentence structure, surrounding-sentence prediction, or word-vector composition.
- Contrastive learning: Contrastive sentence-embedding methods generate two views of each sentence and bring their representations closer through an effective loss function.
- Related contrastive methods: ConSERT and unsup-SimCSE are identified as the methods most related to ESimCSE.The related contrastive methods use data augmentation strategies to construct alternative sentence views.
7 Conclusion and Future Work
The paper combines optimizations for constructing positive and negative pairs into ESimCSE and reports improvements over SimCSE on standard semantic textual similarity tasks. Future work targets a more refined objective and improved performance on both similarity and transfer tasks.
- Conclusion: ESimCSE combines optimizations for constructing positive and negative pairs with SimCSE.
- Conclusion: ESimCSE achieves considerable improvements over SimCSE on standard semantic textual similarity tasks.
- Future work: Future work will design a more refined objective function to improve discrimination between different negative pairs.
- Future work: Future work will also target performance on both semantic textual similarity and transfer tasks.