Source-linked AI summary
CLEAR: Contrastive Learning for Sentence Representation
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, Hao Ma
TL;DR
Sentence-level objectives are rarely studied in pre-trained language models, motivating CLEAR’s combination of masked language modeling, contrastive learning, and multiple sentence augmentations. CLEAR outperforms RoBERTa across GLUE and SentEval, while experiments examine augmentation effects and sources of improvement.
Problem
Most pre-trained language models emphasize word-level objectives, while sentence-level objectives are rarely studied.
Method
CLEAR combines word-level masked language modeling with sentence-level contrastive learning over augmented sentence pairs, using deletion, substitution, and reordering strategies.
Results
+2.2% absolute improvement on 8 GLUE tasks and +5.7% absolute improvement on 7 SentEval semantic textual similarity tasks compared to RoBERTa.
Takeaways & Limitations
Adequate sentence-level supervision improves robustness, while different augmentations learn different features and gains come from both larger batch size and contrastive loss.
Takeaways & Limitations
The hyperparameters may not be optimal, and it remains unclear whether settings transfer across contrastive objectives and pre-training datasets.
Abstract
from arXiv · showhide
Pre-trained language models have proven their unique powers in capturing implicit language features. However, most pre-training approaches focus on the word-level training objective, while sentence-level objectives are rarely studied. In this paper, we propose Contrastive LEArning for sentence Representation (CLEAR), which employs multiple sentence-level augmentation strategies in order to learn a noise-invariant sentence representation. These augmentations include word and span deletion, reordering, and substitution. Furthermore, we investigate the key reasons that make contrastive learning effective through numerous experiments. We observe that different sentence augmentations during pre-training lead to different performance improvements on various downstream tasks. Our approach is shown to outperform multiple existing methods on both SentEval and GLUE benchmarks.
1 Introduction
CLEAR addresses the limited study of sentence-level objectives in language-model pre-training by combining masked language modeling with contrastive learning and sentence augmentations. It outperforms strong baselines on GLUE and SentEval.
- 1 Introduction: Sentence-level objectives remain relatively uncommon because most pre-trained language models focus on word-level noise restoration.The paper positions CLEAR as a response to this gap in pre-training objectives.
- 1 Introduction: CLEAR combines word-level MLM with sentence-level contrastive learning to align embeddings of different augmentations of the same sentence.The MLM objective captures word-level hidden features, while contrastive learning trains recognition of similar meanings.
- 1 Introduction: CLEAR tests four sentence augmentations: random-word deletion, span deletion, synonym substitution, and reordering.These augmentations are proposed to expand the limited set of text augmentations studied for contrastive learning.
- 1 Introduction: +2.2% absolute improvement on 8 GLUE tasks and +5.7% absolute improvement on 7 SentEval semantic textual similarity tasks compared to RoBERTa.The model also outperforms strong baselines including RoBERTa and BERT on both benchmarks.
2 Related Work
Related work spans sentence representation, pre-trained language models, and contrastive learning. CLEAR differs from prior text-contrastive methods through its augmentation coverage and from DeCLUTR through training setup and input requirements.
- 2 Related Work: Sentence representation methods include pooling word embeddings, reconstructing surrounding sentences, selecting correct contexts, and using BERT’s [CLS] token.Sentence-BERT reported that mean pooling of final-layer BERT outputs marginally outperformed the [CLS] representation.
- 2 Related Work: GPT and BERT established influential pre-trained language-model architectures, while later work explored sequence-to-sequence structures and alternative pre-training objectives.These approaches target language generation, performance improvement, or faster pre-training.
- 2 Related Work: Computer-vision contrastive learning aligns representations of differently augmented images, with studies differing mainly in their definitions of image augmentation.NLP work adapted this idea using back-translation or spans from the same document.
- 2 Related Work: DeCLUTR requires 2048-token documents and trains from existing pre-trained models, whereas CLEAR pre-trains models from scratch for direct comparison with existing models.The long-input requirement limits DeCLUTR to a narrower pre-training data setting.
3 Method
CLEAR is a contrastive-learning framework for sentence representation that combines sentence augmentation, an encoder, projection, and contrastive loss with masked language modeling. It tests deletion, reordering, and synonym substitution to construct augmented sentence pairs and optimize both token- and sentence-level objectives.
- Framework: Each original sentence receives two random augmentations, and the two variants form a positive pair while other minibatch instances serve as negatives.The same augmentation is used for both variants when testing an augmentation’s isolated effect.
- Framework: A transformer encoder represents augmented sentences, while a nonlinear projection head maps those representations into a new space for contrastive learning.The sentence representation uses the manually inserted [CLS] token, while contrastive vectors are produced by the projection head.
- Framework: The contrastive objective sums positive-pair losses across a minibatch using cosine similarity, a temperature parameter, and a positive-pair indicator.The indicator selects positive pairs, and the overall loss aggregates their individual losses.
- Framework: CLEAR combines a sentence-level contrastive learning objective with masked language modeling during language-model pre-training.The combined loss targets both token-level and sentence-level features.
- Sentence Augmentations: The four tested augmentation families are word deletion, span deletion, reordering, and synonym substitution.Deletion replaces selected tokens or spans with [DEL], whereas reordering switches sampled spans and substitution replaces sampled words with synonyms.
- Sentence Augmentations: Span deletion is a word-deletion special case that focuses on consecutive words, while deletion noise is intended to preserve robustness despite possible semantic changes.The implementation collapses consecutive [DEL] tokens into one token to reduce augmentation-specific cues.
4 Experiment
The experiments evaluate CLEAR and baselines on GLUE and SentEval, using multiple sentence augmentations and standard pre-training configurations. CLEAR improves GLUE performance over RoBERTa, while contrastive learning substantially benefits semantic textual similarity.
- Evaluation: Models are evaluated mainly on GLUE’s development set, which covers inference, similarity, sentiment, and linguistic acceptability tasks.The study also evaluates semantic textual similarity tasks on SentEval.
- GLUE results: 2.2%: MLM+del-word and MLM+del-span+reorder improve RoBERTa-base’s average GLUE score by 2.2%.The best model exceeds the baseline by 7.0% on CoLA and 8.0% on RTE, and every task’s best result comes from a proposed model.
- GLUE results: Some augmentation combinations are unstable: MLM+subs, MLM+reorder, and MLM+subs+reorder either converge quickly or suffer gradient explosion.The authors interpret this as evidence that these augmentations are too easy to distinguish.
- SentEval results: 5.7%: Adding the contrastive-learning loss yields a 5.7% improvement over the best baseline on SentEval semantic textual similarity tasks.CLS pooling often outperforms mean pooling for the proposed models because contrastive learning directly updates the [CLS] representation.
5 Discussion
The ablation study separates contrastive-learning effects from batch-size effects and examines how augmentations specialize across downstream tasks. Results indicate that contrastive loss contributes beyond larger batches, while different augmentations learn different features.
- Ablation study: The ablation compares CLEAR against Double MLM and Double-batch RoBERTa baselines to test whether gains arise from contrastive loss or larger batches.Double MLM uses two masking patterns, while Double-batch doubles the batch size.
- Ablation study: The ablation uses the smaller WiKiText-103 corpus because of computational-resource limitations.Models are pre-trained for 500 epochs on 64 NVIDIA Tesla V100 32GB GPUs.
- Ablation study: 0.4%: Double-batch RoBERTa-base increases the average GLUE score by 0.4%, confirming that larger batches benefit representation training.The best baseline still underperforms the best proposed model, indicating that the proposed model does not benefit solely from batch size.
- Different Augmentation Learns Different Features: MLM+subs+del-span performs especially well on similarity and paraphrase tasks, achieving the highest scores on QQP and STS and ranking second on MRPC.The authors associate this pattern with synonym substitution and varied span deletion producing similar-meaning sentence views.
- Different Augmentation Learns Different Features: MLM+del-span performs well on inference tasks including MNLI, QNLI, and RTE.The authors relate span deletion to pre-training on similar sentences, narrowing the gap between the pre-training and downstream tasks.
- Different Augmentation Learns Different Features: Different augmentations learn different features, and some are especially effective for particular downstream tasks.The paper proposes task-specific augmentation or adaptive selection of contrastive objectives as future work.
6 Conclusion
CLEAR’s contrastive sentence-representation methods are effective across GLUE and SentEval, with robustness linked to sentence-level supervision. Experiments also show that augmentation choice, batch size, and contrastive loss each contribute to performance.
- CLEAR’s contrastive sentence-representation methods are effective on GLUE and SentEval benchmarks across diverse pre-training corpora.
- Adequate sentence-level supervision makes the pre-trained model more robust.
- Different sentence augmentations learn different features for the model.
- Performance improvement comes from both larger batch size and contrastive loss.