Source-linked AI summary
Self-Guided Contrastive Learning for BERT Sentence Representations
Taeuk Kim, Kang Min Yoo, Sang-goo Lee
TL;DR
The paper addresses the unresolved problem of deriving general sentence embeddings from BERT without labeled data. It proposes self-guided contrastive fine-tuning that directly refines BERT and adapts NT-Xent without data augmentation. Experiments report stronger representations than competitive baselines, efficient inference, and relative robustness to domain shifts, while design choices remain non-optimal for particular environments.
Problem
The best way to derive general-purpose sentence embeddings from BERT without labeled data remains unclear, while naive [CLS] embeddings can perform poorly.
Method
The method fine-tunes BERT self-supervisedly by using intermediate hidden representations as positive samples, customizing NT-Xent, and avoiding data augmentation.
Results
The method outperforms competitive baselines across diverse sentence-related settings and is efficient at inference and relatively robust to domain shifts.
Takeaways & Limitations
The approach enables the usual [CLS] embedding to serve as a sentence vector without post-processing after training.
Takeaways & Limitations
The method’s layer and pooling choices prioritize simplicity, so task-specific subsets or pooling methods may perform better in particular environments.
Abstract
from arXiv · showhide
Although BERT and its variants have reshaped the NLP landscape, it still remains unclear how best to derive sentence embeddings from such pre-trained Transformers. In this work, we propose a contrastive learning method that utilizes self-guidance for improving the quality of BERT sentence representations. Our method fine-tunes BERT in a self-supervised fashion, does not rely on data augmentation, and enables the usual [CLS] token embeddings to function as sentence vectors. Moreover, we redesign the contrastive learning objective (NT-Xent) and apply it to sentence representation learning. We demonstrate with extensive experiments that our approach is more effective than competitive baselines on diverse sentence-related tasks. We also show it is efficient at inference and robust to domain shifts.
1 Introduction
Deriving general-purpose sentence embeddings from BERT without labeled data remains unresolved, with performance highly sensitive to layer and pooling choices. The proposed self-guided contrastive method addresses this by refining BERT directly without data augmentation.
- Without labeled datasets, the best strategy for deriving BERT sentence embeddings remains unclear, and naive [CLS] representations perform poorly.
- 16.71 to 63.19 Spearman correlation (× 100) was observed across BERT(-base) layer and pooling combinations on STS-B.The lowest score was [CLS] at layer 10, while the highest was max pooling at layer 2.
- Self-guided training recycles intermediate BERT representations as positive samples for the final sentence embedding.The method customizes NT-Xent for sentence representation learning and avoids data augmentation.
- The method outperforms competitive baselines across varied environments while improving inference efficiency and robustness to domain shifts.
2 Related Work
Prior sentence-representation methods use contrastive learning or pooling-based adaptations of pre-trained Transformers, often requiring augmentation, supervision, or post-training processing. This work instead revises BERT for sentence embeddings using plain text and the [CLS] vector.
- Contrastive sentence-learning methods commonly require data augmentation, such as back-translation, or prior knowledge about training data.
- Computer-vision contrastive learning provides the NT-Xent framework that this work optimizes for pre-trained Transformer sentence representations.
- SBERT derives embeddings through mean pooling of BERT’s last layer and fine-tunes pooled vectors on natural language inference datasets.
- Other approaches apply pooling or distributional post-processing, whereas this work refines BERT so its [CLS] vector functions as a sentence embedding.
3 Method
The method fine-tunes BERT without labeled data or data augmentation by using fixed intermediate representations as self-guidance for sentence embeddings. It also redesigns NT-Xent to focus training on final sentence vectors and multiple intermediate views.
- 3.1 Contrastive Learning with Self-Guidance: Self-guided contrastive learning uses intermediate BERT hidden representations as positive samples for the final sentence embedding, avoiding data augmentation.These internal representations serve as sentence-specific pivots that the final vectors should approach or move away from.
- 3.1 Contrastive Learning with Self-Guidance: The framework clones BERT into fixed BERTF and trainable BERTT so the training signal does not degenerate during optimization.BERTT is fine-tuned to construct sentence embeddings, while BERTF remains fixed throughout training.
- 3.1 Contrastive Learning with Self-Guidance: For each sentence, the fixed model produces token-level representations across layers, which are pooled into sentence-level views and sampled as guidance vectors.The method uses max pooling and uniform sampling by default; each sampled view is another representation of the same sentence.
- 3.1 Contrastive Learning with Self-Guidance: The final sentence vector is the last-layer [CLS] embedding, while a projection head computes similarities used by the contrastive objective.The similarity function is cosine similarity, and the projection head is an MLP; the matching function pairs each final vector with its corresponding intermediate view.
- 3.2 Learning Objective Optimization: The optimized loss attracts each sentence vector to views of the same sentence and separates it from vectors and views associated with other sentences.A regularizer penalizes excessive distance between BERTF and BERTT, while multiple intermediate views diversify the training signals.
- 3.2 Learning Objective Optimization: The objective is modified to prioritize the final sentence vector, remove an intermediate-view interaction, discard another insignificant factor, and incorporate multiple views.The paper reports that these design decisions improve performance.
4 Experiments
The experiments evaluate self-guided contrastive BERT sentence embeddings across semantic-textual-similarity, multilingual, and SentEval settings, comparing them with unsupervised baselines. Across these evaluations, SG and SG-OPT generally outperform competitive methods, while inference is efficient and supervised fine-tuning remains comparable.
- Semantic Textual Similarity Tasks: The study evaluates BERT sentence embeddings on seven STS datasets, measuring agreement with gold similarity scores using Spearman correlation (× 100).The datasets include STS-B, SICK-R, and STS12-16, each containing sentence pairs with scores from 0 to 5.
- Semantic Textual Similarity Tasks: SG and SG-OPT generally outperform competitive baselines on STS tasks, while naïve [CLS] embeddings and mean pooling are inferior to more sophisticated methods.SG-OPT generally performs better than SG, supporting the effectiveness of optimizing the learning objective.
- Multilingual STS Tasks: MBERT refined with the proposed method outperforms or matches the baseline on English, Spanish, and Arabic multilingual STS evaluations.The method uses English STS-B data for training and evaluates cross-lingual transfer on Spanish, Arabic, and English datasets.
- Multilingual STS Tasks: Cross-lingual transfer is more promising for languages analogous to English, such as Spanish, than for distant languages such as Arabic.The authors report that effectiveness may shrink on distant languages.
- SentEval and Supervised Fine-tuning: SG-OPT improves BERT-like models over usual mean pooling on SentEval and outperforms WK pooling on BERT-base and BERT-large.It is comparable with WK pooling on SBERT-base, suggesting partially overlapping benefits with SBERT training and an additional contribution from WK pooling.
- SentEval and Supervised Fine-tuning: SG-OPT produces comparable performance to original BERT on GLUE validation experiments, indicating little effect on supervised fine-tuning.The reported comparison covers five GLUE subtasks; improvements appear on three tasks.
5 Analysis
The analysis examines why SG-OPT works, its robustness to domain shifts, computational efficiency, and the contributions of its design choices.
- Ablation Study: All modifications to the NT-Xent loss improve STS-task performance, while hyperparameters and the projection head are also important.The ablation study finds that each optimization contributes, and the projection head has a significant role.
- Robustness to Domain Shifts: SG-OPT outperforms Flow across STS-B and seven STS tasks when trained on either STS-B or NLI.This supports relative robustness to domain shifts across both evaluation settings.
- Robustness to Domain Shifts: 1.83 points on STS-B and 1.63 points across all STS tasks are SG-OPT’s losses when training changes from STS-B to NLI.Flow instead loses 12.16 and 4.19 points in the corresponding settings.
- Computational Efficiency: SG-OPT requires moderate training time of < 8 min. and is the most efficient method at inference on STS-B.The comparison uses the same machine and batch size across methods.
- Representation Visualization: SG-OPT encourages positive sentence pairs to align more closely while keeping negative pairs relatively distant in the visualization.The visualization uses 20 positive and 20 negative STS-B validation pairs projected into two dimensions with t-SNE.
6 Discussion
The discussion identifies non-optimal design choices and argues that combining self-guidance with other contrastive-learning techniques may further improve performance.
- Design Choices: The method prioritizes simplicity over fully optimized choices for intermediate layers and pooling.The authors note that these design decisions were made without extensive optimality analysis.
- Design Choices: A specific layer subset or another pooling method may perform better in particular environments, as mean pooling without lower layers improved SentEval results.The authors recommend systematic task-aware optimization of these choices.
- Future Directions: Combining back-translation with self-guidance generally improves performance, suggesting a direction for further contrastive-learning research.The ensemble inserts the original sentence into BERTT and its back-translation into BERTF.
7 Conclusion
The paper presents self-guided contrastive learning as an effective way to improve BERT sentence embeddings without external augmentation, while retaining efficient inference and relative domain robustness.
- Conclusion: Self-guided contrastive learning improves BERT sentence embeddings without relying on data augmentation or back-translation.The method generates higher-quality representations than competitive baselines.
- Conclusion: The method is efficient at inference because it requires no post-processing after training and is relatively robust to domain shifts.These are reported as additional benefits of the proposed approach.
A.1 Hyperparameters
The hyperparameter settings used for the experiments are summarized in Table 8.
- Hyperparameters: Table 8 lists the hyperparameters used in the experiments.
A.2 Specification on Contrastive (BT)
Contrastive (BT) matches the Contrastive (SG) model but creates positive samples through English–German–English back-translation.
- Contrastive (BT) is identical to Contrastive (SG) except for its positive-sample generation procedure.It uses back-translation rather than the SG procedure.
- English training sentences are translated into German with WMT’19 English–German translation and then back-translated into English.The two translation directions use WMT’19 models provided by Ng et al. (2019).
- Beam search is used during decoding for the back-translation process.
A.3 SentEval Configurations
This section specifies SentEval settings and reports configuration changes for SG-OPT, alongside evaluations on GLUE and RoBERTa-based sentence embeddings.
- SentEval Configurations: Table 9 lists the hyperparameters used with the SentEval toolkit.
- SG-OPT modifications: SG-OPT uses concatenated SNLI and MNLI data instead of STS-B for training.
- SG-OPT modifications: SG-OPT does not use the first several PLM layers when generating positive samples and uses mean pooling instead of max pooling.
- GLUE evaluation: GLUE evaluation uses the first 10% of the validation set for validation and the remaining 90% as test data across five subtasks.The benchmark does not officially provide test data.
- Sentence-vector visualization: Figure 6 visualizes sentence vectors computed by SBERT-base.
- RoBERTa evaluation: SG and SG-OPT show competitive performance for RoBERTa sentence embeddings, while back-translation also performs remarkably with RoBERTa-base.