PaperlayerView exact revision on arXiv

Source-linked AI summary

PromptBERT: Improving BERT Sentence Embeddings with Prompts

Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, Qi Zhang

arXiv:2201.04337v2cs.CL

TL;DR

Original BERT produces poor sentence embeddings because static token biases and ineffective layer use limit its representations. PromptBERT uses prompt-based representations and template-denoising contrastive learning, substantially improving sentence-embedding performance across datasets and settings.

  • Problem

    Original BERT performs poorly for sentence embeddings because existing representations suffer from static token embedding bias and ineffective BERT-layer use.

  • Method

    PromptBERT uses prompt-based templates for sentence representations and template-denoising contrastive learning to improve unsupervised sentence embeddings.

  • Results

    Prompt-based methods improve original BERT across all datasets, raising Spearman correlation by more than 10% over pooling methods and surpassing BERT-flow and BERT-whitening with manual templates.

  • Takeaways & Limitations

    Prompt-based sentence representations can substantially improve original BERT on STS and transfer tasks without relying solely on conventional pooling.

  • Takeaways & Limitations

    The templates remain manually generated, while automatically generated templates underperform manual ones.

Abstract

from arXiv · show

We propose PromptBERT, a novel contrastive learning method for learning better sentence representation. We firstly analyze the drawback of current sentence embedding from original BERT and find that it is mainly due to the static token embedding bias and ineffective BERT layers. Then we propose the first prompt-based sentence embeddings method and discuss two prompt representing methods and three prompt searching methods to make BERT achieve better sentence embeddings. Moreover, we propose a novel unsupervised training objective by the technology of template denoising, which substantially shortens the performance gap between the supervised and unsupervised settings. Extensive experiments show the effectiveness of our method. Compared to SimCSE, PromptBert achieves 2.29 and 2.58 points of improvement based on BERT and RoBERTa in the unsupervised setting.

1 Introduction

The introduction argues that original BERT produces poor sentence embeddings because its layers and token embeddings introduce biases, not solely because of anisotropy. It motivates prompt-based representations as a way to reduce embedding bias while exploiting BERT layers and improving contrastive positive-pair construction.

  • Problem: Original BERT performs poorly for sentence embeddings, even underperforming traditional word embeddings such as GloVe.This contrasts with the broader success of pretrained models including BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) in sentence embeddings (Gao et al., 2021b; Yan et al., 2021).
  • Problem: The introduction challenges anisotropy as the primary explanation and identifies damage from original BERT layers plus frequency-, case-, and subword-sensitive token-embedding bias.Prior work links anisotropy to poor performance (Li et al., 2020; Yan et al., 2021; Gao et al., 2021b), but the authors argue that static token embeddings also remain inferior to GloVe.
  • Motivation: Prompt-based templates are proposed to avoid embedding bias while utilizing original BERT layers, yielding reasonable sentence embeddings that can outperform some fine-tuned BERT-based methods.The approach is motivated by prompt-based reformulations of NLP tasks as fill-in-the-blank problems (Brown et al., 2020).
  • Contribution: The approach also applies to fine-tuned settings and uses prompts to seek better positive-pair generation than existing contrastive-learning practices.Existing unsupervised methods suffer from leaking proper positive pairs, while dropout noise has appeared more effective than several data-augmentation methods (Gao et al., 2021b; Yan et al., 2021).

2 Related Work

Sentence embedding research increasingly leverages BERT, with contrastive learning methods achieving state-of-the-art results. However, original BERT performs unsatisfactorily because of anisotropy, motivating post-processing approaches.

  • BERT Sentence Embeddings: Contrastive learning methods achieve state-of-the-art sentence embedding results with BERT in supervised and unsupervised settings.Gao et al. (2021b) proposed a contrastive training objective that uses inner dropout as noise.
  • BERT Sentence Embeddings: Recent work has focused on leveraging BERT for sentence embeddings across supervised and unsupervised settings.Several studies achieved strong performance with BERT (Gao et al., 2021b).
  • Limitations of Original BERT: Original BERT shows unsatisfactory sentence embedding performance, partly because anisotropy makes sentence pairs highly similar.Post-processing methods reduce this anisotropy (Su et al., 2021).

3 Rethinking the Sentence Embeddings of the Original BERT

The analysis attributes poor original-BERT sentence embeddings primarily to ineffective BERT layers and static token embedding biases, rather than anisotropy alone. Removing biased tokens substantially improves performance, although this remedy can omit meaningful words in short sentences.

  • 3 Rethinking the Sentence Embeddings of the Original BERT: The analysis concludes that ineffective BERT layers and static token embedding biases, rather than anisotropy itself, are the main causes of poor semantic similarity.The anisotropy measure is based on average cosine similarity between sentence embeddings, with isotropic representations expected to approach 0.
  • Observation 1: Original BERT layers fail to improve the performance: BERT layers significantly harm sentence-embedding performance in bert-base-uncased and roberta-base, while bert-base-cased gains only 0.28.The degradation is not explained by sentence-level anisotropy: last-layer averaging can be more isotropic yet perform worse than static-token averaging.
  • Observation 2: Embedding biases harms the sentence embeddings performance: Token embeddings are highly biased by frequency, subword structure, and case, forming distinct regions associated with these irrelevant attributes.High-frequency tokens cluster while low-frequency tokens disperse; begin-of-word tokens are more frequency-sensitive in BERT, whereas subwords are more sensitive in RoBERTa.
  • Observation 2: Embedding biases harms the sentence embeddings performance: Static-token anisotropy is not a reliable explanation for embedding bias: roberta-base is isotropic with 0.0235 average cosine similarity yet remains biased.Only bert-base-uncased has highly anisotropic static token embeddings among the three examined pretrained models.
  • Observation 2: Embedding biases harms the sentence embeddings performance: Removing static embedding biases improves Spearman correlation by 9.22, 7.08, and 11.76 points across the three pretrained models.The gains come from simply removing selected tokens; the resulting RoBERTa representation also outperforms BERT-flow and BERT-whitening while using only static token embeddings.
  • Observation 2: Embedding biases harms the sentence embeddings performance: Manually removing embedding biases is inadequate for very short sentences because it may omit meaningful words.This limitation applies even though bias removal generally improves sentence-embedding performance.

4 Prompt Based Sentence Embeddings

PromptBERT reformulates sentence embedding as masked-language modeling with prompts, using [MASK]-based representations to exploit BERT layers and avoid static token-embedding bias. It further introduces template search and template-denoised contrastive learning to construct stronger positive pairs.

  • 4 Prompt Based Sentence Embeddings: PromptBERT reframes sentence embedding as masked-language modeling, leveraging BERT’s pretrained knowledge while avoiding static token-embedding bias through prompt-based sentence representation.The approach is inspired by Brown et al. (2020).
  • 4 Prompt Based Sentence Embeddings: The method addresses prompt representation and prompt discovery, then builds a prompt-based contrastive learning method rather than using MLM classification labels as sentence outputs.Sentence embeddings are vectors, not label tokens predicted by the MLM classification head.
  • 4.1 Represent Sentence with the Prompt: Given a template such as “[X] means [MASK]”, PromptBERT maps the input sentence to the template and feeds it through a pretrained model to obtain a representation.One representation method uses the hidden vector of the [MASK] token; the alternative averages top-k MLM-predicted token embeddings by probability.
  • 4.1 Represent Sentence with the Prompt: PromptBERT selects the [MASK]-hidden-vector method because token-weight averaging retains static embedding bias and makes BERT harder to fine-tune downstream.The token-mapping alternative is described as more conventional but suffers from both disadvantages.
  • 4.2 Prompt Search: Prompt search combines manual greedy search, T5-based generation, and OptiPrompt; manual search separates relationship and prefix tokens, while T5 generation uses 500 word-definition templates.The methods are evaluated with Spearman correlation on the STS-B development set; T5-based generation follows Gao et al. (2021a), and OptiPrompt follows Zhong et al. (2021).
  • 4.2 Prompt Search: 80.90 versus 73.44 STS-B Spearman shows that an optimized continuous template outperforms its manual initialization under frozen-BERT unsupervised contrastive training.OptiPrompt initializes continuous templates from manual-template static token embeddings and optimizes them with unsupervised contrastive learning.
  • 4.3 Prompt Based Contrastive Learning with Template Denoising: Template denoising treats different templates for one sentence as positive views, subtracting template bias before contrastive training to reduce template-specific representation effects.The objective uses temperature τ and mini-batch size N, following prompt-based positive-instance construction.

5 Experiments

Experiments evaluate PromptBERT on seven STS datasets in non-fine-tuned and fine-tuned BERT settings. Prompt-based representations improve original BERT, while prompt-based contrastive learning outperforms prior methods and yields stable unsupervised training.

  • Experimental setup: Prompt-based methods are evaluated on seven common STS datasets, whose sentence pairs receive semantic-similarity scores from 0 to 5.The datasets comprise STS tasks from 2012–2016, STS-B, and SICK-R, downloaded using SentEval.
  • Non-fine-tuned BERT: More than 10% improvement in Spearman correlation over BERT pooling methods is achieved by prompt-based representations across all non-fine-tuned STS datasets.The experiments compare against averaging the last layer and averaging the first and last layers, as well as BERT-flow and BERT-whitening.
  • Fine-tuned BERT: Prompt-based contrastive learning outperforms prior contrastive and earlier sentence-embedding methods in fine-tuned BERT experiments, while narrowing the unsupervised–supervised performance gap.The evaluation uses BERT and RoBERTa under unsupervised and supervised settings; unsupervised results are trained with 10 random seeds because contrastive learning was unstable.
  • Training-objective analysis: PromptBERT produces more stable unsupervised results than SimCSE and achieves the best and most stable performance among same-template, different-template, and template-denoising objectives.The objective comparison uses 10 random runs and varies only how positive pairs are generated during training.

6 Discussion

The discussion finds that template denoising reduces template bias and improves predicted token relevance, while PromptBERT produces more stable unsupervised contrastive-learning results than SimCSE.

  • Template denoising: Template denoising efficiently removes bias from the prompt templates in original BERT.This mechanism supports better masked-token predictions and sentence embeddings.
  • Template denoising: PromptBERT’s template denoising removes unrelated predicted tokens and improves the quality of top-k tokens used for sentence embeddings.It removes tokens such as “nothing,no,yes” and encourages more sentence-related predictions.
  • Result stability: 3.14% is the best-to-worst result gap for SimCSE across 10 random seeds, compared with only 0.53 for PromptBERT.The comparison uses reproduced unsupervised SimCSE-BERTbase results and demonstrates greater stability for PromptBERT.

7 Conclusion

The paper argues that original BERT’s weak sentence embeddings stem from inappropriate representation methods that introduce static token embedding bias and underuse BERT layers. It addresses these limitations with a prompt-based sentence embedding method that improves BERT’s sentence-embedding performance.

  • 7 Conclusion: The paper attributes original BERT’s poor sentence embeddings to inappropriate sentence representation methods.These methods cause static token embedding bias and fail to effectively use the original BERT layer.
  • 7 Conclusion: The proposed prompt-based sentence embedding method better leverages BERT for sentence embeddings.The conclusion states that this approach helps original BERT achieve impressive sentence-embedding performance.

8 Limitation · A Static Token Embeddings Biases · A.1 Eliminating Biases by Removing Tokens

The paper’s methods perform reasonably in unsupervised and supervised settings, but template construction remains a limitation because manual templates outperform automatically generated T5 templates. The appendix details token-removal procedures for addressing static token-embedding bias across BERT and RoBERTa variants.

  • 8 Limitation: Manual templates still outperform automatically generated T5 templates, limiting the current method’s automation despite reasonable unsupervised and supervised performance.The authors leave development of a better automatic template-generation mechanism for future work.
  • 8 Limitation: Continuous templates verify that prompts are effective for sentence embeddings, although carefully designed automatic templates may improve performance further.This expectation is presented as future work rather than as a demonstrated result.
  • A Static Token Embeddings Biases: The appendix investigates static token-embedding bias by deleting tokens from bert-base-uncased, bert-base-cased, and roberta-base.The implementation covers frequency, subword, and case-based token-removal strategies.
  • A.1 Eliminating Biases by Removing Tokens: For frequency-based removal, the procedure deletes the top 36 frequent tokens according to the settings of Yan et al. (2021).The removed frequency tokens are listed in Table 11.
  • A.1 Eliminating Biases by Removing Tokens: For subword-based removal, the procedure deletes all subword tokens, identified as the yellow tokens in Figure 2.This is applied as a direct token-removal strategy.
  • A.1 Eliminating Biases by Removing Tokens: For case-based removal, only SICK contains upper- and lowercase sentences, so those sentences are lowercased to remove uppercase tokens.The procedure is therefore specific to the SICK dataset’s casing variation.

A.2 Eliminating Biases by Pre-training

The section attributes static token-embedding biases primarily to the MLM classification head and examines how weight tying contributes to those biases. It compares tied and untied BERT-like models and visualizes their embedding distributions across frequency, subword, and capitalization characteristics.

  • Bias Analysis: Static token-embedding bias primarily originates from the MLM classification head, whose tied weights transform [MASK] representations into token probabilities.Weight tying between static token embeddings and the MLM classification head causes the embeddings to suffer from these biases.
  • Embedding Visualization: Figure 2 visualizes the untied model’s static embeddings and MLM-head weights alongside the tied model’s static embeddings, marking frequency, subword, and capitalization patterns.Darker colors indicate higher token frequency; yellow marks subwords and red marks tokens containing capital letters.
  • Pre-training Setup: Two BERT-like models were pre-trained with MLM for 125k steps using batch size 2k, differing only in whether token-embedding and MLM-head weights were tied.

B Training Details

Training uses manually designed prompts, with OptPrompt additionally learning template embeddings while BERT remains frozen; fine-tuned and unsupervised settings use distinct template-denoising procedures and fixed sequence-length constraints.

  • Training configuration: OptPrompt initializes template embeddings from the manual prompt, then trains them with frozen BERT using the unsupervised task, with batch size 256, learning rate 3e-5, 5 epochs, and validation every 1000 steps.The fine-tuned setting uses the same training data as Gao et al. (2021b).
  • Template design: For non-fine-tuned training, the manual template is “This sentence: “[X]” means [MASK]”; fine-tuned models use manually searched templates selected according to STS-B development performance.The fine-tuned setting limits maximum sentence length to 32.
  • Template denoising: Unsupervised training uses two templates with template denoising, whereas prediction uses one template without denoising; supervised training uses template denoising with the same template for contrastive learning.The manual templates for fine-tuned models are listed in Table 13.

C Transfer Tasks

PromptBERT is evaluated on seven transfer tasks using SentEval’s default configurations. Its RoBERTa-based method improves over SimCSE by 2.52 points in unsupervised models and 0.92 points in supervised models.

  • Transfer-task setup: The transfer evaluation covers MR, CR, SUBJ, MPQA, SST-2, TREC, and MRPC under SentEval’s default configurations.Results are reported in Table 15.
  • Transfer-task results: PromptBERT’s RoBERTa-based method improves over SimCSE by 2.52 points for unsupervised models and 0.92 points for supervised models.The comparison covers the transfer-task results reported in Table 15.
  • Transfer-task results: Table 15 presents the transfer-task results for PromptBERT and SimCSE.The evaluation compares the RoBERTa-based method in unsupervised and supervised settings.
Loading 2201.04337v2…