PaperlayerView exact revision on arXiv

Source-linked AI summary

WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach

Junjie Huang, Duyu Tang, Wanjun Zhong, Shuai Lu, Linjun Shou, Ming Gong, Daxin Jiang, Nan Duan

arXiv:2104.01767v3cs.CL

TL;DR

Unsupervised sentence embeddings are valuable for sentence matching and retrieval when large amounts of supervised data are unavailable. The paper evaluates pretrained-model embeddings across four models and seven semantic datasets, finding consistent benefits from token averaging, combining bottom and top layers, and whitening.

  • Problem

    Unsupervised sentence embeddings are valuable for matching and retrieval, especially when large amounts of supervised data are unavailable.

    Source: §1 · p. 1

  • Method

    The study examines token pooling, layer combinations, and linear whitening across four pretrained models and seven semantic-textual-similarity datasets.

    Source: §1 · p. 1

  • Results

    Averaging all token representations, combining bottom and top layers, and whitening consistently improve sentence embeddings across the reported experiments.

    Source: §4.2 · p. 3, §6 · p. 5

  • Takeaways & Limitations

    The paper concludes that token averaging, combining the bottom and top layers, and whitening are simple, effective strategies for deriving sentence embeddings from pretrained models.

    Source: §6 · p. 5

Abstract

from arXiv · show

Producing the embedding of a sentence in an unsupervised way is valuable to natural language matching and retrieval problems in practice. In this work, we conduct a thorough examination of pretrained model based unsupervised sentence embeddings. We study on four pretrained models and conduct massive experiments on seven datasets regarding sentence semantics. We have there main findings. First, averaging all tokens is better than only using [CLS] vector. Second, combining both top andbottom layers is better than only using top layers. Lastly, an easy whitening-based vector normalization strategy with less than 10 lines of code consistently boosts the performance.

1 Introduction

The paper investigates unsupervised sentence embeddings from pretrained language models for settings where supervised data are scarce, examining token pooling, layer combinations, and simple linear transformation. Across four models and seven semantic-textual-similarity datasets, it reports three consistent findings about pooling, layer choice, and whitening.

1 Introduction

  • The study asks whether token vectors beyond [CLS] help, whether a better layer combination exists, and whether a simple linear transform can address anisotropic embeddings.These questions are motivated by the value of unsupervised embeddings for matching and retrieval when large amounts of supervised data are unavailable.Source: §1 · p. 1
  • The experiments examine four pretrained language models on seven datasets concerning semantic textual similarity.The four models are BERT-base, RoBERTa-base, DistilBERT, and LaBSE.Source: §1 · p. 1
  • Averaging all token representations consistently outperforms using only the [CLS] embedding, and whitening consistently improves sentence-embedding performance.The reported third finding is that combining bottom and top layers performs better than using the top two layers.Source: §1 · p. 1

2 Transformer-based PLMs

Transformer-based pretrained language models represent input sequences with hidden vectors at multiple encoder layers. The paper uses four such models that differ in architecture and pretraining objectives.

2 Transformer-based PLMs

  • For an input sequence, a transformer-based pretrained model produces hidden representations at each encoder layer, with per-token embeddings at each layer.The initial hidden representation corresponds to non-contextual word-piece embeddings.Source: §2 · p. 1
  • The study uses BERT-base, RoBERTa-base, DistilBERT, and LaBSE as its four transformer-based pretrained models.The models vary in architecture and pretraining objectives.Source: §2 · pp. 1–2
  • BERT-base, RoBERTa-base, and LaBSE have twelve transformer layers, while DistilBERT has six.LaBSE is pretrained with a translation-ranking task that brings parallel-sentence embeddings closer; the other three models do not use that task.Source: §2 · p. 2

3 WhiteningBERT

WhiteningBERT derives sentence embeddings by pooling token representations, combining layer representations, and applying a whitening transformation. The section describes how each of these three strategies constructs or transforms the embedding.

3.1 [CLS] Token v.s. Average Tokens

  • A sentence embedding can use the last-layer [CLS] vector or the average of all last-layer token vectors, including [CLS].The average-token strategy is contrasted with selecting the first token's representation.Source: §3.1 · p. 2

3.2 Layer Combination

  • Layer combination forms a sentence embedding by adding the sentence-level representations computed separately from selected layers.For example, the paper describes combining layers one and twelve by adding their vector representations.Source: §3.2 · p. 2

3.3 Whitening

  • Whitening linearly transforms a set of sentence embeddings using its mean and covariance eigendecomposition.The embedding set contains N sentence vectors of dimension d; the covariance is expressed through its eigenvalues and orthogonal eigenvectors.Source: §3.3 · p. 2

4 Experiment

The experiments evaluate unsupervised sentence embeddings across seven STS datasets, examining token pooling, layer combinations, and whitening. The results favor averaging tokens and whitening, while layer-combination findings depend on the comparison and model scope.

4 Experiment

  • The experiment evaluates unsupervised semantic textual similarity and reports the best ways to derive pretrained-model sentence embeddings.The task predicts sentence-pair similarity without direct supervision.Source: §4 · p. 2

4.1 Experiment Settings

  • The evaluation uses seven STS datasets, cosine similarity for predictions, and averaged Spearman rank correlation as its final score.The experiments also compare against five representative unsupervised sentence-embedding models.Source: §4.1 · p. 2

4.2 Overall Results

  • Averaging last-layer token representations outperforms using only the [CLS] token across all four pretrained models.The authors attribute this to the single [CLS] embedding not conveying enough semantic information for sentence representation.Source: §4.2 · p. 3
  • Adding layer-one and last-layer token representations outperforms using either layer alone across pretrained models.Table 1 reports Spearman rank correlation (ρ × 100); its caption defines AVG/CLS, layer, and whitening settings.Source: §4.2 · p. 3, Table 1 · p. 3
  • Whitening consistently improves STS performance, with LaBSE showing the smallest gain among the four pretrained models.The authors attribute LaBSE's smaller improvement to its translation-ranking pretraining and resulting intrinsic representation ability.Source: §4.2 · p. 3

4.3 Analysis of Layer Combination

  • Layer one plus the last layer is not the best two-layer combination for every pretrained model, though the authors consider it a simple, satisfactory choice.Figure 1 encodes each two-layer combination's average correlation across seven STS tasks, with redder cells indicating better performance.Source: §4.3 · p. 4, Figure 1 · p. 4
  • On BERT-base, three layers perform best, with L1+L2+L12 the strongest combination among the layer-count groups.Maximum correlation rises and then falls as more layers are combined; the authors conclude that three layers suffice.Source: §4.3 · p. 4, Figure 2 · p. 4

5 Related works

Prior unsupervised sentence-embedding work either further trains pretrained embeddings or uses weighted averages based on word features. The authors note the extra training or feature requirements and mention concurrent work on whitening.

Related approaches

  • Prior approaches either further train pretrained embeddings or use weighted averages based on word features.Examples include Li et al. (2020), Zhang et al. (2020), and Mu and Viswanath (2018) for further training, and Arora et al. (2017), Ethayarajh (2018), Yang et al. (2019), and Wang and Kuo (2020) for weighted averages.Source: §5 · p. 4
  • The authors say these approaches require further training or additional features, limiting direct application of their embeddings in real-world scenarios.The cited feature-based approaches use word features, while the other category uses unlabeled natural language inference datasets for training.Source: §5 · p. 4

Concurrent work

  • The authors note that Su et al. (2021) also explored whitening sentence embeddings, with its arXiv release one week earlier.This is identified as work concurrent with their own.Source: §5 · pp. 4–5

6 Conclusion

The conclusion presents three empirical findings on unsupervised sentence embeddings from pretrained language models: token averaging, combining bottom and top layers, and whitening.

Conclusion

  • Averaging all token representations consistently yields better sentence representations than using the [CLS] embedding.Source: §6 · p. 5
  • Combining bottom-layer and top-layer embeddings outperforms using the top two layers.Source: §6 · p. 5
  • Whitening sentence embeddings consistently boosts performance.Source: §6 · p. 5

A.1 More Results of WhiteningBERT

Whitening improves average performance across seven STS tasks for the tested pretrained models; the tables describe the evaluation metric and embedding configuration.

A.1 More Results of WhiteningBERT

  • Average performance on seven STS tasks improves after whitening is incorporated across the tested pretrained models.Table 2 reports Spearman’s rank correlation coefficient (ρ×100) for token-averaged embeddings combining layer one and the top layer.Source: §A.1 · p. 6, Table 2 · p. 7
  • Table 2 compares results with and without whitening using Spearman’s rank correlation coefficient (ρ×100).The embeddings use token averaging and combine layer one with the top layer.Source: Table 2 · p. 7
  • Table 3 reports RTE results using token-averaged embeddings, layer one plus the last layer, and whitening.Source: Table 3 · p. 7

A.2 Comparison with GPT-3

The section describes WhiteningBERT’s RTE evaluation setup and mentions GPT-3’s zero-shot classification capability. It reports using BERT on the RTE development set, with predictions based on cosine similarity and a manually set threshold.

A.2 Comparison with GPT-3

  • WhiteningBERT with BERT is evaluated on the RTE development set.Source: §A.2 · p. 7
  • Sentence-pair labels are predicted from cosine similarity using a manually set threshold of 0.5.Source: §A.2 · p. 7
  • GPT-3 is described as capable of zero-shot classification, while the reported evaluation uses WhiteningBERT with BERT.Source: §A.2 · p. 7

A.3 Code for Whitening

The section identifies Figure 3 as PyTorch code for the whitening strategy.

A.3 Code for Whitening

Related papers

Loading 2104.01767v3…