PaperlayerView exact revision on arXiv

Source-linked AI summary

SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models

Bin Wang, C. -C. Jay Kuo

arXiv:2002.06652v2cs.CLcs.LGcs.MM

TL;DR

High-quality sentence embeddings from BERT-based word models remain difficult to obtain, despite BERT’s strong NLP performance and layer-specific linguistic information. The paper studies layer-wise representation geometry and proposes SBERT-WK, a training-free embedding method evaluated across similarity and supervised tasks. SBERT-WK achieves state-of-the-art performance, including strong results without fine-tuning.

  • Problem

    Generating high-quality sentence representations from BERT-based word models remains an open problem relevant to transferring sentence-level knowledge to downstream tasks.

  • Method

    SBERT-WK dissects BERT-based word models through geometric subspace analysis, fusing layer-wise information and requiring no further training.

  • Results

    SBERT-WK achieves state-of-the-art performance across semantic textual similarity and downstream supervised evaluations, ranking among the top three on all eight supervised tasks.

  • Takeaways & Limitations

    SBERT-WK provides an efficient sentence embedding approach that uses 768-dimensional vectors and performs strongly without fine-tuning.

Abstract

from arXiv · show

Sentence embedding is an important research topic in natural language processing (NLP) since it can transfer knowledge to downstream tasks. Meanwhile, a contextualized word representation, called BERT, achieves the state-of-the-art performance in quite a few NLP tasks. Yet, it is an open problem to generate a high quality sentence representation from BERT-based word models. It was shown in previous study that different layers of BERT capture different linguistic properties. This allows us to fusion information across layers to find better sentence representation. In this work, we study the layer-wise pattern of the word representation of deep contextualized models. Then, we propose a new sentence embedding method by dissecting BERT-based word models through geometric analysis of the space spanned by the word representation. It is called the SBERT-WK method. No further training is required in SBERT-WK. We evaluate SBERT-WK on semantic textual similarity and downstream supervised tasks. Furthermore, ten sentence-level probing tasks are presented for detailed linguistic analysis. Experiments show that SBERT-WK achieves the state-of-the-art performance. Our codes are publicly available.

I. INTRODUCTION

The paper asks how to obtain high-quality universal sentence representations from BERT-based contextualized word models. It studies layer-wise representation patterns and proposes SBERT-WK as a training-free solution evaluated across linguistic and downstream tasks.

  • Motivation: Different BERT layers encode different linguistic properties, motivating information fusion across layers for stronger sentence representations.Intermediate layers are described as more transferable, while higher layers capture more high-level semantic information.
  • Observations: The paper observes that word-representation evolution across layers is related to word content and uses this pattern to identify salient and informative words.Richer-information words vary more across layers, whereas less informative token representations change more gradually.
  • Motivation: BERT-based contextualized word models provide strong task performance, but deriving high-quality sentence embeddings from them remains an open problem.Existing approaches commonly use the last layer or [CLS], while SBERT uses additional supervised training on sentence pairs.
  • Contribution: SBERT-WK derives sentence embeddings by geometrically dissecting the space learned by deep contextualized word models, without requiring further training.The method is intended to retain the strengths of contextualized models and is compatible with models such as BERT and RoBERTa.
  • Evaluation: The method is evaluated on eight downstream tasks, seven semantic textual similarity tasks, and sentence-level probing tasks for linguistic analysis.The paper reports state-of-the-art performance in these evaluations.

B. Universal Sentence Embedding

SBERT-WK combines contextualized representations with subspace analysis to construct generic sentence embeddings. Its design is motivated by layer-evolution patterns and word-dependent variation, and it is compared with existing embedding methods.

  • Universal Sentence Embedding: SBERT-WK combines parameterized contextualized models with non-parameterized subspace analysis rather than relying on a universal sentence-level training objective.The method analyzes the manifold learned by deep contextualized models to compute sentence embeddings.
  • Word Representation Evolution: The method measures cosine similarity between representations at neighboring transformer layers to quantify how word representations evolve across depth.The analysis uses 1-hop and N-hop neighbors and averages similarity maps across isolated word representations from STS-Benchmark sentences.
  • Word Representation Evolution: Nearby layers generally have high similarity except near the beginning and end, while intermediate layers are flatter and more stable for BERT, SBERT, and RoBERTa.Consecutive-layer cosine similarity exceeds 0.9 for these models; XLNET shows a different evolution pattern.
  • Word Importance: The study finds that more informative words vary more across layers, with variance positively correlated with inverse document frequency.Words are grouped by variance, and the reported correlation is statistically significant.

IV. PROPOSED SBERT-WK METHOD

SBERT-WK determines sentence embeddings in two stages: it integrates word representations across layers, then weights the resulting word representations by importance.

  • The method section is organized around these two steps and is illustrated by the proposed-model diagram.
  • SBERT-WK first determines a unified word representation by integrating representations across layers using alignment and novelty properties.
  • It then produces the sentence embedding by taking a weighted average of unified word representations according to word importance.

A. Unified Word Representation Determination

The unified word representation combines layer-specific representations of each word. Its layer weights are based on inverse alignment with neighboring vectors and the novelty of each layer’s information.

  • Layer-weighted integration: For each word, SBERT-WK integrates representations from different layers into a unified representation using layer weights α_i.
  • Inverse Alignment Measure: Inverse alignment measures how well a layer’s word representation aligns with neighboring word vectors using average pairwise cosine similarity.
  • Inverse Alignment Measure: The neighboring matrix uses embedding dimension d and a window size m around the target word representation.
  • Inverse Alignment Measure: Representations that align well with neighboring vectors receive smaller inverse-alignment weights because they provide less additional information.

2) Novelty Measure:

The novelty measure quantifies information in a word representation that lies outside the subspace spanned by representations from neighboring layers.

  • Novelty Measure: The method decomposes each layer’s word representation into a component inside the neighboring subspace and an orthogonal component.
  • Novelty Measure: It treats the orthogonal component as novel information and uses its magnitude as the novelty score.
  • Novelty Measure: Singular value decomposition of the neighboring matrix provides an orthogonal basis for computing the component outside the neighboring subspace.

3) Unified Word Representation:

SBERT-WK combines alignment- and novelty-based layer weights to form a unified contextualized word representation, then uses word-level variance to weight words in the sentence.

  • The method measures new information in each layer’s word representation using alignment and novelty components, which can be combined through a weight parameter ω.
  • The experiments compare novelty weight ω = 0, inverse alignment weight ω = 1, and combined weight ω = 0.5.
  • The unified word representation is computed as a weighted sum across layer representations and is treated as the new contextualized representation for the word.
  • Word importance is estimated from the variance of values on the offset-1 diagonal of each word’s cross-layer cosine-similarity matrix.
  • Words that evolve faster across layers receive higher weights because their larger variances indicate greater importance.

C. Computational Complexity

SBERT-WK’s main computational burden is decomposing the context window, with QR factorization used in experiments as a faster alternative to SVD.

  • C. Computational Complexity: SVD decomposes the context window matrix C into UΣV, with U providing the orthogonal basis for geometric analysis.C has size d × 2m, where d is the word-embedding size and 2m is the window size.
  • C. Computational Complexity: QR factorization concatenates the center-word representation with C, producing 2m + 1 word representations for factorization.The resulting matrix is denoted ˜C and is factorized as ˜C = QR.
  • C. Computational Complexity: The columns of Q form an orthonormal basis, while R contains word-representation weights under that basis.The center word’s representation is encoded in the final column of R, and its newly introduced direction is the final column of Q.
  • C. Computational Complexity: The novelty weight is derived from the final component of the center word’s coefficient vector under the QR basis.This component is associated with the new direction introduced by the center word into the context.
  • C. Computational Complexity: QR factorization has complexity O(d(2m + 1)^2), is two times faster than SVD, and shows little practical performance difference.SVD complexity is O(8dm^2) in the stated setting where d is much larger than m.

V. EXPERIMENTS

The experiments assess SBERT-WK across semantic similarity, supervised transfer, and probing tasks using BERT- and RoBERTa-based models and multiple baselines.

  • V. EXPERIMENTS: SBERT-WK is evaluated on semantic textual similarity, supervised downstream, and probing tasks to test embedding quality, transfer, and linguistic properties.The experiments use the SentEval toolkit and build the method on BERT and RoBERTa models fine-tuned with natural language inference data.
  • V. EXPERIMENTS: The benchmark compares SBERT-WK with 10 parameterized and non-parameterized sentence-embedding methods.Baselines include averaged GloVe and BERT representations, BERT [CLS], SIF, GEM, p-mean, and Skip-Thought.
  • V. EXPERIMENTS: STS12–STS16 evaluation uses cosine similarity between sentence pairs and reports Pearson and Spearman correlation coefficients.The datasets contain sentence pairs with semantic-relatedness labels from 0 to 5.
  • V. EXPERIMENTS: STS-Benchmark and SICK-R provide supervised semantic-relatedness evaluation distinct from the unsupervised STS12–STS16 setting.STS-Benchmark contains 8,628 sentences from captions, news, and forums.
  • V. EXPERIMENTS: Direct BERT representations perform poorly: [CLS] averages 38.9/41.2 correlation, while averaging BERT embeddings averages 61.5/60.6.Both settings underperform non-parameterized methods such as SIF.
  • V. EXPERIMENTS: SBERT-WK improves textual-similarity performance by a significant margin while using 768-dimensional vectors instead of InferSent’s 4096 dimensions.It is slightly inferior to InferSent on SICK-R but performs substantially better on the other tasks.
  • V. EXPERIMENTS: RoBERTa does not show obvious improvement over BERT on STS datasets, and a 24-layer model generally performs worse than the base model.The authors suggest different hyperparameters or fine-tuning schemes may be needed for the large model.

B. Supervised Downstream Tasks

SBERT-WK is tested on eight supervised downstream tasks and achieves strong transfer performance without task-specific fine-tuning.

  • B. Supervised Downstream Tasks: The eight downstream tasks cover sentiment, subjectivity, opinion polarity, question type, paraphrase, and natural-language-inference classification.They include MR, CR, SUBJ, MPQA, SST2, TREC, MRPC, and SICK-E.
  • B. Supervised Downstream Tasks: The evaluation uses an MLP with one hidden layer of 50 neurons, batch size 64, Adam optimization, and four training epochs.Cross-validation procedures vary by dataset, with standard validation used for SST2.
  • B. Supervised Downstream Tasks: SBERT-WK achieves good performance without fine-tuning, ranks first on 5 of 8 tasks, and ranks among the top three on the remaining tasks.SRoBERTa-WK-base achieves the best averaged supervised-task performance at 88.8%.
  • B. Supervised Downstream Tasks: SBERT-WK-base outperforms SBERT-base by 6.8% on the supervised-task comparison.USE is particularly suited to TREC, while InferSent-based methods outperform SBERT-WK on SICK-E.
  • B. Supervised Downstream Tasks: BERT [CLS] performs poorly on semantic similarity but well on classification, where an MLP can assign different importance to embedding dimensions.Cosine similarity in semantic-similarity tasks treats all dimensions equally.

C. Probing Tasks

Probing experiments examine 10 surface, syntactic, and semantic properties, showing that SBERT-WK improves over SBERT across all tasks while balancing shallow and deep information.

  • C. Probing Tasks: The 10 probing tasks span surface, syntactic, and semantic information to provide finer-grained analysis than downstream-task performance alone.Tasks include sentence length, word content, tree depth, constituent prediction, tense, number agreement, and word-order judgments.
  • C. Probing Tasks: The probing evaluation uses the same one-hidden-layer MLP setting as supervised tasks, with 50 neurons, batch size 64, Adam, and four epochs.Standard validation is used, including the same MLP for WC to support comparison with prior work.
  • C. Probing Tasks: SBERT-WK outperforms SBERT on all probing tasks and achieves the best result on 4 of 10 tasks.The method merges information from different layers, which carry different balances of surface and semantic properties.
  • C. Probing Tasks: Good word-content performance alone does not guarantee satisfactory sentence embeddings; high-level semantic meaning must also be considered.Although p-means performs best on WC, it does not achieve the best downstream-task performance.
  • C. Probing Tasks: SBERT-WK’s improvement over SBERT across probing tasks is associated with substantial improvement on TREC, which correlates with many probing tasks.The paper reports that SBERT-WK works well on TREC relative to the SBERT baseline.
  • C. Probing Tasks: SBERT-WK can obtain similar or better probing results than BERT despite SBERT’s sentence-level training emphasis.The paper contrasts SBERT’s sentence-pair objective with BERT’s word-, phrase-, and intersentence-level objectives.

D. Ablation and Sensitivity Study

The ablation study shows that Alignment, Novelty, and Token Importance each improve the plain SBERT model, while attention maps indicate greater focus on informative words.

  • 1) Ablation study of each module’s contribution:: All three components—Alignment, Novelty, and Token Importance—contribute to SBERT-WK’s performance.Adding Alignment and Novelty alone improves performance by 1.86 and 2.49, respectively.
  • 1) Ablation study of each module’s contribution:: The ultimate performance gain over plain SBERT reaches 3.56.
  • 1) Ablation study of each module’s contribution:: Attention heat maps show the word importance module placing greater focus on informative words.

2) Sensitivity to window size and layer selection:

SBERT-WK remains robust across tested window sizes and starting layers, with m = 2 selected as the default for performance and computational efficiency. CPU inference adds a modest QR overhead to the SBERT baseline.

  • 2) Sensitivity to window size and layer selection:: SBERT-WK is robust to different context window sizes m and starting layers lS.The study tests m values from 1 to 4 and lS values from 0 to 6.
  • 2) Sensitivity to window size and layer selection:: m = 2 is selected as the default window size by balancing performance and computational efficiency.
  • 8.59 ms of QR overhead plus 168.67 ms for the SBERT baseline gives SBERT-WK’s reported CPU inference time.The overhead is about 5% compared with the baseline BERT model.
  • SBERT-WK requires no further training and is evaluated across a wide range of tasks.
  • The paper identifies theoretical subspace analysis of deep contextualized models as a future direction for understanding their effectiveness.
Loading 2002.06652v2…