PaperlayerView exact revision on arXiv

Source-linked AI summary

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Nils Reimers, Iryna Gurevych

arXiv:1908.10084v1cs.CL

TL;DR

BERT’s pairwise architecture makes large-scale sentence similarity search computationally expensive. SBERT derives comparable sentence embeddings with siamese and triplet networks, reducing search over 10,000 sentences from about 65 hours to about 5 seconds while maintaining BERT’s accuracy.

  • Problem

    BERT’s cross-encoder requires roughly 50 million pairwise inference computations and about 65 hours to find the most similar pair among 10,000 sentences.

  • Method

    SBERT fine-tunes BERT with siamese and triplet network architectures to produce fixed-sized sentence embeddings comparable with cosine similarity.

  • Results

    About 5 seconds replaces 65 hours for finding the most similar pair among 10,000 sentences with SBERT, while benchmark evaluations show improvements over other state-of-the-art sentence embedding methods.

  • Takeaways & Limitations

    SBERT enables semantic similarity search and clustering tasks that are computationally infeasible to model directly with BERT.

  • Takeaways & Limitations

    SBERT’s performance drops by about 7 Spearman-correlation points in cross-topic argument similarity evaluation.

Abstract

from arXiv · show

BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences are fed into the network, which causes a massive computational overhead: Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours) with BERT. The construction of BERT makes it unsuitable for semantic similarity search as well as for unsupervised tasks like clustering. In this publication, we present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT. We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning tasks, where it outperforms other state-of-the-art sentence embeddings methods.

1 Introduction

The paper introduces Sentence-BERT (SBERT), a siamese/triplet modification of BERT that produces semantically meaningful sentence embeddings for similarity comparison, clustering, and semantic search. SBERT makes large-scale similarity computations far more efficient and achieves strong transfer performance after fine-tuning on NLI data.

  • Motivation and contribution: SBERT uses siamese and triplet networks to derive fixed-size, semantically meaningful sentence embeddings.Semantically similar sentences are close in vector space, enabling efficient comparison with cosine similarity or distance measures.
  • Motivation and contribution: 49 995 000 inference computations are required to find the highest-similarity pair among n = 10 000 sentences with BERT.BERT’s cross-encoder passes both sentences through the transformer, creating too many pairwise combinations for large-scale search.
  • Motivation and contribution: 65 hours with BERT is reduced to about 5 seconds for computing 10,000 SBERT sentence embeddings, plus ~0.01 seconds for cosine-similarity.With optimized index structures, finding the most similar Quora question is reduced from 50 hours to a few milliseconds.
  • Motivation and contribution: 11.7 points is SBERT’s improvement over InferSent across seven Semantic Textual Similarity tasks, while the improvement over Universal Sentence Encoder is 5.5 points.These results follow fine-tuning SBERT on natural language inference data.
  • Motivation and contribution: SBERT can be adapted to specific tasks and sets new state-of-the-art performance on argument similarity and Wikipedia-section sentence-triplet datasets.The evaluated datasets distinguish argument similarity and sentences drawn from different sections of a Wikipedia article.

2 Related Work

Related work establishes BERT and RoBERTa as strong sentence-pair regression models but highlights the difficulty of deriving independent sentence embeddings and the computational limits of cross-encoders. SBERT instead fine-tunes pretrained BERT and RoBERTa to produce useful sentence embeddings with substantially shorter training time.

  • BERT and RoBERTa: BERT is a pretrained transformer that achieved state-of-the-art results across NLP tasks, including sentence-pair regression and the STS benchmark.For sentence-pair regression, BERT processes both sentences together, separated by a special [SEP] token.
  • BERT and RoBERTa: RoBERTa further improved BERT through small pretraining adaptations, while XLNet generally produced worse results than BERT.These comparisons concern the reported sentence-embedding evaluation context.
  • Sentence embedding methods: BERT does not compute independent sentence embeddings, so prior work derived fixed-size vectors by averaging outputs or using the special CLS token.These approaches passed single sentences through BERT before constructing the vector representation.
  • Sentence embedding methods: Prior sentence embedding methods included Skip-Thought’s encoder-decoder training and InferSent’s supervised siamese BiLSTM with max-pooling.Skip-Thought predicts surrounding sentences, whereas InferSent uses labeled natural language inference datasets.
  • SBERT’s distinction: SBERT fine-tunes pretrained BERT and RoBERTa rather than training from random initialization, completing tuning in less than 20 minutes while outperforming comparable sentence embedding methods.The paper presents this as a substantial reduction in required training time.

3 Model

SBERT modifies pretrained BERT/RoBERTa with pooling and siamese or triplet fine-tuning to produce semantically meaningful sentence embeddings comparable by cosine similarity. Its training objectives include classification, regression, and triplet losses, selected according to available training data.

  • Pooling: SBERT adds pooling over BERT/RoBERTa outputs to derive fixed-sized sentence embeddings, with CLS, MEAN, and MAX strategies evaluated and MEAN as default.The MEAN strategy computes the mean of all output vectors, while MAX computes a max-over-time pooling.
  • Network structures: Siamese and triplet network structures fine-tune BERT/RoBERTa weights so sentence embeddings become semantically meaningful and comparable with cosine similarity.The network structure depends on the available training data.
  • Classification objective: Classification combines embeddings u and v with |u−v|, applies trainable W_t ∈ R^(3n×k), and optimizes cross-entropy loss.Here n is the sentence-embedding dimension and k is the number of labels.
  • Regression objective: Regression computes cosine similarity between sentence embeddings u and v and uses mean-squared-error loss.This inference architecture is also used with the regression objective function.
  • Triplet objective: Triplet loss makes anchor-positive distance smaller than anchor-negative distance, using Euclidean distance with margin ϵ = 1 in the experiments.The margin ensures the positive embedding is at least ϵ closer to the anchor than the negative embedding.

4 Evaluation - Semantic Textual Similarity

SBERT is evaluated on semantic textual similarity using cosine similarity between sentence embeddings, including unsupervised STS datasets, the supervised STS benchmark, and argumentative similarity. It substantially improves over direct BERT embeddings and several baselines, while cross-topic generalization remains challenging.

  • Unsupervised STS: SBERT evaluates STS without STS-specific training data, comparing sentence embeddings with cosine similarity against gold semantic-relatedness labels.The evaluation covers STS tasks from 2012–2016, the STS benchmark, and SICK-Relatedness; correlations are measured with Spearman’s rank correlation.
  • Unsupervised STS: 54.81 average correlation from averaged BERT embeddings and 29.19 from CLS-token outputs are both below average GloVe embeddings.The passage characterizes direct BERT outputs as poor-performing sentence representations.
  • Unsupervised STS: Siamese fine-tuning substantially improves correlation, outperforming InferSent and Universal Sentence Encoder except on SICK-R.Universal Sentence Encoder performs better on SICK-R, plausibly because its training data better matches that dataset’s news, question-answer, and discussion-forum content.
  • STS benchmark: On supervised STSb, NLI pretraining followed by STSb fine-tuning improves performance by 1-2 points, while BERT and RoBERTa show no significant difference.SBERT uses the STSb training set with a regression objective and cosine similarity at prediction time; systems use 10 random seeds.
  • Argument Facet Similarity: In 10-fold AFS evaluation, SBERT performs nearly on-par with BERT, but cross-topic evaluation produces a drop of about 7 points Spearman correlation.AFS requires matching claims and reasoning across argumentative excerpts, making unseen-topic sentence-vector comparison more difficult than BERT’s direct attention-based comparison.
  • Wikipedia triplets: SBERT clearly outperforms Dor et al.’s BiLSTM on Wikipedia triplet evaluation using accuracy over 222,957 test triplets.SBERT is trained for one epoch on about 1.8 Million training triplets, with positives required to be closer to anchors than negatives.

5 Evaluation - SentEval

SentEval evaluates sentence embeddings by training logistic regression classifiers in 10-fold cross-validation, while SBERT is assessed as a representation rather than a transfer-learning method. SBERT achieves the best performance on 5 of 7 tasks, improves average performance by about 2 percentage points over InferSent and Universal Sentence Encoder, and captures sentiment particularly well.

  • Evaluation setup: SentEval trains logistic regression classifiers on sentence embeddings and computes prediction accuracy on test folds in a 10-fold cross-validation setup.The toolkit evaluates sentence embeddings as features for various tasks.
  • Evaluation setup: SBERT is not intended for transfer learning, although SentEval provides an impression of its quality across seven sentence-classification tasks.The authors consider fine-tuning all BERT layers for new tasks more suitable.
  • Overall results: SBERT achieves the best performance in 5 out of 7 tasks and raises average performance by about 2 percentage points over InferSent and Universal Sentence Encoder.Despite transfer learning not being its intended use, SBERT outperforms other state-of-the-art sentence-embedding methods on SentEval.
  • Task-level results: SBERT shows large improvements over InferSent and Universal Sentence Encoder on all sentiment tasks: MR, CR, and SST.These results suggest that SBERT sentence embeddings capture sentiment information well.
  • Task-level results: Universal Sentence Encoder significantly outperforms SBERT only on TREC, likely because its question-answering pretraining benefits question-type classification.Average BERT embeddings and BERT CLS-token outputs perform decently on SentEval but slightly worse than InferSent or Universal Sentence Encoder; the fine-tuned siamese setup achieves a new state-of-the-art for SentEval.

6 Ablation Study

The ablation study evaluates pooling strategies and classification-objective concatenation methods across SBERT training settings. Pooling has minor impact for NLI classification but large impact for STS regression, while the element-wise difference is most important for classification training.

  • Study design: The study compares MEAN, MAX, and CLS pooling, evaluates concatenation methods for classification, and averages results across 10 random seeds.SBERT-base is trained on SNLI and Multi-NLI for classification or on STS benchmark training data for regression; performance is measured on the STS benchmark development split.
  • Classification objective: For NLI classification, pooling strategy has a minor impact, whereas concatenation mode has a much larger impact.Concatenation scores are reported with MEAN pooling.
  • Classification objective: The element-wise difference |u −v| is the most important classification component, while adding element-wise u ∗v decreases performance.InferSent and Universal Sentence Encoder use (u, v, |u −v|, u ∗v), but SBERT benefits most from the element-wise difference; concatenation affects only softmax-classifier training, not inference.
  • Regression objective: For regression training, pooling strategy has a large impact, with MAX performing significantly worse than MEAN or CLS-token pooling.This contrasts with InferSent, where MAX pooling was found beneficial for its BiLSTM layer.

7 Computational Efficiency

Section 7 compares sentence-embedding computation speed across SBERT, average GloVe, InferSent, and Universal Sentence Encoder. SBERT benefits substantially from GPU execution and smart batching, although InferSent remains faster on CPU.

  • Comparison setup: The comparison evaluates SBERT against average GloVe embeddings, InferSent, and Universal Sentence Encoder on sentences from the STS benchmark.SBERT and InferSent use PyTorch, Universal Sentence Encoder uses TensorFlow Hub, and average GloVe uses Python dictionary lookups with NumPy.
  • Method: Smart batching groups sentences with similar lengths to improve sentence-embedding computation.The strategy was implemented for SBERT and measured alongside the compared embedding methods.
  • CPU performance: 65% faster: InferSent exceeds SBERT on CPU because its single BiLSTM layer is simpler than BERT’s 12 stacked transformer layers.The reported CPU difference is attributed to the simpler network architecture used by InferSent.
  • GPU performance: 9% faster: SBERT with smart batching exceeds InferSent on GPU, and 55% faster: it exceeds Universal Sentence Encoder.Transformer networks provide computational efficiency on GPUs in this comparison.
  • Batching impact: 89% on CPU and 48% on GPU: smart batching achieves these speed-ups for sentence embedding computation.The speed-up figures are reported for the batching strategy across the two hardware settings.

8 Conclusion

The conclusion identifies out-of-the-box BERT sentence vectors as unsuitable for common similarity measures, and presents SBERT as a more accurate and computationally efficient alternative. SBERT improves over state-of-the-art sentence embeddings, while RoBERTa offers no significant experimental improvement.

  • 8 Conclusion: BERT’s sentence vectors perform below average GloVe embeddings on seven STS tasks and are unsuitable for cosine similarity.The conclusion attributes this limitation to BERT’s out-of-the-box sentence representations.
  • 8 Conclusion: SBERT fine-tunes BERT using siamese or triplet networks and significantly improves over state-of-the-art sentence embedding methods.The method was evaluated on various common benchmarks.
  • 8 Conclusion: Replacing BERT with RoBERTa did not yield a significant improvement in the experiments.This comparison concerns the evaluated sentence embedding systems.
  • 8 Conclusion: 9% faster than InferSent and 55% faster than Universal Sentence Encoder, SBERT is computationally efficient on a GPU.These speed comparisons support using SBERT for tasks that are computationally infeasible with BERT.
  • 8 Conclusion: 65 hours with BERT versus about 5 seconds with SBERT, clustering 10,000 sentences becomes computationally feasible.BERT requires around 50 Million sentence combinations for hierarchical clustering of 10,000 sentences.
Loading 1908.10084v1…