PaperlayerView exact revision on arXiv

Source-linked AI summary

Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks

Nandan Thakur, Nils Reimers, Johannes Daxenberger, Iryna Gurevych

arXiv:2010.08240v2cs.CL

TL;DR

Cross-encoders are accurate but computationally expensive, while bi-encoders are efficient yet require substantial task-specific training data. Augmented SBERT uses a cross-encoder to soft-label additional sentence pairs for bi-encoder training, improving performance by up to 6 points in-domain and up to 37 points for domain adaptation.

  • Problem

    Cross-encoders are too slow for many applications, while bi-encoders need substantial training data and fine-tuning to achieve competitive performance.

  • Method

    Augmented SBERT uses a cross-encoder to label additional sentence pairs and adds them to the bi-encoder’s training data.

  • Results

    Augmented SBERT improves performance by 1 to 6 percentage points on four in-domain tasks and by up to 37 percentage points for domain adaptation.

  • Takeaways & Limitations

    Selecting suitable sentence pairs is crucial, with BM25 sampling providing the best trade-off between performance gain and computational complexity.

  • Takeaways & Limitations

    KDE sampling is computationally inefficient because it labels many randomly drawn samples that are later discarded.

Abstract

from arXiv · show

There are two approaches for pairwise sentence scoring: Cross-encoders, which perform full-attention over the input pair, and Bi-encoders, which map each input independently to a dense vector space. While cross-encoders often achieve higher performance, they are too slow for many practical use cases. Bi-encoders, on the other hand, require substantial training data and fine-tuning over the target task to achieve competitive performance. We present a simple yet efficient data augmentation strategy called Augmented SBERT, where we use the cross-encoder to label a larger set of input pairs to augment the training data for the bi-encoder. We show that, in this process, selecting the sentence pairs is non-trivial and crucial for the success of the method. We evaluate our approach on multiple tasks (in-domain) as well as on a domain adaptation task. Augmented SBERT achieves an improvement of up to 6 points for in-domain and of up to 37 points for domain adaptation tasks compared to the original bi-encoder performance.

1 Introduction

Pairwise sentence scoring balances cross-encoder accuracy against bi-encoder efficiency. Augmented SBERT addresses the bi-encoder’s data and performance gap by using a cross-encoder to expand training data, improving results in-domain and under domain adaptation.

  • Bi-encoders encode sentences independently into dense vectors, enabling efficient indexing and comparison but typically achieving lower performance than cross-encoders.Their performance gap is largest when little training data is available because independent mapping requires sufficient fine-tuning examples.
  • Augmented SBERT uses a BERT cross-encoder to label new input pairs, then fine-tunes the SBERT bi-encoder on the enlarged training set.The method’s success depends crucially on selecting suitable sentence pairs for soft-labeling.
  • Across four diverse in-domain tasks, AugSBERT consistently improves over the state-of-the-art SBERT bi-encoder by 1 to 6 percentage points.The tasks are argument similarity, semantic textual similarity, duplicate question detection, and news paraphrase identification.
  • In domain adaptation, AugSBERT achieves a performance increase of up to 37 percentage points.The approach addresses the bi-encoder’s difficulty mapping unseen-domain terminology to a sensible vector space.

2 Related Work

Related work contrasts cross-, bi-, and poly-encoder architectures and situates AugSBERT among semi-supervised and synthetic-data approaches. The paper’s distinguishing emphasis is that sentence-pair sampling strongly affects augmentation performance.

  • BERT cross-encoders jointly encode sentence pairs with full attention, whereas SBERT independently encodes inputs into fixed-sized sentence embeddings.
  • Poly-encoders occupy an intermediate position by separately encoding candidate and context, then applying attention between inputs only at the top layer.They retain separate representations while avoiding cross-encoders’ full self-attention across the inputs.
  • DiPair also uses a cross-encoder to annotate unlabeled pairs for bi-encoder fine-tuning, but AugSBERT focuses on sampling techniques while keeping the architecture constant.
  • AugSBERT builds on semi-supervised learning and differs from paraphrase-generation methods by generating suitable sentence pairs through sampling.
  • Randomly combining sentences is insufficient because appropriate pair sampling has a decisive impact on performance.

3 Methods

Augmented SBERT creates weakly labeled sentence pairs with a cross-encoder and merges them with gold data to train a bi-encoder. The method evaluates multiple sampling strategies for in-domain and domain-adaptation settings, where sampling quality and computational cost matter.

  • 3.1 Augmented SBERT: AugSBERT samples sentence pairs, labels them with a pretrained cross-encoder, merges the resulting silver dataset with gold data, and trains the bi-encoder on the combined set.
  • Pair Sampling Strategies: Because all n × (n −1)/2 sentence combinations are costly to annotate, selecting an effective subset is crucial for performance.
  • Random Sampling (RS): Random sampling mostly produces dissimilar negative pairs, heavily skewing the silver dataset’s label distribution.
  • Kernel Density Estimation (KDE): KDE sampling retains pairs to match the gold dataset’s label distribution, but it is computationally inefficient because many randomly labeled samples are discarded.
  • Sampling Strategies: BM25 retrieves lexically similar sentences efficiently, while semantic-search sampling retrieves cosine-similar sentences and BM25-S.S. combines lexical and semantic similarity.BM25-S.S. captures both similarity types but skews the label distribution toward negative pairs.
  • Domain Adaptation: For domain adaptation, a source-domain fine-tuned cross-encoder labels target-domain pairs before the bi-encoder is trained on those labeled target examples.The strategy is motivated by limited annotated data in new domains and the bi-encoder’s difficulty with unseen terminology.

4 Datasets

The paper evaluates sentence-pair scoring across regression and classification datasets, including English and Spanish single-domain tasks, argument similarity, and multi-domain duplicate-question datasets for domain adaptation.

  • Single-domain datasets: The in-domain evaluation covers semantic textual similarity, argument similarity, duplicate question detection, and news paraphrase identification.The first two are regression tasks, while the latter two are binary classification tasks.
  • Single-domain datasets: Spanish STS measures sentence similarity from 0 to 5, with 0 meaning no semantic overlap and 5 meaning identical content; scores are normalized to [0, 1].Training and development use SemEval STS 2014 and 2015, while testing uses SemEval STS 2017.
  • Single-domain datasets: The BWS Argument Similarity Dataset was created to address limitations in existing sentence-pair selection and sampling procedures.It uses continuous argument-similarity annotations derived through comparative preference judgments and Best-Worst Scaling.
  • Dataset splits: Cross-topic argument experiments train on topics T1–T5, develop on T6, and test on unseen topics T7 and T8.These experiments differ from domain adaptation because they fine-tune in-domain rather than on target-domain data.
  • Multi-domain datasets: The multi-domain evaluation uses AskUbuntu, Quora, Sprint, and SuperUser datasets for duplicate-question domain adaptation.Negative pairs are added by random sampling for datasets without explicitly labeled negatives, except Quora.

5 Experimental Setup

The experiments compare sentence-embedding and augmentation strategies across in-domain and domain-adaptation settings, using task-specific metrics and repeated random seeds.

  • Models: The setup uses PyTorch, Hugging Face Transformers, and sentence-transformers, with bert-base-uncased for English and bert-base-multilingual-cased for Spanish.AugSBERT models retain computational speeds identical to SBERT models.
  • Models: Cross-encoders are fine-tuned with a linear sigmoid layer over the [CLS] representation to output scores from 0 to 1.The reported optimal configuration uses learning rate 1 × 10^-5, hidden-layer sizes in {200, 400}, and batch size 16.
  • Models: SBERT bi-encoders are fine-tuned with batch size 16, learning rate 2 × 10^-5, and AdamW.The paper initially evaluates additional bi-encoder hyperparameters listed in the appendix.
  • Evaluation: In-domain regression is evaluated with Spearman’s rank correlation ρ × 100, while classification uses positive-class F1.Domain adaptation uses AUC(0.05), the area under the true-positive-rate curve for false-positive rates from 0 to 0.05.
  • Evaluation: In-domain experiments are generally repeated with 10 random seeds, and results are reported as mean scores with standard deviations.The evaluation also uses development-set threshold selection for classification tasks.
  • Baselines: Baselines include Jaccard similarity, a majority-label classifier, Universal Sentence Encoder, and NLPAug synonym replacement.AugSBERT is compared with synonym replacement as a straightforward text-augmentation strategy.

6 Results and Discussion

AugSBERT consistently improves SBERT on in-domain and domain-adaptation tasks, but its gains depend strongly on how sentence pairs are sampled. Domain direction also matters: generic-to-specific transfer is more beneficial than specific-to-generic transfer.

  • In-Domain Experiments: 4.5–9.1 points: plain SBERT underperforms the cross-encoder across all in-domain tasks.Seed optimization narrows the gap to 2.8–8.2 points, but does not eliminate it.
  • In-Domain Experiments: 1–6 percentage points: AugSBERT improves over SBERT across four diverse in-domain tasks.The tasks are argument similarity, semantic textual similarity, duplicate question detection, and news paraphrase identification.
  • In-Domain Experiments: AugSBERT gives only a minor improvement on the BWS cross-topic split but shows stronger performance on known topics.The reported explanation is that unseen topics make mapping similar arguments into a useful vector space more difficult.
  • Pairwise Sampling: Random sampling usually decreases performance, whereas BM25 and KDE produce the best AugSBERT results.BM25 offers similar score distributions to the gold data and is computationally efficient; combined BM25 plus semantic search can perform worse.
  • Domain Adaptation: Up to 37 points: AugSBERT improves over out-of-domain SBERT on the Sprint target dataset.In a few cases, AugSBERT also outperforms SBERT trained on gold in-domain target data.
  • Domain Adaptation: Generic-to-specific domain transfer benefits AugSBERT more than specific-to-generic transfer.The paper attributes this pattern to generic source data covering diverse topics that transfer better through cross-encoder labeling.

7 Conclusion

AugSBERT augments bi-encoder training with cross-encoder soft labels, improving performance when suitable sentence pairs are selected. It yields gains in both in-domain and domain-adaptation settings, while BM25 provides the best reported efficiency–performance trade-off.

  • Conclusion: AugSBERT uses a stronger cross-encoder to soft-label new sentence pairs and add them to the bi-encoder training set.The resulting model is fine-tuned on the combined original and weakly labeled data.
  • Conclusion: Up to 6 points: AugSBERT improves performance in in-domain experiments.The method improves across the evaluated in-domain tasks.
  • Conclusion: Randomly selecting sentence pairs fails to produce a performance gain, making pair selection crucial for soft-labeling.The conclusion identifies naive random selection as ineffective.
  • Conclusion: BM25 sampling provides the best trade-off between performance gain and computational complexity.This conclusion follows the comparison of several sampling strategies.
  • Conclusion: Up to 37 points: AugSBERT improves over an SBERT model trained purely on source-domain data for domain adaptation.The approach soft-labels data from the target domain before training the bi-encoder.

B.2 Density Distribution Analysis

The density analysis compares BWS and Spanish-STS distributions and shows that dataset construction can create different similarity-score profiles. Spanish-STS is concentrated toward extreme scores, whereas BWS is less biased.

  • Density Distribution Analysis: Figure 5 compares the density distributions of the BWS Argument Similarity dataset and Spanish-STS.The comparison concerns how sentence-pair similarity scores are distributed across the datasets.
  • Density Distribution Analysis: Spanish-STS contains many pairs near both ends of the similarity scale, indicating selection bias from pre-sampling.The passage links the extreme-score concentration to the dataset creation process.
  • Density Distribution Analysis: BWS has fewer pairs near either end of the similarity scale than Spanish-STS.Its distribution is described as less biased and more consistent with randomly paired arguments from real-world data.

D Seed Optimization

The paper evaluates seed optimization as a way to improve model selection and reduce wasted computation, and separately tests top-k choices in retrieval-based sampling. Small top-k values generally work best.

  • Seed Optimization: Five random seeds are used for in-domain training, with the best development-set model selected and the complete setup repeated ten times.The procedure improves robustness but can be computationally expensive.
  • Seed Optimization: Development-set performance at an intermediate training point is compared with final development-set rankings to identify potentially poor runs early.This evaluates whether bad runs can be stopped before completing training.
  • Top-k Analysis: Small k values yield predominantly positive pairs, whereas larger k values make negative pairs dominant.This pattern is described for BM25 and semantic-search sampling.
  • Top-k Analysis: Top-k values of 3 or 5 produce optimal scores for most experiments, while the overall impact of k is rather small.The tested values are 3, 5, 7, 9, 12, and 18.

F Computational Efficiency vs. Size of Silver Datasets

AugSBERT’s efficiency depends on the size and selection of the silver-pair dataset. Smaller datasets reduce cross-encoder labeling and bi-encoder training overhead, while sampling strategies differ substantially in performance and cost.

  • Smaller silver datasets are preferable because larger sets increase cross-encoder labeling and subsequent bi-encoder training overhead.The method requires weakly labeling sentence pairs before retraining the bi-encoder.
  • BM25 generally offers the best average performance across tasks while remaining computationally efficient.
  • Random sampling is inefficient and often performs worse, whereas KDE can achieve optimal scores with large datasets but at greater computational cost.
  • Semantic Search requires additional bi-encoder training, creating extra computational overhead.

G Development Set Performances

Development-set performance is reported separately for in-domain and domain-adaptation experiments, using task-appropriate metrics and highlighted best AugSBERT or transfer strategies. The supplied passages specify reporting conventions and annotation examples but do not provide the underlying score values.

  • In-domain performances: In-domain development results are summarized for semantic textual similarity, argument similarity, duplicate question detection, and news paraphrase identification.
  • Sampling comparisons: The development-set tables distinguish sampling strategies and report the best AugSBERT strategy or smallest effective silver dataset where specified.The tables cover BM25, semantic-search, and other sampling summaries across regression and classification datasets.
  • Metric conventions: Regression tasks use Spearman’s rank correlation ρ × 100, while classification tasks use positive-class F1 scores.Table 17 uses Jaccard similarity and majority-label baselines for regression and classification tasks, respectively.
  • Domain adaptation: Domain-adaptation development experiments report AUC(0.05), with the best transfer strategy highlighted and cross-domain evaluation used except for SBERT in-domain.
  • Argument similarity: The argument-similarity annotation example treats differently worded arguments as similar when they express the same meaning.The example concerns claims about toxic chemicals from fracking contaminating or polluting basic environmental resources and food supplies.
Loading 2010.08240v2…