PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation of BERT and ALBERT Sentence Embedding Performance on Downstream NLP Tasks

Hyunjin Choi, Judong Kim, Seongho Joe, Youngjune Gwon

arXiv:2101.10642v1cs.CLcs.AI

TL;DR

Finding effective sentence embeddings from contextualized language models remains an active research problem, and [CLS] and average pooling each have limitations. The paper evaluates SBERT, SALBERT, and CNN variants on STS and NLI. CNN improves ALBERT models more than BERT models on STS, while ALBERT sentence embeddings remain highly competitive despite fewer parameters.

  • Problem

    Sentence embeddings from contextualized language models remain an active research problem, and [CLS] vectors and average-pooled token embeddings have limitations.

    Source: §I · p. 1

  • Method

    The paper adapts SBERT’s siamese and triplet structures to ALBERT as SALBERT, tests CNN sentence-embedding layers, and evaluates models on STS and NLI.

    Source: §I · p. 2

  • Results

    CNN improves ALBERT models by up to 8 points in STS Spearman’s rank correlation, versus 1 point for BERT models; ALBERT sentence embeddings remain highly competitive despite fewer parameters.

    Source: §I · p. 2, §V · p. 5

  • Takeaways & Limitations

    Within the evaluated downstream tasks, ALBERT sentence embeddings are highly competitive with BERT despite significantly fewer model parameters.

    Source: §I · p. 2

Abstract

from arXiv · show

Contextualized representations from a pre-trained language model are central to achieve a high performance on downstream NLP task. The pre-trained BERT and A Lite BERT (ALBERT) models can be fine-tuned to give state-ofthe-art results in sentence-pair regressions such as semantic textual similarity (STS) and natural language inference (NLI). Although BERT-based models yield the [CLS] token vector as a reasonable sentence embedding, the search for an optimal sentence embedding scheme remains an active research area in computational linguistics. This paper explores on sentence embedding models for BERT and ALBERT. In particular, we take a modified BERT network with siamese and triplet network structures called Sentence-BERT (SBERT) and replace BERT with ALBERT to create Sentence-ALBERT (SALBERT). We also experiment with an outer CNN sentence-embedding network for SBERT and SALBERT. We evaluate performances of all sentence-embedding models considered using the STS and NLI datasets. The empirical results indicate that our CNN architecture improves ALBERT models substantially more than BERT models for STS benchmark. Despite significantly fewer model parameters, ALBERT sentence embedding is highly competitive to BERT in downstream NLP evaluations.

I. INTRODUCTION

The paper addresses the ongoing search for effective sentence embeddings from contextualized language models, where [CLS] and average pooling have limitations. It evaluates SBERT, SALBERT, and CNN variants, finding larger STS gains from CNN for ALBERT than for BERT.

I. INTRODUCTION

  • BERT uses masked language modeling and next-sentence prediction, while ALBERT uses sentence-order prediction for multi-sentence inputs.The paper situates these models within the ongoing research problem of computing sentence embeddings from contextualized language models.Source: §I · p. 1
  • [CLS] vectors lack independently computed sentence embeddings, while average-pooled token embeddings also have limitations.The text also notes that multi-sentence semantic-search tasks can be time-consuming.Source: §I · p. 1
  • The study adapts SBERT’s siamese and triplet structures to ALBERT as SALBERT and also tests CNNs in place of average pooling.SBERT compares sentences using cosine similarity at run time.Source: §I · p. 2
  • CNN improves ALBERT models by up to 8 points in STS Spearman’s rank correlation, versus 1 point for BERT models.Despite significantly fewer parameters, ALBERT sentence embeddings are highly competitive to BERT in downstream evaluations.Source: §I · p. 2

II. RELATED WORK

Prior sentence-embedding work spans neural architectures and training strategies, while BERT and ALBERT offer contextual token representations that can be pooled or adapted for sentence comparison. SBERT is a particularly relevant approach because its inference uses cosine similarity rather than more elaborate pairwise fine-tuning mechanisms.

II. RELATED WORK

  • BERT and ALBERT provide contextual token representations, with [CLS] or average pooling forming fixed-length sentence vectors.ALBERT incorporates parameter-reduction techniques and uses sentence-order prediction rather than BERT’s next-sentence prediction.Source: §II · p. 2
  • Earlier sentence encoders include CNNs, BiLSTMs, recursive tensor networks, tree-LSTMs, and memory-augmented NSE.Later work explored self-attention pooling and fine-grained attention.Source: §II · p. 2
  • Supervised transfer tasks such as NLI can improve sentence embeddings, and training-task choice significantly affects embedding quality.SNLI is identified as suitable for training sentence embeddings.Source: §II · p. 2
  • SBERT replaces elaborate pairwise inference mechanisms with cosine similarity, reducing a reported search from 65 hours with BERT to about 5 seconds with little accuracy impact.Its siamese and triplet structures produce sentence embeddings for comparison.Source: §II · p. 2

III. MODELS

The models turn contextual token outputs from BERT or ALBERT into fixed-size sentence embeddings for sentence-level regression tasks such as STS.

III. MODELS

  • BERT or ALBERT can produce up to 512 contextual token vectors of 768 dimensions each.These contextualized word embeddings can be fine-tuned for downstream tasks.Source: §III · p. 2
  • Sentence-level regressions such as STS require fixed-size sentence embeddings.The section describes sentence-embedding models for both BERT and ALBERT.Source: §III · p. 2
  • The models described in this section derive sentence embeddings from BERT and ALBERT token outputs.The passage introduces the models after distinguishing token-level outputs from the fixed-size vectors needed for sentence-level regression.Source: §III · p. 2

A. The [CLS] token embedding

The section presents [CLS] as a sentence vector based on pretraining context prediction and average pooling as an alternative that summarizes token embeddings.

A. The [CLS] token embedding

  • [CLS] is the straightforward sentence vector, used during pretraining to predict sentence-level context for BERT NSP or ALBERT SOP.Its self-attention mechanism summarizes information from other tokens, and fine-tuning is expected to make it more task-specific.Source: A. The [CLS] token embedding · pp. 2–3

B. Pooled token embeddings

  • The [CLS] vector supports pretraining context prediction, whereas pooled token embeddings provide an alternative fixed-size representation.The alternatives are presented as sentence-embedding models for BERT and ALBERT.Source: A. The [CLS] token embedding · p. 2, B. Pooled token embeddings · p. 3
  • Average pooling converts token embeddings into a fixed-length sentence vector, chosen here over max pooling.The text characterizes max pooling as selecting important features rather than a representative summary.Source: B. Pooled token embeddings · p. 3

C. Sentence-BERT (SBERT)

SBERT uses siamese and triplet structures to produce sentence embeddings; the section extends this design to ALBERT and CNN-based sentence encoders.

C. Sentence-BERT (SBERT)

  • SBERT average-pools paired BERT outputs into fixed-size sentence embeddings, then uses their embeddings and element-wise difference for classification or regression.Its siamese architecture allows sentence comparison using cosine similarity at run time.Source: C. Sentence-BERT (SBERT) · p. 3

D. Sentence-ALBERT (SALBERT)

E. CNN-SBERT

  • CNN-SBERT uses convolutional layers with tanh activation and interleaved pooling to convert token embeddings into a fixed-size sentence embedding.All pooling layers use max pooling except the final average-pooling layer.Source: E. CNN-SBERT · p. 3

F. CNN-SALBERT

CNN-SALBERT adopts CNN-SBERT’s convolutional architecture, and the models are evaluated on STS and NLI benchmarks.

F. CNN-SALBERT

  • CNN-SALBERT uses the same CNN architecture as CNN-SBERT.Fig. 2 depicts the architecture used by both models.Source: F. CNN-SALBERT · p. 3, Figure 2 · p. 3
  • Fig. 2 defines B, T, and H as mini-batch size, number of tokens, and transformer hidden size, respectively.Source: Figure 2 · p. 3

IV. EXPERIMENTS

  • The evaluation uses cosine similarity and Pearson and Spearman rank coefficients to assess sentence-embedding similarity against dataset ground-truth labels.Source: §IV · p. 3

A. Datasets and tasks

The experiments fine-tune sentence-embedding models on STSb, MultiNLI, and SNLI, covering semantic similarity and natural language inference.

1) Semantic Textual Similarity benchmark:

2) Multi-genre Natural Language Inference:

  • MultiNLI is a 433k-pair entailment corpus for classification, with genre diversity that supports cross-genre generalization evaluation.Its labels are contradiction, entailment, and neutral.Source: 2) Multi-genre Natural Language… · p. 4

3) Stanford Natural Language Inference:

B. Training

Training uses BERT and ALBERT base models with GLUE fine-tuning and different procedures for STSb and NLI.

B. Training

  • The study uses BERT and ALBERT base models, fine-tunes [CLS] and average-pooled embeddings with GLUE, and trains SBERT-family models with Adam and linear warmup.The learning rates are 3 × 10−5 for GLUE, 2 × 10−5 for SBERT and SALBERT, and 1 × 10−5 for CNN-SBERT and CNN-SALBERT.Source: B. Training · p. 4

1) STSb:

  • STSb training uses a siamese network for 10 epochs with batch size 32.Source: 1) STSb · p. 4

2) NLI (MultiNLI + SNLI):

  • NLI training uses the siamese architecture with a cross-entropy softmax classifier for one epoch and batch size 16.SBERT and SALBERT are optimized on the three-way softmax loss using MultiNLI and SNLI data.Source: 2) NLI (MultiNLI + SNLI) · p. 4, B. Training · p. 4

3) NLI + STSb:

The section compares fine-tuning sources for STSb and reports that combining STSb and NLI training gives the best STSb results.

3) NLI + STSb:

  • After fine-tuning on NLI, the models were trained on the STS benchmark with a batch size of 32.Source: 3) NLI + STSb · p. 4

1) Effect of fine-tuning:

  • Fine-tuning on both STSb and NLI produced the best STSb results; Table I compares STS-only, NLI-only, and combined training.The table evaluates sentence embeddings on STSb.Source: 1) Effect of fine-tuning · p. 4, Table I · p. 4
  • Fine-tuning improved performance over no fine-tuning, and [CLS] embeddings performed poorly without fine-tuning.STSb-related training gave reasonably good results, while NLI-only training yielded suboptimal performance.Source: 1) Effect of fine-tuning · p. 4

2) Model comparison:

The comparison finds that CNN sentence embeddings perform best overall, while CNN improves ALBERT models more than BERT models on STSb.

2) Model comparison:

  • Pooling token embeddings outperformed [CLS], siamese structure helped further, and CNN-based models performed best overall on STSb.Source: 2) Model comparison · p. 5

3) Performance of ALBERT:

  • ALBERT sentence-embedding models generally scored lower than BERT on STSb; after STSb fine-tuning, ALBERT was better only for [CLS] and average-pooled embeddings.SALBERT trailed SBERT, but CNN brought their performance closer; CNN-SALBERT remained slightly inferior to CNN-SBERT.Source: 3) Performance of ALBERT · p. 5

4) Effect of CNN:

  • CNN improved ALBERT models by up to 8 points in Spearman’s rank correlation on STSb, versus 1 point for BERT models.The authors propose that CNN may alleviate ALBERT’s observed instability from parameter sharing.Source: 4) Effect of CNN · p. 5

5) Evaluation of STS12–STS16 Tasks:

Across STS tasks, CNN’s gains were larger for ALBERT than BERT, and the conclusion reports strong STS improvements for ALBERT despite its smaller parameter count.

5) Evaluation of STS12–STS16 Tasks:

  • Across STS12–STS16, SBERT averaged 84.49 Spearman correlation and CNN-SBERT averaged 84.34, while CNN-SALBERT’s average improved by 2 points.CNN had almost no effect on BERT-based sentence embeddings in these tasks.Source: 5) Evaluation of STS12–STS16 Tasks · p. 5

V. CONCLUSION AND FUTURE WORK

  • The CNN architecture improved ALBERT models by up to 0.08 in Spearman’s rank correlation on the STS benchmark, versus 0.01 for BERT models.The conclusion also describes ALBERT sentence embeddings as highly competitive to BERT despite significantly fewer parameters.Source: §V · p. 5
  • Future work includes evaluating ALBERT-large and ALBERT-xlarge and optimizing num_hidden_groups, which was set to 1 for the reported ALBERT results.Source: §V · p. 5
Loading 2101.10642v1…