PaperlayerView exact revision on arXiv

Source-linked AI summary

Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models

Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, Yinfei Yang

arXiv:2108.08877v3cs.CL

TL;DR

The paper addresses how to obtain useful sentence embeddings from T5 encoder-decoder models, which are otherwise difficult to use efficiently for retrieval and clustering. It proposes three ST5 architectures, contrastive fine-tuning, and the SentGLUE benchmark, finding strong transfer performance for encoder-only models, better textual similarity performance for encoder-decoder models, and gains from scaling.

  • Problem

    T5 performs strongly on text-to-text tasks, but effective extraction of sentence representations from its encoder-decoder architecture remains unclear, despite the broad utility of sentence embeddings.

  • Method

    The paper builds ST5 from pre-trained T5 models with three sentence-embedding architectures, two-stage contrastive fine-tuning, and evaluation on SentEval, STS, and SentGLUE.

  • Results

    Encoder-only ST5 models show strong transfer performance, encoder-decoder models perform better on textual similarity tasks, and increasing model size greatly improves sentence embedding quality.

  • Takeaways & Limitations

    Scaling and improved pre-trained text-to-text models may yield further gains in sentence embedding quality, while architecture choice matters across transfer and similarity tasks.

  • Takeaways & Limitations

    The submitted experiments for MNLI-m and MNLI-mm comparisons involving SBERT RoBERTa Large and SimCSE RoBERTa Base were still running.

Abstract

from arXiv · show

We provide the first exploration of sentence embeddings from text-to-text transformers (T5). Sentence embeddings are broadly useful for language processing tasks. While T5 achieves impressive performance on language tasks cast as sequence-to-sequence mapping problems, it is unclear how to produce sentence embeddings from encoder-decoder models. We investigate three methods for extracting T5 sentence embeddings: two utilize only the T5 encoder and one uses the full T5 encoder-decoder model. To support our investigation, we establish a new sentence representation transfer benchmark, SentGLUE, which extends the SentEval toolkit to nine tasks from the GLUE benchmark. Our encoder-only models outperforms Sentence-BERT and SimCSE sentence embeddings on both SentEval and SentGLUE transfer tasks, including semantic textual similarity (STS). Scaling up T5 from millions to billions of parameters is found to produce consistent further improvements. Finally, our encoder-decoder method achieves a new state-of-the-art on STS when using sentence embeddings. Our models are released at https://tfhub.dev/google/collections/sentence-t5/1.

1 Introduction

Sentence embeddings offer compact representations for many language tasks, but T5’s encoder-decoder design makes efficient sentence representation extraction unclear. The paper investigates three ST5 architectures, introduces SentGLUE, and studies scaling and fine-tuning.

  • Sentence embeddings provide compact meaning representations for classification, question-answering, retrieval, bitext mining, and semantic similarity tasks.
  • T5 is difficult to apply efficiently to retrieval and clustering because scoring each query-candidate pair requires full cross-attention inference.Sentence embeddings instead support efficient retrieval and clustering.
  • The paper extracts sentence representations using the encoder’s first token, mean encoder pooling, or the decoder’s first token.
  • The study evaluates raw and fine-tuned T5 representations on SentEval and semantic textual similarity tasks, using NLI and ReQA contrastive learning.
  • The work studies scaling sentence embedding models to 11 billion parameters and reports strong transfer and STS performance from encoder-only and encoder-decoder ST5 variants.
  • The paper introduces SentGLUE, extending SentEval to nine GLUE tasks for sentence-representation transfer evaluation.

2 Text-to-Text Transfer Transformers (T5)

T5 is an encoder-decoder transformer pretrained through span corruption and designed to solve tasks as text-to-text mappings. Despite its success across NLP tasks, extracting high-quality text representations from T5 remains unexplored.

  • T5 frames diverse tasks as simple text-to-text mapping problems.
  • T5 uses an encoder-decoder transformer architecture pretrained on an unsupervised span corruption task.
  • How to extract high-quality text representations from T5 remains unexplored despite its successful application to numerous NLP tasks.

3 Sentence T5

Sentence T5 extracts sentence embeddings from T5 using encoder-only pooling or the first decoder output, then trains shared dual encoders with contrastive objectives.

  • Sentence representation strategies: Three strategies extract sentence representations: encoder first-token pooling, encoder mean pooling, and first-token pooling from the decoder.The decoder method feeds the sentence through the encoder and a start symbol as the first decoder input.
  • Sentence representation strategies: ST5-Enc mean averages encoder outputs across all input tokens, while ST5-Enc first uses the encoder output of the first token.
  • Sentence representation strategies: The encoder-decoder method assumes the first decoder output can capture whole-sentence semantics because the decoder attends to the encoded input.
  • Dual encoder architecture: The dual encoder uses two shared-weight transformer modules, followed by projection and L2 normalization to produce fixed-length embeddings.The transformer modules can use either encoder-only or encoder-decoder architectures.
  • Contrastive learning: Contrastive training pairs each input with a related positive example and treats other batch examples as negatives, pulling positives closer and pushing negatives away.The loss is implemented with an in-batch sampled softmax; additional negatives can also be supplied.
  • Contrastive learning: Preliminary experiments found NLI classification fine-tuning inferior to contrastive learning for sentence-encoder fine-tuning.
  • Contrastive learning: Two-stage training first uses mined Community QA pairs and then fine-tunes on human-annotated NLI sentence pairs.

4 Experimental Setup

The experiments evaluate T5 sentence embeddings on transfer and semantic textual similarity tasks, using SentEval protocols and specified training infrastructure and hyperparameters.

  • Evaluation: The evaluation uses SentEval's 7 transfer tasks and 7 semantic textual similarity tasks.
  • Evaluation: Transfer performance is measured by linear-classifier performance using sentence embeddings as features.
  • Evaluation: STS performance is measured by the correlation between embedding cosine similarities and human-annotated similarity scores.The reported STS setting aggregates subsets and uses Spearman’s correlation.
  • Implementation: Models are implemented in JAX, trained on Cloud TPU-v8, and initialized from public T5 checkpoints.
  • Implementation: Training uses Adafactor with learning rate 0.001, linear decay after 10% of training steps, and a softmax temperature of 0.01.Fine-tuning batch sizes are 512 for NLI and 2048 for Community QA.

5 Experimental Goals

The experiments ask how to extract T5 sentence representations, how raw embeddings transfer to downstream tasks, how contrastive training helps, and whether larger models improve them.

  • Research questions: Q1 asks which method best extracts sentence representations from T5.
  • Research questions: Q2 asks how well raw T5 sentence embeddings perform on downstream tasks.
  • Research questions: Q3 asks how much contrastive sentence-embedding tasks such as NLI and QA improve T5 sentence embeddings.
  • Research questions: Q4 asks whether scaling model capacity improves sentence representations.
  • Comparison plan: The study compares transfer and STS performance across model and training configurations, including SBERT, SRoBERTa, and SimCSE.

6 Results

The experiments compare raw and fine-tuned T5 sentence embeddings across transfer and STS tasks, pooling strategies, architectures, model sizes, and training data. Encoder-only mean pooling is strongest for transfer, while encoder-decoder models are strongest for STS.

  • Raw T5 sentence embeddings: First-token representations from either the encoder or decoder perform much worse than encoder-only mean pooling across all SentEval tasks.T5 does not reserve its first token as a CLS-style placeholder or train it specifically to represent whole-sentence semantics.
  • Raw T5 sentence embeddings: Mean pooling of T5 encoder outputs greatly outperforms average BERT embeddings and exceeds fine-tuned SimCSE-RoBERTa on transfer tasks without fine-tuning.The authors attribute possible gains to T5’s larger pretraining corpus and inclusion of downstream tasks, while noting that five SentEval tasks were not included in GLUE.
  • Raw T5 sentence embeddings: 55.97 is the average STS score for raw T5 mean pooling, slightly exceeding BERT mean pooling but remaining below supervised fine-tuned models.The authors relate the weaker raw STS performance to anisotropy and embedding collapse, which impair distance-based metrics.
  • Fine-tuning T5 sentence embeddings: 0.57 is the transfer-performance gain of the best fine-tuned ST5 model over the best raw T5 sentence embeddings.After fine-tuning, transfer performance is consistent across the remaining embedding-extraction strategies, so the first-token model is discarded.
  • Fine-tuning T5 sentence embeddings: NLI fine-tuning significantly improves ST5 performance on STS tasks compared with raw T5 embeddings.The result supports contrastive learning as a way to mitigate embedding collapse in T5-style models.
  • Fine-tuning T5 sentence embeddings: Additional Community QA training before NLI fine-tuning brings a large performance boost on both transfer and STS tasks.The authors suggest that continued contrastive learning on additional semi-structured data may further improve sentence embedding quality.
  • Encoder-only vs. encoder-decoder: Encoder-only models consistently outperform encoder-decoder models on transfer tasks as ST5 scales from Base to Large, 3B, and 11B.This supports building ST5 on the T5 encoder for strong transfer performance, although the authors frame decoder-related explanations as hypotheses.
  • Encoder-only vs. encoder-decoder: 84.11 is the STS Spearman correlation for ST5 encoder-decoder Large, improving over SimCSE-RoBERTa Large’s 83.76 and outperforming encoder-only models on all STS tasks.The decoder may add useful parameters or function as an additional attention-pooling layer over encoder outputs.

7 Scaling up Sentence T5

Scaling T5 models consistently improves sentence-embedding transfer and similarity performance, although larger capacity alone does not resolve embedding collapse and fixed embedding sizes may limit STS gains.

  • Transfer performance consistently improves as T5 scales, supporting larger pre-trained models for sentence embeddings.
  • Increasing capacity alone does not mitigate embedding collapse, because T5 pre-training does not require avoiding anisotropy.The paper highlights contrastive loss or regularization as possible ways to address this mismatch with similarity and retrieval goals.
  • The ST5 11B encoder-only model reaches an average transfer score of 91.08, versus 90.45 for ST5 Large.
  • The ST5 11B encoder-decoder model reaches an STS score of 84.94, outperforming the ST5 Large model.
  • STS gains from scaling 3B to 11B are smaller than gains from Large to 3B, possibly because embedding sizes remain fixed.
  • Scaling reduces uniformity loss with only a slight increase in alignment loss for both encoder and encoder-decoder models.Lower values indicate better performance for both losses.

8 SentGLUE Evaluation

SentGLUE extends sentence-representation evaluation to nine GLUE tasks under a fixed-embedding constraint, showing strong performance from scaled ST5 encoder-only models.

  • SentGLUE extends the sentence evaluation toolkit to nine GLUE challenge tasks, including classification, inference, and similarity tasks.
  • WNLI is excluded because existing sentence-embedding models were found too challenged by that task.
  • Each input in SentGLUE must be independently encoded into a fixed embedding before additional prediction layers operate on it.
  • ST5-Enc Base outperforms SBERT-RoBERTa Base and SimCSE-RoBERTa Base on every SentGLUE task except CoLA and MNLI.
  • ST5 Enc 11B achieves the best overall performance, with aggregate sentence-embedding results approaching those of T5 base cross-attention models.

9 Conclusion

The paper finds that encoder-only ST5 models transfer strongly, encoder-decoder models perform better on textual similarity, and scaling improves sentence-embedding quality.

  • The paper proposes three T5 sentence-encoder architectures and a two-stage contrastive learning method for fine-tuning ST5.
  • Encoder-only models show strong transfer performance, while encoder-decoder models perform better on textual similarity tasks.
  • Scaling model size greatly improves sentence-embedding quality across the reported experiments.
  • MNLI-m and MNLI-mm results for SBERT RoBERTa Large and SimCSE RoBERTa Base were still running at submission time.

A Model Inference

ST5 inference speed varies substantially by hardware and batch configuration, with TPU providing the fastest reported execution and large gains over cross-attention-based inference.

  • ST5 inference speed is evaluated across model sizes, sequence lengths, batch sizes, and hardware platforms.
  • On Cloud TPU-v8, ST5 11B reaches 274 examples per second with sequence length 128 and batch size 1024.
  • On four V100 GPUs, ST5 11B reaches 27 examples per second with sequence length 128 and batch size 1024.
  • On CPU, ST5 11B reaches 0.5 examples per second with batch size 512.
  • Sentence-embedding models remain faster than cross-attention models whose computation time increases quadratically with the number of examples.
Loading 2108.08877v3…