PaperlayerView exact revision on arXiv

Source-linked AI summary

DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations

John Giorgi, Osvald Nitski, Bo Wang, Gary Bader

arXiv:2006.03659v4cs.CLcs.LG

TL;DR

High-performing universal sentence embeddings depend on labelled data, limiting their applicability across languages and domains. DeCLUTR introduces a self-supervised contrastive objective for extending transformer pretraining without labels, and reports that it closes the performance gap with supervised methods while improving with model and data size.

  • Problem

    High-performing universal sentence-embedding methods require labelled data, limiting their usefulness where labelled training data is not abundant.

  • Method

    DeCLUTR extends MLM pretraining with a self-supervised contrastive objective that brings embeddings of nearby text spans from the same document closer together.

  • Results

    DeCLUTR closes the performance gap with existing methods requiring human-labelled data when extending transformer-based language-model pretraining.

  • Takeaways & Limitations

    Embedding quality can be further improved by increasing model and unlabelled training-set size.

  • Takeaways & Limitations

    The sampling procedure assumes that nearby positive spans are more likely to be semantically similar, and evaluation excludes SNLI when comparing supervised approaches.

Abstract

from arXiv · show

Sentence embeddings are an important component of many natural language processing (NLP) systems. Like word embeddings, sentence embeddings are typically learned on large text corpora and then transferred to various downstream tasks, such as clustering and retrieval. Unlike word embeddings, the highest performing solutions for learning sentence embeddings require labelled data, limiting their usefulness to languages and domains where labelled data is abundant. In this paper, we present DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations. Inspired by recent advances in deep metric learning (DML), we carefully design a self-supervised objective for learning universal sentence embeddings that does not require labelled training data. When used to extend the pretraining of transformer-based language models, our approach closes the performance gap between unsupervised and supervised pretraining for universal sentence encoders. Importantly, our experiments suggest that the quality of the learned embeddings scale with both the number of trainable parameters and the amount of unlabelled training data. Our code and pretrained models are publicly available and can be easily adapted to new domains or used to embed unseen text.

1 Introduction

The paper addresses the limited availability of labelled data for high-performing universal sentence embeddings by proposing a self-supervised contrastive objective. It extends transformer pretraining without labels and reports strong transfer performance, with embedding quality scaling with model and data size.

  • Motivation: High-performing universal sentence encoders require labelled data, limiting their usefulness for languages and domains where labelled data is scarce.Sentence embeddings support downstream tasks including clustering and retrieval.
  • Approach: DeCLUTR proposes a self-supervised sentence-level objective that can pretrain transformer language models alongside MLM without labelled data.The objective is inspired by contrastive metric learning and targets generalized sentence- and paragraph-length embeddings.
  • Approach: The objective trains encoders to bring embeddings of textual segments sampled from nearby within the same document closer together.Contrastive learning treats other minibatch embeddings as negatives.
  • Results: DeCLUTR achieves state-of-the-art results on SentEval, a benchmark containing 28 tasks for evaluating universal sentence embeddings.The reported evaluation concerns transfer performance across sentence-embedding tasks.
  • Results: Embedding quality scales with model and data size, suggesting that larger encoders and more unlabelled text may improve performance.The paper presents this as an experimental finding and a possible route for further improvement.
  • Availability: The authors open-source the solution and provide instructions for training on new data or embedding unseen text.The release is intended to support adaptation to new domains and languages.

2 Related Work

Prior universal sentence-embedding methods include supervised, semi-supervised, and unsupervised approaches, with leading systems relying on labelled NLI data. DeCLUTR instead uses self-supervised contrastive pretraining with flexible span sampling that extends beyond strictly adjacent sentences.

  • Method categories: Universal sentence-embedding methods are grouped by whether their pretraining uses labelled data: supervised or semi-supervised versus unsupervised.This distinction concerns the use of labelled data during pretraining.
  • Supervised methods: The highest-performing universal sentence encoders are pretrained on labelled SNLI and MultiNLI natural language inference datasets.NLI classifies premise–hypothesis pairs as entailment, contradiction, or neutral.
  • Unsupervised methods: Unsupervised Skip-Thoughts and FastSent predict words in neighbouring sentences, but their generative objectives are computationally expensive and reconstruct surface form.The paper notes that surface-form reconstruction may capture information irrelevant to sentence meaning.
  • DeCLUTR: DeCLUTR extends transformer language-model pretraining with a self-supervised objective, avoiding dependence on labelled data.This permits use of unlabelled text without restriction to languages or domains with abundant labels.
  • DeCLUTR: Unlike QuickThoughts, DeCLUTR samples segments up to paragraph length, allows multiple positives, and permits adjacent, overlapping, or subsuming spans.These sampling choices broaden the span relationships used for contrastive training.

3 Model

DeCLUTR learns textual representations by contrasting nearby spans from the same document, using a transformer encoder, pooling, and a contrastive loss. Its sampling design creates diverse positives and both easy and hard negatives, while continued MLM pretraining combines the contrastive and MLM objectives.

  • DeCLUTR maximizes agreement between spans sampled from nearby within the same document.
  • Data loading: The data loader samples paired anchor-positive spans from each document, with multiple anchors and positives allowed per document.The method defines A anchor spans per document and P positive spans per anchor.
  • Encoder and pooling: A transformer language model encodes spans, while a pooling function maps token representations to fixed-length embeddings.The authors use mean pooling, which they report performs well.
  • Contrastive objective: The contrastive task identifies each anchor’s positive embedding among other minibatch embeddings using an InfoNCE or NT-Xent loss.Similarity is cosine similarity, and τ is the temperature hyperparameter.
  • Span sampling: Sampling uses paragraph-length spans and permits adjacent, overlapping, or subsuming positives, producing semantically related positives plus easy and hard negatives.Span lengths are clipped between 32 and 512 tokens; positives are sampled nearby, while negatives come from other or the same document.
  • Continued MLM pretraining: Continued pretraining sums the MLM and contrastive losses, using pretrained RoBERTa-base and DistilRoBERTa models to form DeCLUTR-base and DeCLUTR-small.

4 Experimental setup

The experiments train and compare DeCLUTR models using OpenWebText and evaluate them against supervised, unsupervised, and weak baselines on SentEval. The setup also controls for parameter scale, training data, and possible SNLI contamination.

  • Dataset: 497,868 OpenWebText documents with at least 2048 tokens form the unsupervised training corpus.The passage contrasts this corpus with baselines trained using up to 1 million labelled SNLI and MultiNLI sentence pairs.
  • Model configurations: Table 1 reports trainable parameter counts and sentence embedding dimensions for DeCLUTR-small and DeCLUTR-base.The two models are continued-pretraining versions of DistilRoBERTa and RoBERTa-base, respectively.
  • Training: Training uses minibatches of 16, two anchors, and two positives per anchor for each document, with τ = 5 × 10^-2 and AdamW optimization.
  • Evaluation: SentEval evaluates 18 downstream tasks and ten probing tasks covering transfer performance and encoded linguistic properties.Downstream examples include sentiment analysis, natural language inference, paraphrase detection, and image-caption retrieval.
  • Evaluation controls: Average downstream scores exclude SNLI when comparing supervised approaches because SNLI is itself a SentEval task.This avoids train-test contamination in those comparisons.
  • Baselines: Comparisons include InferSent, Google’s USE, Sentence Transformers, averaged GloVe, and fastText word vectors.The principal sentence-embedding comparisons use current models, with architecture and parameter-count matching for selected baselines.

5 Results

DeCLUTR improves downstream sentence-embedding performance over underlying pretrained models and remains competitive with existing methods without labelled training data. Ablations show benefits from multiple anchors, combined MLM and contrastive objectives, and larger datasets or models.

  • 5.1 Comparison to baselines: +6% average downstream performance is obtained by DeCLUTR-base over its underlying pretrained model, while DeCLUTR-small gains +4%.
  • 5.1 Comparison to baselines: DeCLUTR-base matches or outperforms average performance of existing methods without hand-labelled training data.
  • 5.1 Comparison to baselines: DeCLUTR preserves probing-task performance comparable to the underlying pretrained model while achieving high downstream performance.Sentence Transformers scores approximately 10% lower on probing tasks than its pretrained model.
  • 5.2 Ablation of the sampling procedure: Multiple anchors per document substantially improve learned embeddings, whereas multiple positives per anchor have minimal impact.The authors attribute the anchor effect to a more difficult contrastive objective and find complementary benefits from allowing adjacent and subsuming positives.
  • 5.3 Training objective, train set size and model capacity: Combining MLM with the contrastive objective improves performance over either objective alone and improves monotonically as training-set size increases.
  • 5.3 Training objective, train set size and model capacity: Embedding quality scales with model capacity and training-data size, enabling improvement through larger encoders or more unlabelled text.

6 Discussion and conclusion

The paper proposes a self-supervised objective for universal sentence embeddings that requires no labelled training data and evaluates it on SentEval. Extending transformer pretraining with this objective closes the gap with methods using human-labelled data, while larger models and datasets may further improve quality.

  • The proposed self-supervised objective learns universal sentence embeddings without requiring labelled training data.
  • On SentEval’s 28 tasks, the objective was effective for evaluating transferability and linguistic properties of sentence representations.SentEval contains 28 tasks designed for these evaluations.
  • When used to extend transformer-based language-model pretraining, the objective closes the performance gap with methods requiring human-labelled data.
  • Embedding quality scaled with model and training-set size, suggesting improvements from larger encoders or more unlabelled text.
  • The model and code are publicly released for extension to new domains and non-English languages.

A Pretrained transformers make poor universal sentence encoders

The section evaluates pretrained transformers as universal sentence encoders and finds that NSP or SOP training does not reliably improve CLS embeddings over a model trained without either objective. Mean pooling generally performs better than CLS pooling.

  • BERT, ALBERT, and RoBERTa were evaluated using NSP, SOP, or neither sentence-level loss, respectively.
  • CLS embeddings from models trained with NSP or SOP did not outperform embeddings from a model trained without either loss.
  • These CLS embeddings sometimes failed to outperform a bag-of-words baseline on SentEval.
  • Mean pooling of token embeddings outperformed pooling through the CLS classification token.

B Examples of sampled spans

The sampling procedure constructs anchor-positive and anchor-negative span pairs, illustrating several possible relationships between sampled positives and anchors and contrasting hard and easy negatives.

  • Sampled positives may be adjacent to, overlap with, or be subsumed by the anchor.
  • Each anchor-positive pair is shown with a hard negative from the same document and an easy negative from another document.
  • Minibatches are composed of random documents, providing the sampling context for these pair constructions.

C SentEval evaluation details

SentEval evaluates fixed-length sentence embeddings across downstream and probing tasks using standardized training and evaluation procedures. Table 4 compares sentence-level losses and pooling strategies, while semantic-similarity tasks use embeddings without further training.

  • SentEval evaluates fixed-length sentence embeddings across 18 downstream tasks and 10 probing tasks.
  • Its interface standardizes training, evaluation, and hyperparameters across sentence-embedding methods.
  • Most tasks train a logistic-regression or multi-layer-perceptron model using produced sentence embeddings as input.
  • Table 4 compares NSP, SOP, and neither-loss models under CLS and mean pooling on SentEval’s validation tasks.
  • Semantic text-similarity tasks use the embeddings as-is without further training, unlike typical GLUE evaluations that fine-tune the sentence-embedding model.

C.1 SentEval tasks

SentEval evaluates fixed-length sentence embeddings across downstream transfer tasks and probing tasks that test encoded linguistic properties. The benchmark spans classification, similarity, retrieval, and structural or lexical probes.

  • Benchmark structure: SentEval contains 18 downstream tasks and ten probing tasks for evaluating sentence representations.Downstream tasks assess transferability, while probing tasks assess which linguistic properties are encoded.
  • Downstream tasks: Downstream tasks include binary and multi-class classification, entailment and semantic relatedness, semantic textual similarity, paraphrase detection, and caption-image retrieval.The benchmark covers sentiment, natural language inference, similarity, paraphrase, and cross-modal retrieval settings.
  • Downstream tasks: Caption-image retrieval ranks images for text queries and captions for image queries using pretrained 2048-dimensional image embeddings.Both subtasks use data from COCO and evaluate relevance rankings.
  • Probing tasks: Probing tasks test lexical, syntactic, semantic, and grammatical properties such as word content, tree depth, bigram shift, tense, number agreement, and semantic oddity.Other probes include top constituents, coordinate inversion, and sentence length.
  • Probing tasks: Several probing tasks are binary or multi-class classification problems targeting word order, syntax, tense, grammatical number, and lexical substitution.The listed probes include object number, semantic odd man out, and coordinate inversion in addition to the other properties.

C.2 Computing an average score

Average probing scores are computed by averaging task accuracies, while average downstream scores use each task’s reported metric or a specified aggregation for multi-score tasks.

  • Probing scores: Average probing scores are the arithmetic mean of the accuracies from ten probing tasks.Each probing task reports a simple accuracy.
  • Downstream scores: For SICK-R, STS-B, and STS12–STS16, the downstream average uses the reported Spearman correlation or mean Spearman correlation.The procedure uses the mean correlation for benchmark groups with multiple subtasks.
  • Downstream scores: For MRPC, the downstream average uses the mean of the reported accuracy and F1 score.Both metrics contribute equally to the task’s aggregate score.
  • Downstream scores: For Caption-Image Retrieval, the downstream average uses the mean Recall@K across K ∈{1, 5, 10} for image and caption retrieval.This aggregates six Recall@K scores in total.
  • Downstream scores: For all other downstream tasks, the procedure uses the reported accuracy.This is the fallback aggregation rule when another metric is not specified.
Loading 2006.03659v4…