PaperlayerView exact revision on arXiv

Source-linked AI summary

ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer

Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, Weiran Xu

arXiv:2105.11741v1cs.CLcs.AI

TL;DR

BERT-derived sentence representations collapse and perform poorly on STS, while labeled supervision is costly or unavailable. ConSERT fine-tunes BERT with self-supervised contrastive learning over augmented sentence views. It improves STS performance, reaches new state-of-the-art results with NLI supervision, and remains effective with limited unlabeled data.

  • Problem

    BERT-derived native sentence representations collapse into a small region, producing high similarity even for unrelated sentences, while human annotation is costly and often unavailable.

  • Method

    ConSERT fine-tunes BERT on unlabeled target-distribution texts using contrastive learning and sentence augmentations.

  • Results

    8% relative performance gain over BERTlarge-flow is achieved across 6 STS datasets, while supervised variants achieve new state-of-the-art performance.

  • Takeaways & Limitations

    ConSERT transfers sentence representations without extra inference structure and remains robust when only small amounts of unlabeled target data are available.

  • Takeaways & Limitations

    BERT-flow, a compared unsupervised approach, introduces extra model structures and requires specialized implementation, potentially limiting application.

Abstract

from arXiv · show

Learning high-quality sentence representations benefits a wide range of natural language processing tasks. Though BERT-based pre-trained language models achieve high performance on many downstream tasks, the native derived sentence representations are proved to be collapsed and thus produce a poor performance on the semantic textual similarity (STS) tasks. In this paper, we present ConSERT, a Contrastive Framework for Self-Supervised Sentence Representation Transfer, that adopts contrastive learning to fine-tune BERT in an unsupervised and effective way. By making use of unlabeled texts, ConSERT solves the collapse issue of BERT-derived sentence representations and make them more applicable for downstream tasks. Experiments on STS datasets demonstrate that ConSERT achieves an 8\% relative improvement over the previous state-of-the-art, even comparable to the supervised SBERT-NLI. And when further incorporating NLI supervision, we achieve new state-of-the-art performance on STS tasks. Moreover, ConSERT obtains comparable results with only 1000 samples available, showing its robustness in data scarcity scenarios.

1 Introduction

BERT-derived native sentence representations collapse into a narrow similarity range, limiting STS use when labeled data are costly or unavailable. ConSERT addresses this with contrastive learning and augmentation, improving transfer under unsupervised, supervised, and data-scarce settings.

  • Motivation: BERT sentence representations are collapsed, mapping nearly all sentences into a small area and producing uniformly high similarity scores.On STS, almost all sentence pairs receive scores between 0.6 and 1.0, including pairs judged unrelated by humans.
  • Motivation: BERT’s anisotropic word space lets high-frequency words dominate averaged sentence representations, biasing them against real semantics.High-frequency words cluster near the origin while low-frequency words disperse sparsely.
  • Motivation: Human annotation is costly and often unavailable, motivating methods that transfer sentence representations with less labeled data.Traditional approaches usually fine-tune BERT with additional supervision.
  • Approach: ConSERT uses contrastive learning to pull augmented views of the same sentence together and push views from different sentences apart.The framework reshapes BERT’s sentence representation space and uses adversarial attack, token shuffling, cutoff, and dropout augmentations.
  • Results: 35% relative performance gain over BERT is achieved using only 1,000 unlabeled texts, with training taking a few minutes on one V100 GPU.ConSERT adds no inference structure, retains BERT’s parameter size, and uses augmentations with minimal semantic impact.
  • Results: ConSERT achieves new state-of-the-art performance with NLI supervision and demonstrates robustness in data scarcity scenarios.The contributions also include analysis of transferred representations.

2 Related Work

Prior work includes supervised, self-supervised, unsupervised, and contrastive approaches to sentence representation learning. BERT-flow improves comparability through a Gaussian latent space but introduces extra structures and specialized implementation.

  • Supervised Approaches: Supervised approaches train sentence encoders with NLI datasets, while Universal Sentence Encoder combines Transformer architecture with SNLI-based supervision.SBERT is also identified as a supervised sentence representation approach.
  • Self-supervised Objectives: BERT pre-training includes next sentence prediction, but prior work finds NSP weak and contributes little to final performance.Other self-supervised objectives were subsequently proposed for BERT-like sentence encoders.
  • Unsupervised Approaches: BERT-flow maps BERT embeddings into a standard Gaussian latent space where embeddings are more suitable for comparison.The approach is presented as an unsupervised method for improving BERT embedding geometry.
  • Unsupervised Approaches: BERT-flow introduces extra model structures and requires specialized implementation, potentially limiting application.This limitation motivates simpler approaches that preserve BERT’s native inference structure.
  • Contrastive Learning: Contrastive learning distinguishes views of the same object from views of other objects using transformed inputs.Visual methods commonly use transformations such as cropping, rotation, and cutout.
  • Contrastive Learning: NLP studies apply contrastive learning to textual representation learning, including methods that maximize mutual information or adapt visual contrastive frameworks.The cited approaches include IS-BERT and CERT.

3 Approach

ConSERT fine-tunes a BERT-like encoder with contrastive learning, using augmented views of unlabeled texts to reshape sentence representations. The framework combines data augmentation, shared encoding, contrastive loss, and optional NLI supervision.

  • General Framework: ConSERT fine-tunes a BERT-like model on an unsupervised target dataset to make sentence representations more task-relevant for downstream tasks.The framework is designed for sentence representation transfer from unlabeled texts drawn from the target distribution.
  • General Framework: The framework generates augmented views, encodes them with a shared BERT encoder, and applies contrastive loss to their sentence representations.During training, average pooling of the final-layer token embeddings produces sentence representations.
  • General Framework: Two transformations produce two token-embedding views for each input, which BERT encodes into representations through average pooling.For input x, the transformations are written as e_i = T1(x) and e_j = T2(x), with e_i and e_j in R^L×d.
  • General Framework: NT-Xent trains each augmented representation to identify its counterpart among 2(N − 1) in-batch negatives.A mini-batch of N texts yields 2N representations, and cosine similarity and temperature determine the contrastive scores.
  • Data Augmentation Strategies: The four augmentation strategies are adversarial attack, token shuffling, cutoff, and dropout.Adversarial attack is applicable only with supervision because it relies on supervised loss to compute perturbations; token shuffling changes position ids, while cutoff erases tokens or feature dimensions and dropout independently zeros embedding elements.
  • Incorporating Supervision Signals: NLI supervision distinguishes contradiction, entailment, and neutral relations between sentence pairs and can be combined with contrastive learning in three training schedules.The schedules are joint, supervised-then-unsupervised transfer, and joint-then-unsupervised transfer.

4 Experiments

ConSERT is evaluated on STS datasets under unsupervised and supervised settings, using Spearman correlation between sentence-representation similarities and human scores. It improves over BERT-based baselines and reaches state-of-the-art performance with additional supervision.

  • 4.1 Setups: The experiments evaluate ConSERT on STS12–STS16, STSb, and SICK-R using unlabeled target-dataset texts for unsupervised fine-tuning.Each sample contains a sentence pair and a human similarity score; the seven datasets are mixed for unsupervised training.
  • 4.1 Setups: Evaluation reports Spearman correlation between cosine similarities of sentence representations and human-annotated gold scores.Representations are obtained by averaging token embeddings from BERT’s last two layers.
  • 4.2 Unsupervised Results: 8% relative gain raises ConSERTlarge from 70.76 to 76.45 over BERTlarge-flow across six STS datasets.ConSERTlarge also exceeds InferSent and Universal Sentence Encoder, while remaining comparable to SBERTlarge-NLI at 76.55.
  • 4.2 Unsupervised Results: ConSERTbase improves 3.17 points over BERTbase-flow, increasing from 69.57 to 72.74.Both methods improve the representation space compared with GloVe and native BERT embeddings.
  • 4.3 Supervised Results: With NLI supervision, ConSERTjoint gains 2.88 points over reimplemented SBERTbase-NLI, while ConSERTlarge joint gains 2.70.Further transfer using STS unlabeled texts adds 1.84 points over ConSERTlarge joint and 2.92 over BERTlarge-flow.

5 Qualitative Analysis

Qualitative analyses examine why ConSERT improves BERT representations, how augmentation choices affect performance, and how robust the method is to data and optimization settings. The results implicate frequent-token dominance, favor token-level augmentations, and show strong few-shot performance.

  • 5.1 Analysis of BERT Embedding Space: Removing 34 frequent tokens raises BERT’s average STS Spearman correlation from 53.86 to 61.66, while ConSERT changes by less than 0.3.This supports the claim that ConSERT reduces the influence of common tokens by reshaping BERT’s embedding space.
  • 5.2 Effect of Data Augmentation Strategy: Shuffle and Token Cutoff are the most effective augmentation strategies, with Shuffle slightly outperforming Token Cutoff.They outperform Feature Cutoff and Dropout, likely because token-level changes produce hard examples related to STS tasks.
  • 5.2 Effect of Data Augmentation Strategy: Combining Shuffle with Feature Cutoff achieves the best augmentation result, while Feature Cutoff and Dropout each improve roughly 4 points over None-None.These strategies are described as complementary and useful for modeling internal noise invariance.
  • 5.2 Effect of Data Augmentation Strategy: Even without augmentation, contrastive training improves BERT from 53.86 to 63.84 by pushing representations apart.The authors attribute this result to correcting collapse in BERT’s native representation space.
  • 5.4 Few-shot Experiments: With 1,000 unlabeled texts, ConSERT achieves results comparable to models trained on the full dataset in both unsupervised and supervised settings.The few-shot study uses 1, 10, 100, 1,000, and 10,000 texts, plus all 89,192 available texts.
  • 5.5 Influence of Temperature: The optimal NT-Xent temperature is 0.1, with the best range approximately 0.08–0.12.Performance is sensitive to temperature; both excessively small and large values perform poorly.
  • 5.5 Influence of Batch Size: Larger batch sizes improve performance modestly and accelerate training, but require more GPU memory.The analysis reports both Spearman correlation and corresponding training steps.

6 Conclusion

The paper concludes that ConSERT transfers BERT sentence representations through self-supervised contrastive learning without extra inference structure. Experiments report state-of-the-art STS performance and robustness when unlabeled data are scarce.

  • 6 Conclusion: ConSERT is a self-supervised contrastive framework for transferring sentence representations to downstream tasks without adding structure to the encoder.The framework is described as easy to implement for any encoder.
  • 6 Conclusion: On various STS datasets, both unsupervised and supervised ConSERT methods achieve new state-of-the-art performance.Few-shot experiments also indicate robustness in data-scarcity scenarios.

Broader Impact

ConSERT is presented as useful for practical sentence-representation transfer, particularly when labeled data are unavailable or scarce. Its unlabeled training data should remain ethical, unbiased, and closely related to downstream tasks.

  • Broader Impact: Users can fine-tune BERT with a few unlabeled texts from the target distribution in a self-supervised manner.The authors state that this avoids bias from human annotations, while random-sampling augmentations have little probability of introducing extra biases.
  • Broader Impact: Unlabeled training texts can still introduce data bias, so they should be ethical, unbiased, and closely related to downstream tasks.This is the paper’s stated practical boundary for applying ConSERT.
Loading 2105.11741v1…