PaperlayerView exact revision on arXiv

Source-linked AI summary

A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectives

Nils Rethmeier, Isabelle Augenstein

arXiv:2102.12982v1cs.CLcs.AIcs.CV

TL;DR

Contrastive NLP’s benefits and best practices remain less established than contrastive pretraining in computer vision. This primer surveys supervised and self-supervised methods and reports benefits across several NLP applications, including language modeling, zero-shot learning, and data efficiency.

  • Problem

    The benefits and best practices of contrastive pretraining in NLP are comparatively less established than in computer vision.

    Source: §1 · p. 1

  • Method

    The primer synthesizes recent supervised and self-supervised contrastive NLP methods, design lessons, and connections to energy-based models.

    Source: §1 · p. 2

  • Results

    Reviewed contrastive NLP methods show performance and data-efficiency benefits across zero-shot and few-shot learning, language modeling, and rare-event prediction.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The paper identifies open challenges and underrepresented directions for future contrastive NLP research.

    Source: §5 · p. 6

Abstract

from arXiv · show

Modern natural language processing (NLP) methods employ self-supervised pretraining objectives such as masked language modeling to boost the performance of various application tasks. These pretraining methods are frequently extended with recurrence, adversarial or linguistic property masking, and more recently with contrastive learning objectives. Contrastive self-supervised training objectives enabled recent successes in image representation pretraining by learning to contrast input-input pairs of augmented images as either similar or dissimilar. However, in NLP, automated creation of text input augmentations is still very challenging because a single token can invert the meaning of a sentence. For this reason, some contrastive NLP pretraining methods contrast over input-label pairs, rather than over input-input pairs, using methods from Metric Learning and Energy Based Models. In this survey, we summarize recent self-supervised and supervised contrastive NLP pretraining methods and describe where they are used to improve language modeling, few or zero-shot learning, pretraining data-efficiency and specific NLP end-tasks. We introduce key contrastive learning concepts with lessons learned from prior research and structure works by applications and cross-field relations. Finally, we point to open challenges and future directions for contrastive NLP to encourage bringing contrastive NLP pretraining closer to recent successes in image representation pretraining.

1 Introduction

Contrastive pretraining supports supervised and self-supervised representation learning, but its benefits and best practices in NLP remain less established than in computer vision. This primer surveys methods, applications, design lessons, and open research directions.

1 Introduction

  • NLP contrastive pretraining’s benefits and best practices remain comparatively less established than those in computer vision.The introduction points to emerging evidence of performance and data-efficiency benefits.Source: §1 · p. 1
  • Figure 1 groups contrastive-pretraining types with associated works and marks text-image contrastive works.Source: Figure 1 · p. 1
  • Reviewed methods report benefits for zero-shot and few-shot learning, language modeling, data efficiency, and rare-event prediction.Examples include task-agnostic self-supervision for language modeling and supervised contrastive pretraining for unseen text classes.Source: §1 · pp. 1–2
  • The primer surveys supervised and self-supervised methods, explains links to energy-based models, and identifies challenges and future directions.Source: §1 · p. 2

2 Contrastive Learning Concepts and Benefits

Contrastive learning compares similar and dissimilar pairs, using either different views of text or text-label pairs. The section outlines NCE, sampling lessons, and connections to other learning frameworks, alongside the computational trade-offs and reported benefits of input-output and input-input methods.

2 Contrastive Learning Concepts and Benefits

  • Contrastive methods learn from positive pairs representing different views of one instance and negative pairs drawn from different instances; NLP also uses text-label pairs.In input-label pairs, the label embedding represents a short-text description of the label.Source: §2 · p. 2

2.1 Noise Contrastive Estimation (NCE)

  • NCE is used by most NLP contrastive approaches and can be understood as a sub-sampled softmax with K negative samples.The section describes binary and ranking variants; a compatibility score measures how well a text input matches another sample.Source: §2.1 · p. 2
  • More negatives improve performance only when sampled from the same context, while long-tail positive and negative samples are reported to aid rare-class prediction and zero-shot performance.Weak negatives can impair learning of conceptually related classes.Source: §2.1 · p. 2

2.2 Contrastive Learning as Mutual Information Maximization, Inverse Data Generation and Energy Based Models:

  • Contrastive learning relates to mutual-information maximization, inverse data generation, metric learning, and energy-based models.InfoNCE is described as maximizing a lower bound on mutual information between data views.Source: §2.2 · pp. 2–3
  • Input-output methods encode text and labels separately, allowing many label views and large-scale negative sampling with relatively little computation.They can use sampled input words as pseudo-labels for self-supervised pretraining and reuse the pretrained NCE classifier for zero-shot transfer.Source: §2.2 · p. 3, Figure 2 · p. 3
  • Input-input methods contrast original texts with augmented texts, but encoding those augmentations is computationally intensive.Clark et al. [2020] replace a subset of input words with vocabulary-sampled words for augmentation.Source: §2.2 · p. 3, Figure 3 · p. 3
  • Supervised contrastive pretraining supports zero-shot and improved few-shot prediction, while self-supervised methods report zero-shot learning and stronger long-tail performance, including with 6–60MB of pretraining text.Input-output methods can also increase self-supervised signals instead of adding more pretraining data.Source: §2.2 · p. 4

3 Self- or Supervised Contrastive Pretraining

The primer surveys self- and supervised contrastive pretraining methods, from automated text views and sentence prediction to input-label, distillation, and cross-modal approaches. It describes reported benefits including zero-shot and few-shot learning, rare-event prediction, and improved performance on specific tasks.

3.1 Self-supervised Contrastive Pretraining

  • Automated input-input methods combine text augmentations, with Qu et al. finding mixtures most useful when views differ and using consistency and contrastive losses to balance semantic consistency and view difference.Their augmentation-embedding memory bank increases the number of sampled negative augmentations.Source: §3.1 · p. 4
  • For contrastive NSP, random negative sentences reduce computation but sacrifice text generation, while BERT studies found same-document negatives critical and removal of its original NSP task improved downstream performance.Surrounding-sentence prediction also uses out-of-window sentences from the same paragraph as hard negatives.Source: §3.1 · p. 4
  • Random input words used as pseudo-labels support unsupervised zero-shot prediction and improve few-shot and rare-concept learning despite a very small pretraining corpus.Separate label and text encoders allow many negative labels to be computed while encoding the input text once.Source: §3.1 · pp. 4–5
  • CoDIR distills a pretrained teacher into a smaller student by contrasting their representations of matching and different texts, using same-topic inputs to mine hard negatives.It uses cosine similarity between mean-pooled Transformer layer representations with InfoNCE.Source: §3.1 · p. 5

3.2 Supervised Contrastive Pretraining

  • Contrastive text continuation scoring can improve language-model generation perplexity, while contrastive attention losses in abstractive summarization emphasize summary-relevant tokens but require a more complex dual-head network.The generation approach is computationally expensive.Source: §3.1 · p. 5, §3.2 · p. 6

3.1 Self-supervised Contrastive Pretraining

  • CLIP and ALIGN use image-caption contrastive pretraining for zero-shot image classification and cross-modal generation; in CLIP, a data-efficient CNN text encoder yields 3-fold faster zero-shot prediction convergence than a Transformer encoder.Source: §3.1 · p. 5

3.2 Supervised Contrastive Pretraining

  • Supervised input-label pretraining represents labels with textual descriptions, enabling zero-shot prediction on unseen classes; reported methods also boost few-shot learning, and self-supervision improves rare-event learning and convergence speed.Task-specific contrastive learning also produces strong performance increases in unsupervised and supervised commonsense reasoning.Source: §3.2 · pp. 5–6
  • Supervised cross-modal methods represent unseen classes with text descriptions, while contrastive learning on limited stock-series data improves the fluency and non-repetitiveness of generated text.A contrastive sentence corpus for machine-translation word-sense disambiguation is built by identifying ambiguous words and mining replacement candidates.Source: §3.2 · p. 6

4 Challenges and Potential Directions

The primer identifies challenges in negative sampling and text augmentation, alongside under-researched applications and directions for cross-modal generation, model fusion, and commonsense learning.

Challenges

  • Contrastive methods often need many negatives, but the useful balance of hard, medium, and easy samples—and the role of diverse negatives—depends on the task and remains underexplored.The primer notes that negative difficulty may matter especially for generalization in open-class pretraining.Source: §4 · p. 6

Challenge and directions: text augmentation quality and efficiency

  • Text augmentation methods can be computationally expensive or non-robust, and many self-supervised approaches still rely on pretrained Transformers, leaving efficient and scalable alternatives desirable.Input-output methods avoid some of these requirements, while scalable augmentation and approaches without negative sampling are proposed as directions.Source: §4 · p. 6

Challenge: under-researched applications

  • Contrastive text generation and abstractive summarization are identified as under-researched applications.The primer points to Momentum Contrast as a possible route for improving contrastive summarization.Source: §4 · p. 6

Direction: cross-modal generation

  • Generating text from non-text inputs remains an under-researched direction, with stock-series descriptions offered as an example of contrastive data-to-text learning using limited data.The primer also points to cross-modal work reporting multiplied zero-shot learning efficiency.Source: §4 · p. 6

Direction: contrastive (language) model fusion

  • Contrastive language-model distillation is proposed as a basis for future work that fuses multiple language models or transfers knowledge between them.The cited distillation work compresses a large language model.Source: §4 · p. 6

Direction: commonsense contrastive learning

  • The primer suggests adapting contrastive word-sense disambiguation dataset construction to automatically mine inputs for contrastive pronoun learning.The proposed connection is between the dataset method of Raganato et al. and the pronoun-learning method of Klein and Nabi.Source: §4 · p. 6

5 Conclusion

The conclusion summarizes the primer’s survey of contrastive learning concepts and cross-field relations, its organization of methods as self- or supervised, and its identification of challenges and future directions.

5 Conclusion

  • The primer surveys contrastive learning concepts and cross-field relations, organizes methods as self- or supervised learning, and highlights challenges and future research directions.Source: §5 · p. 6

Related papers

Loading 2102.12982v1…