PaperlayerView exact revision on arXiv

Source-linked AI summary

Entailment as Few-Shot Learner

Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, Hao Ma

arXiv:2104.14690v1cs.CLcs.AI

TL;DR

Few-shot NLP remains difficult because adapting models across many tasks, domains, and languages requires labeled data. EFL reformulates classification and regression as textual entailment, and evaluations report 12% average improvement over existing few-shot methods.

  • Problem

    Few-shot learning addresses NLP settings where only a small number of labeled examples are available across many tasks, domains, and languages.

  • Method

    EFL reformulates classification and regression tasks as textual entailment by converting class labels into natural-language descriptions, with optional MNLI training and contrastive augmentation.

  • Results

    12% average improvement over various few-shot learning methods was reported across evaluations spanning 18 NLP tasks.

  • Takeaways & Limitations

    EFL improves few-shot performance across varied NLP tasks and shows significantly better Macro-F1 performance on some low-resource classes.

  • Takeaways & Limitations

    EFL does not extend to generative tasks such as translation or summarization.

Abstract

from arXiv · show

Large pre-trained language models (LMs) have demonstrated remarkable ability as few-shot learners. However, their success hinges largely on scaling model parameters to a degree that makes it challenging to train and serve. In this paper, we propose a new approach, named as EFL, that can turn small LMs into better few-shot learners. The key idea of this approach is to reformulate potential NLP task into an entailment one, and then fine-tune the model with as little as 8 examples. We further demonstrate our proposed method can be: (i) naturally combined with an unsupervised contrastive learning-based data augmentation method; (ii) easily extended to multilingual few-shot learning. A systematic evaluation on 18 standard NLP tasks demonstrates that this approach improves the various existing SOTA few-shot learning methods by 12\%, and yields competitive few-shot performance with 500 times larger models, such as GPT-3.

1 Introduction

Few-shot learning addresses the need to adapt language models across many tasks, domains, and languages with very little labeled data. EFL reformulates NLP tasks as textual entailment with fine-grained label descriptions, improving few-shot performance while avoiding reliance on extremely large models.

  • Motivation: Few-shot learning targets NLP settings where only a small number of labeled examples are available.The motivation is the large number of domains, tasks, and languages requiring adaptation.
  • Motivation: GPT-3 achieved strong prompted few-shot performance, but its capabilities depended heavily on scaling to 175 billion parameters.Its performance improved with as few as 16 examples per class, while the parameter scale made training and serving challenging.
  • EFL: Compared with prompt-based methods, EFL uses entailment tasks rather than cloze questions and fine-grained label descriptions rather than one task description.This design is illustrated as the distinction between the prompt-based and entailment-based approaches in Figure 1.
  • EFL: EFL reformulates NLP tasks as textual entailment by pairing each input with a natural-language description of the candidate label.A sentiment example becomes an entailment decision between the input and a sentence such as “This is a great movie.”
  • Results: 12% improvement over various few-shot learning methods was obtained across evaluations spanning GLUE, SNLI, BoolQ, and other sentence-classification tasks.The method also reported 1.9pt improvement with full training data and average 19pt improvement for multilingual few-shot learning.
  • Extensions: EFL can combine entailment reformulation with contrastive-learning data augmentation and extend to multilingual few-shot learning.The authors also argue that strong performance does not require a prohibitively large language model.

2 Related Work

Prior few-shot methods use demonstrations, cloze reformulations, intermediate training, or general data and optimization techniques. EFL differs from prompt-based approaches by using textual entailment and class-specific label descriptions, while related intermediate-training approaches can require extensive task sweeping.

  • Language modeling with demonstrations: GPT-style methods enable few-shot learning by supplying task descriptions and annotated examples as demonstrations.This paradigm has been applied to classification, question answering, commonsense knowledge mining, and probing.
  • Scope: EFL does not extend to generative tasks such as translation or summarization.This scope boundary is stated explicitly in the related-work discussion.
  • Intermediate training: Intermediate training across 11 tasks may require 110 fine-tuning runs to identify the best source task, limiting its task-agnostic character.The approach also requires knowledge of the downstream task when selecting a source task.
  • Task reformulation: Cloze-based methods reuse masked-language-modeling heads to predict labels from reformulated questions, but performance can be limited by distribution mismatch.These methods include LM prompt, PET, and LM-BFF.
  • Intermediate training: Intermediate training improves downstream performance by first training on data-rich supervised tasks before few-shot adaptation.The cited approach builds on pre-training with supervised source tasks and later matching-based few-shot learning.
  • General techniques: General techniques include fine-tuning optimization and regularization, semi-supervised data augmentation, and supervised contrastive learning.The authors anticipate these techniques are largely complementary to EFL.
  • EFL: EFL reformulates NLP tasks as textual entailment and provides label-specific descriptions instead of a single task description.These are the stated differences from existing prompt-based few-shot methods.

3 Framework of Entailment Training

EFL reformulates NLP tasks as textual entailment, using label descriptions to create a unified sentence-pair framework for few-shot learning. The framework supports classification and regression while reusing entailment-trained language-model parameters.

  • Design considerations: Label descriptions should remain semantically aligned with the task because unrelated descriptions degrade performance, whereas closely matched descriptions keep it relatively stable.The label-description study uses K = 8.
  • Scope and objective: The framework targets task-agnostic generalization from limited data across discrete classification and continuous regression label spaces.The default setting uses K = 8 training examples per class.
  • Entailment reformulation: EFL converts NLP classification and regression tasks into textual entailment by pairing each input with a natural-language label description.The model predicts whether the input entails the label description.
  • Unified framework: The framework consolidates one-sentence and sentence-pair tasks into the unified template softmax[WM(xin)[CLS]].The input xin contains the original sentence or sentence pair followed by a label description.
  • Data construction: For multi-class tasks, EFL assigns one label description p_k to each class k ∈ Y and reshapes class-specific data into entailment examples.The transformed examples pair each input with its corresponding class description.
  • Data construction: The reshaped training procedure samples K examples from alternative classes, while inference evaluates each candidate label description.This requires |Y| forward passes, unlike standard fine-tuning’s single forward pass.

4 Experiment

Across 18 NLP tasks, the experiments evaluate EFL against standard fine-tuning and other few-shot methods, including effects of training-data size and model size. EFL improves few-shot performance, especially with limited data, while entailment pre-training and contrastive augmentation contribute to its effectiveness.

  • 4.1 Data Statistics: 18 NLP tasks are evaluated across sentiment, topic classification, natural language inference, paraphrase, similarity, and question-answering settings.The benchmark includes GLUE, SNLI, BoolQ, and additional sentence-classification tasks.
  • 4.3 Main Results: At K=8, EFL achieves average 8.2% improvements across 15 tasks, with improvements reaching up to 55% over existing methods.In full-data training, EFL shows around 1.9pt average improvement over standard fine-tuning of RoBERTa-Large.
  • 4.3 Main Results: Removing entailment pre-training significantly reduces EFL performance in the 8-shot setting, although the gap narrows as K increases to 16 or 32.EFL without entailment training can outperform LM-BFF and standard fine-tuning once K exceeds 16, and performs similarly to EFL at K=32 in additional ablations.
  • 4.3 Main Results: Macro-F1 results show EFL performs significantly better than other methods on unbalanced multi-class benchmarks, indicating stronger performance on some low-resource classes.Macro-F1 averages F1 across classes and is reported alongside accuracy for these datasets.
  • 4.4 Impact of Training Data Scale: EFL improves more when annotated samples are scarce: on SST-2, K=8 performs as well as K=256, while on QNLI performance keeps improving with K.Figure 3 compares standard fine-tuning, LM-BFF, EFL, and EFL without entailment pre-training as K changes.
  • 4.5 Impact of Model Size: Model-size effects differ by method: LM-BFF and EFL improve with larger pre-trained models, whereas standard fine-tuning performs better with smaller models on SST-2 and QNLI.The comparison uses RoBERTa-base with 125M parameters and RoBERTa-Large with 355M parameters.

5 Optimizations

The framework extends EFL with unsupervised contrastive augmentation and multilingual few-shot learning. UCA improves performance across tasks, while EFL generalizes from English supervision to multilingual evaluation.

  • 5.1 Unsupervised Contrastive Learning: UCA constructs positive and negative sentence pairs from limited annotated data using sentence-level augmentation.Positive pairs preserve meaning through augmentation, while negatives come from mismatched or aggressively altered sentences.
  • 5.1 Unsupervised Contrastive Learning: 2.7pt average improvement across 15 tasks accompanies significant gains on 12 tasks with UCA.QQP improves by 14pt, and CoLA benefits from negative examples created through aggressive deletion or reordering.
  • 5.1 Unsupervised Contrastive Learning: UCA improves all 8 evaluated EFL tasks and yields average 3 to 4pt gains when combined with different training methods.The gains extend to standard fine-tuning and LM-BFF, especially on sentence-pair tasks.
  • 5.2 Multilingual Few-shot Learning: Multilingual few-shot learning requires generalization to both unseen test data and unseen languages.The multilingual setting uses language-specific test datasets while retaining the few-shot training setup.
  • 5.2 Multilingual Few-shot Learning: EFL can evaluate multilingual data using English label descriptions without translating them into target languages.The method follows the monolingual training construction and uses translated test benchmarks with XLM-R.
  • 5.2 Multilingual Few-shot Learning: 80.7 average accuracy versus 61.9 for standard fine-tuning across 7 multilingual tasks demonstrates EFL’s multilingual effectiveness.Translating training examples raises standard fine-tuning from 61.9 to 67.0, but EFL only increases from 80.7 to 81.3.

6 Conclusion

The paper concludes that EFL combines textual-entailment reformulation with unsupervised contrastive augmentation to improve few-shot learning. It reports gains of up to 55% and 12% on average, while identifying better label descriptions and entailment training tasks as future directions.

  • 6 Conclusion: EFL reformulates classification and regression tasks as textual entailment and adds unsupervised contrastive data augmentation.These are presented as the paper’s two simple and effective few-shot learning methods.
  • 6 Conclusion: 12% average improvement and up to 55% improvement are reported across systematic evaluations against various few-shot learning methods.The conclusion summarizes the reported gains across the paper’s evaluations.
  • 6 Conclusion: Future work includes selecting label descriptions with reinforcement learning and creating more effective entailment training tasks than MNLI.These directions target label-description choice and the source entailment task used for training.

A Experiment Details

The experiments specify fixed optimization settings and UCA augmentation rules, including distinct positive and negative perturbation strengths. Prompts and label descriptions are documented in Table 10.

  • Optimization settings: Few-shot experiments and ablations use learning rate 1e-5, batch size 8, 10 epochs, and standard Adam.Full-data experiments instead use batch size 32, weight decay 0.1, 10 epochs, and linear decay with warmup ratio 0.06.
  • UCA setup: UCA randomly augments 8 data examples for each class.This is the stated default setup for unsupervised contrastive augmentation.
  • UCA setup: Positive augmentation uses relatively mild character, span, word-deletion, and word-reordering perturbations.The listed probabilities are 10%, 10%, 40%, and 40%, with 15% deletion rates for characters or words.
  • UCA setup: Negative augmentation uses stronger deletion and reordering operations than positive augmentation.Negative generation raises deletion rates to 40% for characters and words and applies larger span perturbations.
  • Sample construction: For one-sentence tasks, positive samples pair the original input with an augmented sentence or the label description.The two pair orders are sampled with equal probability, while negative data combines down-sampled and UCA-generated negatives.
  • Sample construction: For sentence-pair tasks, positive samples replace either sentence with its augmented version.The two replacement configurations are sampled with equal probability, and negatives follow the one-sentence construction.
  • Prompt specification: Table 10 documents the prompts and label descriptions used for prompt-based fine-tuning and EFL.The table provides the experiment’s task-specific textual formulations.

B Benchmark

The benchmark covers 12 named datasets spanning GLUE, SNLI, IMDB, Yelp, AG News, and related sentence-classification tasks. It includes both single-sentence and sentence-pair evaluation settings.

  • Datasets: The benchmark includes 8 GLUE datasets: CoLA, SST-2, MPRC, QQP, STS-B, MNLI, QNLI, and RTE.It also includes IMDB, Yelp, AG News, and SNLI.
  • Task coverage: The evaluated datasets span linguistic acceptability, sentiment, paraphrase, similarity, inference, question entailment, topic classification, and review classification.These task types are represented by the named benchmark datasets.
Loading 2104.14690v1…