PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving In-Context Few-Shot Learning via Self-Supervised Training

Mingda Chen, Jingfei Du, Ramakanth Pasunuru, Todor Mihaylov, Srini Iyer, Veselin Stoyanov, Zornitsa Kozareva

arXiv:2205.01703v2cs.CL

TL;DR

In-context few-shot learning uses a few examples to solve unseen tasks, but standard self-supervised pretraining objectives are not specifically adapted to that setting. The paper inserts self-supervised training with downstream-like task formats between pretraining and evaluation; across two benchmarks, its best model outperforms strong baselines on average.

  • Problem

    Self-supervised pretraining objectives are not typically adapted specifically for in-context few-shot learning, where models solve unseen tasks by conditioning on a few examples.

    Source: §1 · p. 1

  • Method

    The paper fine-tunes pretrained checkpoints on datasets built from four self-supervised objectives formatted similarly to downstream tasks, then evaluates them on benchmarks.

    Source: §1 · p. 1

  • Results

    Across SuperGLUE and Natural-Instructions, self-supervised training achieves the best average performance; the best 1.3-billion-parameter model beats initial pretrained models and two strong baselines on average.

    Source: §1 · p. 1, §4.3 · p. 6

  • Takeaways & Limitations

    The analysis finds that training-data amount, task diversity, and template similarity affect performance, while human-annotated supervision complements self-supervision.

    Source: §1 · p. 1, §6 · p. 9

    The data-volume analysis aims to show rough performance trends, not estimate the exact training-data requirement.

    Source: §5.2 · p. 7

Abstract

from arXiv · show

Self-supervised pretraining has made few-shot learning possible for many NLP tasks. But the pretraining objectives are not typically adapted specifically for in-context few-shot learning. In this paper, we propose to use self-supervision in an intermediate training stage between pretraining and downstream few-shot usage with the goal to teach the model to perform in-context few shot learning. We propose and evaluate four self-supervised objectives on two benchmarks. We find that the intermediate self-supervision stage produces models that outperform strong baselines. Ablation study shows that several factors affect the downstream performance, such as the amount of training data and the diversity of the self-supervised objectives. Human-annotated cross-task supervision and self-supervision are complementary. Qualitative analysis suggests that the self-supervised-trained models are better at following task requirements.

1 Introduction

The paper proposes intermediate self-supervised training with objectives designed to resemble downstream task formats, targeting in-context few-shot learning. Its best setting outperforms initial pretrained models and two strong baselines on average, with analyses identifying factors that affect performance.

1 Introduction

  • Prior work has improved in-context few-shot learning through inference-time example selection, ordering, or output rescoring, and through human-annotated resources.The paper instead focuses on self-supervised objectives.Source: §1 · p. 1
  • The authors hypothesize that self-supervised objectives closer to downstream task formats can improve in-context few-shot learning, and use them in an intermediate stage after pretraining.They construct similarly formatted self-supervised datasets, fine-tune pretrained checkpoints, and evaluate the models on benchmarks.Source: §1 · p. 1
  • Experiments use four self-supervised objectives and evaluate 13 tasks across SuperGLUE’s discriminative tasks and Natural-Instructions’ generative tasks.The objectives include masked word prediction and classification tasks related to next sentence prediction (Devlin et al., 2019).Source: §1 · p. 1
  • The best 1.3-billion-parameter self-supervised model outperforms initial pretrained models and two strong baselines on average.The experiments compare pretrained models with 125 million and 1.3 billion parameters.Source: §1 · p. 1
  • The analysis finds that training-data amount, task diversity, and template similarity affect performance; human-annotated supervision is complementary, and trained models better follow instructions.Benefits from more training data diminish; adding objectives may not help without greater task diversity.Source: §1 · p. 1

2 Related Work

Prior work addresses few-shot learning through inference-time procedures, fine-tuning, adapted pretraining, and intermediate training. This paper distinguishes its focus as self-supervised training for in-context few-shot learning without downstream-example fine-tuning.

2 Related Work

  • Prior in-context few-shot work adjusts inference through output rescoring, example selection, and example ordering, or uses human-annotated datasets and resources.Related approaches include instructions and retrieved examples.Source: §2 · p. 2
  • Few-shot fine-tuning research designs templates, optimizes soft prompts, or unifies task formats, whereas this work targets in-context learning without fine-tuning on downstream examples.The cited unified formats cover question answering, textual entailment, and other tasks.Source: §2 · p. 2
  • Prior few-shot pretraining adapts resources or targets specific tasks; this work uses plain text, unlike human-annotated resources, and differs from self-supervised meta-learning for classification.Intermediate fine-tuning has also used human-annotated or crosslingual supervision, while Rubino and Sumita (2020) studied intermediate self-supervision for machine translation quality estimation.Source: §2 · p. 2

3 Method

The method trains pretrained language models on self-supervised tasks formatted as input–output examples and multi-example instances, then applies four task designs to teach prediction and comparison across examples.

3 Method

  • Training instances concatenate examples from one task; causal attention lets earlier examples act as demonstrations, and cross-entropy loss is computed on output tokens.Examples are built from consecutive sentences using task-specific rules, then sampled into instances up to 2048 tokens.Source: §3 · p. 2, Figure 2 · p. 3, Figure 1 · p. 3

3.1 Next Sentence Generation

  • Next sentence generation (NSG) uses preceding sentences as input and the last sentence as the target to generate.The task uses language modeling as a self-supervised objective.Source: §3.1 · p. 3

3.2 Masked Word Prediction

  • Masked word prediction (MWP) replaces 1–20 input words with a special token and trains the model to recover them.The masked-out words serve as the output text.Source: §3.2 · p. 3

3.3 Last Phrase Prediction

  • Last phrase prediction (LPP) asks models to predict a sentence’s final phrase through generation or binary classification, with the phrase identified from function words.For classification, candidate answers are drawn from phrases extracted using the same function words, making negatives more challenging.Source: §3.3 · pp. 3–4

3.4 Classification

  • The classification task combines original, shuffled, or cross-document sentence inputs and assigns labels that models must infer by comparing examples.Instances combine two or three input types; cross-document variants include replacing some or all sentences with text from another document.Source: §3.4 · p. 4

4 Experiment

The experiments compare intermediate self-supervised training with pretrained and cross-task baselines on SuperGLUE and Natural-Instructions. The proposed training achieves the best average performance on both benchmarks.

4.1 Training Setup

  • Training uses about 1 million instances from four domains, with 250,000 instances per self-supervised task.The 125M model trains for 10 epochs; the 1.3B model trains for 5 epochs.Source: §4.1 · p. 4

4.2 Evaluation Setup

  • Evaluation covers SuperGLUE understanding tasks and Natural-Instructions generation tasks, using few-shot demonstrations and benchmark-specific evaluation procedures.SuperGLUE uses official development sets, random training examples, five runs, and perplexity-based label ranking; Natural-Instructions evaluates four task categories with 100 examples per task and greedy decoding.Source: §4.2 · p. 5
  • SuperGLUE templates are changed to resemble the input-output format of the self-supervised tasks, including a MultiRC template modeled on classification LPP.The MultiRC template uses language-model perplexities to rank labels.Source: §4.2 · p. 5, Table 1 · p. 5
  • The baselines include pretrained models, extra language-model training, cross-task fine-tuning, and same-task fine-tuning as an approximate upper bound.CrossTask estimates performance from human-annotated datasets, while SameTask uses evaluation-task training sets.Source: §4.2 · p. 5

4.3 Results

  • The proposed self-supervised training achieves the best average performance on both SuperGLUE and Natural-Instructions.The results are presented in the SuperGLUE and Natural-Instructions tables.Source: §4.3 · p. 6, Table 2 · p. 6, Table 3 · p. 6

2. ExtraLM and NewTemplate show similar perfor-

ExtraLM and NewTemplate perform similarly to pretrained checkpoints, suggesting the reported improvements are unlikely to come from additional training alone or template changes. CrossTask performs worse than the pretrained checkpoints on both benchmarks, likely because its training and evaluation tasks differ.

2. ExtraLM and NewTemplate show similar perfor-

  • Figure 4 reports 1.3B model averages on both benchmarks while varying the amount of self-supervised training data.Source: Figure 4 · p. 6
  • ExtraLM and NewTemplate perform similarly to pretrained checkpoints, making additional training and template changes unlikely explanations for the improvements.The comparison concerns ExtraLM's additional language-model training and evaluation with new templates.Source: §2 · p. 6
  • CrossTask performs worse than pretrained checkpoints on both benchmarks, likely reflecting differences between training and evaluation tasks.Source: §2 · p. 6

5 Analysis

The analysis finds that self-supervised training data amount and task diversity affect downstream performance. It also examines task combinations, template alignment, shot settings, complementarity with human supervision, and generated outputs.

5.1 Effect of Amount of Data

  • Improvements become clearer as self-supervised training data increases, then plateau around 100% of the data; at 1%, performance is worse.The authors describe these results as rough trends, not an exact estimate of the data required.Source: §5.1 · pp. 6–7

5.2 Effect of Individual Self-Supervised Tasks

  • Combining all four tasks yields the biggest improvements for most tasks, while individual benefits vary and classification learning helps SuperGLUE but hurts Natural-Instructions.The authors suggest task similarity may explain individual benefits and hypothesize that SuperGLUE’s ranking-based format favors classification-related training.Source: §5.2 · p. 7

5.3 Effect of More Self-Supervised Tasks

  • Adding denoising autoencoding and gap sentence generation does not improve downstream performance, which the authors attribute to insufficient task diversity.The added tasks share similarities with existing tasks, including gap sentence generation’s inference style with masked word prediction.Source: Table 6 · p. 7, §5.3 · p. 8

5.4 Effect of Few-Shot Templates

  • Random labels hurt performance, while templates similar to those used in self-supervised training give better SuperGLUE results.The random-label result suggests models learn more than few-shot templates alone.Source: Table 7 · p. 7, Table 8 · p. 7, §5.4 · p. 8

5.5 Zero-Shot vs. One-Shot vs. Few-Shot

  • Self-supervised training helps most in the few-shot setting and does not help in zero-shot, suggesting improved in-context learning capabilities.Table 11 reports average SuperGLUE performance for zero-shot, one-shot, and few-shot settings.Source: §5.5 · p. 8, Table 11 · p. 9

5.6 Combine Self-Supervision with Cross-Task Human-Supervision

  • Combining self-supervised and human-annotated CrossTask tasks yields better average performance, indicating that the supervision types are complementary.Table 9 reports Natural-Instructions results for the combined tasks.Source: Table 9 · p. 7, §5.6 · p. 9

5.7 Generation Examples

  • Generated examples suggest self-supervised models use task-input information more effectively and follow task requirements better than pretrained models.Examples include less semantic drift and answers more relevant to the question, although one self-supervised answer only partially matches the reference.Source: Table 10 · p. 8, §5.7 · p. 9

6 Conclusion

The paper evaluates four self-supervised objectives in an intermediate training stage and reports better average performance than strong baselines. Its analysis identifies factors associated with downstream performance and finds complementarity with human-annotated data.

6 Conclusion

  • Four self-supervised objectives are evaluated on two few-shot benchmarks in an intermediate stage between pretraining and downstream evaluation.The benchmarks are SuperGLUE and Natural-Instructions.Source: §6 · p. 9
  • Models trained with the objectives achieve the best average performance compared with strong baselines.The conclusion summarizes the average comparison without specifying numerical values.Source: §6 · p. 9
  • Downstream performance varies with training-data amount and task diversity; self-supervised tasks complement human-annotated datasets, and trained models better follow task requirements.Source: §6 · p. 9

A.1 Additional Details for LPP and Classification Tasks

The appendix lists label strings used for binary and three-way classification tasks and for LPP.

A.1 Additional Details for LPP and Classification Tasks

  • LPP uses paired positive and negative strings selected from Yes/No, Y/N, True/False, and T/F.The positive string is randomly chosen from Yes, Y, True, or T; the other string in its pair is used for the negative label.Source: §A.1 · p. 15
  • Three-way classification uses one of five label-string sets: Positive/Negative/Neutral, True/False/Neither, T/F/N, Yes/No/Unknown, or Y/N/U.Source: §A.1 · p. 15

A.2 Dataset Statistics

The appendix points to tables reporting dataset statistics for both benchmarks.

A.2 Dataset Statistics

  • Dataset statistics for SuperGLUE and Natural-Instructions are reported in Tables 12 and 13, respectively.Source: §A.2 · p. 15

A.3 Training Details

The appendix specifies the training setup, datasets, templates, and hyperparameters, then examines data scale and the effects of individual self-supervised tasks. Combining all four tasks gives the largest improvements for most tasks, while task-specific gains vary across downstream benchmarks.

A.3 Training Details

A.4 More Details for Natural-Instructions

  • The Natural-Instructions sources include nine named datasets, and the cross-task training setting uses eight specified subtasks.The listed sources include CosmosQA, DROP, MultiRC, and Winogrande; the cross-task subtasks include question generation, answer generation, and question typing.Source: §A.4 · p. 15

A.5 List of Function Words for the Last Phrase Prediction Task

  • The last-phrase prediction task uses a listed set of function words to identify the last phrase.The appendix enumerates words including “the,” “for,” “during,” and “without.”Source: §A.5 · p. 15

A.6 Templates for SuperGLUE

  • SuperGLUE templates were altered to resemble the formats of self-supervised tasks, with template variables drawing values from data fields.The templates also mark text used to compute language-model perplexities for ranking labels.Source: §A.6 · p. 15, Table 14 · p. 17

A.7 Hyperparameters

  • Hyperparameters were tuned on development-set performance, with learning rates from 1e-7 to 5e-5 and attention dropout of 0.0 or 0.1.Source: §A.7 · p. 15

A.8 Effect of Amount of Data

  • Figure 5 reports 125M- and 1.3B-model performance on SuperGLUE and Natural-Instructions across 1%, 10%, 20%, 50%, and 200% of training examples.Source: §A.8 · p. 15, Figure 5 · p. 15

A.9 Effect of Individual Self-Supervised Task to Downstream Tasks

  • Combining all four self-supervised tasks gives the largest improvements for most tasks, suggesting that the tasks are mostly complementary.The comparison covers SuperGLUE and Natural-Instructions results when tasks are excluded.Source: §A.9 · p. 15, Table 17 · p. 17, Table 18 · p. 17
  • Task-specific gains likely reflect semantic similarity; CL helps SuperGLUE but hurts Natural-Instructions, while NSG and CL have the largest effects across the benchmarks.The authors suggest SuperGLUE favors classification-related training and Natural-Instructions generation-related training; they also point to differences in task formulation and inference style for NSG and CL.Source: §A.9 · pp. 15–16
Loading 2205.01703v2…