PaperlayerView exact revision on arXiv

Source-linked AI summary

CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP

Qinyuan Ye, Bill Yuchen Lin, Xiang Ren

arXiv:2104.08835v2cs.CLcs.LG

TL;DR

The paper asks whether models can acquire cross-task generalization to improve few-shot learning across diverse NLP tasks. It introduces the CROSSFIT challenge and a 160-task NLP Few-shot Gym with standardized task partitions, learning stages, and evaluation. Results show that upstream learning can improve unseen-task performance, while the choice of seen tasks significantly influences transfer.

  • Problem

    Existing few-shot methods mainly improve generalization within individual tasks, leaving cross-task generalization across diverse NLP tasks relatively underexplored.

  • Method

    CROSSFIT evaluates upstream learning on seen tasks followed by few-shot learning on unseen tasks, using standardized protocols and a 160-task unified text-to-text task repository.

  • Results

    51.47% of test tasks significantly improved by more than 5% relative to direct fine-tuning, while 12.60% became worse by more than 5%.

  • Takeaways & Limitations

    Cross-task generalization can be acquired through multi-task learning and meta-learning, and seen-task selection influences few-shot performance on unseen tasks.

  • Takeaways & Limitations

    Meta-learning comparisons were constrained by one inner-loop update, limited hyperparameter and seed tuning, and transformer architectures unlike typical meta-learning settings.

Abstract

from arXiv · show

Humans can learn a new language task efficiently with only few examples, by leveraging their knowledge obtained when learning prior tasks. In this paper, we explore whether and how such cross-task generalization ability can be acquired, and further applied to build better few-shot learners across diverse NLP tasks. We introduce CrossFit, a problem setup for studying cross-task generalization ability, which standardizes seen/unseen task partitions, data access during different learning stages, and the evaluation protocols. To instantiate different seen/unseen task partitions in CrossFit and facilitate in-depth analysis, we present the NLP Few-shot Gym, a repository of 160 diverse few-shot NLP tasks created from open-access NLP datasets and converted to a unified text-to-text format. Our analysis reveals that the few-shot learning ability on unseen tasks can be improved via an upstream learning stage using a set of seen tasks. We also observe that the selection of upstream learning tasks can significantly influence few-shot performance on unseen tasks, asking further analysis on task similarity and transferability.

1 Introduction

The paper asks whether models can acquire human-like cross-task generalization for few-shot learning, beyond improving performance within individual tasks. It introduces CROSSFIT and the NLP Few-shot Gym to study this ability across diverse NLP tasks.

  • Large labeled datasets are expensive, and fine-tuned models can generalize poorly to small out-of-distribution perturbations.
  • Cross-task generalization means learning a new task efficiently from experience with previous tasks, rather than learning each task in isolation.
  • CROSSFIT standardizes the training pipeline, data access, and evaluation protocol for acquiring and evaluating cross-task generalization.
  • The NLP Few-shot Gym provides 160 diverse few-shot NLP tasks gathered from open-access resources and converted to a unified text-to-text format.
  • The challenge evaluates upstream learning on seen tasks followed by few-shot learning on unseen tasks across standardized seen/unseen partitions.
  • Multi-task learning and three meta-learning algorithms improve few-shot performance on unseen tasks, while task selection and upstream data size substantially affect outcomes.

2 Related Work

Prior NLP work improves few-shot learning within individual tasks or applies meta-learning to narrow task distributions. This paper instead studies transfer across diverse NLP goals using a unified text-to-text representation.

  • Few-shot Fine-tuning: Few-shot fine-tuning methods such as PET and demonstrations improve task-specific learning, but downstream tasks are generally learned in isolation.
  • Meta-learning in NLP: Existing NLP meta-learning studies commonly use synthetic subtasks or narrow distributions such as one task family or language.
  • Meta-learning in NLP: CROSSFIT explores meta-learning across diverse NLP goals, including classification, question answering, and conditional generation.
  • Unifying NLP Task Formats: Prior work has unified task formats to facilitate knowledge transfer, including DecaNLP, UFO-Entail, EFL, and UnifiedQA.
  • Unifying NLP Task Formats: Following T5, this work adopts a unified text-to-text format and expands the repository to 160 tasks to broaden coverage.

3 The CROSSFIT Challenge

CROSSFIT defines cross-task generalization as upstream learning on seen tasks followed by few-shot adaptation to unseen tasks. It uses standardized partitions and Average Relative Gain to compare diverse task outcomes.

  • A CROSSFIT system learns cross-task generalization from seen tasks and adapts to new unseen tasks efficiently.
  • Each task is defined as T = (Dtrain, Ddev, Dtest), with annotated text-to-text examples and small training and development sets in the few-shot setting.
  • The benchmark partitions a repository of tasks into non-overlapping Ttrain, Tdev, and Ttest sets to test different generalization scenarios.
  • During upstream learning, models access training-task Dtrain and Ddev plus Tdev for validation, while Dtest remains unavailable; they then perform few-shot learning on Ttest.
  • Because task metrics differ, CROSSFIT uses seven metrics and Average Relative Gain to average relative performance changes across test tasks.
  • ARG compares upstream methods with direct fine-tuning and averages each test task’s relative performance change across heterogeneous metrics.

4 NLP Few-shot Gym

The NLP Few-shot Gym supplies a broad, standardized task repository for CROSSFIT. It contains 160 filtered datasets, converted to text-to-text format and organized into multiple task partitions.

  • The repository contains 160 few-shot tasks covering a wide range of NLP applications in a unified text-to-text format.
  • Candidate datasets are filtered to English monolingual tasks without information retrieval, sequence labeling, or extremely long document inputs.
  • A Unified Text-to-Text Format: All datasets are converted into a unified text-to-text format, including premise-hypothesis inputs for natural language inference and question-context inputs for reading comprehension.
  • Few-shot sampling uses 16 training examples per class for classification and regression, and 32 training examples for other task types.
  • Five random seeds produce an effective repository size of 160 × 5 = 800 task instances, while the paper reports the repository as 160 tasks.
  • Task Ontology and Partitions: The authors manually classify the tasks into an ontology and design eight partitions of (Ttrain, Tdev, Ttest) for varied evaluation settings.

5 Methods to CROSSFIT

CROSSFIT evaluates whether upstream learning on seen tasks improves few-shot adaptation to unseen tasks under standardized partitions and protocols. The methods include direct fine-tuning, multi-task learning, and meta-learning approaches such as MAML, first-order MAML, and Reptile.

  • CROSSFIT setup: CROSSFIT evaluates upstream learning methods across standardized (Ttrain,Tdev,Ttest) task partitions before few-shot adaptation on test tasks.The study uses BART-Base as its main text-to-text model and reports confirmatory experiments with other model sizes in the appendix.
  • Baselines: Direct fine-tuning trains separately on each test task and serves as the ARG=0% baseline.It uses Dtrain for fine-tuning, Ddev for hyperparameter tuning, and Dtest for evaluation without Ttrain or Tdev upstream learning.
  • Upstream learning methods: Multi-task learning combines source-target examples from Ttrain, then uses the resulting checkpoint to initialize direct fine-tuning on each test task.Its performance gain over direct fine-tuning contributes to the overall ARG score.
  • Upstream learning methods: MAML learns an initialization that adapts quickly to new tasks through an inner update on Bsupport and an outer update using Bquery.The implementation uses one inner-loop gradient update because of the large size of pretrained text-to-text models.
  • Meta-learning variants: First-order MAML approximates the outer-loop gradient with respect to fast weights, while Reptile uses the parameter difference θ′ − θ for memory-efficient first-order meta-learning.Both methods avoid second-order optimization or use a first-order approximation to improve training efficiency or stability.

6 Empirical Analysis

Upstream learning generally improves cross-task few-shot generalization, but gains depend on task selection and can include negative transfer. The analyses also find that more upstream data does not necessarily improve transfer and that simple multi-task learning remains difficult to beat.

  • Overall Performance: 51.47% of test tasks improved by more than 5% relative to direct fine-tuning, while 12.60% worsened by more than 5%.Across upstream methods and task partitions, 35.93% of tasks were relatively unaffected within ±5%.
  • Correlated Performance Gains: Multi-task learning gains correlated with MAML gains on the Random partition, with Spearman correlation 0.66 and p value 0.0015.The authors suggest different upstream methods capture similar inductive bias from Ttrain.
  • Method Comparison: Simple multi-task learning was hard to beat by the evaluated meta-learning methods.The paper attributes possible explanations to limited inner-loop updates, untuned sensitivity, transformer complexity, and diverse upstream tasks.
  • Negative Transfer: Upstream learning sometimes produced negative gains, including on Glue-COLA and Domain Crawl, consistent with a possible forgetting-related explanation.The paper frames this explanation as a conjecture and identifies negative-transfer mitigation as future work.
  • Q2: Task Selection: Non-classification and classification upstream tasks were equivalently helpful in ARG scores for fixed classification test tasks.This indicates transfer can occur even when upstream and test tasks differ substantially in format and goal.
  • Q2: Task Selection: Performance varied across different Ttrain sets, but no consistent task type predicted better performance for a specific test task.The case studies compare tasks appearing in multiple partitions, including AI2_ARC and Race-High.
  • Q3: Upstream Data Size: Upstream data enlargement produced only 4% ARG improvement, so more labeled data did not necessarily yield better cross-task generalization.The experiments multiplied original upstream-task data by 2, 4, and 8 times.
  • Additional Analysis: Applying PET after upstream learning generally deteriorated performance compared with direct fine-tuning.The authors hypothesize PET may not directly apply to the encoder-decoder language models used.

7 Conclusion and Future Work

The paper introduces CrossFit and the NLP Few-shot Gym to study cross-task generalization, finding that upstream learning can improve unseen-task few-shot performance. It also identifies task selection and related meta-problems as important directions for future work.

  • CrossFit standardizes the training pipeline, data access, and evaluation protocol for studying cross-task generalization.
  • The NLP Few-shot Gym provides 160 diverse few-shot NLP tasks for evaluating CrossFit across different scenarios.
  • Multi-task learning and meta-learning can acquire cross-task generalization, while seen-task selection influences few-shot performance on unseen tasks.
  • Future work includes selecting appropriate source tasks, task augmentation, knowledge accumulation, and avoiding catastrophic forgetting.

A Selected Tasks in NLP Few-shot Gym

The NLP Few-shot Gym contains diverse tasks organized by task name, ontology, and reference. The selected examples span classification, question answering, and dialogue-related categories.

  • Table 3 organizes NLP Few-shot Gym entries by task name, ontology, and reference.
  • The selected tasks include fact checking, closed-book question answering, and dialogue.
  • The table also includes natural language inference, other classification tasks, multiple-choice question answering, and emotion classification.

B.1 Partition 1. Random

The random partition divides tasks into train, development, and test sets, with multiple listed configurations covering diverse NLP task families. The task lists include classification, question answering, language understanding, and generation tasks.

  • Partition 1. Random: The partition specifications explicitly separate tasks into train, dev, and test splits.
  • Partition 1. Random: The listed splits include classification and sentiment tasks such as MRPC, emotion, hate-speech, and polarity datasets.
  • Partition 1. Random: The configurations also contain language, dialogue, summarization, rewriting, and linguistic-agreement tasks.
  • Partition 1. Random: Question-answering and reasoning tasks include CommonsenseQA, HotpotQA, FreebaseQA, MathQA, OpenBookQA, and related datasets.

C Additional Results and Analysis

Additional experiments examine whether upstream learning transfers beyond few-shot settings, scales to larger models, and combines with PET. Benefits persist in medium-resource settings, while PET deteriorates average performance.

  • Beyond Few-shot Settings: Upstream learning benefits extend to medium-resource settings with up to 2048 training examples, but diminish for resource-rich tasks such as MNLI.
  • Larger Pre-trained Models: The larger-model experiments use T5-v1.1-Base (248M) and BARTLarge (406M), considering only first-order methods because second-order optimization was computationally impossible.
  • Larger Pre-trained Models: Except for FoMAML+T5-v1.1-Base, upstream learning consistently improves few-shot performance on Ttest with larger pre-trained models.
  • Pattern-Exploiting Training: PET applied to BART-Base improves CoLA and MRPC but deteriorates performance on average across the remaining GLUE tasks and upstream-learning checkpoints.
  • Pattern-Exploiting Training: The authors stop further PET investigation because adapting PET methods to autoregressive models is outside the study’s scope.

D Reproducibility

Experiments use Hugging Face Transformers, with higher for higher-order optimization in the meta-learning approach, and are open-sourced. Few-shot fine-tuning follows specified learning-rate, batch-size, update, warmup, and evaluation practices, while compute and model sizes are documented.

  • Implementation: Experiments use Hugging Face Transformers and the higher library for higher-order optimization in the meta-learning approach.The code is included in supplementary materials and open-sourced on GitHub.
  • Hyper-parameters: Few-shot fine-tuning selects learning rates from {1e − 5, 2e−5, 5e−5} and batch sizes from {2, 4, 8} using Ddev performance.Training uses 1000 total updates, 100 warmup updates, and evaluation every 100 steps.
  • Infrastructure and Runtime: Upstream learning runs on one Quadro RTX 8000, with jobs finishing within 3 hours on average.Fine-tuning uses one GPU selected from five NVIDIA Quadro or GeForce models based on availability.
  • Experimental Design: Figure 9 presents combining upstream learning with pattern-exploiting training.
  • Infrastructure and Runtime: Fine-tuning one few-shot task, including hyperparameter tuning for five random samples, takes approximately 4 hours on average.
  • Number of Parameters: BART-Base, T5-v1.1-Base, and BARTLarge contain 139 million, 246 million, and 406 million parameters, respectively.
Loading 2104.08835v2…