PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Few-Shot Learn Across Diverse Natural Language Classification Tasks

Trapit Bansal, Rishikesh Jha, Andrew McCallum

arXiv:1911.03863v3cs.CLcs.LG

TL;DR

Fine-tuning pre-trained transformers still requires substantial labeled data, while meta-learning has often been evaluated on simulated or limited-diversity tasks. LEOPARD generates task-dependent classifier initializations for diverse NLP tasks; across 17 unseen tasks, it outperforms baselines, with a 14.60% average relative accuracy gain at 4 examples per label.

  • Problem

    Pre-trained transformers still need large amounts of task-specific labeled data for fine-tuning, and meta-learning applications have often focused on simulated or less diverse tasks.

    Source: §1 · p. 1

  • Method

    LEOPARD combines MAML with a parameter generator that produces task-dependent initial softmax parameters for tasks with disjoint label spaces.

    Source: §1 · p. 2, §3.2 · pp. 3–4

  • Results

    Across 17 unseen NLP tasks, LEOPARD yields a 14.60% average relative gain in accuracy for k = 4 examples per label.

    Source: §4.3 · p. 7, §4.3.1 · p. 8

  • Takeaways & Limitations

    LEOPARD learns more general-purpose parameters that better prime a model to solve completely new tasks with few examples.

    Source: §6 · p. 10

    With few examples, performance still lags behind human-level performance.

    Source: §6 · p. 10

Abstract

from arXiv · show

Self-supervised pre-training of transformer models has shown enormous success in improving performance on a number of downstream tasks. However, fine-tuning on a new task still requires large amounts of task-specific labelled data to achieve good performance. We consider this problem of learning to generalize to new tasks with few examples as a meta-learning problem. While meta-learning has shown tremendous progress in recent years, its application is still limited to simulated problems or problems with limited diversity across tasks. We develop a novel method, LEOPARD, which enables optimization-based meta-learning across tasks with different number of classes, and evaluate different methods on generalization to diverse NLP classification tasks. LEOPARD is trained with the state-of-the-art transformer architecture and shows better generalization to tasks not seen at all during training, with as few as 4 examples per label. Across 17 NLP tasks, including diverse domains of entity typing, natural language inference, sentiment analysis, and several other text classification tasks, we show that LEOPARD learns better initial parameters for few-shot learning than self-supervised pre-training or multi-task training, outperforming many strong baselines, for example, yielding 14.5% average relative gain in accuracy on unseen tasks with only 4 examples per label.

1 Introduction

Few-shot NLP aims to solve new tasks from limited labeled examples, but pre-trained transformers still need substantial task-specific data, and prior meta-learning work has often focused on simulated or less diverse tasks. LEOPARD addresses diverse label spaces with generated task-specific classifier initializations and evaluates generalization to unseen tasks.

1 Introduction

  • Learning to solve a new NLP task from k labeled examples matters where labeled data is difficult or expensive to obtain.Self-supervised transformer pre-training helps downstream tasks, but fine-tuning still requires large amounts of task-specific data for good performance.Source: §1 · p. 1
  • Prior meta-learning applications have often used simulated tasks, while this work investigates generalization beyond training tasks across NLP tasks with different numbers of labels.Because NLP tasks share a natural-language input space, the paper investigates whether meta-learning can generalize to unseen tasks.Source: §1 · p. 1
  • MAML shares model parameters, including softmax layers, which poses a barrier across tasks with disjoint label spaces; multi-task learning requires a new classifier for each task, while metric-based methods do not adapt as labeled examples increase.The paper presents LEOPARD as addressing these concerns with task-dependent initial softmax parameters.Source: §1 · p. 2
  • LEOPARD couples MAML with a parameter generator that produces task-dependent initial softmax parameters, enabling meta-learning across tasks with disjoint label spaces.The paper evaluates generalization to unseen tasks and new domains across entity typing, natural language inference, sentiment classification, and other text classification tasks.Source: §1 · p. 2

2 Background

The background frames few-shot learning as episodic training across tasks, followed by adaptation to a new task. It describes MAML as learning an initialization that can be quickly adapted through gradient descent.

2 Background

  • Episodes sample a task, use k examples per label for training and other examples for validation, and use validation error to adjust the model across tasks.The trained procedure is then evaluated on a new task using k examples per label for adaptation.Source: §2 · p. 2
  • MAML learns an initial parameter point that can be adapted to a new task through a few gradient-descent steps.Its inner loop adapts parameters on the task objective, while the outer loop refines the initialization across tasks.Source: §2 · p. 2
  • The paper names LEOPARD as a method for generating softmax parameters for diverse classification, and notes that gradient-based inner-loop learning can approach supervised performance with large data.The episodic meta-objective uses task validation-set error to adjust model parameters.Source: §2 · pp. 2–3

3 Model

LEOPARD combines a shared Transformer encoder, a task-conditioned softmax parameter generator, and MAML-based adaptation to support classification tasks with different numbers of classes. It separates task-agnostic parameters shared across tasks from task-specific parameters adapted within each task.

3 Model

  • LEOPARD combines a shared encoder, a training-data-conditioned softmax generator, and MAML-based adaptation for NLP classification tasks with different class counts.The model distinguishes task-agnostic parameters shared across tasks from task-specific parameters adapted per task.Source: §3 · p. 3

3.1 Text Encoder

  • BERT-base encodes sentence sequences, using the final-layer CLS representation as the fixed-dimensional input feature.The encoder handles sentence pairs and phrases by encoding them as sequences.Source: §3.1 · p. 3

3.2 Generating Softmax Parameters for Task-specific Classification

  • A shared generator forms class representations from encoded examples and produces task-dependent softmax weights and biases for each class.Unlike multi-task learning, LEOPARD does not introduce separate task-specific parameters; generated softmax parameters can instead be adapted with SGD.Source: §3.2 · pp. 3–4

3.3 Learning to Adapt Efficiently

  • To avoid overfitting examples used to generate the classifier, LEOPARD adapts on later support batches and updates task-agnostic parameters using validation error.The method uses a first-order approximation for outer-loop optimization and per-layer inner-loop learning rates; the number of task-specific Transformer layers is a hyperparameter.Source: §3.3 · p. 5

4 Experiments

The experiments test few-shot transfer across diverse NLP classification tasks, including tasks and domains withheld during training. They compare LEOPARD with transfer-learning and meta-learning baselines and examine how its performance depends on model components and training tasks.

4.1 Training Tasks

  • LEOPARD is evaluated on 17 target sentence-classification tasks, using GLUE tasks for training and including NLI and entity-typing targets.Training tasks include MNLI, SST-2, QNLI, QQP, MRPC, RTE, and SNLI.Source: §4 · p. 5, §4.1 · p. 6

4.2 Evaluation and Baselines

  • Evaluation uses k=4, 8, or 16 examples per label, then measures performance on each target task’s full test set across 10 sampled training sets.Baselines include pretrained BERT, multi-task BERT variants, and Prototypical BERT.Source: §4.2 · p. 6

4.3 Results

  • 14.60%, 10.83%, and 11.16% average relative accuracy gains on unseen tasks at k=4, 8, and 16, respectively; LEOPARD is significantly better than other models on average.The values are reported for the three k-shot settings across 17 target NLP tasks.Source: §4.3 · p. 7, Table 1 · p. 7

4.3.1 Generalization Beyond Training Tasks

  • On unseen tasks, LEOPARD outperforms baselines on average, with its greatest improvements at four examples; BERT performs better on some tasks, including Disaster classification.The authors identify the Disaster result as possible negative transfer from training tasks.Source: §4.3.1 · p. 8

4.3.2 Few-Shot Domain Transfer

  • LEOPARD performs better than baselines across sentiment domains, while MT-BERT models perform better on SciTail NLI.The related SST training task has phrase-level rather than sentence-level sentiment labels.Source: §4.3.2 · p. 8

4.4 Ablation Study

  • A zero-initialized softmax without the generator performs worse on new tasks; adapting all parameters helps new tasks, while adapting fewer helps NLI.The ablations use CoNLL-2003, Amazon DVD sentiment, and SciTail; the outer loop still optimizes all parameters.Source: §4.4 · pp. 8–9, Table 3 · p. 9
  • When training tasks are held out one by one, LEOPARD’s target-task performance is more consistent than MT-BERT’s, though removing tasks often lowers LEOPARD performance.The figure encodes relative performance change for each target task when a training task is held out, compared with training on all tasks.Source: Figure 2 · p. 9, §4.4 · p. 9

5 Related Work

Prior meta-learning work spans optimization-based, model-based, and metric-learning approaches, but many applications use simulated or narrowly defined tasks. This paper situates its contribution among efforts toward more diverse NLP meta-learning and transfer learning.

Approaches to Meta-Learning

Meta-Learning in NLP

  • Many prior meta-learning applications use simulated datasets with limited diversity or specialize in a task, while this work trains and evaluates on diverse real NLP tasks.The paper also contrasts its scope with work on GLUE tasks that does not consider unseen tasks.Source: §5 · p. 10

Transfer Learning

6 Conclusions

LEOPARD is a meta-learning approach that learns parameters to better prime models for solving entirely new tasks from few examples. Few-shot performance still lags behind human-level performance.

6 Conclusions

  • LEOPARD learns general-purpose parameters that better prime models to solve completely new tasks with few examples.The conclusion describes LEOPARD as a meta-learning approach.Source: §6 · p. 10
  • The authors identify bridging the remaining gap to human-level performance as a goal for demonstrating general linguistic intelligence.Source: §6 · p. 10
  • The conclusion presents meta-learning as a strong contender for pursuing general linguistic intelligence.Source: §6 · p. 10

A Datasets

The datasets span several NLP classification settings, and the appendix describes task augmentation, input formatting, validation choices, and dataset statistics. For few-shot experiments, it also reports balancing some datasets and sampling ten datasets for each k-shot setting.

A Datasets

  • For datasets with more than two classes, every pair of labels becomes a separate training task to increase task diversity and per-label examples per batch.Source: A Datasets · p. 13
  • SST-2 inputs pair the containing sentence with the phrase to classify, and Table 4 illustrates one input per entity mention.The example includes a sentence with two mentions, followed by a model input for each mention.Source: A Datasets · pp. 13–14, Table 4
  • For meta-learning methods, validation uses additional datasets, including Amazon review categorization and sentiment tasks.MT-BERT instead uses development data from its training task for validation.Source: A Datasets · p. 14
  • Table 5 reports statistics for all analyzed datasets; severely imbalanced Political Bias and Audience datasets are balanced for training, with ten datasets sampled per k-shot setting.A dash marks data unavailable or unused in the study.Source: Table 5 · p. 14

A.1 Test Datasets

The test datasets cover entity typing, sentiment and rating classification, varied social-media text classification, and science-question entailment.

A.1 Test Datasets

  • Entity typing uses CoNLL-2003 news text and MIT-Restaurant queries, treating each mention as a labeled example.Source: §A.1 · p. 14
  • Sentiment classification uses binary sentiment labels from Amazon Reviews in the Books, DVD, Electronics, and Kitchen & Housewares domains.Source: §A.1 · p. 14
  • Rating classification uses three Amazon Reviews classes: rating ≤2, rating = 4, and rating = 5.Source: §A.1 · p. 14
  • Text classification includes tweets about airlines and disasters, emotional text, and social-media messages about politicians’ audience, bias, or message.The passage characterizes these tasks as differing from training tasks in labels and input domain.Source: §A.1 · p. 15
  • SciTail supplies the natural language inference test dataset, with entailment examples derived from science questions.Source: §A.1 · p. 15

B Additional Results

The additional results describe domain-transfer accuracy on sentiment tasks, development accuracy on training tasks, and target-task performance across training-task choices. The reported effect of training-task choice decreases as the number of shots increases.

B Additional Results

C Hyperparameters

The appendix reports model-specific hyperparameter choices, including shot-specific fine-tuning settings and LEOPARD’s learned per-layer learning rates. Figure 3 also compares the consistency of LEOPARD and MT-BERT across held-out training tasks.

C Hyperparameters

  • Fine-tuning iterations and batch size were tuned separately for each shot setting, and all methods used a warm-up value of 0.1.The appendix also reports model-specific epoch and batch-size selections for MT-BERT variants and BERTbase.Source: C Hyperparameters · p. 15, Table 8 · p. 16
  • LEOPARD uses learned per-layer learning rates with SGD, with 150 epochs for 4-shot and 100 epochs for both 8-shot and 16-shot..Source: C Hyperparameters · p. 15, Table 8 · p. 16
  • LEOPARD’s performance is more consistent than MT-BERT’s across held-out training tasks, indicating more generalized initial parameters from meta-training.Figure 3 encodes relative performance changes by target task and held-out training task.Source: Figure 3 · p. 16
Loading 1911.03863v3…