PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Generate Task-Specific Adapters from Task Description

Qinyuan Ye, Xiang Ren

arXiv:2101.00420v2cs.CLcs.LG

TL;DR

Direct fine-tuning of text-to-text models can learn from individual examples rather than tasks as a whole. HYPTER generates task-specific adapters from task descriptions, and reports an 11.3% comparative improvement in Competence@90 on ZEST with BART-Large over fine-tuning.

  • Problem

    Direct fine-tuning learns from individual source-target examples rather than explicitly treating multiple examples as a task, while task-level methods for classification do not directly apply to text-to-text models.

    Source: §1 · p. 1

  • Method

    HYPTER uses a hypernetwork to generate lightweight task-specific adapters from task descriptions, and trains it while evaluating multiple examples within each task.

    Source: §1 · p. 2

  • Results

    11.3% comparative improvement in Competence@90 on ZEST is reported for HYPTER with BART-Large over direct fine-tuning.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that HYPTER improves text-to-text transformers' generalization to unseen tasks and demonstrate effectiveness on two datasets.

    Source: §7 · p. 5

    Performance improves with 75% or 100% of task examples, but not with fewer examples, suggesting sufficient examples per task are needed for effective adapters.

    Source: §5.2 · p. 5

Abstract

from arXiv · show

Pre-trained text-to-text transformers such as BART have achieved impressive performance across a range of NLP tasks. Recent study further shows that they can learn to generalize to novel tasks, by including task descriptions as part of the source sequence and training the model with (source, target) examples. At test time, these fine-tuned models can make inferences on new tasks using the new task descriptions as part of the input. However, this approach has potential limitations, as the model learns to solve individual (source, target) examples (i.e., at the instance level), instead of learning to solve tasks by taking all examples within a task as a whole (i.e., at the task level). To this end, we introduce Hypter, a framework that improves text-to-text transformer's generalization ability to unseen tasks by training a hypernetwork to generate task-specific, light-weight adapters from task descriptions. Experiments on ZEST dataset and a synthetic SQuAD dataset demonstrate that Hypter improves upon fine-tuning baselines. Notably, when using BART-Large as the main network, Hypter brings 11.3% comparative improvement on ZEST dataset.

1 Introduction

Text-to-text models can generalize to new tasks when task descriptions accompany source inputs, but direct fine-tuning still learns from individual examples rather than tasks as a whole. HYPTER generates task-specific adapters from descriptions and improves over fine-tuning baselines, including on ZEST.

1 Introduction

  • Direct fine-tuning can be sensitive to heuristically designed task descriptions and learns from individual examples instead of multiple examples within a task.The paper also notes that classification-focused task-level methods do not directly apply to text-to-text models, while generating an entire large model from a description is infeasible.Source: §1 · p. 1
  • HYPTER uses a hypernetwork to generate lightweight, task-specific adapters from task descriptions, enabling task-level learning across multiple within-task examples.The adapters are inserted into transformer layers.Source: §1 · p. 2
  • 11.3% comparative improvement in Competence@90 on ZEST is reported for HYPTER with BART-Large versus direct fine-tuning.The reported absolute improvement is 0.45%.Source: §1 · p. 2
  • The paper evaluates HYPTER on ZEST and synthetic SQuAD and reports improvements over direct fine-tuning baselines.Source: §1 · p. 2

2 Problem Definition

The paper studies inference on an unseen task from its natural-language description, using training tasks that pair descriptions with source-target examples. At test time, the model must predict targets for the new task without further training.

2 Problem Definition

  • A task is represented as a description d paired with a set D of source-target examples.In the text-to-text formulation, both sources and targets are text sequences.Source: §2 · p. 2
  • At training time, both each task description and its examples are available; at test time, only an unseen description is given.Source: §2 · p. 2
  • The model is expected to predict the correct target for an input from the unseen task without further training.Source: §2 · p. 2

3 Background: Adapters

The paper uses lightweight adapters in transformer layers for parameter-efficient transfer learning, with one adapter after the multi-headed attention in each layer.

3 Background: Adapters

  • Adapters are lightweight modules placed in transformer layers for parameter-efficient transfer learning.Source: §3 · p. 2
  • The paper's simplified design adds exactly one adapter after the multi-headed attention in each transformer layer.Source: §3 · p. 2
  • Each adapter has two linear layers separated by a nonlinear activation, with down- and up-projection parameters.Source: §3 · p. 2

4 Method

HYPTER combines a pretrained text-to-text main network with a hypernetwork that generates task-specific adapters from task descriptions. Training first fine-tunes the main network, then freezes it while optimizing the hypernetwork on task examples; at inference, a new description generates adapters for an unseen task.

4 Method

  • HYPTER pairs a pretrained text-to-text main network with a hypernetwork that generates adapters for it.The main network is instantiated with BART-Base or BART-Large.Source: §4 · pp. 2–3
  • A RoBERTa-Base encoder maps the task description to a latent representation, and layer-specific decoders generate each adapter's parameters.The generated parameters are reshaped into the adapter's down- and up-projection parameters.Source: §4 · p. 3
  • Training first fine-tunes the main network on concatenated task descriptions and inputs, then freezes it while the hypernetwork learns from sampled task minibatches.The second stage computes cross-entropy loss on generated targets and back-propagates it to update the hypernetwork.Source: §4 · p. 3
  • At test time, the hypernetwork uses an unseen task description to generate adapters for inference on that task.Source: §4 · p. 3

5 Experiments

Experiments compare HYPTER with a direct fine-tuning baseline on ZEST and Synthetic SQuAD, finding improved results overall while showing that data availability affects performance.

5.1 Experiment Setup

  • ZEST supplies generalized-question task descriptions and multiple source-target examples; Synthetic SQuAD groups questions by question-type bigrams.The Synthetic SQuAD dataset uses the 100 most frequent question types, with 80/10/10 type splits and out-of-domain examples from Natural Questions and NewsQA.Source: §5.1 · p. 3
  • The baseline is the first-stage text-to-text transformer, and seven-seed averages and early stopping help control randomness and extra-training effects.The ZEST dev table reports means and standard deviations over seven runs; the baseline is tuned with early stopping so HYPTER's improvement is not due to additional training.Source: §5.1 · pp. 3–4, Table 1 · p. 4

5.2 Results

  • On ZEST test, BART-Large Competence@90 rises from 3.98 to 4.43 (11.3% relative improvement), while BART-Base rises from 2.23 to 2.53.Table 2 reports ZEST test performance from the official leaderboard.Source: Table 2 · p. 4, §5.2 · p. 4
  • Synthetic SQuAD improvements are 0.74% with BART-Base and 0.41% with BART-Large; out-of-domain results are comparable or better except BART-Large underperforms BART-Base on NewsQA.Table 3 includes NQ and NewsQA as out-of-domain test data.Source: Table 3 · p. 4, §5.2 · p. 4
  • Performance improvements were significant (p<0.05) in multiple settings, including ZEST dev with C@75.Table 1 reports ZEST dev performance over seven runs and marks paired-test significance.Source: §5.2 · p. 4, Table 1 · p. 4
  • On ZEST, HYPTER beats the BART-Large fine-tuning baseline in base and paraphrase categories, with corrected predictions spanning four error types.The authors report two “n/a” predictions flipped to correct answers for one base-category task, reducing recall errors.Source: §5.2 · p. 4
  • HYPTER improves ZEST performance with 75% or 100% of tasks, but not with fewer tasks or fewer examples per task.The study uses BART-Large on ZEST; the authors suggest sufficient tasks and within-task examples are needed to generate effective adapters.Source: §5.2 · pp. 4–5, Figure 3 · p. 4

6 Related Work

Related work covers zero-shot learning through task and instruction descriptions, parameter-efficient transformer adapters, and hypernetworks that generate model parameters.

6 Related Work

  • Task descriptions and instructions are framed as a zero-shot learning route to generalizing to unseen tasks at inference.Other cross-task approaches unify input-output formats, including relation extraction and named entity recognition as machine reading comprehension.Source: §6 · p. 5
  • Transformer adapters use bottleneck layers for parameter-efficient transfer and have been applied to multilingual NER, QA, and commonsense reasoning.The cited work places adapter layers after attention and feed-forward layers.Source: §6 · p. 5
  • Hypernetworks generate another network’s weights; UDapter applies this idea to multilingual dependency parsing, whereas HYPTER targets text-described tasks without restricting task format or subtask relations.The comparison contrasts dependency parsing and cross-lingual tasks with general text-to-text tasks and tasks described in text.Source: §6 · p. 5

7 Conclusion

The conclusion presents HYPTER as a framework for unseen-task generalization through generated adapters, while preserving the main model’s general task-solving ability. It reports effectiveness on two datasets and identifies broader task coverage and further robustness work as future directions.

7 Conclusion

  • HYPTER generates task-specific adapters while freezing main-model parameters to maintain general task-solving ability.The framework is aimed at improving text-to-text transformers’ generalization to unseen tasks.Source: §7 · p. 5
  • HYPTER’s effectiveness is demonstrated on two datasets.Source: §7 · p. 5
  • Future work includes compositional instructions, robust fine-tuning for unseen data, and a large, diverse NLP task dataset.Source: §7 · p. 5

A Dataset Details

The appendix describes the ZEST resources and the construction and task partitions of the synthetic SQuAD dataset.

A Dataset Details

  • ZEST’s dataset is released publicly, and its leaderboard is hosted for submissions.Source: A Dataset Details · p. 8
  • The synthetic dataset is built from processed SQuAD, Natural Questions, and NewsQA data from MRQA Shared Task 2019.The authors provide a reconstruction script and list the task bigrams and their train/dev/test partition.Source: A Dataset Details · p. 8
  • Synthetic-task bigrams are assigned to train, dev, and test partitions.The listing gives examples under each partition, including “why were” for train, “what month” for dev, and “where were” for test.Source: A Dataset Details · p. 8

B Training Details

Training details specify the hardware and search settings for fine-tuning the text-to-text model and training the hypernetwork.

B Training Details

  • Experiments use Transformers and run on one GPU, selected from three NVIDIA models depending on availability.The listed GPUs are Quadro RTX 8000, Quadro RTX 6000, and GeForce RTX 2080 Ti.Source: B Training Details · p. 8
  • Text-to-text fine-tuning searches learning rates of 1e-5, 3e-5, and 5e-5 and dataset-specific epoch counts, with batch size fixed at 32.Epoch candidates are 5, 10, 15, 20, and 30 for ZEST; for synthetic SQuAD they are 10, 20, 30, 50, and 100.Source: B Training Details · p. 8
  • Hypernetwork training runs up to 100 epochs and greedily selects adapter width, learning rate, task batch size, and example batch size using dev performance.The searched values are d ∈ {4,8,16,32}, α ∈ {3e-6, 1e-5, 3e-5, 1e-4}, and b and b′ ∈ {4,8,16,32}.Source: B Training Details · p. 8

C Additional Baseline

The additional baseline compares fine-tuning with randomly initialized adapters against fine-tuning without them; the reported tests find no significant difference.

C Additional Baseline

D Dev Set Performance of Models Submitted to ZEST Leaderboard

The section reports dev performance for models submitted to the ZEST leaderboard and discusses the parameter cost of HYPTER and its intended role as an alternative for medium-sized models.

D Dev Set Performance of Models Submitted to ZEST Leaderboard

E Discussion

  • HYPTER's efficacy comes at the cost of introducing new parameters in the hypernetwork.The authors identify generating task-specific adapters from descriptions and task-level training as the contribution.Source: E Discussion · p. 8
  • The authors present HYPTER as an alternative for augmenting a medium-sized pretrained model, while noting that larger models may achieve better generalization to unseen tasks.Source: E Discussion · p. 8

Related papers

Loading 2101.00420v2…