PaperlayerView exact revision on arXiv

Source-linked AI summary

Parameter-Efficient Transfer Learning for NLP

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, Sylvain Gelly

arXiv:1902.00751v2cs.LGcs.CLstat.ML

TL;DR

When tasks arrive one after another, fine-tuning requires a new model for each, creating a parameter-efficiency challenge. This paper proposes adapter modules for transfer learning and reports near-state-of-the-art text-classification performance, with GLUE results nearly matching full fine-tuning using fewer task-specific parameters.

  • Problem

    In online settings with many sequential tasks, the goal is to perform well across them without training an entirely new model for each task.

    Source: §1 · p. 1

  • Method

    Adapters add modules to a pre-trained network and train only task-specific parameters while keeping the original parameters fixed.

    Source: §1 · p. 2

  • Results

    Across diverse text-classification tasks, adapters attained near-state-of-the-art performance; on GLUE, they almost matched fully fine-tuned BERT using 3% task-specific parameters versus 100%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach supports adding tasks incrementally without revisiting previous ones, yielding an extensible model for sequential tasks.

    Source: §1 · p. 1

    The classification evaluation omits WNLI because no current algorithm beats the majority-class baseline.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task. As an alternative, we propose transfer with adapter modules. Adapter modules yield a compact and extensible model; they add only a few trainable parameters per task, and new tasks can be added without revisiting previous ones. The parameters of the original network remain fixed, yielding a high degree of parameter sharing. To demonstrate adapter's effectiveness, we transfer the recently proposed BERT Transformer model to 26 diverse text classification tasks, including the GLUE benchmark. Adapters attain near state-of-the-art performance, whilst adding only a few parameters per task. On GLUE, we attain within 0.4% of the performance of full fine-tuning, adding only 3.6% parameters per task. By contrast, fine-tuning trains 100% of the parameters per task.

1. Introduction

Adapter tuning addresses the online task-stream setting by adding compact, extensible modules to a pre-trained network while keeping its original parameters fixed. On GLUE, it almost matches fully fine-tuned BERT with 3% task-specific parameters versus fine-tuning’s 100%.

1. Introduction

  • For tasks arriving in a stream, adapters provide compact downstream models that can be extended incrementally without forgetting previous tasks.The introduction identifies customer tasks arriving in sequence as a motivating setting.Source: §1 · p. 1
  • On GLUE, adapter tuning almost matches fully fine-tuned BERT using 3% task-specific parameters, versus 100% for fine-tuning.The paper also reports similar results on 17 additional public text datasets and SQuAD extractive question answering, and near state-of-the-art text-classification performance overall.Source: Figure 1 · p. 1, §1 · p. 2
  • Adapters add bottleneck modules between pre-trained network layers; the original weights stay fixed while only the small task-specific parameters are trained.The initial adapter parameters are set so the augmented network resembles the original, and fixing shared weights allows new tasks to be added without affecting earlier ones.Source: §1 · p. 2
  • Unlike multi-task learning, adapter tuning does not require simultaneous access to all tasks; frozen shared parameters give previous tasks perfect memory with few task-specific parameters.The paper distinguishes adapters from continual-learning systems because tasks do not interact and shared parameters remain frozen.Source: §1 · p. 2

2. Adapter tuning for NLP

Adapter tuning adapts pretrained models with small task-specific modules while freezing original weights, enabling sequential downstream-task training and parameter sharing. For Transformers, the paper instantiates this approach with bottleneck adapters and task-specific layer-normalization parameters.

2. Adapter tuning for NLP

  • The strategy targets good performance, sequential task training without simultaneous access to all datasets, and few additional parameters per task.These properties support high parameter sharing when models are trained on many downstream tasks.Source: §2 · p. 2
  • Adapter tuning injects randomly initialized task-specific layers while freezing the original network weights, unlike fine-tuning, which co-trains the original weights and new top layer.The frozen parameters can therefore be shared across tasks.Source: §2 · p. 2
  • Near-identity initialization preserves the original network at training start; adapters can later alter activations, vary in influence, or fail when initialized too far from identity.The modules are intended to be small relative to the original layers, so model size grows slowly as tasks are added.Source: §2 · p. 2

2.1. Instantiation for Transformer Networks

  • The Transformer design inserts two serial adapters after each attention and feedforward sub-layer projection, before the skip connection and following layer normalization.The authors report that this simple design performed as well as the more complex alternatives they tested across many datasets.Source: §2.1 · p. 3, Figure 2 · p. 3
  • Each bottleneck projects d-dimensional features to m dimensions and back, adding 2md + d + m parameters per layer; using m ≪ d adds around 0.5–8% of original-model parameters.The bottleneck dimension trades off performance against parameter efficiency, and an internal skip connection with near-zero projection weights gives approximate identity initialization.Source: §2.1 · p. 3
  • Adapter tuning also trains task-specific layer-normalization parameters, but tuning layer normalization alone is insufficient for good performance.Figure 2 depicts the adapter, layer-normalization parameters, and final classification layer as trained on downstream data.Source: §2.1 · p. 3, Figure 2 · p. 3

3. Experiments

Across GLUE and additional classification tasks, adapters approach full fine-tuning with substantially fewer parameters; on SQuAD, they likewise achieve comparable performance with fewer trained parameters.

3.1. Experimental Settings

  • The experiments use pretrained BERT with a linear classification head on the first-token classification embedding, and compare adapters with full fine-tuning, which scales to N times the model parameters for N tasks.For GLUE, the reported metrics come from the benchmark submission website; other classification tasks use test-set accuracy.Source: §3.1 · pp. 3–4

3.2. GLUE benchmark

  • GLUE mean score is 80.0 for adapters versus 80.4 for full fine-tuning, while solving all Table 1 tasks uses 1.3× versus 9× total BERT parameters.The experiments use BERTLARGE with 24 layers and 330M parameters; adapter sizes are fixed or selected per task from {8, 64, 256}. WNLI is omitted because no current algorithm beats the majority-class baseline.Source: §3.2 · p. 4, §3.3 · p. 4, Table 1 · p. 5

3.3. Additional Classification Tasks

  • Across 17 additional tasks, adapters finish 0.4% behind full fine-tuning while using 1.19× total parameters versus 17×; the AutoML baseline performs worse on average.Variable fine-tuning performs slightly better than full fine-tuning while training 52% of the network on average, for 9.9× total parameters. The tasks span 900–330k training examples and 2–157 classes.Source: §3.3 · p. 4, Table 2 · p. 5

3.4. Parameter/Performance trade-off

  • Across both task suites, adapters maintain good performance across sizes two orders of magnitude smaller than fine-tuning, while fewer-layer fine-tuning hurts GLUE more than the additional tasks.Some additional tasks benefit from training fewer layers, so their fine-tuning performance decays less as fewer layers are trained.Source: §3.4 · p. 5, Figure 3 · p. 6
  • On MNLIm, size-64 adapters use approximately 2M parameters for 83.7% ± 0.1% validation accuracy, versus 77.8% ± 0.1% for top-layer fine-tuning with approximately 9M parameters.Full fine-tuning reaches 84.4% ± 0.02% on MNLIm. Tuning layer-normalization parameters alone uses 40k parameters but lowers performance by approximately 3.5% on CoLA and 4% on MNLI.Source: §3.4 · p. 5, Figure 4 · p. 6

3.5. SQuAD Extractive Question Answering

  • 90.4% F1 with size-64 adapters (2% parameters) compares with 90.7 F1 for fine-tuning; size-2 adapters (0.1%) still reach 89.9 F1.The comparison is on SQuAD v1.1 question answering, which selects an answer span from a Wikipedia paragraph.Source: §3.5 · p. 6, Figure 5 · p. 7

3.6. Analysis and Discussion

  • Removing any one layer’s adapters causes at most a 2% performance drop, but removing all lowers MNLI and CoLA to majority-class scores of 37% and 69%; lower-layer removals matter less.The analysis concludes that adapters automatically prioritize higher layers, whose features are more task-specific than lower-layer features.Source: §3.6 · pp. 6–7
  • 86.2%, 85.8%, and 85.7% are mean validation accuracies for adapter sizes 8, 64, and 256, respectively; performance is robust to initialization scales below 10^-2.Tested architectural changes—including normalization, more layers, alternate activations, and alternate insertion schemes—did not significantly improve on the original bottleneck adapter.Source: §3.6 · p. 7

4. Related Work

Related work frames adapters against feature-based transfer, full-model fine-tuning, multi-task and continual learning, and convolutional adapters in vision. It also distinguishes adapters’ write-to-inner-layers approach from methods that read inner-layer representations.

Pre-trained text representations

  • Pre-trained representations, from word embeddings to sentence and paragraph embeddings, are commonly used as downstream features and may be fine-tuned (Mikolov et al., 2013).ELMo uses internal representations but only reads from inner layers, whereas adapters write to inner layers to re-configure feature processing through the network (Peters et al., 2018).Source: §4 · p. 7

Fine-tuning

Multi-task Learning

  • Multi-task learning shares parameters and yields one model for multiple problems, but requires simultaneous access to tasks during training.Examples span text processing, multilingual models, semantic parsing, machine translation, and question answering (Peng et al., 2017; Johnson et al., 2017).Source: §4 · p. 8

Continual Learning

  • Continual learning addresses sequential tasks, but deep networks can catastrophically forget previous tasks; Progressive Networks avoid forgetting with a new column per task, growing parameters linearly (Rusu et al., 2016).Adapters are small, so their models scale more favorably than Progressive Networks (s11.p8#393a2d).Source: §4 · p. 8

Transfer Learning in Vision

  • Vision adapters add small convolutional layers to ResNets or VGG nets and increase model size by 11% per task, while bottleneck adapters can be smaller and still perform well (Rebuffi et al., 2017; 2018; Rosenfeld & Tsotsos, 2018).The vision adapters use 1 × 1 convolutions alongside networks that typically use 3 × 3 convolutions.Source: §4 · p. 8

Concurrent work

  • Concurrent work proposes Projected Attention Layers for BERT, using a different architecture and jointly fine-tuning BERT across GLUE tasks (Stickland & Murray, 2019).Sina Semnani (2019) empirically compared bottleneck adapters and PALs on SQuAD v2.0 (Rajpurkar et al., 2018).Source: §4 · p. 8

Supplementary Material for Parameter-Efficient Transfer Learning for NLP · A. Additional Text Classification Tasks

This supplementary subsection documents additional text-classification tasks and the resources and settings used with them, including task statistics and references, training epochs, baseline search spaces, and text-embedding options.

A. Additional Text Classification Tasks

  • Table 3 documents statistics for the additional text-classification tasks.Its caption also says it provides references for those tasks.Source: Table 3 · p. 11
  • Table 3 provides references for the additional text-classification tasks.Source: Table 3 · p. 11
  • Table 4 reports the number of training epochs selected for the additional classification tasks.Source: Table 4 · p. 11
  • Table 5 specifies the search space of baseline models for the additional text-classification tasks.Source: Table 5 · p. 12
  • Table 6 presents options for text-input embedding modules, which are pre-trained text-embedding tables.Source: Table 6 · p. 12
  • For publicly distributed embedding modules, Table 6 provides their TensorFlow Hub handles.Source: Table 6 · p. 12

B. Learning Rate Robustness

The section tests adapter and fine-tuning robustness to learning rate, selecting the best hyperparameters for each method at each rate.

B. Learning Rate Robustness

  • Experiments span learning rates from 2 · 10−5 to 10−3, with the best hyperparameters selected separately for each method at each rate.Source: B. Learning Rate Robustness · p. 12
  • Figure 7 shows the best-performing models at different learning rates, with error bars indicating the s.e.m. across three random seeds.Source: Figure 7 · p. 13

Related papers

Loading 1902.00751v2…