Source-linked AI summary
Parameter-Efficient Transfer Learning for NLP
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, Sylvain Gelly
TL;DR
Training a new full model for every downstream NLP task is parameter-inefficient. This paper introduces adapter modules that keep the pre-trained network fixed, achieving near-full-fine-tuning performance with far fewer task-specific parameters.
Problem
Online multi-task transfer requires performing well as tasks arrive without training an entire new model for each task.
Method
Adapter tuning adds task-specific modules between layers while keeping the pre-trained network parameters fixed and training only the adapters.
Results
Across diverse text-classification tasks, adapters nearly match fully fine-tuned BERT on GLUE while using only 3% task-specific parameters.
Takeaways & Limitations
Adapter-based tuning provides a compact, extensible model that supports incremental task addition without sacrificing performance.
Takeaways & Limitations
The evaluation omits WNLI because no current algorithm beats the majority-class baseline.
Abstract
from arXiv · showhide
Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task. As an alternative, we propose transfer with adapter modules. Adapter modules yield a compact and extensible model; they add only a few trainable parameters per task, and new tasks can be added without revisiting previous ones. The parameters of the original network remain fixed, yielding a high degree of parameter sharing. To demonstrate adapter's effectiveness, we transfer the recently proposed BERT Transformer model to 26 diverse text classification tasks, including the GLUE benchmark. Adapters attain near state-of-the-art performance, whilst adding only a few parameters per task. On GLUE, we attain within 0.4% of the performance of full fine-tuning, adding only 3.6% parameters per task. By contrast, fine-tuning trains 100% of the parameters per task.
1. Introduction
The paper proposes adapter-based transfer for online streams of NLP tasks, using compact, extensible downstream models instead of training an entire model per task. Adapters nearly match fully fine-tuned BERT on GLUE while requiring only 3% task-specific parameters.
- Problem and motivation: The online setting requires strong performance across tasks arriving sequentially without training an entire new model for every task.The motivation is especially relevant when models must serve many customer tasks in sequence.
- Method: Adapters produce compact, extensible models that add few parameters per task and can be trained incrementally without forgetting previous tasks.Unlike multi-task learning, adapter-based tuning does not require simultaneous access to all tasks.
- Method: Adapter modules are added between layers of a pre-trained network, while fine-tuning adjusts the original parameters for each task.Adapter tuning instead introduces new task-specific modules while preserving the pre-trained network’s parameters.
- Results: 3% task-specific parameters are sufficient for adapter tuning to almost match fully fine-tuned BERT on the GLUE benchmark.The paper identifies a simple bottleneck adapter architecture and its integration with the base model as key contributions.
- Results: Adapters attain near state-of-the-art performance in text classification while remaining parameter-efficient.The evaluation covers a large and diverse set of text classification tasks.
2. Adapter tuning for NLP
Adapter tuning adds compact, task-specific modules to a large text model while preserving good performance, sequential task training, and limited per-task parameter growth. The proposed bottleneck adapters use near-identity initialization and are inserted throughout Transformer layers to enable parameter-efficient adaptation.
- Core properties: Adapter tuning targets good performance, sequential downstream-task training, and only a small number of additional parameters per task.Sequential training does not require simultaneous access to all datasets.
- Adapter modules: The method adds task-trained parameters to the model, providing a more general architectural modification than changing only the top layer during vanilla fine-tuning.Different upstream and downstream label spaces and losses motivate modifying the architecture.
- Adapter design: Adapters use a small bottleneck and near-identity initialization, so model size grows slowly across tasks while the original network is initially unaffected.The bottleneck keeps adapters small relative to the original layers and supports stable training.
- Transformer integration: Each Transformer layer receives two serial adapters after its attention and feedforward sub-layers, with skip-connections and layer normalization around the sub-layers.This placement follows the Transformer’s attention and feedforward structure.
- Parameter efficiency: 2md + d + m parameters are added per layer, and choosing m ≪ d limits growth to around 0.5–8% of the original model’s parameters per task.The bottleneck projects d-dimensional features to m dimensions, applies a nonlinearity, and projects them back to d dimensions.
- Task-specific parameters: New layer-normalization parameters are trained per task, but training layer-normalization parameters alone is insufficient for good performance.The method therefore combines task-specific normalization parameters with adapter modules.
3. Experiments
Experiments show that adapter tuning transfers BERT effectively across GLUE, additional classification tasks, and SQuAD while using substantially fewer task-specific parameters than full fine-tuning. Adapters retain strong performance across sizes and appear to prioritize higher network layers.
- GLUE: 80.0 mean GLUE score versus 80.4 for full fine-tuning, while adapters require 1.3× total BERT parameters instead of 9×.Adapter size varies by dataset: 256 for MNLI and 8 for RTE; fixed size 64 yields 79.6 mean accuracy.
- Additional classification tasks: 0.4% behind full fine-tuning on additional classification tasks, while fine-tuning requires 17× BERTBASE parameters.The benchmark spans diverse datasets with 900–330k training examples, 2–157 classes, and 57–1.9k-character average texts.
- Parameter efficiency: Two orders of magnitude fewer parameters than fine-tuning still yield good adapter performance across GLUE and additional-task parameter/performance trade-offs.On GLUE, performance decreases dramatically when fewer layers are fine-tuned, whereas some additional tasks benefit from training fewer layers.
- Layer-wise comparison: 83.7% ± 0.1% validation accuracy with approximately 2M adapter parameters exceeds 77.8% ± 0.1% from approximately 9M top-layer fine-tuning parameters on MNLIm.Full fine-tuning reaches 84.4% ± 0.02% on MNLIm, and a similar trend appears on CoLA.
- Question answering: 89.9 SQuAD F1 is attained by adapters of size 2, corresponding to 0.1% parameters.SQuAD performs well even with very small adapters.
- Analysis and extensions: 2% is the largest performance drop from removing any single layer’s adapters, whereas removing all adapters reduces performance to 37% on MNLI and 69% on CoLA.Removing adapters from layers 0–4 on MNLI barely affects performance, indicating that adapters prioritize higher layers; architectural extensions did not significantly improve performance.
4. Related Work
Related work includes pre-trained representations, full-model fine-tuning, multi-task and continual learning, and transfer methods in vision. Adapters differ by writing to inner layers while preserving parameter sharing across tasks.
- Pre-trained text representations: Pre-trained textual representations are trained on large corpora and used as features for downstream NLP models, sometimes with downstream fine-tuning.Brown clusters are a classic example of pre-trained representations.
- Pre-trained text representations: ELMo reads from inner layers of a pre-trained network, whereas adapters write to inner layers and reconfigure feature processing through the entire network.This distinguishes adapters from contextual feature extraction strategies based on sequence-model representations.
- Fine-tuning: Fine-tuning entire pre-trained models is popular in NLP, including Transformer networks trained with a Masked Language Model loss for question answering and text classification.Fine-tuning also avoids task-specific model design.
- Multi-task Learning: Multi-task learning trains on tasks simultaneously, sharing lower-layer parameters while using specialized higher layers to exploit task regularities.This approach has been applied across many NLP systems.
- Continual Learning: Continual learning trains on a sequence of tasks, but retraining deep networks can cause catastrophic forgetting of previous tasks.Proposed techniques mitigate forgetting, while adapter-based models scale more favorably because their parameters grow linearly with the number of tasks and adapters are very small.
- Transfer Learning in Vision: Vision transfer commonly fine-tunes ImageNet-pre-trained models, while concurrent BERT work introduced Projected Attention Layers with a similar role to adapters.The concurrent approach used a different architecture and jointly fine-tuned BERT on all GLUE tasks.
Supplementary Material for Parameter-Efficient Transfer Learning for NLP · A. Additional Text Classification Tasks
This supplementary section documents additional text classification tasks and the experimental setup used to evaluate them. It covers task statistics, training epochs, baseline-model search spaces, and text-input embedding options.
- A. Additional Text Classification Tasks: Table 3 provides statistics and references for the additional text classification tasks.
- A. Additional Text Classification Tasks: Together, these materials define the task references and key configuration choices for the additional classification experiments.
- A. Additional Text Classification Tasks: Table 4 reports the number of training epochs selected for the additional classification tasks.
- A. Additional Text Classification Tasks: Table 5 specifies the search space of baseline models for the additional text classification tasks.
- A. Additional Text Classification Tasks: Table 6 lists options for text-input embedding modules based on pre-trained text embedding tables.
- A. Additional Text Classification Tasks: The embedding modules are publicly distributed through the TensorFlow Hub service.The passage provides the TensorFlow Hub URL.
B. Learning Rate Robustness
The study tests adapters’ and fine-tuning’s robustness across learning rates from 2 · 10^-5 to 10^-3, selecting the best hyperparameters for each method at each rate. Figure 7 reports the best-performing models across rates, with error bars showing s.e.m. over three random seeds.
- B. Learning Rate Robustness: Learning rates ranged from 2 · 10^-5 to 10^-3, with the best hyperparameters selected separately for adapters and fine-tuning at each rate.The experiments directly evaluate both methods’ learning-rate robustness.
- B. Learning Rate Robustness: Figure 7 compares the best-performing models at different learning rates, with error bars indicating s.e.m. across three random seeds.The figure summarizes performance variation over the tested learning-rate range.