PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation

Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jia-Wei Low, Lidong Bing, Luo Si

arXiv:2106.03164v1cs.CL

TL;DR

Prior work emphasized adapters’ parameter efficiency but provided limited evidence about their adaptation effectiveness. This paper compares adapter-based tuning with fine-tuning across downstream settings and finds that adapters better preserve pretrained representations while performing particularly well in low-resource and cross-lingual tasks and remaining less sensitive to learning rates.

  • Problem

    Prior work largely focused on adapter-based tuning’s parameter efficiency, leaving its effectiveness insufficiently investigated.

  • Method

    The paper empirically compares adapter-based tuning and fine-tuning across monolingual and cross-lingual downstream tasks and uses representation-similarity analysis to examine forgetting.

  • Results

    Adapter-based tuning tends to outperform fine-tuning on low-resource and cross-lingual tasks, while showing higher stability and lower sensitivity to learning rates.

  • Takeaways & Limitations

    Adapter-based tuning can improve adaptation effectiveness beyond parameter savings, particularly when data are limited or tasks are cross-lingual.

  • Takeaways & Limitations

    Test-result access was limited to two submissions per day, making a large number of experimental runs inconvenient.

Abstract

from arXiv · show

Adapter-based tuning has recently arisen as an alternative to fine-tuning. It works by adding light-weight adapter modules to a pretrained language model (PrLM) and only updating the parameters of adapter modules when learning on a downstream task. As such, it adds only a few trainable parameters per new task, allowing a high degree of parameter sharing. Prior studies have shown that adapter-based tuning often achieves comparable results to fine-tuning. However, existing work only focuses on the parameter-efficient aspect of adapter-based tuning while lacking further investigation on its effectiveness. In this paper, we study the latter. We first show that adapter-based tuning better mitigates forgetting issues than fine-tuning since it yields representations with less deviation from those generated by the initial PrLM. We then empirically compare the two tuning methods on several downstream NLP tasks and settings. We demonstrate that 1) adapter-based tuning outperforms fine-tuning on low-resource and cross-lingual tasks; 2) it is more robust to overfitting and less sensitive to changes in learning rates.

1 Introduction

Adapter-based tuning was introduced as a parameter-efficient alternative to fine-tuning, and this paper investigates whether its frozen-and-learnable structure also improves adaptation effectiveness. Across settings, adapters perform especially well in low-resource and cross-lingual tasks and are more stable across learning rates.

  • Adapter-based tuning updates lightweight modules while freezing the pretrained language model, adding few task-specific parameters and enabling parameter sharing.Adapters are inserted between transformer layers; prior work found comparable performance to full fine-tuning despite using fewer trainable parameters.
  • The paper addresses the underexplored effectiveness of adapter-based tuning beyond its parameter-efficiency.It empirically studies whether alternating frozen and learnable layers improve model performance and mitigate forgetting.
  • Adapter-based tuning yields better results than fine-tuning for monolingual adaptation in low-resource settings, especially on more domain-specific tasks.Its performance advantage becomes less significant as the number of training samples increases.
  • Adapter-based tuning tends to outperform fine-tuning on zero-shot cross-lingual tasks under different amounts of training data.
  • Adapter-based tuning demonstrates higher stability and better generalization ability, with lower sensitivity to learning-rate changes than fine-tuning.

2 Adapter Better Regularizes Tuning

Adapter-based tuning inserts lightweight networks into a pretrained language model and updates only selected task-specific components while keeping the original model largely frozen. The resulting representations deviate less from the pretrained model than those produced by fine-tuning, supporting better regularization through reduced forgetting.

  • 2.1 Adapter-based Tuning: Adapter-based tuning inserts lightweight neural networks between transformer layers and updates adapter parameters while keeping pretrained-language-model weights frozen.The standard configuration also updates normalization layers and the final classification layer.
  • 2.1 Adapter-based Tuning: Each adapter first down-projects the hidden vector from dimension d to m, where m < d, then up-projects it back to dimension d with a skip connection.
  • 2.2 Representation Similarity: RSA compares representation spaces before and after tuning by correlating pairwise-similarity matrices computed from the same sampled inputs.The analysis uses 5,000 randomly sampled development-set tokens from BERT-base representations at each layer.
  • 2.2 Representation Similarity: Representation changes generally arise in the network’s top layers for both methods, which are more task-relevant.
  • 2.2 Representation Similarity: Adapter-based tuning produces representations with less deviation from BERT-base than fine-tuning at each layer.The paper interprets this pattern as evidence that adapters better regularize tuning by mitigating forgetting.

3 Monolingual Adaptation

Across monolingual adaptation experiments, adapter-based tuning generally performs better than fine-tuning in low-resource settings, while the methods are similar with abundant data. Its advantage is strongest for domain-specific tasks and diminishes as training data increases.

  • 3.1 TAE: Adapter-based tuning consistently outperforms fine-tuning on low-resource TAE tasks, while both methods perform similarly on high-resource tasks.Low-resource tasks contain fewer than 5k training examples.
  • 3.1 TAE: 1.9% average improvement is achieved by adapter-based tuning over fine-tuning across four low-resource TAE tasks.Results average five runs with different random seeds.
  • 3.3 Discussions: With increasing training samples, adapter-based tuning’s advantage becomes less significant as fine-tuning gradually catches up.Figure 3 reports test-performance averages across five runs with different random seeds.
  • 3.2 GLUE Low-resource Adaptation: Adapter-based tuning outperforms fine-tuning most of the time on GLUE under both 1k and 5k training-example settings.In the 1k setting, average gains are 2.5% for BERT and 0.7% for RoBERTa.
  • 3.3 Discussions: The authors attribute adapters’ low-resource advantage to fine-tuning’s more severe overfitting, whereas fine-tuning’s greater model capacity matters more in high-resource settings.Fine-tuning updates substantially more parameters than adapter-based tuning.
  • 3.3 Discussions: Adapter-based tuning gains are larger on TAE than GLUE, with RoBERTa-base improvements of 1.9% on TAE versus 0.7% and 0.4% on GLUE 1k and 5k.TAE is more domain-specific and has less overlap with RoBERTa-base’s pretraining corpus.

4 Cross-lingual Adaptation

In zero-shot cross-lingual transfer, adapter-based tuning significantly outperforms fine-tuning across UD-POS, Wikiann NER, and XNLI, including high-resource and reduced-data settings.

  • Adapter-based tuning significantly outperforms fine-tuning on all three evaluation groupings for UD-POS, Wikiann NER, and XNLI.The groupings are All, Target, and Distant language averages.
  • 2.06% and 3.71% are adapter-based tuning’s gains over reported fine-tuning on UD-POS Target and Distant languages.
  • 1.08% and 0.8% are the corresponding Wikiann NER gains on Target and Distant languages.
  • 0.87% is the adapter-based tuning gain over fine-tuning on XNLI for both Target and Distant languages.
  • Adapters retain consistent improvements over fine-tuning on XNLI when trained with 5%, 10%, or 20% of the original training data.

5 Analysis

The analysis examines adapter capacity, learning-rate stability, overfitting, and loss landscapes, finding that adapter behavior depends on task conditions while generally remaining more stable than fine-tuning.

  • Adapter Hidden Size: The optimal adapter hidden size depends on both the task domain and training-set size.Larger sizes help some domain-specific or full-data settings, but increasing size does not consistently improve low-data XNLI results.
  • Learning Rate Robustness: Adapter-based tuning is more stable across a wider range of learning rates, while fine-tuning often collapses above 4e-5 on RoBERTa-base.Figure 4 compares rates from 2e-5 through 1e-4 across 20 random-seed runs.
  • Overfitting and Generalization: Fine-tuning can easily overfit on both low- and high-resource tasks, whereas adapter-based tuning is more robust to overfitting.The comparison uses CoLA and MRPC as low-resource tasks and QNLI and SST-2 as high-resource tasks.
  • Overfitting and Generalization: Mean dev-set results consistently favor adapters, and their mean-to-best gaps are smaller across evaluation steps.These results indicate greater stability during training.
  • Loss Landscapes: Adapter-based tuning produces wider, flatter minima than fine-tuning in the examined BERT-base loss landscapes.The landscapes are evaluated on CoLA and SST-2 by interpolating between pre- and post-tuning weights.

6 Related Work

Prior work established fine-tuning as effective but parameter-inefficient and showed that adapters offer comparable performance with fewer task-specific parameters, while low-resource fine-tuning can overfit.

  • Fine-tuning pretrained language models requires a new set of weights for each downstream task.
  • Adapter-based tuning was proposed to address fine-tuning’s parameter inefficiency while maintaining comparable performance.
  • Fine-tuning on a few samples can overfit and generalize poorly, making results unstable in low-resource settings.

7 Conclusion

The paper argues that adapter-based tuning should be evaluated not only for parameter efficiency but also for effectiveness across adaptation settings.

  • Adapter-based tuning better regularizes learning and tends to outperform fine-tuning on low-resource and cross-lingual tasks.
  • Adapter-based tuning is more stable than fine-tuning under different learning rates.

A.1 Datasets

The appendix describes the datasets and evaluation splits used for TAE, GLUE, and representation or overfitting analyses. Low-resource GLUE settings are created by sampling 1k or 5k training examples.

  • TAE dataset statistics are reported for the tasks evaluated in Section 3.1.
  • GLUE low-resource settings use randomly sampled 1k and 5k examples from each original training set.Each setting uses 1k remaining training examples for validation and the original validation set as held-out test data.
  • RSA and overfitting analyses use the original training and development sets to expose behavior under high- and low-resource conditions.

A.2 Experimental Details

Experiments use standard HuggingFace implementations, Adam optimization, and linear learning-rate scheduling, with separate hyperparameter tuning for fine-tuning and adapters. Xtreme-task tuning uses English data for selecting learning rates and adapter sizes.

  • Models use HuggingFace implementations, Adam optimization, a linear learning-rate scheduler, and a task-specific feedforward prediction layer.Each experiment was performed on a single V100 GPU.
  • Both tuning methods train for a fixed number of epochs and evaluate the epoch-end model with the best validation performance.
  • Fine-tuning and adapter experiments use separately specified learning-rate, batch-size, and adapter-hidden-size searches across TAE and GLUE.The adapter searches include hidden sizes 64, 128, and 256, while fine-tuning searches include GLUE learning rates from 1e-5 to 5e-5 and batch sizes 16 or 32.
  • The GLUE analysis experiments reuse the same hyperparameter settings described for the main GLUE experiments.
  • Detailed TAE and GLUE hyperparameter settings are provided in Table 10.
  • For UD-POS, Wikiann NER, and XNLI, learning rates are tuned per task, adapter hidden sizes are selected across tasks, and English data is used for tuning.The candidate learning rates range from 1e-5 to 5e-5, and candidate adapter hidden sizes are 64, 128, and 256.

A.3 Additional Results

Additional analyses examine representation similarity, overfitting, and cross-lingual performance across GLUE, XNLI, POS tagging, and NER. They report greater overfitting robustness for adapters and provide detailed per-language results and hyperparameters.

  • Overfitting analysis: Adapter-based tuning is more robust to overfitting than fine-tuning across both high-resource and low-resource GLUE tasks.Figure 9 evaluates QNLI, SST-2, CoLA, and MRPC as training steps increase.
  • Overfitting analysis: On XNLI trained with 10% of the original English data, fine-tuning overfits more easily while adapter-based tuning remains more robust.Figure 10 plots average development loss and accuracy across target languages excluding English.
  • Cross-lingual results: Detailed cross-lingual results cover zero-shot POS accuracy, NER F1, and XNLI accuracy under full-data and 5%, 10%, and 20% training conditions.Tables 12–15 report results by target language and training-data proportion.
  • Representation analysis: RSA compares pre- and post-tuning representations for BERT-base on CoLA, MRPC, and MNLI, and for XLMR-large on distant-language Wikiann NER.Higher RSA scores indicate greater similarity between representation spaces before and after tuning.
  • Supporting details: The appendix also supplies dataset statistics for TAE and GLUE and hyperparameter tables for TAE, GLUE, and Xtreme tasks.
Loading 2106.03164v1…