PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuning

Zhen-Ru Zhang, Chuanqi Tan, Haiyang Xu, Chengyu Wang, Jun Huang, Songfang Huang

arXiv:2305.15212v1cs.CL

TL;DR

Fine-tuning all parameters of large pre-trained models is prohibitively expensive, motivating parameter-efficient approaches. APT adapts prefix vectors through token- and layer-level gates; experiments report improvements over original prefix tuning and support variable prefixes, while the evaluated design is limited to encoder models.

  • Problem

    Fixed-length prefixes are used across layers even though language-model layers represent different aspects of input features.

    Source: §1 · p. 1

  • Method

    APT uses token-level gated weight assignment and layer-level scaling to dynamically customize prefixes at each layer.

    Source: §1 · p. 1, §3.2 · p. 2

  • Results

    APT improves performance over original prefix tuning, and gate-based probing supports the effectiveness of variable prefixes.

    Source: §1 · p. 1, §5 · p. 5

  • Takeaways & Limitations

    The experiments and analysis support adaptive prefix tuning as a promising strategy for parameter-efficient fine-tuning.

    Source: §5 · p. 5

    APT is evaluated on encoder models; designs for decoder-only and encoder-decoder architectures are lacking.

    Source: Limitations · p. 5

Abstract

from arXiv · show

Fine-tuning large pre-trained language models on various downstream tasks with whole parameters is prohibitively expensive. Hence, Parameter-efficient fine-tuning has attracted attention that only optimizes a few task-specific parameters with the frozen pre-trained model. In this work, we focus on prefix tuning, which only optimizes continuous prefix vectors (i.e. pseudo tokens) inserted into Transformer layers. Based on the observation that the learned syntax and semantics representation varies a lot at different layers, we argue that the adaptive prefix will be further tailored to each layer than the fixed one, enabling the fine-tuning more effective and efficient. Thus, we propose Adaptive Prefix Tuning (APT) to adjust the prefix in terms of both fine-grained token level and coarse-grained layer level with a gate mechanism. Experiments on the SuperGLUE and NER datasets show the effectiveness of APT. In addition, taking the gate as a probing, we validate the efficiency and effectiveness of the variable prefix.

1 Introduction

Parameter-efficient learning freezes the pre-trained model and tunes a small number of task-specific parameters, while prefix tuning adds continuous pseudo-token vectors at each layer. APT adapts prefixes through token- and layer-level gates, motivated by differences in layer representations, and experiments report its effectiveness on sentence and token classification tasks.

1 Introduction

  • Prefix tuning prepends continuous task-specific pseudo-token vectors to each layer and can achieve comparable or superior performance with 0.1-3% of parameters.This contrasts with vanilla fine-tuning, which adjusts all model parameters.Source: §1 · p. 1
  • Layer representations vary: bottom layers tend toward concrete phrase-level features, while top layers tend toward abstract semantic information.The authors motivate adaptive prefixes as a way to flexibly emphasize layer information for different downstream tasks.Source: §1 · p. 1
  • APT uses token-level gated weights and layer-level scaling to balance task-specific prefix tokens with original input tokens at each layer.The proposed gates adjust prefixes at fine-grained token and coarse-grained layer levels.Source: §1 · p. 1
  • Experiments against prefix tuning validate APT on sentence and token classification tasks in both full-data and low-resource settings.The learned gate is also used to probe necessary parameters by layer and guide variable-prefix tuning.Source: §1 · p. 1

2 Related Works

Parameter-efficient methods freeze most pre-trained parameters while tuning a small task-specific subset; related approaches include adapters and prompt or prefix tuning. Adaptable Adapters select layers with a hard decision, whereas APT aims to softly filter required information at each layer.

2 Related Works

  • Adapter tuning inserts two task-specific modules after attention and feed-forward networks, while prompt tuning and Prefix-Tuning train soft prompts at inputs or hidden states.The passage reports adapter tuning with 2-4% tuned parameters and Liu et al. (2022) matching fine-tuning with 0.1%-3%.Source: §2 · p. 2
  • Parameter-efficient learning freezes the pre-trained model and tunes only a small number of task-specific parameters for downstream tasks.The related-work passage frames these methods as lightweight alternatives to prohibitively expensive whole-model fine-tuning.Source: §2 · p. 2
  • Adaptable Adapters select beneficial layers and learn task-specific activation functions, while APT softly filters required information at each layer rather than making a hard module-addition choice.The cited work uses an adapter framework, unlike APT's prefix-tuning framework.Source: §2 · p. 2

3 Methodology

Prefix tuning adds trainable pseudo-token keys and values to each Transformer layer while keeping the pretrained model largely fixed. Adaptive Prefix Tuning (APT) varies prefix influence both token by token and by layer.

3.1 Prefix Tuning

  • Prefix tuning prepends pseudo tokens to each language-model layer by concatenating their keys and values with the original attention items.The prefix has length l, and its key and value matrices have dimension d.Source: §3.1 · p. 2

3.2 Adaptive Prefix Tuning

  • APT addresses fixed prefix lengths with fine-grained token gates and a coarse-grained layer-level weight to customize prefix influence.The token gates are calculated from the previous layer’s hidden states; the layer-level weight balances prefix and original-input information.Source: §3.2 · p. 2

4 Experiments

Experiments compare APT with vanilla fine-tuning and P-Tuning v2 across SuperGLUE and NER tasks. Results, ablations, and probing analyses examine APT’s performance and the role of its gates and variable prefixes.

4.1 Experimental Setup

  • Experiments cover five SuperGLUE and three NER tasks, comparing APT with vanilla fine-tuning and P-Tuning v2 across BERT and RoBERTa variants, plus DeBERTa-xlarge on NER.Source: §4.1 · p. 3

4.2 Results

  • APT improves over P-Tuning v2 by 1.5% and 0.7% on BERT-base SuperGLUE and NER, respectively, and by 1.8% and 1.4% on BERT-large.Source: §4.2 · p. 3
  • APT also gains 1.5% on RoBERTa-large SuperGLUE and 0.2% on NER, and 0.8% on average for DeBERTa-xlarge NER; it is comparable or better than vanilla fine-tuning on some tasks.Source: §4.2 · pp. 3–4
  • In low-resource SuperGLUE, APT exceeds P-Tuning v2 by 4.2%/3.4% at 16-shot and 2.9%/3.6% at 32-shot for BERT-base/BERT-large, respectively.Source: §4.2 · p. 4

4.3 Ablation Study

  • Removing any gate strategy hurts performance; layer-level scaling helps slightly more than token-level gating overall, while prior-layer context is especially useful on SuperGLUE.Source: §4.3 · p. 4, Table 3 · p. 4

4.4 Discussion

  • Figure 2’s gates emphasize bottom layers for CoNLL04 but higher layers for COPA, consistent with the distinction between token-level NER and semantic reasoning.Source: §4.4 · p. 4, Figure 2 · p. 4

Does variable prefix work better than fixed one?

5 Conclusion

The paper presents Adaptive Prefix Tuning (APT), whose token- and layer-level gates improve on original prefix tuning; its gate analysis also supports variable prefixes over fixed ones.

5 Conclusion

  • APT uses token-level and layer-level gates to adapt prefixes.The paper presents this as its proposed approach to prefix tuning.Source: §5 · p. 5
  • APT achieves improved performance over original prefix tuning.Source: §5 · p. 5
  • The gate's learned weight distribution is used as a probe, and the analysis validates variable prefixes over fixed ones.Source: §5 · p. 5

Limitations

The reported APT design is limited to encoder models, leaving other architectures and broader parameter-efficient approaches for future work.

Limitations

  • The authors adapt APT on encoder models.Source: Limitations · p. 5
  • They lack designs for decoder-only and encoder-decoder architectures.Source: Limitations · p. 5
  • They suggest generalizing the key idea to other parameter-efficient learning approaches and exploring a unified solution.Source: Limitations · p. 5

A Experimental Details

The experiments use specified SuperGLUE low-resource sampling, a grid-searched training setup, and four pretrained encoder models.

A Experimental Details

  • Low-resource SuperGLUE datasets use 16- and 32-shot samples from five fixed seeds, with the full development set treated as test data.Average performance is reported in Table 2.Source: A Experimental Details · p. 7
  • The setup grid-searches learning rate, epochs, batch size, and seeds, while matching APT prefix length to P-Tuning v2.Low-resource batch size is 2; Eq. (4) uses the first input token's hidden state as the previous-layer representation.Source: A Experimental Details · p. 7
  • Experiments use BERT-base (110M), BERT-large (335M), RoBERTa-large (355M), and DeBERTa-xlarge (750M) on V100 or A100 GPUs.Source: A Experimental Details · p. 7

B Further Ablation Results

Further ablations on BERT-large and RoBERTa-large report effects consistent with those on BERT-base; the supplied figure caption identifies a separate prefix-length comparison.

B Further Ablation Results

  • The beneficial impact of three strategies and the observation is generally consistent with BERT-base results.The passage locates these further ablations on BERT-large and RoBERTa-large in Table 5.Source: B Further Ablation Results · p. 7
  • The further ablation results cover BERT-large and RoBERTa-large.Source: B Further Ablation Results · p. 7
  • Figure 3 compares APT and PT-2 across prefix lengths on COPA and WSC using BERT-large.Source: Figure 3 · p. 7

C Prefix Length

Prefix length is an important hyperparameter for APT, which outperforms P-Tuning v2 across most tested prefix lengths and has a wider range of lengths with better performance.

C Prefix Length

  • APT is superior to P-Tuning v2 in most tested prefix-length settings.Figure 3 compares their performance over a range of prefix lengths.Source: C Prefix Length · p. 7
  • APT has a relatively wider range of prefix lengths at which it achieves better performance than P-Tuning v2.Source: C Prefix Length · p. 7
  • Prefix length is an important hyperparameter for both APT and prefix tuning.Source: C Prefix Length · p. 7

D Scientific Artifacts

The experiments cover SuperGLUE and named entity recognition tasks using several pretrained language models; Table 5 reports ablations on BERT-large and RoBERTa-large.

D Scientific Artifacts

  • The evaluation includes five SuperGLUE tasks and three named entity recognition tasks.The named entity recognition datasets are CoNLL03, CoNLL04, and OntoNotes 5.0.Source: D Scientific Artifacts · p. 7
  • Experiments use BERT-base and BERT-large, RoBERTa-large, and DeBERTa-xlarge pretrained models.Source: D Scientific Artifacts · p. 7
  • Table 5 presents ablations on BERT-large and RoBERTa-large for two gate mechanisms and the use of previous-layer hidden states.Bold marks the best score.Source: Table 5 · p. 8

Related papers

Loading 2305.15212v1…