PaperlayerView exact revision on arXiv

Source-linked AI summary

XPrompt: Exploring the Extreme of Prompt Tuning

Fang Ma, Chen Zhang, Lei Ren, Jingang Wang, Qifan Wang, Wei Wu, Xiaojun Quan, Dawei Song

arXiv:2210.04457v1cs.CLcs.LG

TL;DR

Fine-tuning large PLMs is parameter-inefficient, and XPrompt targets improved prompt-tuning performance relative to fine-tuning. It uses hierarchical pruning to remove negative prompt tokens; on SuperGLUE, it improves prompt tuning across tasks and model scales, closing the gap at moderate scales.

  • Problem

    Fine-tuning large PLMs is parameter-inefficient because gradients and optimizer states must be stored for trainable parameters, while XPrompt targets prompt tuning’s performance gap with fine-tuning.

    Source: §1 · pp. 1–2

  • Method

    XPrompt uses hierarchical structured pruning to remove negative prompt tokens at token and piece levels, then rewinds and retrains the identified positive prompts.

    Source: §1 · p. 2

  • Results

    XPrompt significantly improves prompt-tuning methods across tasks and model scales, closing the gap with fine-tuning at moderate scales and exceeding it on most tasks at large scales.

    Source: §1 · p. 2

  • Takeaways & Limitations

    XPrompt yields a more parameter-efficient prompt at an extremely small scale while maintaining competitive effectiveness.

    Source: §8 · p. 9

    Finding optimal compression ratios without trial training remains under-explored, and multi-task, out-of-domain, and prompt-ensembling scenarios remain for future study.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Prompt tuning learns soft prompts to condition frozen Pre-trained Language Models (PLMs) for performing downstream tasks in a parameter-efficient manner. While prompt tuning has gradually reached the performance level of fine-tuning as the model scale increases, there is still a large performance gap between prompt tuning and fine-tuning for models of moderate and small scales (typically less than 11B parameters). In this paper, we empirically show that the trained prompt tokens can have a negative impact on a downstream task and thus degrade its performance. To bridge the gap, we propose a novel Prompt tuning model with an eXtremely small scale (XPrompt) under the regime of lottery tickets hypothesis. Specifically, XPrompt eliminates the negative prompt tokens at different granularity levels through a hierarchical structured pruning, yielding a more parameter-efficient prompt yet with a competitive performance. Comprehensive experiments are carried out on SuperGLUE tasks, and the extensive results indicate that XPrompt is able to close the performance gap at smaller model scales.

1 Introduction

Fine-tuning large PLMs is parameter-inefficient, while prompt tuning leaves a performance gap at smaller model scales. XPrompt prunes negative prompt tokens hierarchically and reports improved SuperGLUE performance across tasks and scales.

1 Introduction

  • Fine-tuning is parameter-inefficient because gradients and optimizer states must be stored for trainable PLM parameters, while prompt tuning updates only soft prompts but still trails fine-tuning at smaller scales.Prompt tuning uses a much smaller set of tunable parameters than fine-tuning.Source: §1 · p. 1
  • The approach follows the lottery tickets hypothesis: task-trained prompts may include tokens that contribute negatively, alongside positive prompt tokens.The paper frames positive prompt tokens as winning tickets and negative ones as losing tickets.Source: §1 · pp. 1–2
  • XPrompt removes negative prompt tokens first at token level, then at piece level, and rewinds the surviving prompts to retrain them.The method targets a more parameter-efficient prompt through hierarchical structured pruning.Source: §1 · p. 2
  • Across tasks and model scales, XPrompt significantly improves prompt tuning; it closes the gap to fine-tuning at moderate scales and exceeds fine-tuning on most tasks at large scales.The experiments cover high- and low-resource SuperGLUE scenarios.Source: §1 · p. 2

2 Related Work

Related work covers the resource demands of PLMs, prompt-based alternatives and the lottery tickets hypothesis that motivates XPrompt’s pruning approach.

2.1 Pre-trained Language Models

  • Large PLMs make fine-tuning resource-intensive because it maintains model parameters, and separate task-specific fine-tuned models must be stored.The related-work discussion also notes computational expense as model parameter counts grow.Source: §2.1 · p. 2

2.2 Prompt Learning in NLP

  • Prompt learning adds tokens to inputs; soft-prompt tuning updates only prompt embeddings, but moderate-scale models retain a gap to fine-tuning.SPOT initializes target-task prompts from source tasks, while HyperPrompt uses hypernetworks and tunes all parameters for multi-task learning.Source: §2.2 · pp. 2–3

2.3 Lottery Ticket Hypothesis

  • The lottery tickets hypothesis holds that an over-parameterized network can contain a subnetwork that matches the original network’s test accuracy after no more training iterations.Prior work reports winning tickets in PLMs and cases where their generalization can exceed that of the full model.Source: §2.3 · p. 3

3 Preliminary

Prompt tuning prepends trainable soft prompts to inputs and optimizes those prompts for text-generation tasks. The paper motivates XPrompt by the possibility that some trained prompt tokens harm task performance.

3 Preliminary

  • Prompt tuning prepends m soft-prompt embeddings to n input-token embeddings and maximizes label likelihood by optimizing only the prompt embeddings.T5’s model parameters remain fixed in this formulation.Source: §3 · p. 3
  • Because prompt tuning can trail fine-tuning at small and moderate scales, the authors hypothesize that some trained soft-prompt tokens may negatively affect task performance.They propose hierarchical structured pruning to identify useful prompts and address the gap.Source: §3 · p. 3

4 XPROMPT

XPROMPT tunes soft prompts, prunes low-importance tokens and embedding pieces hierarchically, then rewinds and retrains the selected prompts. This process targets negative prompt components while balancing effectiveness and efficiency.

4 XPROMPT

  • XPROMPT repeatedly prunes at token and piece levels to identify prompt components at different compression ratios, then rewinds and retrains the selected prompts.The method uses weight rewinding to retrain prompts after pruning.Source: §4 · pp. 3–4

4.1 Prompt Tuning

  • The initial prompt-tuning stage learns target-task embeddings for all soft prompt tokens, which initialize hierarchical pruning.Source: §4.1 · p. 4

4.2 Hierarchical Structured Pruning

  • Token-level pruning first removes low-importance prompt tokens; piece-level pruning then targets negative pieces remaining within each token.The two levels are intended to trade off effectiveness and efficiency.Source: §4.2 · p. 4, §4.2.1 · p. 4

4.2.1 Token-level Pruning

  • Token importance is the expected sensitivity of model outputs to token masks; low-importance tokens are treated as likely negative and pruned.A zero mask value means the corresponding token is pruned.Source: §4.2.1 · p. 4

4.2.2 Piece-level Pruning

  • Piece-level pruning divides each token embedding into equal-scale pieces, scores them, and removes low-importance pieces that may contribute little or negatively.Pruning is repeated at different compression ratios.Source: §4.2.2 · pp. 4–5

4.3 Rewinding

  • Rewinding resets selected prompt parameters to their post-prompt-tuning values, masks pruned prompts, and retrains using the original prompt-tuning strategy.Source: §4.3 · p. 5

5 Experiments

Experiments evaluate XPROMPT on SuperGLUE tasks using validation results after fixed-step training. Comparisons include fine-tuning and prompt-based baselines, with implementation and training settings specified.

5.1 Datasets

  • The evaluation covers SuperGLUE datasets in high- and low-resource settings, selecting the best checkpoint on validation after a fixed number of training steps.Test access is restricted, so results are reported on the validation set.Source: §5.1 · p. 5

5.2 Baselines

  • The baselines include standard T5 fine-tuning, vanilla prompt tuning, P-Tuning and P-TuningV2, and Prefix-Tuning.Fine-tuning updates all pretrained parameters separately for each target task.Source: §5.2 · p. 5

5.3 Implementation

  • XPROMPT uses OpenPrompt, converts SuperGLUE datasets to text-to-text format, and omits prepended task names from inputs.Experiments use T5 Large, XL, and XXL; prompts are trained for 100 epochs with 20 tokens and 16 pieces per token.Source: §5.3 · p. 5

6 Results

In high-resource SuperGLUE results, XPROMPT improves prompt tuning and brings it to fine-tuning performance at T5-XL, while exceeding fine-tuning on most tasks at T5-XXL. In 32-shot results, it also improves on prompt-tuning baselines on several tasks.

6.1 Results on High-resource Scenarios

  • 3.26%, 2.96%, and 1.88% are XPROMPT’s average-score improvements over vanilla Prompt-Tuning on T5-Large, T5-XL, and T5-XXL, respectively.The reported gains span all three model scales.Source: §6.1 · p. 6
  • XPROMPT matches fine-tuning on all tasks with T5-XL and exceeds it on most tasks with T5-XXL.Table 1 reports the main results across seven SuperGLUE tasks.Source: §6.1 · p. 6, Table 1 · p. 6
  • On T5-XXL, XPROMPT’s scores exceed fine-tuning on WSC, CB, RTE, BoolQ, and MultiRC by +1.91%, +0.0%, +2.84%, +0.47%, and +0.30%, respectively.The reported scores are 97.11%, 100.00%, 94.94%, 90.87%, and 88.90%, respectively.Source: §6.1 · p. 6

6.2 Results on Low-resource Scenarios

  • In 32-shot T5-XL results, XPROMPT outperforms same-scale baselines on BoolQ, WiC, and RTE; its WiC score is 62.85%, 2.04% above Prompt-Tuning.The setting uses 32 randomly selected training examples per task and reports full development-set results.Source: §6.2 · pp. 6–7, Table 2 · p. 6

7 Analysis and Discussion

The analysis examines whether pruning can remove harmful prompt components while retaining useful ones. It also evaluates parameter efficiency, pruning granularity, prompt length, and initialization and transfer choices.

We identify both positive and negative prompts

  • With only 15% positive subprompts, XPROMPT performs 4.8% better than full prompt tuning.The remaining prompt tokens or pieces are treated as winning tickets after low-importance elements are pruned.Source: We identify both positive and negative… · p. 7
  • XPROMPT performs best among the compared variants, while reversed XPROMPT performs significantly worse than the other prompt-tuning variants.Reversed XPROMPT uses low-score prompt tokens and pieces; the comparison also includes random masking and a prompt-length-matched control.Source: We identify both positive and negative… · p. 7, Figure 5 · p. 7

7.2 Parameter Efficiency

  • XPROMPT further reduces Prompt-Tuning’s tunable-parameter count through hierarchical structured pruning.The comparison covers T5-XL with 20 prompt tokens.Source: §7.2 · p. 7, Table 3 · p. 7

7.3 Granularity of Pruning

  • Token-level and piece-level pruning each outperform vanilla Prompt-Tuning, and combining both levels outperforms either level alone.These ablations cover four SuperGLUE tasks.Source: §7.3 · p. 8, Table 4 · p. 8

7.4 Prompt Length

  • Increasing XPROMPT prompt length beyond 20 tokens yields only marginal gains.The T5-XL comparison tests prompt lengths of 10, 20, and 100 tokens.Source: §7.4 · p. 8, Table 5 · p. 8

7.5 Prompt Initialization and Transfer

  • Sampled-vocabulary initialization performs best, while XPROMPT Transfer outperforms TaskTransfer without a rewinding stage.The transfer comparison attributes the gains to pruning and rewinding.Source: §7.5 · p. 8, Table 6 · p. 8, Table 7 · p. 8

8 Conclusions

The paper targets prompt tuning’s performance gap to fine-tuning at small and moderate scales, proposing XPrompt’s hierarchical pruning to separate positive from negative prompts at token and piece levels.

8 Conclusions

  • XPrompt applies hierarchical structured pruning at token and piece levels to separate positive prompts from negative ones, targeting the gap between prompt tuning and fine-tuning at small and moderate scales.The paper reports that the resulting prompts are extremely small yet retain competitive effectiveness.Source: §8 · pp. 8–9

Limitations

The authors identify compression-ratio selection and additional prompt-tuning scenarios as open areas for future research.

Limitations

  • Pruning requires rewinding at different compression ratios, leaving optimal ratio selection without trial training under-explored; multi-task, domain-shift, and prompt-ensembling scenarios also remain future work.Source: Limitations · p. 9

A More Results of P-TuningV2

The supplementary results compare XPrompt with P-Tuning methods, examine WSC importance-score distributions, and describe transfer from a source task to a target task.

A More Results of P-TuningV2

B Token and Piece Importance Score Distribution

C XPROMPT Transfer

  • XPROMPT Transfer trains prompts on a source task, uses them to initialize target-task prompts, then trains on the target; unlike SPOT, it offers more cross-task information to prompts.Source: C XPROMPT Transfer · p. 13, Figure 8 · p. 13

D Importance Scores Visualization

The visualization section shows importance scores and pruning on WSC, while the accompanying benchmark materials describe the seven evaluated SuperGLUE tasks, their metrics, and prompt templates.

D Importance Scores Visualization

E SuperGLUE Statistics, Metrics and Soft Prompt Templates

Related papers

Loading 2210.04457v1…