PaperlayerView exact revision on arXiv

Source-linked AI summary

IDPG: An Instance-Dependent Prompt Generation Method

Zhuofeng Wu, Sinong Wang, Jiatao Gu, Rui Hou, Yuxiao Dong, V. G. Vinod Vydiswaran, Hao Ma

arXiv:2204.04497v1cs.CLcs.LG

TL;DR

Large pretrained language models make full fine-tuning costly, while prompt-tuning methods commonly rely on fixed task-specific prompts. IDPG generates prompts conditioned on each input instance. Across ten NLU tasks, it outperforms task-specific prompt-tuning baselines and is comparable to adapter-based methods with fewer parameters.

  • Problem

    Full-model fine-tuning requires updating and storing all parameters for each downstream task, and existing prompt-tuning methods use task-specific prompts that may not fit varied pretraining prefixes.

    Source: §1 · p. 1

  • Method

    IDPG uses a lightweight conditional generator to create a prompt for each input instance while tuning a small number of parameters.

    Source: §1 · p. 2

  • Results

    Across ten NLU tasks, IDPG outperforms task-specific prompt-tuning methods by 1.6–3.1 points and offers comparable Adapter-based performance with 134K versus 1.55M parameters.

    Source: §1 · p. 2

  • Takeaways & Limitations

    IDPG provides an instance-dependent prompt-tuning approach that the paper reports as consistently improving over prompt-tuning methods and performing on par with lightweight adapter tuning.

    Source: §6 · p. 8

    Inference requires two forward passes of the pretrained language model, first to generate M(xi) and then for classification.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Prompt tuning is a new, efficient NLP transfer learning paradigm that adds a task-specific prompt in each input instance during the model training stage. It freezes the pre-trained language model and only optimizes a few task-specific prompts. In this paper, we propose a conditional prompt generation method to generate prompts for each input instance, referred to as the Instance-Dependent Prompt Generation (IDPG). Unlike traditional prompt tuning methods that use a fixed prompt, IDPG introduces a lightweight and trainable component to generate prompts based on each input sentence. Extensive experiments on ten natural language understanding (NLU) tasks show that the proposed strategy consistently outperforms various prompt tuning baselines and is on par with other efficient transfer learning methods such as Compacter while tuning far fewer model parameters.

1 Introduction

Full-model fine-tuning becomes costly as pretrained language models grow, while existing prompt tuning uses fixed task-specific prompts. IDPG generates prompts per input instance and is reported to improve on prompt-tuning baselines across ten NLU tasks.

1 Introduction

  • Full-model fine-tuning requires updating and storing all parameters for each downstream task, becoming costly as pretrained models grow.The passage gives BERT at 330M parameters and GPT-3 at 175B as examples.Source: §1 · p. 1
  • Existing prompt-tuning methods use task-specific prompts, which may be mismatched with the varied prefixes in pretraining data.The paper asks whether input-dependent prompts could smooth the domain difference.Source: §1 · pp. 1–2
  • IDPG generates a prompt for each input instance instead of using a fixed task-specific prompt, with a lightweight conditional generator.Its generator can degenerate to traditional prompt tuning when W is zero and only the bias is trained.Source: §1 · p. 2
  • IDPG trains 134K parameters per task, about 0.04% of a pretrained model such as RoBERTa-Large.Source: §1 · p. 2
  • Across ten natural language understanding (NLU) tasks, IDPG consistently outperforms task-specific prompt-tuning methods by 1.6–3.1 points and offers comparable Adapter-based performance using 134K rather than 1.55M parameters.Source: §1 · p. 2

2 Preliminary

Manual prompting inserts predefined words into each input, whereas prompt tuning uses trainable soft prompts. The latter converts prompt search into continuous optimization with randomly initialized embeddings.

2.1 Manual Prompt

  • Manual prompting inserts predefined label words into each input sentence, then uses the resulting sentence representation for task-specific classification.The cited examples report benefits from designed prompts in few-shot settings and entailment reformulation in low- and high-resource settings.Source: §2.1 · p. 2

2.2 Prompt Tuning

  • Prompt tuning, prefix tuning, and P-tuning prepend trainable soft prompts to single-sentence or sentence-pair inputs.The prompt is an embedding table of randomly initialized tensors without corresponding real text, apart from LM-BFF and EFL.Source: §2.2 · p. 2

3 Instance-Dependent Prompt Generation (IDPG)

IDPG generates prompts conditioned on each input representation, combining an instance-dependent component with a task-specific component. The section describes parameter reduction through PHM layers and multi-layer prompt designs that share generator weights.

3 Instance-Dependent Prompt Generation (IDPG)

  • The method section introduces IDPG and its model optimizations, with Figure 1 illustrating manual prompting, prompt tuning, and the proposed method.The figure marks trainable modules in red and frozen modules in blue.Source: §3 · p. 2, Figure 1 · p. 3

3.1 Instance-Dependent Generation

  • IDPG conditions prompt generation on the input sequence representation M(xi), rather than only on the task.The generated prompt is inserted with the input sequence for inference during fine-tuning.Source: §3.1 · p. 3
  • The generator projects each sentence representation through a bottleneck and back to produce a prompt combining instance-dependent output with a task-specific bias.Its parameter count is m(d +1)+td(m+1), and choosing m much smaller than d controls the added parameters.Source: §3.1 · p. 3
  • Inference uses two forward passes of the pretrained language model: one to obtain M(xi), then another for classification.The method relies on the input sentence representation to generate prompts.Source: §3.1 · p. 3

3.2.1 Parameterized Hypercomplex Multiplication (PHM) Layers

  • PHM replaces a dense weight matrix with a sum of Kronecker products of smaller matrices to reduce generator parameters.The passage describes n as a user-defined positive integer dividing m and d; reducing parameters is the purpose of the optimization.Source: §3.2.1 · p. 4
  • For the stated RoBERTa-Large generator configuration, sharing A_i across two PHM layers reduces parameters from 1.5M to 105K.The configuration uses d = 1024, m = 256, n = 16, and prompt length t = 5.Source: §3.2.1 · p. 4

3.2.2 Multi-layer Prompt Tuning

  • Single-layer prompt tuning is described as limited in capturing deep context and in generalizing to long input sequences.Source: §3.2.2 · p. 4
  • M-IDPG prepends generated prompts at each transformer layer, sharing generator weights while keeping each layer's bias specific.Independent generators would substantially increase training parameters; the shared-weight design is presented as a way to balance parameter count and performance.Source: §3.2.2 · p. 4

4 Experiment Results

Across full-data and few-shot evaluations, IDPG methods generally outperform prompt-tuning baselines, with performance comparable to efficient adapter methods at a lower parameter count. The experiments also examine generator design, prompt placement, sentence representations, and how prompts affect sentence similarity.

4.2 Performance in high-resource scenario

  • M-IDPG-PHM beats prompt tuning and Ptuning v2 by average 3.1pt and 1.6pt, respectively, except on RTE.It performs slightly worse than Compacter and Adapter across ten tasks, with most of the gap from RTE and QQP; it matches or beats Compacter on seven tasks. Gains are most prominent in single-sentence classification, and PHM and GloVe variants perform comparably to their larger or contextual alternatives.Source: Table 1 · p. 5, §4.2 · p. 5

4.3 Efficiency

  • M-IDPG-PHM uses 134K trainable parameters and performs on par with Compacter.The comparison excludes classification-head parameters.Source: §4.3 · pp. 5–6, Table 2 · p. 6

4.4 Performance in low-resource scenario

  • At K=100, M-IDPG-PHM exceeds traditional prompt tuning by 2.5pt and multi-layer P-Tuning v2 by 0.5pt; gains persist at K=500 and K=1000.Performance sometimes has high variance at small K, including 4.6 on MPQA at K=100; the authors suspect bad initialization.Source: §4.4 · p. 6

4.5.1 Sentence Encoder: GloVe or LMs?

  • Using GloVe sentence representations to generate prompts sacrifices little performance over ten tasks and still outperforms prompt tuning and P-tuning v2.The authors conclude that IDPG does not rely on a strong contextual pretrained language model for its sentence encoder.Source: §4.5.1 · p. 6

4.5.2 Prompt Generator: PHM or DNN?

  • Replacing DNN layers with PHM layers adds only 0.1pt average performance while saving 87K parameters.The ablation supports PHM layers as an efficient prompt-generator design.Source: §4.5.2 · p. 6

4.5.3 Multi-layer Architecture Exploration

  • In the M version, hidden size 16 reaches 92.0 versus 92.1 for size 256; input choice makes no significant difference, while results follow S < M < L.The authors hypothesize that size 16 can already store useful instance information and that larger hidden sizes may be less efficient.Source: §4.5.3 · p. 7, Figure 2 · p. 7

4.5.4 Prompt Insertion: Single-layer or Multi-layer?

  • Multi-layer IDPG significantly outperforms single-layer IDPG in both evaluation settings; supplementary training has a larger impact on the single-layer model.Source: §4.5.4 · pp. 7–8

4.5.5 How Prompts Help?; 4.5.6 IDPG Scalability

5 Related Work

Prior work reduces fine-tuning costs with adapters and continuous prompts, while IDPG extends prompt optimization from task-specific to instance-specific generation.

5 Related Work

  • Adapters insert and train bottleneck layers between frozen transformer layers; on GLUE, adapters come within 0.4% of full fine-tuning while training 3.6% of parameters per task.Compacter replaces projector matrices with sums of Kronecker products to reduce parameters while maintaining overall performance.Source: §5 · p. 8
  • Continuous prompt methods address discrete prompt-search challenges by optimizing trainable embeddings, while hand-crafted and searched prompts are also explored for generation and few-shot settings.The passage discusses GPT-3 hand-crafted prompts and LM-BFF prompt searching in few-shot settings.Source: §5 · p. 8
  • IDPG is presented as the first prompt generator that is both task-specific and instance-specific, unlike earlier continuous-prompt methods focused on task-specific optimization.Earlier approaches prepend trainable embeddings at transformer layers or interleave them in the input embedding layer.Source: §5 · p. 8

6 Conclusion and Discussion

IDPG generates instance-dependent prompts and applies Parameterized Hypercomplex Multiplication to keep its generator lightweight. It reports consistent gains over prompt-tuning methods and performance on par with Compacter using a similar amount of trainable parameters.

6 Conclusion and Discussion

  • IDPG consistently improves on prompt-tuning methods and is on par with Compacter while using a similar amount of trainable parameters.The conclusion describes the instance-dependent prompt as robust to data variance and identifies Parameterized Hypercomplex Multiplication (PHM) as the parameter-shrinking mechanism.Source: §6 · p. 8
  • IDPG introduces instance-dependent prompts, which the authors describe as robust to data variance, and uses PHM to shrink the prompt generator.The paper presents this lightweight generator as a new research angle for prompt tuning of pre-trained language models.Source: §6 · p. 8

A.1.1 Training hyperparameters

The appendix identifies hyperparameters for supplementary training, fine-tuning, and few-shot fine-tuning; transformer fine-tuning follows official instructions with warmup and a fixed epoch count.

A.1.1 Training hyperparameters

  • Table 4 presents hyperparameters for supplementary training, fine-tuning, and few-shot fine-tuning.Source: Table 4 · p. 11
  • Transformer fine-tuning methods RoBERTa and EFL use a polynomial learning-rate scheduler, warm up for 6% of total steps, and train for 10 epochs.The authors state that they follow the methods’ official training instructions.Source: §A.1.1 · p. 11

A.1.2 Model hyperparameters

The appendix details parameter counts for the compared tuning methods and IDPG variants, including PHM and DNN generators and GloVe-based input.

A.1.2 Model hyperparameters

  • The appendix states that it reports detailed model hyperparameters for each method and explains how the Table 2 parameter counts are computed.Source: §A.1.2 · p. 11
  • S-IDPG-PHM uses 105K parameters, while S-IDPG-DNN uses 1.5M parameters.Both use a generator hidden size of 256 and prompt length 5; the PHM configuration specifies n = 16.Source: §A.1.2 · p. 11
  • The listed baselines use 149.25K parameters for Compacter, 1.55M for Adapter, 5K for prompt-tuning, 134K for prompt-tuning-134, and 120K for P-tuning v2.The passage provides the component calculations and settings used to derive these totals.Source: §A.1.2 · p. 11
  • M-IDPG-PHM uses 134K parameters, compared with 216K for M-IDPG-DNN and 141K for M-IDPG-PHM-GloVe.These configurations use prompt length 5; the PHM variants also specify their hidden and generator sizes.Source: §A.1.2 · p. 12

A.2 Datasets

The appendix reports dataset and detailed-result materials, then examines supplementary training, generator architecture, prompt position, similarity distributions, prompt length, and potential risks. Its reported findings include sensitivity to distribution differences and prompt position, architecture-dependent voting results, and IDPG stability with longer prompts and larger models.

A.2 Datasets

A.3 Detailed results for Multi-layer Architecture Exploration

  • Table 6 reports main results across transfer-learning methods, averaged over five runs, while the appendix uses a broader learning-rate search for one M-IDPG-PHM setting.The broader search is why its m = 16, previous-layer-input result is slightly higher than in Table 1; Table 6 reports task-specific metrics.Source: §A.3 · p. 12, Table 6 · p. 13

A.4 Supplementary Training for Single-layer IDPG

  • When supplementary and downstream data distributions differ, supplementary training may harm downstream performance; the authors balance sentence lengths by repositioning prompts and reordering MNLI examples.MNLI’s first sentence is 9.8 tokens longer on average than its second, versus a 0.6-token difference in MRPC; doubling MNLI through sentence reordering is used during intermediate training.Source: Figure 4 · p. 12, §A.4 · p. 13, Figure 5 · p. 13

A.5.1 Generator Architecture Exploration

  • Adding LayerNorm slightly improves voting results, residual performs slightly worse, and combining residual with LayerNorm performs significantly more poorly.The generator variants apply residual addition, LayerNorm, or both; the mixed model normalizes embeddings before and after addition.Source: §A.5.1 · p. 14

A.5.2 Prompt Position

  • For downstream tasks, position 0 works best for single-sentence tasks and position 1 for sentence-pair tasks; in supplementary training, IDPG favors position 0 while standard prompt tuning favors position 4.The authors hypothesize position 1 better aligns sentence pairs.Source: §A.5.2 · p. 14, Figure 6 · p. 14

A.6 Cosine Similarity Distributions in STS-B

  • The appendix presents STS-B cosine-similarity distributions for k = 100 and k = 300, with Figure 7 describing top-k pair counts by group.Source: §A.6 · p. 14, Figure 7 · p. 15

A.7 Ablation Study on Prompt Length

  • IDPG remains stable as prompt length increases and models grow larger.The comparison covers several prompt-tuning methods, but the supplied passage gives no numeric values.Source: §A.7 · p. 14, Figure 8 · p. 15

A.8 Potential Risks

  • Whether IDPG works well when its online-stored backbone model is attacked remains unknown, and the authors urge caution in high-stakes use without comprehensive testing.Source: §A.8 · pp. 14–15
Loading 2204.04497v1…