PaperlayerView exact revision on arXiv

Source-linked AI summary

PromptAttack: Prompt-based Attack for Language Models via Gradient Search

Yundi Shi, Piji Li, Changchun Yin, Zhaoyang Han, Lu Zhou, Zhe Liu

arXiv:2209.01882v1cs.CLcs.AIcs.CR

TL;DR

Prompt learning is a lighter alternative to fine-tuning, but the security of prompt-based models and their templates has been little studied. PromptAttack automatically constructs malicious templates, and experiments report effectiveness across three datasets and three models, including few-shot settings.

  • Problem

    Prompt-based model security, particularly the vulnerability of templates, has received little study despite prompt learning's role as a lighter alternative to fine-tuning.

    Source: §1 · pp. 1–3

  • Method

    PromptAttack constructs malicious templates by selecting label words, searching candidate tokens with a gradient-search variant, and choosing replacements by random replacement or beam search.

    Source: §1 · p. 3

  • Results

    Experiments across three datasets and three pre-trained models establish PromptAttack's effectiveness, and the attack remains effective in few-shot cases.

    Source: §1 · p. 3, §5 · p. 11

  • Takeaways & Limitations

    Template choice is a security-relevant attack surface: a malicious template can greatly reduce prediction accuracy.

    Source: §1 · p. 3, §5 · p. 11

    PromptAttack targets only discrete prompts and is a white-box attack requiring access to all model parameters.

    Source: §5 · p. 11

Abstract

from arXiv · show

As the pre-trained language models (PLMs) continue to grow, so do the hardware and data requirements for fine-tuning PLMs. Therefore, the researchers have come up with a lighter method called \textit{Prompt Learning}. However, during the investigations, we observe that the prompt learning methods are vulnerable and can easily be attacked by some illegally constructed prompts, resulting in classification errors, and serious security problems for PLMs. Most of the current research ignores the security issue of prompt-based methods. Therefore, in this paper, we propose a malicious prompt template construction method (\textbf{PromptAttack}) to probe the security performance of PLMs. Several unfriendly template construction approaches are investigated to guide the model to misclassify the task. Extensive experiments on three datasets and three PLMs prove the effectiveness of our proposed approach PromptAttack. We also conduct experiments to verify that our method is applicable in few-shot scenarios.

1 Introduction

Prompt learning offers a lighter alternative to fine-tuning, but the security of prompt-based models—especially their templates—has received little study. The paper proposes PromptAttack to construct malicious templates and evaluates its effectiveness across datasets, models, and few-shot settings.

1 Introduction

  • As PLMs grow, fine-tuning's hardware and data requirements rise, motivating prompt learning as a lighter alternative.Prompt learning aims to use models consistently with pre-training tasks without fine-tuning all parameters.Source: §1 · pp. 1–2
  • Prompt models use cloze templates to elicit masked-token predictions, then map predicted words to task labels with a verbalizer.Templates may be designed manually or constructed automatically; designing an efficient manual template requires substantial experimentation, knowledge, and cost.Source: §1 · p. 2
  • Prompt-based model security remains under-evaluated, although template choice affects predictions and a malicious template can greatly reduce accuracy.The authors state that almost no other papers had considered this security problem.Source: §1 · pp. 2–3
  • PromptAttack searches for candidate tokens, replaces template tokens by random replacement or beam search, and selects the template with the lowest training accuracy.It uses GPT-2 to generate a more readable template with lower perplexity.Source: §1 · p. 3
  • Experiments on three datasets and three pre-trained models establish PromptAttack's effectiveness, and the attack remains effective in few-shot cases.The introduction reports the few-shot finding alongside the broader evaluation.Source: §1 · p. 3

2 Related work

Related work covers prompt-learning approaches ranging from manual and discrete templates to continuous and multimodal prompts, alongside adversarial and backdoor attacks. The PromptAttack overview depicts template formation, masked-token prediction, and label comparison.

2.1 Prompt Learning

  • Prompt learning spans manual PET [12], automatic verbalizer search [11], discrete AutoPrompt [13], continuous prefix/P-tuning [6] [8], and multimodal prompts [18] [19].The passage presents these as representative prompt methods.Source: §2.1 · p. 3
  • PromptAttack's overview shows template parts forming an input, followed by masked-token prediction, label mapping, and comparison with the real label.The figure caption describes this pipeline.Source: Figure 2 · p. 4

2.2 Attack methods

  • Adversarial attacks target test-time inputs [1] [5], while backdoor attacks poison training data [4]; AToP and BToP address prompt methods [17].The passage distinguishes adversarial attacks at testing from backdoor attacks during training.Source: §2.2 · p. 4

3 Methodology

PromptAttack constructs shared trigger tokens to lower the predicted probability of the true label, then selects token sequences for malicious templates. It also automatically maps labels to words.

3.1 Overview

  • PromptAttack selects shared trigger tokens through gradient search, then chooses the sequence with the best attack result by random replacement or beam search.The template combines the input, trigger tokens, and a masked token.Source: §3.1 · pp. 4–5

3.2 Select candidate words based on gradient search

  • Trigger-token search iteratively targets a lower true-label probability and keeps the k vocabulary tokens with the smallest first-order approximation scores as candidates.The score approximates the loss change from replacing a trigger position; its gradient is obtained by backpropagation.Source: §3.2 · p. 5

3.3 Selection of token sequences

  • Three strategies—random replacement, beam search, and GPT-2 generation—select token sequences for the final template.The selected sequence is used in training and testing.Source: §3.3 · p. 5

Random replacement strategy.

  • Random replacement tests candidate words at a randomly chosen position, keeps the least accurate training sequence for each iteration, and selects the lowest-accuracy test sequence overall.The method may leave some [MASK] positions unreplaced, especially in few-shot settings.Source: Random replacement strategy · p. 5, Beam search method · p. 6

Beam search method.

  • Beam search expands candidate sequences position by position and retains the k sequences with the best attack effect at each step.The example uses 100 candidates, beamwidth 2, and three trigger tokens, retaining the two lowest-accuracy templates after each step.Source: Beam search method · p. 6, Figure 3 · p. 7

Generate token sequences using GPT-2.

  • GPT-2 is used because random replacement and beam search do not account for the fluency of generated token sequences.GPT-2 is trained on the training text; sequences are generated from candidate first tokens, and the worst training-set performer is assessed on test accuracy and perplexity.Source: Generate token sequences using GPT-2 · pp. 6–7
  • The GPT-2 approach is reported to enhance template readability and increase concealment of the generated malicious template.Source: Generate token sequences using GPT-2 · p. 7

3.4 Automatic selection of label mapping

  • Automatic label mapping scores words for their relationship to each label and selects the k highest-scoring words for each label’s word set.A logistic classifier predicts labels from contexts containing a [MASK] token; word embeddings replace the context representation to obtain the scores.Source: §3.4 · p. 7

4 Experiments

The experiments test PromptAttack on three sentiment datasets and three pretrained models, then examine cross-dataset performance, template readability, and few-shot use. Results report successful attacks across datasets and few-shot settings, alongside a readability–attack-performance trade-off for GPT-2 templates.

4.1 Datasets

  • The evaluation covers SST-2 movie-review sentences, IMDB data, and Amazon Video_Games review summaries, with sentiment labels derived from emotion or ratings.SST-2 and IMDB use 0/1 negative/positive labels; Amazon uses 1.0–2.0 ratings as negative and 4.0–5.0 as positive.Source: §4 · p. 8, §4.1 · p. 8

4.2 Setup

  • Experiments use Bert_base, Roberta_base, and Roberta_large, compare against Autoprompt, and measure accuracy drop under matched conditions.Each dataset uses 24,000 balanced training examples and approximately 1,000 balanced test examples.Source: §4.2 · p. 8
  • The setup uses five template tokens, three label words, and 100 candidate trigger tokens; the label-mapping procedure selects the top three words.Training and test batch sizes are 24 and 48, respectively.Source: §4.2 · p. 8, §4.3 · p. 9

Selection results of words corresponding to labels.

  • The top-three label words differed across models, but label 0 words were mostly derogatory and label 1 words mostly commendatory.The label words were selected after 50 iterations, with the table reporting mappings for SST-2.Source: Selection results of words… · p. 9, §4.3 · p. 9, Table 1 · p. 9
  • On SST-2, both strategies reduced accuracy across pretrained models; beam search on Roberta_large was strongest, with reported attack success rates of 47.36% and 56.77% for the two strategies.The attack effect was reported as better for Roberta models than for Bert.Source: Table 2 · p. 9, Selection results of words… · p. 9

Results of SST-2/IMDB/Amazon dataset on Roberta_large.

Validation of the method of GPT-2.

Experimental results in the case of few-shot.

5 Conclusion and future work

PromptAttack targets the overlooked security of prompt templates; experiments on three sentiment datasets and three PLMs support its effectiveness and few-shot applicability. The authors identify discrete prompts and white-box access to all model parameters as current boundaries.

  • Experiments across three sentiment datasets and three pre-trained language models verify PromptAttack's effectiveness and applicability in few-shot settings.Source: §5 · p. 11
  • PromptAttack addresses the security of prompt models by targeting their vulnerable template component.Source: §5 · p. 11
  • The method targets only discrete prompts, and its white-box setting requires access to all model parameters.Source: §5 · p. 11

Related papers

Loading 2209.01882v1…