PaperlayerView exact revision on arXiv

Source-linked AI summary

An LLM can Fool Itself: A Prompt-Based Adversarial Attack

Xilie Xu, Keyi Kong, Ning Liu, Lizhen Cui, Di Wang, Jingfeng Zhang, Mohan Kankanhalli

arXiv:2310.13345v1cs.CR

TL;DR

LLM applications require reliable adversarial-robustness evaluation, especially in safety-critical settings. PromptAttack has the victim model generate semantically preserved adversarial samples through a structured prompt, and it substantially outperforms AdvGLUE variants across evaluated Llama2 and GPT-3.5 settings.

  • Problem

    Existing robustness audits can be ineffective for black-box LLMs and computationally expensive, limiting practical evaluation of adversarial robustness.

  • Method

    PromptAttack uses original input, attack objective, and attack guidance to make a victim LLM generate semantically preserved adversarial samples, with few-shot and ensemble enhancements.

  • Results

    PromptAttack consistently achieves higher attack success rates than AdvGLUE and AdvGLUE++ on Llama2 and GPT-3.5, including a 42.18% GPT-3.5 ASR increase on SST-2.

  • Takeaways & Limitations

    A black-box LLM can generate adversarial samples that fool itself, and even a simple emoji can induce an incorrect GPT-3.5 prediction.

Abstract

from arXiv · show

The wide-ranging applications of large language models (LLMs), especially in safety-critical domains, necessitate the proper evaluation of the LLM's adversarial robustness. This paper proposes an efficient tool to audit the LLM's adversarial robustness via a prompt-based adversarial attack (PromptAttack). PromptAttack converts adversarial textual attacks into an attack prompt that can cause the victim LLM to output the adversarial sample to fool itself. The attack prompt is composed of three important components: (1) original input (OI) including the original sample and its ground-truth label, (2) attack objective (AO) illustrating a task description of generating a new sample that can fool itself without changing the semantic meaning, and (3) attack guidance (AG) containing the perturbation instructions to guide the LLM on how to complete the task by perturbing the original sample at character, word, and sentence levels, respectively. Besides, we use a fidelity filter to ensure that PromptAttack maintains the original semantic meanings of the adversarial examples. Further, we enhance the attack power of PromptAttack by ensembling adversarial examples at different perturbation levels. Comprehensive empirical results using Llama2 and GPT-3.5 validate that PromptAttack consistently yields a much higher attack success rate compared to AdvGLUE and AdvGLUE++. Interesting findings include that a simple emoji can easily mislead GPT-3.5 to make wrong predictions.

1 Introduction

PromptAttack addresses inefficient and ineffective robustness audits for black-box LLMs by having a victim model generate semantically preserved adversarial samples through a structured attack prompt. Experiments show substantially stronger attacks than AdvGLUE variants while requiring only a few victim queries.

  • LLM adversarial robustness matters for reliability in safety-critical applications such as medicine and industrial control.
  • AdvGLUE and AdvGLUE++ are ineffective and inefficient for evaluating black-box victim LLMs because they transfer attacks generated against other models and require substantial computation.
  • PromptAttack converts each labeled example into an attack prompt containing original input, attack objective, and attack guidance for generating a semantically preserved misclassified sample.
  • A fidelity filter uses word modification ratio and BERTScore to retain adversarial samples that preserve the original semantic meaning.
  • Few-shot examples and ensembling across perturbation levels are proposed to improve the quality and attack power of generated adversarial samples.
  • 42.18% and 24.85% ASR gains over AdvGLUE baselines occur for GPT-3.5 on SST-2 and QQP, respectively.The SST-2 ASR rises from 33.04% to 75.23%, while QQP rises from 14.76% to 39.61%.

2 Related Work

Related work frames adversarial attacks as a broad security concern and situates PromptAttack among robustness benchmarks for language models and wider LLM reliability problems.

  • Adversarial attacks perturb inputs to induce incorrect classifications and motivate robustness evaluation before deployment in safety-critical domains.
  • PromptAttack is illustrated as generating an adversarial sample by adding the emoji “:)”, which can fool GPT-3.5.
  • AdvGLUE and AdvGLUE++ are datasets for evaluating language-model and LLM robustness using adversarial samples generated against BERT-based models and other attacks.
  • LLM reliability research includes hallucination, jailbreak-related privacy leakage, and other risks associated with generated content.

3 Prompt-Based Adversarial Attack

PromptAttack structures self-generated adversarial-example creation around labeled inputs, explicit objectives, and perturbation guidance, then filters outputs for fidelity and strengthens generation with few-shot examples and ensembles.

  • 3.1 Framework of PromptAttack: PromptAttack converts adversarial textual attacks into an attack prompt with original input, attack objective, and attack guidance.
  • 3.1 Framework of PromptAttack: The original input represents each dataset example, including sentence types, sentence contents, and its ground-truth label.
  • 3.1 Framework of PromptAttack: The attack objective requires changing one target sentence while preserving semantic meaning and causing an incorrect classification.
  • 3.1 Framework of PromptAttack: Attack guidance specifies output formatting and perturbation instructions at character, word, and sentence levels.
  • 3.1 Framework of PromptAttack: Character-level instructions create typos or append extraneous characters, while word-level instructions substitute, delete, or add semantically neutral words.
  • 3.1 Framework of PromptAttack: Sentence-level instructions append meaningless handles or paraphrase the input, extending established adversarial perturbation strategies.
  • 3.2 Fidelity Filter: The fidelity filter uses word modification ratio and BERTScore thresholds to reject generated samples whose semantics change substantially.
  • 3.3 Enhancing PromptAttack: Few-shot and ensemble strategies are introduced to strengthen PromptAttack, with few-shot examples helping LLMs interpret perturbation instructions and generate stronger samples.

4 Experiments

The experiments evaluate PromptAttack on GLUE tasks using Llama2 and GPT-3.5, comparing it with AdvGLUE and AdvGLUE++. PromptAttack achieves stronger attack performance, while results vary with victim model, perturbation, task-description, fidelity threshold, and transfer setting.

  • Robustness Evaluation on GLUE Dataset: PromptAttack significantly outperforms AdvGLUE and AdvGLUE++ across all GLUE tasks and can efficiently audit LLM adversarial robustness.The evaluation uses filtered adversarial samples and ASR as the attack-power measure.
  • Extensive Empirical Results: GPT-3.5 is reported as more adversarially robust than Llama2 because it attains lower ASR under PromptAttack, while Llama2-13B appears more vulnerable than Llama2-7B.The transfer experiments also report successful attacks against other victim LLMs.
  • Extensive Empirical Results: Sentence-level perturbations are stronger than character- and word-level perturbations, and the few-shot strategy further increases ASR against GPT-3.5.These findings are reported for perturbation instructions evaluated in Table 4.
  • Extensive Empirical Results: Task-description effects differ by victim: role-oriented descriptions yield lower ASR than task-oriented descriptions, while few-shot effects differ between GPT-3.5 and Llama2.Few-shot descriptions are more robust than zero-shot descriptions for GPT-3.5 but produce much higher ASR for Llama2.
  • Extensive Empirical Results: At BERTScore threshold τ2 = 0.95 on QNLI, PromptAttack-FS-EN reaches almost 48% ASR, whereas AdvGLUE and AdvGLUE++ remain below 10%.This comparison demonstrates high attack power under a stringent fidelity threshold.

5 Conclusions

The paper concludes that PromptAttack uses a structured attack prompt to make an LLM generate adversarial samples against itself, with fidelity filtering and enhancement strategies supporting robustness evaluation. It frames this approach as an efficient alternative purpose-built for auditing LLM adversarial robustness rather than improving downstream task performance.

  • 5 Conclusions: PromptAttack combines original input, attack objective, and attack guidance to ask a victim LLM to generate an adversarial sample that fools itself.The attack prompt is provided as a template for automatically generating prompts from data points.
  • 5 Conclusions: A fidelity filter preserves original semantics, while few-shot and ensemble strategies increase PromptAttack’s attack power.The filter uses fidelity criteria, and the strategies are designed to strengthen generated adversarial samples.
  • 5 Conclusions: PromptAttack consistently achieves state-of-the-art attack success rates on GLUE and is presented as an effective, efficient tool for auditing LLM adversarial robustness.Its prompt-based design differs from prompt-based learning and prompt engineering aimed at downstream task performance.

B.1 GLUE Dataset

The GLUE evaluation covers several classification and inference tasks with task-specific sentence types and labels. PromptAttack is applied to selected sentence fields in QQP, RTE, and QNLI.

  • SST-2 is a binary sentiment task that classifies one sentence as positive or negative.
  • QQP is a binary task that determines whether two questions are duplicate or not duplicate.PromptAttack perturbs only question1 in the experiments.
  • MNLI evaluates whether a hypothesis logically follows from a premise, with matched and mismatched versions.
  • RTE is a binary task that determines whether two sentences express entailment or not entailment.The paper applies PromptAttack to the specified sentence fields in this task.
  • QNLI asks whether an answer to a question can be found in a sentence, using entailment and not entailment labels.PromptAttack is applied only to the question-type sentence in QNLI.

B.2 BERTScore

BERTScore measures semantic fidelity between original and adversarial sentences, and its threshold controls which adversarial samples enter the robustness evaluation. At high thresholds, PromptAttack achieves higher ASR across the reported GPT-3.5 tasks.

  • BERTScore h_bert(x, x̃) measures similarity between an original sentence and its adversarial variant using contextual word embeddings.The score is defined over the interval [0, 1].
  • Figure 4 evaluates ASR against τ2 in MNLI-m, QQP, and RTE using GPT-3.5.
  • The fidelity threshold τ2 filters adversarial samples whose semantic meaning has changed significantly.τ2 is calculated as the average BERTScore of AdvGLUE adversarial samples for each task.
  • PromptAttack obtains higher ASR at high BERTScore thresholds across the reported tasks, indicating high-fidelity adversarial samples can remain effective.
  • In RTE, AdvGLUE++ exceeds PromptAttack when τ2 ≤0.85, but the paper excludes low-fidelity comparisons because semantic meaning is significantly changed.

B.3 ASR without Fidelity Filter

The paper reports that PromptAttack outperforms AdvGLUE++ even without the fidelity filter, but argues that unfiltered ASR is not meaningful when adversarial samples change the original semantics.

  • PromptAttack yields higher ASR than AdvGLUE++ without the fidelity filter.
  • The paper considers unfiltered ASR meaningless because low-BERTScore adversarial samples can significantly change semantic meaning.
  • Llama2 shows extremely high ASR standard deviation in tasks such as MNLI-mm and QNLI.The paper attributes this to divergent zero-shot and few-shot ASR results in those tasks.

B.5 ASR Evaluated via Different Types of Task Descriptions

Task-description format affects attack success differently across victim models. Few-shot descriptions generally improve GPT-3.5 results but consistently reduce Llama2 results relative to zero-shot descriptions.

  • GPT-3.5 achieves lower ASR with zero-shot than few-shot task descriptions in most tasks.This pattern is consistent with the cited prior conclusion.
  • Llama2 consistently achieves lower ASR with zero-shot than few-shot task descriptions.
  • The paper conjectures that few-shot examples can confuse Llama2 about how to solve the task, degrading its few-shot inference performance.

B.6 Attack Transferability

PromptAttack adversarial samples transfer across victim models, including BERT-based models, while exposing differences in robustness between fine-tuned classifiers and prompted LLMs. An ensemble of perturbation variants supports transfer evaluation, and adversarial training is associated with lower attack success rates in robust BERT-based models.

  • Transfer evaluation: PromptAttack adversarial samples generated against GPT-3.5 and Llama2-7B were transferred to other LLMs and evaluated using attack success rate (ASR).Table 6 transfers GPT-3.5-generated samples to Llama2-7B and Llama2-13B; Table 7 transfers Llama2-7B-generated samples to Llama2-13B and GPT-3.5.
  • Transfer evaluation: Nine adversarial variants per data point were generated with different perturbation instructions, and the variant that fooled the transfer victim was selected.This ensemble strategy evaluates attack transferability across perturbation levels.
  • BERT-based transfer: PromptAttack was also transferable to BERT-based models, whose robust versions consistently produced lower ASR than standard versions.The evaluation used standard and adversarially trained BERT-based models.
  • Robustness comparison: BERT-based models achieved lower ASR than GPT-3.5, suggesting greater robustness in these evaluations.The paper attributes this difference to task-specific fine-tuning for BERT-based models versus prompt-based task solving by LLMs.

B.7 Extensive Examples

The examples illustrate both the semantic risks and model-specific failure modes of PromptAttack. Low BERTScore corresponds to changed meanings, while few-shot guidance can produce visibly malformed Llama2 outputs that the fidelity filter removes.

  • GPT-3.5 examples: GPT-3.5 adversarial examples for SST-2 can be reproduced with a fixed model version, zero temperature, and a specified sentiment-classification prompt.The examples are reported in Table 17.
  • Fidelity examples: BERTScore below 0.85 is associated with substantially different semantic meanings between adversarial samples and their originals.Five such examples are sampled from the RTE task in AdvGLUE++.
  • Llama2 examples: Llama2-7B outputs under PromptAttack-FS-EN repeatedly used two sentences joined by a meaningless arrow pattern, copying the few-shot guidance format.These outputs were easily identified as low fidelity and filtered out, producing low ASR against Llama2.
Loading 2310.13345v1…