Source-linked AI summary
PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, Xing Xie
TL;DR
PromptRobust addresses the largely unexplored robustness of LLMs to prompt perturbations, which can affect multiple samples when prompts are reused. It constructs and evaluates a benchmark using realistic perturbations across attack levels, tasks, datasets, and models, then analyzes vulnerabilities and transferability. The results show that current LLMs are not robust enough to adversarial prompts, motivating guidance for more robust prompt engineering.
Problem
Robustness to prompt perturbations is largely unexplored, despite prompts being reused across multiple samples and LLMs being used in safety-critical and decision-making domains.
Method
PromptRobust benchmarks character-, word-, sentence-, and semantic-level perturbations across prompt types, models, tasks, and datasets, with analyses of vulnerabilities and transferability.
Results
Current LLMs are not robust enough to adversarial prompts; word-level attacks produce a 39% average performance drop across all tasks.
Takeaways & Limitations
Attention visualization and word-frequency analysis provide guidance for researchers, prompt engineers, and end-users developing more robust prompts.
Takeaways & Limitations
The study relies on sampled rather than full datasets, omits some LLMs and advanced prompt-engineering techniques, and evaluates black-box rather than optimized white-box attacks.
Abstract
from arXiv · showhide
The increasing reliance on Large Language Models (LLMs) across academia and industry necessitates a comprehensive understanding of their robustness to prompts. In response to this vital need, we introduce PromptRobust, a robustness benchmark designed to measure LLMs' resilience to adversarial prompts. This study uses a plethora of adversarial textual attacks targeting prompts across multiple levels: character, word, sentence, and semantic. The adversarial prompts, crafted to mimic plausible user errors like typos or synonyms, aim to evaluate how slight deviations can affect LLM outcomes while maintaining semantic integrity. These prompts are then employed in diverse tasks including sentiment analysis, natural language inference, reading comprehension, machine translation, and math problem-solving. Our study generates 4,788 adversarial prompts, meticulously evaluated over 8 tasks and 13 datasets. Our findings demonstrate that contemporary LLMs are not robust to adversarial prompts. Furthermore, we present a comprehensive analysis to understand the mystery behind prompt robustness and its transferability. We then offer insightful robustness analysis and pragmatic recommendations for prompt composition, beneficial to both researchers and everyday users.
1 Introduction
PromptRobust addresses the largely unexplored robustness of LLMs to prompt perturbations, which can affect multiple samples because one prompt is reused across them. It benchmarks diverse adversarial prompts and analyzes their effects and transferability.
- Motivation: Prompt perturbations can affect multiple clean samples, potentially giving them broader impact than adversarial samples that target individual inputs.A single prompt often instructs an LLM across multiple samples, so changing that prompt may cause repeated incorrect responses.
- Benchmark design: The benchmark includes character-, word-, sentence-, and semantic-level prompt attacks designed to represent both adversarial manipulations and naturally occurring user perturbations.Examples include typos, synonyms, and alternative expressions with the same semantic meaning.
- Evaluation: PromptRobust evaluates robustness across 8 tasks and 13 datasets, spanning applications such as sentiment analysis, reasoning, and other language tasks.
- Findings: 39% average performance drop across all tasks is reported for word-level attacks, which are identified as the most effective attacks.
- Analysis: Attention analysis indicates that adversarial prompts shift LLM focus toward perturbed elements, while transferability between models is reported as successful.The paper also analyzes word-frequency patterns to provide guidance for more robust prompt composition.
- Contribution: PromptRobust is introduced as a systematic benchmark for evaluating and understanding LLM robustness to adversarial prompts.
2 PromptRobust
PromptRobust models prompt attacks as constrained textual perturbations applied to reusable prompts rather than individual samples. It combines multiple prompt formats, attack levels, models, and datasets while checking semantic plausibility.
- 2.1 Prompts and models: The benchmark studies task-oriented and role-oriented prompts in both zero-shot and few-shot settings.Role-oriented prompts frame the model as an entity such as an expert, advisor, or translator; task-oriented prompts explicitly describe the required task.
- 2.1 Prompts and models: The evaluation considers nine LLMs, including Flan-T5-large, Dolly-6B, Vicuna-13B, Llama2-13B-chat, ChatGPT, and GPT-4.
- 2.2 Attacks: Prompt attacks target prompts supplied by users, systems, or developers rather than samples, reflecting their reuse in in-context learning.
- 2.2 Attacks: A prompt attack perturbs P under constraint set C so that model fθ produces incorrect responses across samples in dataset D.This extends universal adversarial perturbation and universal adversarial trigger ideas to prompts.
- 2.2 Attacks: The study uses black-box textual attacks because they are efficient and do not require model gradients, allowing open-source and proprietary LLMs to be targeted.
- 2.2 Attacks: Attacks operate at character, word, sentence, and semantic levels, using typos, synonym substitutions, appended distractions, and meaning-preserving alterations.
- 2.2 Attacks: The adversarial prompts are intended to simulate plausible user errors while preserving semantic integrity; at least 85% of subjects judged generated prompts acceptable.
- 2.3 Tasks and datasets: PromptRobust supports 8 tasks and 13 datasets, with dataset statistics summarized in Table 1.
3 Experiments
The experiments use sampled evaluation sets, clean-prompt baselines, and the normalized Performance Drop Rate to compare attacks, models, and prompt types. Tables 2–4 organize these robustness results.
- Setup: Sampling is used because generating adversarial prompts requires approximately 100 dataset iterations on average, making full-dataset evaluation infeasible.
- Results organization: Table 4 reports Average PDR across different prompt types.
- Setup: The study first measures clean-prompt performance and selects six LLMs whose baseline results are considered satisfactory for subsequent evaluation.The selected models are Flan-T5-large, Vicuna-13B, Llama2-13B-chat, UL2, ChatGPT, and GPT-4.
- Metrics: Performance Drop Rate (PDR) measures relative performance decline after a prompt attack, enabling comparisons across metrics, datasets, and models.A negative PDR indicates that adversarial prompts improve performance.
- Setup: The evaluation can involve 73,248 test instances per model for SST-2 under the described combination of samples, attacks, prompt types, and prompts.
4 Results and analysis
PromptRobust finds that adversarial prompts substantially reduce LLM performance, with vulnerability varying across attack types, models, prompt formats, and datasets. Analyses further connect errors to attention divergence, context-dependent word robustness, model characteristics, and limited transferability across models.
- Attack effectiveness: 33% average performance decline makes word-level attacks the most potent across 13 datasets, followed by 20% drops from character-level attacks in most datasets.Semantic-level attacks are nearly as potent as character-level attacks, while sentence-level attacks are less threatening.
- Model robustness: GPT-4 and UL2 are the most robust models, while Vicuna is consistently the most susceptible across tasks.Robustness varies by dataset: UL2 excels in translation, and ChatGPT is robust on certain natural language inference tasks.
- Prompt formats: Few-shot prompts consistently outperform zero-shot prompts in robustness, while task-oriented and role-oriented prompts trade strengths across datasets.Role-oriented prompts are more robust on SST-2 and QQP, whereas task-oriented prompts are more resilient on MRPC, QNLI, SQuAD V2, and IWSLT.
- Model size and fine-tuning: Larger models are typically more robust than smaller Llama2 models, and fine-tuned models are generally better at handling adversarial prompts.Exceptions occur in which smaller models outperform larger ones.
- Attention analysis: Adversarial prompts can reroute attention from key text segments, producing misclassifications when models attend simultaneously to target text and adversarial content.Attention by Gradient and Attention by Deletion produce similar results; the analysis focuses on Attention by Gradient.
- Transferability: Adversarial prompts show marginal and inconsistent transferability, with some transferred attacks reducing performance and others unexpectedly improving it.Transferability to ChatGPT is better than to T5-large and UL2, while weak transferability may partly explain GPT-4’s observed robustness against ChatGPT-generated prompts.
5 Related work
Prior work evaluates LLM robustness mainly against adversarial or out-of-distribution samples and assesses trustworthiness across several safety-related dimensions. PromptRobust instead addresses robustness to perturbations of the instructions themselves.
- Existing benchmarks such as AdvGLUE and ANLI evaluate language-model robustness to carefully perturbed adversarial samples.
- DecodingTrust evaluates GPT-model trustworthiness through toxicity, stereotype bias, adversarial challenges, and privacy.
- SafetyPrompts and prompt-injection attacks target potentially harmful instructions that may steer models toward outputs misaligned with human values.
- PromptRobust mimics potential user prompts with adversarial perturbations, but its purpose is robustness evaluation rather than actually attacking models.
6 Conclusion and Limitation
PromptRobust evaluates LLM robustness to naturally plausible adversarial prompt perturbations across tasks and models, finding substantial vulnerability and offering analysis of its causes. The study is constrained by sampled data, incomplete model and dataset coverage, and limited prompt-engineering settings.
- PromptRobust evaluates robustness to typos, synonyms, and stylistic differences across various tasks and models.
- Current LLMs show a prevailing lack of robustness to adversarial prompts, with word-level attacks producing a 39% average performance drop across all tasks.
- The study analyzes attention weights associated with erroneous responses on clean and adversarial inputs to investigate the observed vulnerability.
- The evaluation relies on sampled subsets rather than full datasets because of substantial computational requirements.
- The study excludes some LLMs, datasets, and advanced prompt-engineering techniques such as chain-of-thought and tree-of-thought.
- The attacks are black-box; optimized white-box prompt attacks may produce stronger adversarial prompts.
A Details on Adversarial Prompts
The appendix details four prompt types and seven attack strategies designed to create semantically preserved perturbations at character, word, sentence, and semantic levels. Human evaluation and similarity constraints support the intended semantic preservation, while prompt attacks differ from sample attacks in target and universality.
- Prompt types: PromptRobust includes zero-shot, few-shot, role-oriented, and task-oriented prompt types.
- Character-level attacks: Character-level attacks introduce typos through insertions, deletions, replacements, and replications within words.
- Word-level attacks: Word-level attacks replace words with synonyms or contextually similar alternatives while tuning similarity constraints to preserve meaning.TextFooler uses minimum word-synonym cosine similarity of 0.6 and Universal Sentence Encoder similarity of 0.84; BertAttack uses a 0.8 encoder-similarity threshold.
- Sentence-level attacks: Sentence-level attacks append irrelevant or extraneous sentences to distract models from the primary context.
- Semantic-level attacks: Semantic-level attacks translate prompts through languages including Chinese, French, Arabic, Spanish, Japanese, and Korean before translating them back into English.
- Prompt attacks versus sample attacks: Prompt attacks target instructions rather than samples, and one adversarial prompt can affect many samples for a task across datasets.
B Models, Datasets, and Environments
The study evaluates selected open and proprietary LLMs on public datasets spanning language understanding, reasoning, reading comprehension, translation, and mathematics. PromptRobust supports additional datasets, while the experiments use a reproducible software environment on NVIDIA Tesla V100 GPUs.
- Models: The experiments include Flan-T5-large, Vicuna-13B, Flan-UL2, ChatGPT, GPT-4, and other listed LLMs.
- Datasets: PromptRobust uses public datasets including GLUE, MMLU, SQuAD V2, Multi UN, IWSLT 2017, and DeepMind Mathematics.
- Datasets: GLUE covers tasks such as sentiment analysis, grammar correctness, duplicate-sentence detection, and natural-language understanding.
- Datasets: MMLU tests knowledge and problem-solving across 57 multiple-choice tasks from diverse domains.
- Environment: The computational experiments ran on machines equipped with NVIDIA Tesla V100 GPUs with 16GB of memory each.
C Details on Experiments and Results
The experiments evaluate model, prompt, and sample robustness across datasets and attacks, finding that robustness does not clearly track model size and that word-level attacks are especially damaging. Fine-tuning improves resistance, while attacks on both prompts and samples can sharply reduce accuracy.
- Clean-prompt evaluation: Clean-prompt results led the study to focus attacks on T5, Vicuna, Llama2, UL2, ChatGPT, and GPT4.Dolly, Cerebras, and NEXO performed poorly on some datasets, including Dolly’s 0.53% accuracy on QQP versus T5’s 86.67%.
- Model size and fine-tuning: There is no clear correlation between model robustness and size: T5-large is comparable to larger models such as ChatGPT.The authors suggest fine-tuning strategies and training-data memorization may contribute to robustness differences.
- Model size and fine-tuning: Fine-tuned Llama2-chat models fare better against adversarial prompts than non-fine-tuned Llama2-7B models.The comparison covers Llama2-7B and Llama2-7B-chat on SST2 and CoLA.
- Semantic preservation: APDR remains considerably high after excluding adversarial prompts that do not preserve semantics.This result is reported across each evaluated dataset.
- Attacking prompts and samples: Attacking both prompts and samples further reduces LLM performance, with word-level attacks producing more than 49% accuracy drop.Checklist sometimes improved performance, including 69.59% on SST-2 versus the original 63.51%.
D Effectiveness Analysis of LLM Response and Attention Visualization
The analysis examines how adversarial prompts produce incorrect or incoherent responses and uses gradient- and deletion-based attention methods to identify influential words. Sentence-level attacks can sometimes improve performance by sharpening attention to relevant keywords and labels.
- Erroneous response analysis: Adversarial prompts induce misclassification errors and hinder models’ ability to generate meaningful responses.Observed effects include shifted sentiment predictions and incoherent outputs.
- Attention visualization techniques: The gradient analysis sums token-level gradients belonging to each word, then uses the word-gradient l2 norm and min-max normalization to obtain swj.The method first computes token gradients for the input and label.
- Attention visualization techniques: Attention by deletion scores a word by the absolute loss difference between the complete input and the input with that word deleted.The raw score is then min-max normalized into a final word score.
- Sentence-level attacks: Sentence-level attacks occasionally improve performance by directing greater attention to pertinent question keywords and labels.For example, adding “and true is true” increased focus on the “not entailment” label while retaining attention to “minnow” and “duck.”
E Details on Word frequency Analysis
Word-frequency analysis finds that robustness depends on contextual word use rather than the mere presence of particular terms. Robust and vulnerable prompts can share many words, indicating that frequency alone does not determine resilience.
- Frequency analysis: Prompts are classified as vulnerable when performance drops over 10% and robust when the drop is 10% or less.The analysis computes robust word frequency from occurrences in the two prompt categories.
- Frequency findings: In CoLA, prompts containing “acting,” “answering,” and “detection” were more resistant, while “analyze,” “answer,” and “assess” were more susceptible.The analysis used adversarial prompts generated for CoLA and MRPC by T5-large.
- Frequency findings: MRPC showed substantial overlap between words in robust and vulnerable prompts, challenging the idea that specific words inherently determine resilience.Figure 4 presents the word-frequency results for CoLA and MRPC.
- Interpretation: Prompt resilience is tied to contextual word use, with semantic coherence and syntactic structure identified as possible additional factors.The authors position this finding as guidance for research and more resistant prompt construction.
F Defenses
The paper groups defenses against adversarial prompts into training, input preprocessing, and downstream fine-tuning strategies. Proposed measures include adversarial data, diverse-model ensembles, spelling checks, semantic rephrasing, and query-history verification.
- Defense axes: Defense strategies span training-phase methods, input preprocessing, and downstream fine-tuning.The paper presents these as three main axes for robustness enhancement.
- Training-phase strategies: Adversarial data integration aims to reduce susceptibility by exposing models to low-quality or intentionally perturbed inputs during training and fine-tuning.The approach is intended to broaden the range of inputs models encounter.
- Training-phase strategies: Mixture-of-experts ensembles may limit attacks effective against a single model, although prompt attacks on the ensemble could enhance transferability.The proposed ensemble uses independently trained, diverse models and combines their outputs.
- Input preprocessing: Automated spelling verification can counter typographical character-level attacks and sentence-level inconsistencies.The strategy is intended to maintain input fidelity.
- Input preprocessing: Semantic rephrasing and historical-context verification are proposed to filter misleading prompts and flag suspicious input sequences.The former analyzes prompt meaning and intent; the latter tracks recent queries for coordinated attack patterns.
- Visualization platform: A Streamlit web application provides interactive exploration of adversarial prompts across models, datasets, prompt types, and attacks.The application is hosted by Hugging Face and was planned for future release.