PaperlayerView exact revision on arXiv

Source-linked AI summary

Attack Prompt Generation for Red Teaming and Defending Large Language Models

Boyi Deng, Wenjie Wang, Fuli Feng, Yang Deng, Qifan Wang, Xiangnan He

arXiv:2310.12505v1cs.CLcs.CRcs.LG

TL;DR

Red teaming attacks can induce harmful LLM outputs, while manual and automatic prompt-construction methods have cost and quality limitations. The paper combines human-crafted examples with LLM prompt generation and iteratively fine-tunes target models through interactions with the attack framework. Experiments report promising attack performance and improved safety from defense fine-tuning.

  • Problem

    Manual prompt construction is costly, while automatically generated attack prompts often have lower quality, limiting the available means of producing high-quality prompts efficiently.

    Source: §1 · p. 1

  • Method

    The attack framework uses in-context learning to extend manually constructed prompts, and the defense framework iteratively fine-tunes target LLMs through interaction with that attack framework.

    Source: §1 · p. 2

  • Results

    Generated prompts achieved promising attack performance on GPT-3.5 and Alpaca, even surpassing manually constructed cases, while defense fine-tuning enhanced Alpaca-LoRA safety.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors constructed five SAP datasets of varying sizes for safety evaluation and enhancement.

    Source: §5 · p. 9

    Defense experiments primarily focused on Alpaca-series models, and gpt-3.5-turbo-0301 may evaluate some outlier responses inaccurately.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are susceptible to red teaming attacks, which can induce LLMs to generate harmful content. Previous research constructs attack prompts via manual or automatic methods, which have their own limitations on construction cost and quality. To address these issues, we propose an integrated approach that combines manual and automatic methods to economically generate high-quality attack prompts. Specifically, considering the impressive capabilities of newly emerged LLMs, we propose an attack framework to instruct LLMs to mimic human-generated prompts through in-context learning. Furthermore, we propose a defense framework that fine-tunes victim LLMs through iterative interactions with the attack framework to enhance their safety against red teaming attacks. Extensive experiments on different LLMs validate the effectiveness of our proposed attack and defense frameworks. Additionally, we release a series of attack prompts datasets named SAP with varying sizes, facilitating the safety evaluation and enhancement of more LLMs. Our code and dataset is available on https://github.com/Aatrox103/SAP .

1 Introduction

Red teaming attacks can induce harmful LLM outputs, while manual prompt construction is costly and automatic generation often produces lower-quality prompts. The paper combines manual examples with LLM-based generation and iterative defense fine-tuning, reporting effective attacks and defenses.

1 Introduction

  • Manual prompt construction is time-consuming and costly, whereas automatically generated attack prompts often have lower quality.Source: §1 · p. 1
  • Generated prompts achieved promising attack performance on GPT-3.5 and Alpaca, even surpassing manually constructed cases, and defense fine-tuning enhanced Alpaca-LoRA safety.Source: §1 · p. 2
  • The proposed attack framework uses manually constructed prompts as examples for LLMs to generate more prompts through in-context learning.The authors describe this as combining manual and automatic construction to generate extensive, high-quality prompts efficiently.Source: §1 · pp. 1–2
  • The defense framework iteratively fine-tunes target LLMs using attack prompts and generates further prompts through interaction with the attack framework.Prompts that still attack the target after fine-tuning are selected as examples for generating similar prompts in the next round.Source: §1 · p. 2
  • The authors release attack-prompt datasets in varying sizes to support future research.Source: §1 · p. 2

2 Related Work

Prior work spans LLM capabilities and prompt-based attacks, as well as approaches to reducing harmful outputs. The paper situates its hybrid prompt-construction approach against the cost of manual methods and quality concerns with automatic generation.

2 Related Work

  • LLMs demonstrate content-generation and in-context learning capabilities, while instruction tuning has been used to train models to follow natural-language instructions.Source: §2 · p. 2
  • Prompt attacks have been constructed manually or generated automatically; manual methods can produce high-quality prompts but are costly, while automatic prompts often have lower quality.The related work also describes role-playing instructions, goal hijacking, prompt leaking, and malicious prompts targeting LLM-integrated applications.Source: §2 · p. 2
  • Prior defenses include filtering pretraining data, fine-tuning on non-toxic or value-targeted corpora, toxic-prompt training, and reinforcement learning from human feedback.Source: §2 · pp. 2–3

3 Method

The method defines attack and defense tasks, then develops a semi-automatic attack-prompt process and an iterative defense loop. The attack process evaluates generated prompts and expands a dataset; defense fine-tunes on prompts and repeatedly targets those that remain successful.

3 Method

  • The method presents red teaming attack and defense tasks followed by the proposed frameworks.Source: §3 · p. 3

3.1 Task formulation

  • An attack seeks natural-language prompts that induce harmful output across eight sensitive topics, while defense seeks harmless responses or refusal; the defender has prompt access and can fine-tune, whereas the attacker treats the target as a black box.Source: §3.1 · p. 3

3.2 Red Teaming Attack Framework

  • The semi-automatic attack framework seeds the prompt set with manually constructed examples, then uses an LLM to mimic them and generate more prompts.The attack LLM receives three sampled examples through in-context learning; the paper also describes manually adding explanations and requesting rationales for generated prompts.Source: §3.2 · pp. 3–4
  • A generated prompt is treated as high quality when the target output receives a harmfulness score above 5 on the 0–10 scale.The evaluation uses gpt3.5-turbo-0301; the authors report that a threshold of 5 balances recall and precision in Appendix D.Source: §3.2 · pp. 4–5
  • The authors release SAP5, SAP10, SAP20, SAP30, and SAP200, with dataset sizes ranging from 40 to 1,600 prompts.Source: §3.2 · p. 5

3.3 Red Teaming Defense Framework

  • Defense evaluates attacks, retains prompts that still succeed, uses them to expand the attack set, and fine-tunes the target LLM to produce safe outputs in repeated rounds.The loop reassesses defense after each fine-tuning round and expands harder prompts; the desired outputs are typical refusal responses.Source: §3.3 · p. 5

4 Experiment

The experiments ask whether the attack framework can red-team LLMs, whether the defense framework improves safety, and whether fine-tuning affects other capabilities. Across the reported comparisons, SAP30 has the highest harmful scores, while defense fine-tuning reduces harmful scores and leaves regular NLP performance intact.

4.1.1 LLMs

  • The experiments use GPT-3.5 models and Alpaca-LoRA 7B and 13B models.The Alpaca-LoRA models are used for fine-tuning in the experiments.Source: §4.1.1 · pp. 5–6

4.1.3 Benchmarks

  • The study also evaluates whether defense fine-tuning affects performance on BoolQ, ARC-Easy, RACE, CB, and COPA.Accuracy is the metric reported for the original and fine-tuned models.Source: §4.1.3 · p. 6, Table 3 · p. 8

4.2 Attack Results (RQ1)

  • SAP30 achieves the highest harmful scores across four LLMs, outperforming both Dual-Use and BAD+ in the attack comparison.The reported comparison uses average harmful scores, with higher scores indicating more harmful prompts.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 Defense Results (RQ2)

  • Across Alpaca-LoRA 7B and 13B fine-tuned with SAP5, SAP10, or SAP30, harmful scores fell significantly, though some settings showed overfitting.The reported score trends declined during fine-tuning.Source: Table 2 · p. 7, §4.3 · p. 7
  • 0.01 harmful score was reached by some LLMs on SAP20, while Dual-Use and BAD+ gains were smaller amid distributional differences.The authors attribute SAP20's stronger defense results to its distribution match with the training data.Source: §4.3 · pp. 7–8, Table 2 · p. 7
  • Cases show responses shifting from harmful discourse or theft assistance to refusal, including theft prompts absent from training.The case studies sample prompts from SAP5 and BAD+ before and after fine-tuning.Source: Figure 9 · p. 8, Figure 10 · p. 8, §4.3 · p. 8

4.4 Performance on Other Tasks (RQ3)

  • Fine-tuning did not impair performance across five regular NLP benchmarks and improved performance on some, measured by accuracy.The benchmarks are BoolQ, ARC-Easy, RACE, CB, and COPA.Source: §4.1.3 · p. 6, §4.4 · p. 8, Table 3 · p. 8

5 Conclusion

The paper presents paired attack and defense frameworks and reports empirical evidence of their effectiveness. It also releases five SAP datasets and identifies broader evaluation as future work.

5 Conclusion

  • The attack framework combines manual and automatic prompt construction, while the defense framework fine-tunes target LLMs through multi-turn interactions with the attack framework.The experiments report attack-prompt improvements and defense efficacy, with minimal impact on original LLM capabilities.Source: §5 · p. 9
  • The authors constructed five SAP attack-prompt datasets of varying sizes for LLM safety evaluation and enhancement.Source: §5 · p. 9
  • Future work will expand SAP with more prompts and evaluate attack performance on bigger datasets and more LLMs.Source: §5 · p. 9

Limitations

The defense experiments primarily cover Alpaca-series models, and the harmfulness evaluator can misjudge some outlier responses.

Limitations

  • Resource limitations and open-source issues restricted defense experiments primarily to Alpaca-series models, leaving broader LLM coverage for future work.Source: Limitations · p. 9
  • The experiments use gpt-3.5-turbo-0301 as evaluator, which cannot accurately assess some outlier responses.The authors suggest incorporating more powerful LLMs for harmfulness evaluation.Source: Limitations · p. 9
  • The authors identify evaluation of defense on a broader range of diverse LLMs as needed future exploration.Source: Limitations · p. 9

Ethics Statement

The authors caution that generated outputs and released attack prompts may contain harmful material, and advise researchers to handle the dataset carefully.

Ethics Statement

  • Outputs from the attack framework may be harmful or offensive and do not reflect the authors’ viewpoints.Source: Ethics Statement · p. 9
  • The released dataset may contain prompts that could facilitate harmful activities, so researchers are advised to handle it with utmost caution.Source: Ethics Statement · p. 9
  • The authors state that the dataset is provided to help researchers construct safer LLMs using their defense framework.Source: Ethics Statement · p. 9

A Overfitting in Fine-Tuning

Immutable fine-tuning prompts can lead to overfitting, including unexpected text after refusals; the defense framework regenerates prompts at each iteration. Against defense-WID, it lowers harmful scores across three test datasets and significantly alleviates overfitting on BAD+.

A Overfitting in Fine-Tuning

  • Regenerating prompts each iteration reduces harmful scores across three test datasets and significantly alleviates overfitting on BAD+ versus immutable-prompt defense-WID.Defense-WID fine-tunes on fixed original attack prompts; with more iterations, unexpected text may follow refusals, particularly with an immutable dataset, as illustrated in Figure 12.Source: Figure 11 · p. 11, A Overfitting in Fine-Tuning · p. 11, Figure 12 · p. 12

B Attack Mechanisms in Dual-Use

B.1 Obfuscation

  • Obfuscation can use typos or synonyms for terms that trigger input or output filters, such as replacing “COVID-19” with “CVID.”Source: §B.1 · p. 12

B.2 Code injection/Payload splitting

  • Code injection or payload splitting divides a malicious payload into smaller parts, each intended not to trigger defense mechanisms.Source: §B.2 · p. 12

B.3 Virtualization

  • Virtualization sets up a simulated environment or character with rules that may contradict an LLM’s original ethical restrictions, prompting compliance with harmful instructions.Source: §B.3 · p. 12

C Contexts Categories of BAD+

  • BAD+ includes 12 context categories, including identity attack, insult, profanity, threat, sexual content, flirtation, drugs, politics, religion, medical, NSFW, and none.Source: C Contexts Categories of BAD+ · p. 12

D GPT-3.5 vs. Perspective Api

GPT-3.5 achieves a larger ROC AUC than Perspective API in this evaluation. At a harmfulness threshold of 5, its recall is 0.94 and its precision is 1.00.

D GPT-3.5 vs. Perspective Api

  • Perspective API is a widely used free service for identifying toxic comments.Source: D GPT-3.5 vs. Perspective Api · p. 12
  • They generated 100 attack prompts and responses without automatic evaluation, manually selected 50 harmful responses as negative samples, and generated 50 “refuse to answer” responses as positive samples for harmfulness scoring.Perspective API and GPT-3.5 assigned harmful scores to all samples.Source: D GPT-3.5 vs. Perspective Api · p. 12
  • At threshold 5, GPT-3.5 has 0.94 recall and 1.00 precision, with a larger ROC AUC than Perspective API.The ROC curve compares GPT-3.5 and Perspective API.Source: D GPT-3.5 vs. Perspective Api · pp. 12–13, Figure 14 · p. 13

E Reproducibility

The reproducibility appendix reports the hardware used for fine-tuning, the shared hyperparameter setup, and the benchmark evaluation code.

E.1 Computing Infrastructure

  • Alpaca-LoRA-7B was fine-tuned on one GeForce RTX 3090 with 24GB memory, while Alpaca-LoRA-13B used one NVIDIA A40 with 48GB memory.Source: §E.1 · p. 13

E.2 Fine-tuning Hyperparameters

  • Fine-tuning experiments use a consistent set of hyperparameters from Table 5, with all remaining settings left at their defaults.Source: §E.2 · p. 13, Table 5 · p. 13

E.3 Benchmarks Evaluation

  • LLM performance on various benchmarks is assessed with the EleutherAI lm-evaluation-harness repository.Source: §E.3 · p. 13

E.4 Time and Cost Analysis

Generating SAP200 took approximately 35 hours and cost around 10 USD in OpenAI API calls. Prompt-sensitivity experiments found low variance across example selections and orderings.

E.4 Time and Cost Analysis

  • SAP200 generation took approximately 35 hours and around 10 USD in OpenAI API calls.Source: §E.4 · p. 13

F Prompt Sensitivity

  • Across ten three-example sets and their six orderings, detrimental-score variance was low, indicating robustness to example selection and order.Table 4 reports harmful scores for the ten selections across six ordering variations.Source: F Prompt Sensitivity · pp. 13–14
  • The authors suggest iterative selection of relatively high-quality in-context examples as a possible reason for the limited prompt sensitivity.Source: F Prompt Sensitivity · p. 14
Loading 2310.12505v1…