PaperlayerView exact revision on arXiv

Source-linked AI summary

Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts

Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H. Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, Tim Rocktäschel, Roberta Raileanu

arXiv:2402.16822v3cs.CLcs.AIcs.LG

TL;DR

Existing adversarial-prompt discovery methods can require specialized access or human input and often lack diversity. Rainbow Teaming uses quality-diversity search with an LLM-based MAP-Elites archive to generate effective, diverse prompts. It finds hundreds of successful attacks across tested models and domains, while synthetic-data fine-tuning improves robustness without reducing general capabilities or helpfulness.

  • Problem

    Existing adversarial-prompt methods may require attacker-model fine-tuning, white-box target access, or substantial human input, and many lack diversity.

  • Method

    Rainbow Teaming uses quality-diversity search and a MAP-Elites-style archive to generate diverse adversarial prompts through LLM-based mutation and preference evaluation.

  • Results

    Rainbow Teaming found hundreds of adversarial prompts per run with an attack success rate higher than 90% across all tested models, while synthetic-data fine-tuning enhanced robustness without diminishing general capabilities or helpfulness.

  • Takeaways & Limitations

    The method generalizes across safety, question answering, and cybersecurity, and its prompts can support vulnerability diagnosis and robustness improvement.

  • Takeaways & Limitations

    Benchmark comparisons report chat-model scores against base-model scores reported by prior work.

Abstract

from arXiv · show

As large language models (LLMs) become increasingly prevalent across many real-world applications, understanding and enhancing their robustness to adversarial attacks is of paramount importance. Existing methods for identifying adversarial prompts tend to focus on specific domains, lack diversity, or require extensive human annotations. To address these limitations, we present Rainbow Teaming, a novel black-box approach for producing a diverse collection of adversarial prompts. Rainbow Teaming casts adversarial prompt generation as a quality-diversity problem and uses open-ended search to generate prompts that are both effective and diverse. Focusing on the safety domain, we use Rainbow Teaming to target various state-of-the-art LLMs, including the Llama 2 and Llama 3 models. Our approach reveals hundreds of effective adversarial prompts, with an attack success rate exceeding 90% across all tested models. Furthermore, we demonstrate that prompts generated by Rainbow Teaming are highly transferable and that fine-tuning models with synthetic data generated by our method significantly enhances their safety without sacrificing general performance or helpfulness. We additionally explore the versatility of Rainbow Teaming by applying it to question answering and cybersecurity, showcasing its potential to drive robust open-ended self-improvement in a wide range of applications.

1 Introduction

Rainbow Teaming addresses limitations in adversarial-prompt discovery by searching for prompts that are both effective and diverse. It generates broad archives useful for diagnosing vulnerabilities and creating synthetic data for robustness improvements.

  • Motivation: Existing adversarial-prompt methods may require attacker-model fine-tuning, white-box target access, or substantial human input, while often lacking diversity.These limitations reduce their usefulness for vulnerability diagnosis and synthetic-data generation.
  • Approach: Rainbow Teaming casts adversarial-prompt generation as quality-diversity search, directly optimizing attack quality and diversity.The approach is designed to efficiently cover the attack space rather than trade diversity against success rate.
  • Approach: Rainbow Teaming builds a MAP-Elites archive whose dimensions categorize prompts by features such as attack style, risk category, or prompt length.The resulting archive covers combinations of specified features and supports both diagnosis and robustification.
  • Results: Over 90% attack success was achieved across all tested safety-aligned models, with hundreds of adversarial prompts found per run without external data.The experiments targeted Llama 2-chat and Llama 3-Instruct models.
  • Results: Synthetic data from Rainbow Teaming improved resistance to unseen attacks and later search rounds without diminishing general capabilities or helpfulness.The method was also applied to question answering and cybersecurity, uncovering hundreds of effective adversarial prompts in each domain.

2 Background

Rainbow Teaming builds on quality-diversity search and the MAP-Elites algorithm. MAP-Elites maintains high-fitness solutions across a discretized multidimensional feature space.

  • Quality-diversity search: Quality-diversity methods seek solutions that are individually high-performing and collectively diverse.They represent quality with a fitness function and diversity with a feature descriptor mapping solutions into a feature space.
  • MAP-Elites: MAP-Elites tracks the highest-fitness solution in each cell of a multidimensional archive that discretizes the feature space.The archive is initialized with random solutions and iteratively updated through selection, mutation, and evaluation.
  • MAP-Elites: A mutated solution x′ is assigned to the archive cell specified by its descriptor z′ and replaces the occupant when the cell is vacant or x′ has higher fitness.The occupant is called the elite, and repeated updates fill the archive with high-fitness solutions.

3 Rainbow Teaming

Rainbow Teaming adapts MAP-Elites into an LLM-based open-ended search over adversarial prompts. It explicitly targets diverse feature combinations while selecting effective candidates through target responses and preference judgments.

  • Design rationale: Rainbow Teaming treats all possible prompts as the solution space in a quality-diversity search.The method is motivated by the need for diverse attacks for comprehensive diagnosis and safety fine-tuning.
  • Archive: The method stores prompts in a K-dimensional archive, with each cell representing a unique combination of predefined feature categories.Initial solutions may be generated randomly with an LLM or loaded from an existing dataset.
  • Candidate generation: Each iteration samples an archived parent prompt and a target descriptor, then a Mutator LLM generates a candidate prompt for that descriptor.The Mutator applies one mutation per feature, K mutations overall.
  • Evaluation: The Target LLM evaluates the candidate, and a Judge LLM compares it with the elite occupying the same descriptor before storing the winner.The comparison uses effectiveness criteria such as whether the Target response is unsafe.
  • Implementation choices: Rainbow Teaming uses predefined features, a mutation operator, and a preference model, which can be implemented with LLMs or other components.Candidate descriptors are sampled in advance, and high-BLEU candidates are filtered to promote diversity.

4 Rainbow Teaming for Safety

Rainbow Teaming uses quality-diversity search to generate effective, diverse adversarial prompts across safety categories and attack styles. Against multiple instruction-tuned LLMs, it achieves high attack success, transfers across models, and preserves archive diversity through similarity filtering.

  • Safety setup: Rainbow Teaming evaluates safety attacks across risk categories and attack styles using Llama 2 and Llama 3 models among other publicly available LLMs.Risk categories include Criminal Planning, Violence and Hate, and Self-Harm; attack styles include Authority Manipulation and Role Play.
  • Safety setup: The method initializes prompts from scratch, evolves them with risk-category and attack-style mutations, and uses a Llama 2 70B judge to retain the more unsafe response.Attack Success Rate is evaluated with GPT-4 and Llama Guard, while Llama Guard unsafe-response probability biases selection.
  • Main results: 92% approximate ASR is obtained against both Llama 2-chat 7B and Llama 3-Instruct 8B, while Mistral 7B and Vicuna 7B reach 98% after 2000 iterations.Each final archive contains 100 prompts; GPT-4 evaluates these results, which are echoed by Llama Guard.
  • Main results: 90% or higher ASR is achieved across Llama 2-chat models sized 7B, 13B, and 70B.These results show effectiveness across the tested Llama 2-chat model sizes.
  • Baseline comparison: Rainbow Teaming outperforms No Stepping Stones and Same Cell Mutations, highlighting the value of archive stepping stones and cross-category mutations.No Stepping Stones ignores past archive solutions, whereas Same Cell Mutations mutate within each archive cell independently.
  • Transfer: Transferred prompts retain 50% of original-target ASR on average, reach up to 66% ASR on GPT-4o, and perform better from safer to less safe models.Transfer rates vary substantially by model pairing, and the evaluation uses archives generated for each original target.
  • Diversity: A parent-child similarity filter maintains archive linguistic diversity while balancing ASR, discarding approximately 24% of mutated prompts at τ = 0.6.Without filtering, the method reuses highly effective prompts across multiple cells; diversity is measured with self-BLEU, BERTScore, ROGUE-L, and compression ratio.

5 Enhancing Robustness with Synthetic Data

Rainbow Teaming generates synthetic adversarial data that can substantially improve LLM robustness through supervised fine-tuning. This improvement transfers to held-out and out-of-distribution attacks, while general capabilities remain largely preserved.

  • Motivation: Rainbow Teaming offers a low-cost alternative to human-annotated instruction-tuning data by generating diverse synthetic examples targeting model vulnerabilities.The approach is used to improve safety against adversarial prompts.
  • Data and training: 15 archives targeting Llama 2-chat 7B produced 1500 adversarial prompts, split into 1200 training and 300 held-out test prompts.Safe refusal prompts were generated for training before supervised fine-tuning.
  • Safety results: 92% / 95% to 0.3% / 0.7%: fine-tuning reduced attack success rate on held-out prompts as measured by GPT-4 and Llama Guard.The reported reductions compare the model before and after fine-tuning on Rainbow Teaming-generated data.
  • Safety results: 14% to 0%: PAIR attack success rate on JailbreakBench behaviours fell after fine-tuning, indicating improved safety against out-of-distribution attacks.The post-fine-tuning value was measured by Llama Guard.
  • Capabilities: Safety scores increased by 1.5%, while helpfulness dropped by 0.5% after fine-tuning on adversarial prompts.The authors attribute the helpfulness decrease to training exclusively on adversarial prompts.
  • Iterative robustness: 39%: after fine-tuning, the final attack success rate under renewed Rainbow Teaming fell from 92%.The authors suggest alternating synthetic-data collection and adversarial fine-tuning as a route to further robustness.
  • Evaluation caveat: The comparison uses Llama 2-chat scores, whereas Touvron et al. reported base-model scores on the referenced benchmarks.This difference qualifies direct comparisons with the cited prior results.

6 Rainbow Teaming for Other Applications

Rainbow Teaming extends beyond safety to question answering and cybersecurity by adapting archive features, mutation operators, and preference judgments to each domain. Across both applications, it produces diverse or effective adversarial prompt archives.

  • Question answering: In question answering, Rainbow Teaming generates adversarial trivia questions that target incorrect answers from a target model.The question-answering archive uses topic, interrogative word, and question length as diversity features.
  • Question answering: Rainbow Teaming’s question-answering archive uses a 3D feature space covering Topic, Interrogative Word, and Question Length.A Judge compares the target model with a stronger Oracle to determine question fitness.
  • Question-answering results: Rainbow Teaming achieves higher fitness, coverage, and diversity than generating candidate questions from scratch.Reusing previously discovered questions as stepping stones prevents regions of the archive from remaining uncovered.
  • Cybersecurity: In cybersecurity, the method searches for prompts eliciting insecure code or assistance in orchestrating cyberattacks.Its archive combines 10 MITRE attack-tactic categories with 10 equal prompt-length bins.
  • Cybersecurity results: Cybersecurity archives contain 10 × 10 cells fully identified as malicious by CyberSecEval for every tested target model.The archive dimensions correspond to MITRE categories and prompt-length bins.
  • Cybersecurity results: Human evaluation measured attack success rates of 0.94 and 0.92 for Llama 2-chat 7B and CodeLlama 7B Instruct, and 0.8 for both 70B targets.These human-expert rates were lower than the CyberSecEval malicious classifications.

7 Related Work

Rainbow Teaming addresses limitations in prior adversarial-prompt methods by searching for diverse attacks across risk categories and styles. It combines quality-diversity search with LLM mutation and preference-based archive updates.

  • Adversarial attacks on LLMs: Existing automatic red teaming can trade off attack diversity against success rate, collapse into modes, or optimise only a single solution.PAIR and TAP are black-box methods designed around single tasks rather than diverse risk categories and attack styles.
  • Rainbow Teaming: Rainbow Teaming casts adversarial-prompt generation as quality-diversity search to optimise attack quality and diversity jointly.This design aims to cover a broader attack space than single-solution or single-task approaches.
  • Comparison with QDAIF: Unlike QDAIF, Rainbow Teaming targets diverse adversarial prompts, uses a preference-based judge rather than token log probability, and applies parent-child similarity filtering.The similarity filter is intended to preserve linguistic diversity in the archive.

8 Conclusion

The paper presents Rainbow Teaming as a general method for automatically generating diverse adversarial prompts across safety, question answering, and cybersecurity. Its synthetic data improves resistance to further attacks while preserving general performance, supporting open-ended self-improvement with minimal human intervention.

  • Conclusion: Rainbow Teaming uses quality-diversity search to explore potential attacks and produce diverse archives that expose LLM vulnerabilities.Experiments span Llama 2-chat and Llama 3-Instruct across safety, question answering, and cybersecurity.
  • Conclusion: Synthetic data generated by Rainbow Teaming can fine-tune LLMs for greater adversarial robustness without compromising general performance.The authors frame this as continuous, open-ended self-improvement with minimal human intervention.
  • Future work: The authors identify extending Rainbow Teaming to vision and multimodal AI systems as future work.They also discuss incorporating the method into LLM fine-tuning stages.

A Limitations and Broader Impact

Rainbow Teaming combines open-ended quality-diversity search with an archive of adversarial prompts, but its current implementation has fixed features, grid-size constraints, and substantial computational demands. The authors frame it primarily as a tool for identifying and addressing robustness issues.

  • Limitations: The archive uses fixed, pre-defined features and categories, limiting automatic discovery of alternative diversity dimensions.Future work could discover features and categories automatically.
  • Limitations: The number of generated prompts is constrained by the MAP-Elites grid size, although additional categories or multiple values per cell can expand output.The current setting still generates hundreds of adversarial prompts from one run.
  • Limitations: Rainbow Teaming requires extensive computational resources compared with simpler adversarial attack methods.This is identified as a limitation of the approach.
  • Limitations: The method’s undirected, open-ended search is less likely to produce prompts targeting one specific behaviour.The authors also note that this makes the method less likely to be used for malicious purposes.
  • Broader Impact: Rainbow Teaming’s primary value is identifying and addressing LLM robustness issues for responsible development and deployment.The authors view its prompts as a complement to crowd-sourced data.

C.2 Adversarial Training

Rainbow Teaming generates diverse adversarial prompts through directed mutation and quality-diversity search, while its synthetic data can support adversarial training. Across Llama 2 model sizes, the discovered prompts achieve at least 90% attack success rate under two evaluators, and system prompts materially affect robustness.

  • Adversarial Prompt Generation: Directed mutation can produce diverse adversarial prompts from a single common ancestor.Feature descriptors guide alternative mutation paths, including “Criminal Planning” and “Role Play” categories.
  • Adversarial Prompt Generation: 90% or higher ASR is achieved across Llama 2-chat models with 7B, 13B, and 70B parameters using GPT-4 and Llama Guard.The evaluation covers 2000 iterations and reports this pattern across all model sizes.
  • System Prompts: A safety-emphasising system prompt lowers attack success rate to 51% / 74% under GPT-4 and Llama Guard, respectively.The same prompt can make the model overly conservative and occasionally refuse answers.
  • System Prompts: Robustness is highly dependent on the system prompt, even for the safety-aligned Llama 2-chat 7B model.The evaluated system prompts differ in their emphasis on safety and helpfulness.

E.3 Human Evaluation

The human evaluation compares attack success rates across automated and human evaluators on 100 prompt-response pairs. Its reported agreement pattern supports using GPT-4 and Llama Guard as proxies for human evaluation.

  • Human Evaluation: 100 prompts are evaluated by GPT-4, Llama Guard, and human annotators, with inter-evaluator agreement also reported.The human annotations were produced by four authors using the same harm taxonomy as Llama Guard.
  • Human Evaluation: Human-AI agreement matches inter-human agreement, making GPT-4 and Llama Guard good proxies for human evaluations.Results are aggregated over four human annotators.

E.4 Preference Model Ablation

The preference-model ablation shows that pairwise judging supports continued open-ended archive refinement, whereas score-only optimization can exploit its evaluator and stop improving. The section also describes domain-specific evaluations and archive comparisons across safety, question answering, and cybersecurity.

  • Preference Model Ablation: The default preference model is a pairwise Judge LLM that compares candidate and archived responses for harmfulness, using position swapping and few-shot prompting.This variant is called Comparison-based (Judge).
  • Preference Model Ablation: The score-based baseline attains higher Llama Guard ASR but lower GPT-4 ASR, indicating reward hacking against Llama Guard scores.The score-based prompts are also qualitatively lower quality.
  • Preference Model Ablation: The No Judge baseline stops updating after saturating Llama Guard, while the Judge-based variant continues refining archive prompts.Ongoing updates persist even after the archive is filled with successful prompts.
  • Experiments: Rainbow Teaming evaluations cover multiple target models, baselines, fine-tuning before-and-after conditions, archive snapshots, and question-answering projections.The cited figures include Figures 10–14 and comparisons against No Stepping Stones and Same Cell Mutations.
  • Question Answering: In question answering, an Oracle answer is compared with the Target answer; factual questions where the Oracle is better receive fitness 1, while empty cells may retain fitness 0.1 stepping stones.The Oracle is Llama 2-chat 70B and the Target is Llama 2-chat 7B.
  • Cybersecurity: Cybersecurity prompts are assessed through a binary maliciousness Judge using Llama 2-chat 70B for preliminary analysis and CodeLlama-34B Instruct for the final decision.The two-model process classifies generated outputs as benign or malicious.

G Inference Cost Analysis

Rainbow Teaming has substantial computational costs, but the authors argue that its effective and diverse prompts make it competitive with crowd-sourced data. Costs can be amortised through checkpoint reuse and potentially reduced with more capable judging models.

  • Inference costs: Most Rainbow Teaming loops use 2 Mutator, 1 Target, and 4 Judge inference calls, with each call processing a prompt batch.The usual batch size is 16, except 64 for JailbreakBench experiments.
  • Infrastructure: Experiments ran on 128–256 A100 GPUs, and each run typically took around two days.A distributed client-server setup often accelerated runs through parallelised LLM inference.
  • Cost-benefit trade-off: Despite substantial computational cost, the resulting prompts are presented as a competitive alternative to crowd-sourced data because of their effectiveness and diversity.
  • Cost amortisation: Earlier Rainbow Teaming checkpoints can potentially expand the final dataset size by an order of magnitude while amortising costs.
  • Future cost reduction: An accurate judge LLM operating in a single inference call could significantly reduce the method’s judging cost.
Loading 2402.16822v3…