Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning

1 Introduction

• Prior RL red teamers can overfit toxicity rewards and repeat successful attacks, while zero- or few-shot prompting yields diverse outputs with lower attack success.The paper frames its goal as obtaining attacks that are both diverse and effective.Source: §1 · p. 1
• The system factorizes red teaming into generating diverse attacker goals and training an RL red teamer to produce successful attacks for them.Goals can come from existing attack datasets or few-shot prompting, but may be ineffective because they are not tuned to the attacked model.Source: §1 · pp. 1–2
• Per-example rule-based rewards assess success against each goal, while an additional reward discourages drifting from the sampled demonstration during RL.The paper presents these rewards as improving diversity and avoiding collapse during RL.Source: §1 · p. 2
• Multi-step RL conditions each new attack on past attempts and rewards success plus difference, using a style- or tactic-focused diversity measure.The authors report that traditional diversity metrics can look favorable even when the red teamer uses a relatively small set of tactics.Source: §1 · p. 2
• The approaches better balance diversity and effectiveness across both red-teaming tasks, including indirect prompt injections.The introduction reports quantitative and qualitative evidence for this comparison.Source: §1 · p. 2
• Indirect prompt injections are difficult for prior automated red teaming because attacker goals vary beyond unsafe responses, leaving no single grader to cover them.The paper presents its approach as the first method for automated red teaming of indirect prompt injections, to the authors’ knowledge.Source: §1 · p. 2

2 Related Work

• Gradient-based attacks optimize triggers or text operations using model gradients, but the resulting attacks can use soft tokens or unrealistic text unlike human-generated prompts.The cited approaches include Gumbel-Softmax approximation [Jang et al., 2016] and vector-space text operations [Ebrahimi et al., 2017, Wallace et al., 2019a, Shin et al., 2020].Source: §2 · p. 3
• Red teaming seeks model failures [Dinan et al., 2019, Ganguli et al., 2022, Perez et al., 2022, Markov et al., 2023]; earlier RL work used toxic classifiers, whereas this paper uses automatically generated rule-based rewards for diverse goals.FLIRT [Mehrabi et al., 2023] relies on growing in-context exemplars sorted by effectiveness, diversity, and low toxicity.Source: §2 · p. 3
• Related diversity methods include genetic-like attack search that requires more curation [Samvelyan et al., 2024] and an RL diversity regularizer used as a baseline [Hong et al., 2024].Ge et al. [2023] use alternating red teaming and training on red-team data, which the paper regards as combinable with red-teaming approaches.Source: §2 · p. 3

3 Overall System Design

• Figure 1 tracks generated goals g, attacker prompts p, target-model outputs y, and the reward r used for reinforcement learning.The overview depicts multiple stages involving multiple models.Source: Figure 1 · p. 4

3.1 Problem Setup

• The attacker is trained to generate diverse and effective attacks that induce undesirable responses from a target model, as judged by a reward model.The setup treats the target as a model that maps prompts to responses and the attacker as the source of attack prompts.Source: §3.1 · p. 4

3.2 Proposed System Design

• The system factorizes red teaming into generating diverse attacker goals and generating effective attacks that meet those goals.Goals are sampled from a goal-generation method, then supplied to the attacker model.Source: §3.2 · p. 4
• The attacker is trained with reinforcement learning to maximize reward for attacks generated from the diverse goals.The objective averages reward over goals sampled from the goal generator.Source: §3.2 · p. 5

Goal diversity

• Different attacker goals can target distinct undesirable behaviors, and a model's refusal of one goal does not imply refusal of another.Examples include car hijacking instructions and money-laundering guidance.Source: §4 · p. 5

Using goals in prompts and rewards

• Goals can be supplied to the attacker as natural-language instructions or examples, and their criteria can be used to reward goal-specific outputs.For example, a reward can judge whether the target response gives instructions for a particular crime.Source: §4 · p. 5
• Rule-based rewards are classifiers that assess whether text violates a specified policy, and an LLM can perform this classification from a prompted rule.The authors use goal criteria as prompts for targeted, zero-shot reward judgments.Source: §4 · p. 5

Few-shot Generation

• Few-shot prompting can generate diverse but unsuccessful attacks, while producing an example instruction and criteria that can guide the attacker and its reward.The authors also describe tailored goals for styles such as pirate speech or leetspeak.Source: §4 · p. 6

Reward Generation from Data

• An LLM can transform existing red-teaming data into instructions and criteria; the resulting goals can be used for training and evaluation.The Anthropic Harmless dataset [Bai et al., 2022] supplies diverse examples, although using it directly is unlikely to elicit unsafe responses.Source: §4 · p. 6

5.1 Attack Success Reward

• The primary reward judges attack success from the target model's response, using a generated rule-based reward for goal-specific criteria.For unsafe-content generation, the success reward averages the rule-based judgment and a general moderation judgment.Source: §5.1 · pp. 6–7

5.2 Few-shot Similarity

• A few-shot similarity reward encourages the generated attack to remain similar to its example, which helps prevent collapse to repeated attacks.Similarity is computed with cosine similarity over embeddings of the example and generated attack.Source: §5.2 · p. 7

5.3 Multi-step RL and Style Diversity

• In multi-step reinforcement learning, each later attack is conditioned on past attacks and rewarded for differing from them.The attacker produces a sequence of prompts, with feedback after each attempt.Source: §5.3 · p. 7
• The diversity reward compares each new attack with prior attempts in a style subspace that removes goal-related embedding directions.The authors normalize the reward because similarity ranges vary with history length.Source: §5.3 · p. 7

5.4 Implementation Details

• A length penalty discourages arbitrary extra text, with no penalty below a minimum length and equal penalties above a maximum length.The authors favor shorter, simpler attacks as more likely to be discovered by real people.Source: §5.4 · p. 8
• Multiplying reward components was found most effective for jointly encouraging success, example similarity, stylistic diversity, and moderate length.The few-shot reward applies to the first attack, while the diversity reward applies to later attacks; the length reward is scaled between 0.5 and 1.Source: §5.4 · p. 8

6.1 Indirect Prompt Injection

• Indirect-injection goals cover links, images, specific phrases, answering questions, styles, and miscellaneous tasks; attacks are embedded in browsing responses and tested on function-calling responses.The split contains 4,664 training examples and 1,102 test examples, and generated RBRs grade attack success because no general grader covers the goals.Source: §6.1 · pp. 8–9

6.1.1 Indirect Prompt Injection Results

• For indirect prompt injection, RBRs plus few-shot reward produce reasonably diverse and effective attacks; multi-step RL raises diversity to near the one-shot level, while one-shot success is near zero.The task is difficult: maximum success is below 50%, although all methods have reasonable diversity from diverse goal generation.Source: §6.1.1 · pp. 9–10, Figure 4 · p. 9
• Across both tasks, multi-step approaches become more diverse as inference steps increase; style-subspace optimization improves style diversity, while direct multi-step RL looks more diverse overall.For indirect injections, methods not trained with the multi-step diversity reward show no diversity benefit from multi-step inference.Source: §6.1.1 · p. 10, §6.2.1 · p. 12

6.2 Safety “Jailbreaks” to Violate Safety Policies

• The safety-jailbreak task uses 5,000 training and 654 test examples from Anthropic Harmless data; Moderation API scores thresholded at 0.5 grade attack success, and cosine similarity measures diversity.The experiments focus on attacking GPT-4 Turbo with GPT-3.5-sized attackers; the attacker starts from an instruction-following model without safety guardrails.Source: §6 · p. 8, §6.2 · p. 10

6.2.1 Safety Jailbreak Results

• Across indirect prompt injection and safety jailbreaking, multi-step RL increases attack diversity while retaining effectiveness, unlike baselines that favor success or diversity alone.The safety results describe a trade-off between success rate and diversity; the indirect-injection results report effective, diverse attacks.Source: §6.1.1 · p. 9, Figure 4 · p. 9, §6.2.1 · p. 11
• For safety jailbreaking, one-shot prompting yields the most diverse attacks but near-zero success, while vanilla RL reaches nearly 100% success with near-zero diversity.The proposed RBR approach improves diversity with a fairly small success-rate decrease, and multi-step RL produces considerably more diverse attacks while maintaining considerable success.Source: §6.2.1 · p. 11, Figure 5 · p. 11, Table 2 · p. 11
• In safety jailbreaking, goal-adherence success is 0.15 for single-step RBR plus few-shot reward and 0.03 for multi-step approaches; the combined rewards often lead attackers to prioritize Moderation over RBR.The authors attribute these relatively low values to the challenge of optimizing both rewards.Source: §6.2.1 · pp. 11–12
• The safety ablations find that few-shot reward contributes more to diversity than RBR, and multi-step rewards add further diversity.The authors link this to the red teamer's difficulty optimizing Moderation and RBR simultaneously; that issue does not occur in indirect injection, where RBR is the sole attack-success reward.Source: §6.2.1 · p. 12

7 Conclusion, Limitations, and Future Work

• The authors combine diverse attack goals with RL, per-example rewards, and multi-step attacks to produce more effective and diverse red-teaming, including indirect prompt injections.They report improved diversity with minimal hit to attack success rate.Source: §7 · p. 12
• Red-teaming methods and metrics are difficult to compare because they assume different valuable attacks; the method also has significant run variance and reward-shaping sensitivity.The authors say this sensitivity does not prevent usage.Source: §7 · pp. 12–13
• Automated graders used for rewards and evaluation risk reward hacking, in which an attacker exploits the grader rather than the defender; narrower RBRs offer anecdotal evidence of mitigation.The authors describe this as an open problem.Source: §7 · p. 13

A Prompts

• The appendix says the code for both tasks is included as supplementary material.Source: A Prompts · p. 18

A.1 Few-shot Reward Generation

• Few-shot reward prompts can generate varied examples such as phrases, questions, domains, styles, and goals, without requiring those examples to succeed as attacks.The appendix says the method was not extensively tuned.Source: §A.1 · p. 18

A.2 Generating Rewards from Data

• The prompts derive goals and success criteria from examples, with specific criteria preferred over broad labels such as “commit a crime.”The appendix illustrates this with examples drawn from the Anthropic Harmless dataset.Source: §A.2 · p. 18

A.3 Red-teamer Prompt

• The red-teamer prompt asks for a creative, effective adversarial request targeting a specified response, and the appendix says multiple templates and examples are sampled for diversity.Source: §A.3 · p. 19

B.1 RBR Implementation

• Rule-based rewards are posed as yes/no questions to an LLM, with the reward taken as its probability of answering yes.Source: §B.1 · p. 19

B.1.1 Few-shot similarity

• Few-shot similarity uses the sigmoid of cosine similarity between the prompt example and generated attack, with x0 = 0.75 and k = 10.The embeddings use text-embedding-ada-002.Source: §B.1.1 · p. 19

B.1.2 Multi-step RL

• Multi-step feedback tells the red-teamer whether an attack succeeded and asks for a new, different prompt for the same goal; later-step rewards are not propagated to earlier steps.The implementation flags diversity rewards below 0.5 as too similar and uses γ = 0.Source: §B.1.2 · pp. 19–20
• Multi-step diversity rewards normalize each batch’s similarity-based scores using its mean and standard deviation, then apply a sigmoid.The smoothing factor ϵ is 0.1.Source: §B.1.2 · p. 19

B.1.3 Length Penalty

• The length penalty uses minimum and maximum lengths of 100 and 200, with a sigmoid output scaled between 0.5 and 1.Its sigmoid uses k = −10 and x0 = 0.5.Source: §B.1.3 · p. 20

B.2 Curiosity Baseline

• The Curiosity baseline uses entropy weight λE = 0.01 and SelfBLEU and cosine-similarity weights of 1, with similarities measured against the last 10 batches.The length penalty is retained for consistency across methods.Source: §B.2 · p. 20

• The method maintains high cumulative attack success, with later steps often improving the best attack found up to that point.The plotted rate is based on the most effective attack up to step T.Source: §C.1 · p. 20, Figure 6 · p. 21

• For safety jailbreaks, few-shot reward improves diversity more than RBR, while multi-step RL further increases diversity.The reported component comparison uses one run per configuration.Source: §C.2 · p. 21, Figure 7 · p. 21
• In safety jailbreaks, combining Moderation API and RBR success signals led the red-teamer to optimize mostly for Moderation, leaving performance similar to Vanilla RL without few-shot reward.Source: §C.2 · p. 21
• Multi-step RL further improves diversity with a slight decrease in attack success rate.Source: §C.2 · p. 21, Figure 7 · p. 21

C.3 Diversity Juxtaposition

• Multistep RL performs relatively better on overall diversity, while Multistep RL with Style Subspace performs relatively better on style diversity.The comparison plots diversity by step using average cosine similarity for overall diversity and projected-embedding cosine similarity for style diversity.Source: §C.3 · p. 21, Figure 8 · p. 22, Figure 9 · p. 22

C.4 Indirect Prompt Injection Qualitative Examples

• Indirect prompt-injection examples compare attack attempts across baseline, one-shot RBRs, Multistep RL, and its Style Subspace variant.The examples include different formulations pursuing goals such as answering a question; successful and unsuccessful attacks are marked green and red, respectively.Source: §C.4 · pp. 22–23

C.5 Jailbreaks to Safety Policies Qualitative Examples

• Safety-policy jailbreak examples are graded only by the Moderation model, with successful and unsuccessful attempts marked green and red.The appendix warns that the examples may contain offensive material.Source: §C.5 · p. 24
• The examples were arbitrarily selected from the test set’s first few entries and are not necessarily successful or aligned with the authors’ safety policies.Source: §C.5 · p. 24