Evolving Diverse Red-team Language Models in Multi-round Multi-agent Games

Existing approaches

• Human red teams draw on diverse experts but incur high costs, while automated approaches are more efficient and scalable.The introduction says manual annotation can also cause psychological trauma; automated natural-language attacks are distinguished from less interpretable token-search and gradient methods.Source: §1 · pp. 1–2

Research gap

• Single-round attacks against static blue-team targets miss the multi-round interaction patterns of real human-LLM use and limit contextual depth and attack diversity.The authors also identify limited robustness and generalization, and reliance on expert strategies and human-feedback reward models.Source: §1 · p. 2
• Static objectives can produce mode collapse, overfitting, and reward hacking, including stereotyped or non-natural-language attacks that may not reflect real threats.The introduction links these risks to Goodhart Law and says they can leave latent vulnerabilities undetected.Source: §1 · p. 3

Proposed approach

• The paper models red teaming as a multi-agent problem in which opposing teams adapt to one another's strategies over multiple dialogue rounds.It terms this game-theoretic formulation Red Teaming Game (RTG).Source: §1 · p. 3
• Gamified Red-teaming Solver (GRTS) combines diverse policies and meta-game solving to seek approximate Nash equilibrium for both teams.The authors report a spinning-top geometry for RTG and introduce attack-diversity measures to mitigate mode collapse.Source: §1 · pp. 3–4

Human-led red teaming

• Human-led vulnerability discovery relies on manual effort and creativity, with limits on vulnerability diversity and depth from single-round attacks.The cited approaches include manually crafted or supervised test cases, annotators from varied backgrounds, and templates targeting failed responses.Source: §2 · p. 4

Automated red teaming

• Learning-based approaches include generated test cases, adversarial prompt search, classifiers, reinforcement learning, and self-alignment, but retain distinct limitations.The passage notes manual-data dependence, potentially unfluent attacks, binary toxicity classification, and methods that optimize only the blue or red team.Source: §2 · p. 4

Positioning

• The paper contrasts its dynamic multi-round games with prior approaches that optimize only blue-team models, proposing bilateral optimization for more adversarial red teams and safer blue teams.The accompanying game framework separates token-level generation decisions from sentence-level alternating dialogue decisions.Source: §2 · p. 5, Figure 2 · p. 5

3 Problem Formulation

• RTG is a finite-horizon, two-player game in which red teams try to induce toxic blue-team outputs across dialogue rounds, while blue teams defend helpfulness and harmlessness.The model is organized as token-level generation nested within sentence-level dialogue.Source: §3 · p. 5

3.1 Markov Decision Process for Token Generation

• At the token level, MDPTG models sentence generation as sequential decisions over token histories, with policies selecting distributions over next tokens.The state space consists of token combinations, and the token-level objective maximizes cumulative sentence reward.Source: §3.1 · pp. 5–6
• A sentence’s payoff sums token rewards, while the discount factor γ reduces the influence of tokens at more distant positions.The reward model scores toxicity; greater toxicity in a blue-team sentence increases the red-team reward and decreases the blue-team reward.Source: §3.1 · p. 6

3.2 Extensive-form Game in Dialogue

• At the dialogue level, ETGD alternates one sentence from each team per round, with each sentence extending the dialogue history until a terminal state.Sentence-level policies map dialogue histories to distributions over available sentences.Source: §3.2 · p. 7
• ETGD is treated as a zero-sum game whose terminal utilities are associated with the token-generation rewards, and its solution seeks approximate Nash equilibrium policies.An ε-approximate equilibrium allows no unilateral deviation yielding more than additive ε improvement.Source: §3.2 · p. 7
• Solving the full extensive-form game with CFR would require searching the vocabulary at every token-generation step, so the paper instead reformulates RTG as a lower-complexity normal-form meta-game.The meta-game approach is the stated response to the computational cost of directly solving ETGD.Source: §3.2 · p. 7
• Figure 3’s process initializes team populations, trains best responses from multi-round interactions, adds policies to the populations, and solves the resulting meta-game for the next strategy distribution.The solved distribution initializes the following GRTS iteration.Source: Figure 3 · p. 8

4.1 Solving Meta Game of Red-teaming LLMs

• GRTS builds an n × n normal-form meta-game from red- and blue-team policy populations, then iteratively adds best responses and computes meta-strategies from the payoff matrix.The best-response operation uses PPO, and each team responds to the opponent’s population under its current meta-strategy.Source: §4.1 · pp. 8–9
• GRTS’s output is an ε-approximate Nash equilibrium when neither team can gain more than additive ε through unilateral deviation.The paper reports that GRTS iterates toward an approximate equilibrium for RTG.Source: §4.1 · p. 9

4.2 Diversity Measure of Semantic Space

• GRTS represents each policy using semantic-space features derived from its outputs in historical dialogues, then uses those features to measure diversity among red-team policies.The paper presents language-model dialogue representation within a game-theoretic strategy framework as a novel contribution.Source: §4.2 · pp. 9–10
• The diversity-adjusted best response is intended to improve payoff while enhancing the existing population, and the update adds the resulting policy to that population.The update uses a tunable constant τ_t; the paper describes τ_t → 0 as part of the convergence argument.Source: §4.2 · p. 10
• Under the stated concavity and update conditions, GRTS has a GWFP-analogous convergence property to approximate Nash equilibrium in two-player zero-sum or potential games.The paper also states that the diversity measure ensures red-team attack diversity and synchronized optimization of both teams within RTG.Source: §4.2 · p. 10

5.1.1 Training setup

• GRTS trains red and blue populations online with PPO in three-round games, using toxicity scores and a red-team diversity reward in a zero-sum setup.ASR counts toxicity scores above zero, and diversity uses average pairwise n-gram similarity. For evaluation against specific LLMs, the authors select two aggressive red-team policies because the population equilibrium is a defensive solution concept.Source: §5.1.1 · pp. 11–12, §5.1.2 · p. 12, §5.4 · p. 18

5.1.3 The overall game-solving process

• At equilibrium, ASR is approximately 10%; the final red team raises ASR against an untrained blue team from 40% to 70%, while trained blue makes an untrained red team nearly ineffective.In round one, blue-team ASR against training prompts falls from 52% to 1%. Across rounds, high ASR tends to rise and low ASR to fall as dialogue context accumulates.Source: Figure 5 · p. 12, §5.1.3 · p. 14
• Exploitability fell from 6.2 to approximately 0.8 after 15 GRTS iterations, indicating convergence toward an approximate Nash equilibrium.Exploitability measures the gap between the populations’ joint-strategy utility and the utility of approximate-equilibrium strategies.Source: Figure 4 · p. 11, §5.1.3 · p. 13

5.1.4 The geometrical structure of RTG

• Payoff variance and standard deviation first rise and then fall as strategy populations expand and diversify, tracing a spinning-top structure whose narrowing tip corresponds to approximate equilibrium.The authors interpret the broad middle region as non-transitive and argue that diverse strategy populations help traverse it.Source: Figure 4 · p. 11, §5.1.4 · p. 14

5.1.5 The Best Response Iteration for Red Team

• In red-team best-response training, second- and third-round attack prompts surpass the training prompt dataset’s ASR, while first-round ASR and toxicity remain constant.The first round uses training prompts and a fixed blue model; the authors characterize later-round attack performance as exceeding that of human red-teamers.Source: Figure 6 · p. 13, §5.1.5 · p. 15

5.1.6 The Best Response Iteration for Blue Team

• Blue-team best-response training against a dynamically selected red-team population lowers ASR and shifts toxicity toward safer outputs.The resulting blue model shows minimal vulnerability to manually crafted first-round prompts and substantial resilience against the red team.Source: Figure 7 · p. 13, §5.1.6 · pp. 15–16

5.2 Comparison between multi-agent setting with single-agent baselines.

• Multi-agent training maintains greater attack diversity and ASR than single-agent baselines, while its blue-team models improve defense and preserve output diversity.The authors report that Table 3 further supports multi-agent superiority for both teams, though the supplied table caption gives no numerical values.Source: §5.2 · p. 17

5.3 Ablation Study on the Efficacy of Multi-Round Dialogues in the Multi-Agent Setting

• Across 1–5 dialogue rounds at comparable computational effort, multi-round training improves the helpfulness–harmlessness trade-off and reduces alignment tax, motivating the three-round main setting.The authors attribute the reduction to richer, more diverse training trajectories; some loss of instruction-following ability remains unavoidable.Source: Figure 10 · p. 16, §5.3 · p. 17

5.4.1 Attacking Open-source Models

• GRTS significantly outperforms baseline, SFT, and manually crafted prompts when attacking open-source models, including models of at least 7B parameters.The red teams were trained on a fixed or slightly varied 3B blue team, making these evaluations out-of-distribution relative to training.Source: §5.4.1 · pp. 18–19

5.4.2 Policy Level Diversity

• Policies show topic-level diversity and autonomously produce varied attack tactics absent from the fine-tuning and training-prompt datasets.Observed tactics include primising attacks and wild tokens, previously explored with human priors or specialized designs.Source: §5.4.2 · p. 20, Figure 11 · p. 20

5.4.3 Sentence Level Diversity

• Across rounds, red teams tend to persist with attack forms associated with higher ASR but explore alternatives when tactics meet resistance.The analysis covers varied forms and topics, including direct threats, goal hijacking, role-play instructions, reverse exposure, profanity, violence advice, and crime.Source: §5.4.3 · pp. 20–21, Figure 12 · p. 21

6 Conclusions and future work

• RTG models multi-round red teaming as a multi-agent game, while GRTS uses diversity metrics to address mode collapse and guarantees approximate Nash equilibrium convergence.The conclusion identifies increasing attack diversity as a key challenge in red teaming.Source: §6 · p. 21

Appendix; A Preliminaries of Game

• The appendix covers game theory and solvers; its preliminaries describe PSRO as expanding policy populations and identify it as GRTS's basis.The appendix also includes proofs, implementation and evaluation details, and additional empirical results.Source: A Preliminaries of Game · p. 23, Appendix · p. 23

B Additional Empirical Results

• The additional empirical discussion examines how reward modeling, exploration and exploitation, and adversarial training dynamics complicate red teaming.It identifies toxicity-model bias and errors as potential sources of reward hacking and mode collapse.Source: B Additional Empirical Results · p. 23, §B.1 · p. 23

B.1 Why fine-tune a LLM in a vanilla fashion as a red-teamer for safety alignment is a bad idea

• Vanilla red-team fine-tuning faces an exploration–exploitation trade-off, with excessive exploration risking out-of-distribution behavior and excessive exploitation narrowing strategies.The discussion also stresses the need for reward models that reflect human safety preferences.Source: §B.1 · p. 24

B.2 The Paradox of Reward: Challenges and Necessity

• The paper discusses the “Reward is Enough” hypothesis but notes that toxicity-model biases and errors shape the red–blue game and agent behavior.In this setting, the toxicity model supplies the signals received by both teams.Source: §B.2 · p. 24

B.2.1 Human Preferences are not Well-Defined: Ambiguities in Standards of Safety and Toxicity Pose Challenges

• Different toxicity detectors assigned entirely disparate outcomes to similar conversation histories, illustrating ambiguity in safety standards.The comparison used a SafeRLHF-trained toxicity model and Perspective API on conversations involving political news.Source: §B.2.1 · p. 25
• Human preferences may be inconsistent and non-ordinal, so modeling them with a single-scale method such as Bradley–Terry may not make sense.The discussion presents this as an inherent issue even when reward modeling fits collected preference data.Source: §B.2.1 · pp. 24–25

B.2.2 Challenges in Modeling Rewards Through Reverse RL.

• The study trained toxicity models on 30,000-entry and 300,000-entry subsets of the same Safe-RLHF dataset to examine reward-model overfitting.Under the same experimental settings, the smaller-dataset model exhibited some overfitting.Source: §B.2.2 · p. 25
• The adversarial training used the same GRTS algorithm, with the blue reward defined as the negative of toxicity.The comparison examined how the differing toxicity models guided the red–blue game.Source: §B.2.2 · pp. 25–26
• The over-fitted toxicity model produced more unstable adversarial training and more severe mode collapse than normal reward modeling.Normal reward modeling showed slower, more moderate oscillation between the red and blue teams.Source: §B.2.2 · p. 26

B.2.3 Reward is Better Compared to Heuristic Methods.

• Reverse-RL reward modeling is described as better than some human-crafted heuristics, though the toxicity model rewarded a fixed refusal pattern.The pattern began responses with “As an AI Language Model” and then refused to answer.Source: §B.2.3 · p. 26

B.3 Difficulty in balancing between exploration and exploitation

• Balancing exploration and exploitation is difficult in red teaming because the action and state space is vast.The discussion identifies this as a challenge for red-teaming LLMs.Source: §B.3 · p. 26

B.3.1 Excessive exploration leads to Out-of-Distribution issues: It’s a disaster for rewards

• Reward-model distribution shift compounds red-teaming challenges because the model does not continually learn preferences from the policy's shifted output distribution.The passage notes that this can make reward modeling increasingly inaccurate.Source: §B.3.1 · p. 26
• The discussion says some blue-team exploration can diversify agent behavior and proxy for broader human-red-teamer coverage, with pre-train loss added alongside PPO policy loss.The pre-train loss is presented as a way to regulate the blue team's instruction-following behavior.Source: §B.3.1 · p. 26

B.3.2 Excessive exploitation leads to Mode collapse when RLHF destroys the diversity

• Excessive exploitation can collapse a model onto narrow behaviors; models that perform well against fixed opponents may be defeated by others.The discussion describes synthetic-feedback reinforcement as a loop that can reduce exposure to diverse scenarios.Source: §B.3.2 · p. 27
• Suggested responses to mode collapse include varying synthetic feedback through population-based methods and rewarding novelty or diversity.The latter is described as intrinsic motivation to explore new behaviors.Source: §B.3.2 · p. 27

B.4 Asymmetry in the Adversarial Setting

• GRTS uses population-based training and semantic diversity for varied red-team attacks, while diverse blue-team responses are harder to foster because of refusal behavior.The discussion frames this design against the challenge of balancing offensive and defensive progress.Source: §B.4 · p. 27

C Proof of Proposition

• The convergence argument assumes a concave DMS and a deterministic parameter α_t = o(1/log t).The proof also invokes bounded martingale-difference perturbations and a smoothing parameter τ_t tending to zero.Source: C Proof of Proposition · pp. 27–28
• GRTS’s policy sequence converges to a Nash equilibrium in two-player zero-sum or potential games.The proof describes this as analogous to generalized weakened fictitious play.Source: C Proof of Proposition · p. 27
• With progressively decreasing smoothing parameters, GRTS converges almost surely toward generalized weakened fictitious play.The cited account relates the resulting Nash convergence to Leslie’s work and classical game theory.Source: C Proof of Proposition · p. 28

D.1 Preparation: Fine-tuning a Toxicity model and initializing the Red Team model

• The initial red-team model is trained on multi-round adversarial dialogue constructed from BAD, while preference data trains a toxicity model with a 3B backbone and linear score head.The toxicity model and initial red-team model are prepared before the main training runs.Source: §D.1 · p. 28

D.2 Training Algorithms Details

• Training uses Beaver and Open-Spiel frameworks, DeepSpeed ZeRO-3 mixed-precision parallelism, and LoRA with hidden dimension 128 on all linear layers.The reported experiments used an NVIDIA A100 cluster with 80GB GPU memory; training logs reflect adversarial-game dynamics.Source: §D.2.1 · p. 28, §D.2 · p. 28

D.2.1 GRTS Details

• GRTS computes payoff and exploitability information, then iteratively trains red and blue best responses against the opposing population.Each side’s best-response model is initialized from its respective starting model; the red-side initialization is SFT-Red.Source: §D.2.1 · p. 30
• GRTS collects three-round red-blue dialogues per iteration and measures average n-gram diversity by averaging pairwise cosine similarities of sentence n-gram representations.The dialogue prompt begins from the training set, and the chat template is kept consistent with the backbone; the reported best-response process uses approximately 20,000 × 3 dialogue rounds.Source: §D.2.1 · p. 30

D.2.2 Baseline Method Details

• The Fix-Blue baseline holds the blue team at the backbone while PPO trains a variable red team over three-round dialogues.Training is reported to converge after approximately 3 × 20K dialogue rounds; the red-team training approach is described as similar.Source: §D.2.2 · p. 31

D.2.3 Multi-round Ablation Experiments Details

• The ablation varies dialogue rounds from 1 to 5 while keeping the total number of optimized dialogue rounds constant.Other hyperparameters and experimental settings match those used for GRTS.Source: §D.2.3 · p. 31
• At one round, the setup matches a common safety-alignment approach using human-annotated aggressive datasets and toxicity-model-supervised preference learning.Source: §D.2.3 · p. 31

D.3 Evaluation Details

• Evaluation assesses red-team attacks against the same blue model, blue-team security against different red teams and attack prompts, and red-team attacks on out-of-distribution open-source models.Blue-team helpfulness is evaluated with a model reward on Alpaca prompts, and dialogue semantic diversity is measured with an n-gram metric.Source: §D.3 · p. 31
• Attack-prompt clusters are formed by encoding prompts with paraphrase-MiniLM-L6-v2, reducing embeddings to two dimensions with t-SNE, and applying five-category K-means clustering.Source: §D.3 · p. 31

• Supplementary Fig. 13 focuses on attack topics to illustrate sentence-level diversity.Source: §E.1 · p. 31
• After adversarial training, blue-team security is significantly improved, yet red teams can still induce toxic content in diverse ways.Source: §E.1 · pp. 31, 34
• Red teams use story creation and topic digressions to elicit toxic content, including a relatively obscure attack method.Source: §E.1 · p. 34