PaperlayerView exact revision on arXiv

Source-linked AI summary

Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models

Mickel Liu, Liwei Jiang, Yancheng Liang, Simon Shaolei Du, Yejin Choi, Tim Althoff, Natasha Jaques

arXiv:2506.07468v4cs.LGcs.CLcs.MA

TL;DR

Existing safety alignment often separates attack discovery from defense updates, while isolated attackers and defenders can overfit one another. SELF-REDTEAM uses one policy for online attacker–defender self-play, and the authors report broader attack diversity and improved safety, including reduced attack success on RLHF-aligned models.

  • Problem

    Conventional alignment handles attacks against a static model and patches vulnerabilities afterward, while training attackers and defenders in isolation can lead to overfitting and incomplete vulnerability coverage.

    Source: §1 · p. 1

  • Method

    SELF-REDTEAM uses one model for online attacker–defender self-play, with a reward model adjudicating outcomes and hidden chain-of-thought for private planning.

    Source: §1 · pp. 1–2

  • Results

    Across 14 safety benchmarks, SELF-REDTEAM-trained models are substantially more robust than Defender-Only models; on RLHF-aligned models, it reduces attack success rates by up to 95%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present online co-evolution as an approach to adaptive safety alignment and end-to-end MARL training for LLMs.

    Source: §1 · p. 2

    The proof assumes a naïve refusal receives zero reward, treating it as safe but low-quality.

    Source: §3 · p. 3

Abstract

from arXiv · show

Conventional large language model (LLM) safety alignment relies on a reactive, disjoint loop: attackers exploit a static model, then defenders patch exposed vulnerabilities. This sequential setup leads to attackers overfitting obsolete exploits while defenders perpetually lag behind emerging threats. To address this, we introduce Self-RedTeam, the first fully online self-play multi-agent reinforcement learning (MARL) algorithm that continuously co-evolves attacker and defender for robust safety alignment. A single policy self-plays as both attacker and defender, generating adversarial prompts and defending against them, with a reward model adjudicating outcomes. Each role uses hidden chain-of-thought for strategic planning. Grounded in two-player zero-sum game theory, we establish a theoretical safety guarantee: if the game converges to Nash Equilibrium, the defender produces safe responses against any adversarial input. Empirically, Self-RedTeam generalizes across five models from the Llama and Qwen families, uncovering more diverse attacks (+17.80% SBERT) and improving safety of RLHF-trained models by up to 95% across 14 benchmarks. Our work motivates a shift from reactive patching to proactive co-evolution, enabling LLM safety self-improvement via online self-play MARL. Link to code: https://github.com/mickelliu/selfplay-redteaming

1. Introduction

The introduction frames static, sequential attack-and-patch alignment as a lagging, incomplete process and asks whether attackers and defenders can instead co-evolve. SELF-REDTEAM answers with single-policy online self-play, a conditional game-theoretic safety guarantee, and empirical gains across models and benchmarks.

1. Introduction

  • The paper asks whether attackers and defenders can co-evolve adaptively, since training them in isolation risks overfitting and leaves vulnerability coverage ad hoc and incomplete.The described conventional loop attacks a static model and patches exposed vulnerabilities afterward.Source: §1 · p. 1
  • A single model self-plays attacker and defender, using hidden chain-of-thought, a safety reward model, and continuous co-adaptation to emerging attacks and defenses.The paper describes the approach as computationally efficient and a drop-in alternative to standard safety alignment without significant module or compute overhead.Source: §1 · pp. 1–2
  • At Nash equilibrium, the defender is guaranteed to provide safe responses to any attacker prompt, as judged by the reward model.The guarantee is theoretical and conditional on convergence to equilibrium.Source: §1 · p. 2
  • SELF-REDTEAM improves robustness across 14 safety benchmarks and reduces attack success rates by up to 95% on RLHF-aligned models.It also uncovers 17.8% more diverse attacks than Attacker-Only training, measured by semantic similarity.Source: §1 · p. 2
  • SELF-REDTEAM is presented as the first scalable, end-to-end online MARL framework for LLM safety training, extending prior self-play work beyond restricted or offline settings.The introduction contrasts it with RNN-based language-model self-play and offline DPO methods.Source: §1 · p. 2

2. Related Work

Related work spans reinforcement learning, multi-agent self-play, and red-teaming approaches, but the paper distinguishes SELF-REDTEAM by its online, full-parameter co-evolution of attackers and defenders for LLM safety.

2. Related Work

  • Prior multi-agent and self-play work spans cooperative or competitive MARL, offline RL, alignment, data refinement, and verifiable tasks, while resource constraints have motivated lighter or iterative alternatives.Examples in the passage include offline iterative updates, quantization, QLoRA, and RNN architectures.Source: §2 · p. 3
  • SELF-REDTEAM generates experience on the fly for immediate policy updates, uses hidden CoT to conceal reasoning from opponents, and trains full parameters without significant quantization or LoRA.The authors describe it as the first scalable, end-to-end online MARL framework for full-parameter LM safety training.Source: §2 · p. 3
  • Earlier red-teaming approaches often separated attacks from defenses; SELF-REDTEAM instead uses end-to-end online multi-agent RL, alongside zero-sum-game guarantees and empirical gains.The passage also contrasts prior prompt-tuning approaches that could take hours per attack.Source: §2 · p. 3

3. Theoretical Safety Guarantees of LLMs with Zero-Sum Red-Teaming Games

The paper models red-teaming as a two-player zero-sum game in which an attacker minimizes a verifier reward and a defender maximizes it. Under a stated refusal-reward assumption, equilibrium implies safe responses to attacker prompts, though exact equilibrium and zero empirical harmfulness are challenging in practice.

3. Theoretical Safety Guarantees of LLMs with Zero-Sum Red-Teaming Games

  • The attacker proposes a prompt and the defender responds; a reward model scores the pair from -1 to 1, with the attacker minimizing and defender maximizing the score.This defines the two-player zero-sum min–max objective.Source: §3 · p. 3
  • If both policies converge to Nash equilibrium, the defender's response to any attacker prompt has nonnegative reward and is therefore safe under the theorem's criterion.The guarantee is expressed in terms of the reward model's judgment.Source: §3 · p. 3
  • The proof assumes that a naïve refusal receives zero reward, and the authors note exact equilibrium and zero empirical harmfulness are challenging in practice.The assumption treats refusal as safe but low-quality and is described as consistent with safety-finetuning reward models that score helpfulness and harmfulness.Source: §3 · p. 3

4. SELF-REDTEAM: Online Self-Play MARL Safety Training of LLMs

SELF-REDTEAM turns safety training into a game in which one LLM plays both attacker and defender. Its training uses hidden reasoning for strategic planning, while the paper illustrates how self-play attacks remain more dispersed than attacks from a fixed attacker.

4. SELF-REDTEAM: Online Self-Play MARL Safety Training of LLMs

  • One LLM policy plays both attacker and defender in an empirical red-teaming game, taking turns with the attacker moving first.The formulation is motivated by the preceding Nash-equilibrium safety result.Source: §4 · p. 3
  • In the t-SNE visualization of attacks from 1,000 seed prompts, Self-Play attacks are more dispersed, while Attacker-Only attacks converge into a few dominant modes.The caption describes the visualized spatial distribution as semantic clustering of generated attack vectors.Source: Figure 3 · p. 4

4.1. Self-Play Red-Teaming Game

  • Hidden CoT separates private reasoning from the public answer, revealing only the public move to the opponent and encouraging strategic planning through role-specific instructions and format rewards.The approach is intended to encourage CoT reasoning even when the initial policy does not fully support the format.Source: §4.1 · p. 4

A , yCoT

SELF-REDTEAM trains one shared policy in attacker and defender roles, using reward shaping and role-specific online RL to support co-evolution. The training also optionally mixes in self-distilled SFT to preserve conversational fluency.

A , yCoT

  • The attacker rewrites harmful seeds into stealthier attacks and benign seeds into deceptive-looking but innocuous prompts, while the defender responds using private CoT reasoning.Keeping benign prompts in the interaction helps prevent the defender from succeeding through blanket refusals.Source: A , yCoT · p. 4

4.2. Reward Design

  • The zero-sum game reward scores response harmfulness and whether refusal matches query harmfulness, while shaping rewards enforce CoT formatting and attack-seed intent preservation.The attacker is rewarded for eliciting harmful responses or refusals to benign queries; the defender is rewarded for avoiding harmful responses and refusing harmful queries.Source: §4.2 · pp. 4–5

4.3. Self-Play Adversarial Online Training Algorithm

  • SELF-REDTEAM trains a shared attacker–defender policy online with Re++, normalizing token-level advantages separately by role to handle their opposing reward signals.The process generates attacker and defender turns, computes game and shaping rewards, then updates the policy using the role-specific advantages.Source: §4.3 · p. 5
  • Optional self-distilled SFT is optimized jointly with the RL objective to preserve conversational fluency.Game-reward-only optimization may degrade open-ended conversational quality.Source: §4.3 · p. 5

5. Experiment

The experiments evaluate SELF-REDTEAM across safety, benign compliance, attack diversity, and general capabilities, using static and dynamic evaluations. The ablation study reports a safety-utility comparison with safeguarding baselines.

5.2. Evaluation

  • The evaluation spans five Qwen2.5 and Llama3.1 models, static safety and benign-compliance benchmarks, three dynamic jailbreak methods, and general-capability tests.Attack diversity is measured with Self-BLEU and SBERT sentence-embedding similarity; Figure 4 reports training metrics across attack diversity, attacker and defender performance, and CoT-template violations.Source: §5.1 · p. 5, §5.2 · p. 6

5.1. Training Setups

  • The training setup uses WildGuard-7B as judge and samples 26,000 WildJailBreak prompts for RL, balanced between harmful and benign seeds.The auxiliary SFT set contains 30,000 examples from WildJailBreak and HelpSteer3.Source: §5.1 · p. 5

5.3. Ablation Study

  • The study compares attacker-only and defender-only training, self-play with and without CoT, and SFT-augmented variants; SELF-REDTEAM is also compared with LLM-LAT and CircuitBreaker.SELF-REDTEAM and Self-Play + SFT are used interchangeably.Source: §5.3 · p. 7
  • SELF-REDTEAM maintains benign compliance at 0.948 and 0.964 while improving safety, offering a more practical safety-utility trade-off than LLM-LAT and CircuitBreaker.LLM-LAT sharply reduces benign compliance, while CircuitBreaker yields limited safety gains on the reported comparison.Source: §5.3 · p. 7

6. Results and Discussion

SELF-REDTEAM improves safety across model families and benchmarks, including under dynamic attacks, while preserving capability in several settings. Its co-evolving attacker and defender produce more diverse attacks, and hidden chain-of-thought helps reduce over-refusal.

6. Results and Discussion

  • Across 12 benchmarks, SELF-REDTEAM improves all 9 harmful-refusal metrics for Qwen models and 8 of 9 for Llama3.1-8B-IT, with minimal over-refusals.Average relative improvements are 33.4% for Qwen2.5-3B-IT, 39.5% for Qwen2.5-7B-IT, 44.7% for Qwen2.5-14B-IT, and 45.2% for Llama3.1-8B-IT; the abliterated Llama variant also improves across all safety metrics.Source: §6 · p. 7
  • 31.3% average improvement in attack success rates across dynamic-attack settings, with reductions up to 50.7% on PAIR for Qwen2.5-14B-IT.The evaluations cover X-Teaming, PAIR, and AutoDAN; Table 4 presents the dynamic-attack results.Source: §6 · pp. 7–8, Table 4 · p. 8
  • 36.43% average relative ASR reduction across 11 safety benchmarks, plus 17.33% greater safety robustness than Defender-Only + SFT.SELF-REDTEAM also has a higher AlpacaEval-2 LC win rate than that baseline (38.09% vs. 35.50%).Source: §6 · p. 8
  • 24.74 attack perplexity for SELF-REDTEAM versus 8.62 for Attacker-Only, while its attacks span more tactics and maintain greater diversity during training.No SELF-REDTEAM tactic exceeds 30%, compared with 86.3% of Attacker-Only attacks concentrated on one tactic; effective strategy counts are 4.6 vs. 1.9.Source: §6 · p. 8
  • Self-play defender performance improves through co-adaptation as attacker success declines, whereas Defender-Only’s near-perfect harmlessness overfits static attacks and fails to generalize out of distribution.Attacker-Only maintains high attack success against a fixed defender but shows an approximately 60% drop in revision faithfulness.Source: §6 · p. 8
  • Hidden CoT restores benign compliance for Llama3.1-8B-IT and Llama3.1-8B-IT-AB to 0.936 and 0.806, respectively, while preserving safety.Without CoT, compliance is 0.528 and 0.470, respectively; Qwen2.5-14B gains are modest (0.969 vs. 0.948).Source: §6 · pp. 8–9
  • Self-play takes approximately 44–48% more time than Defender-Only due to online prompt generation, but Self-Play + SFT matches Defender-Only + SFT total time.On Llama-3.1-8B-IT, the respective times are 3h 32m and 3h 35m; both methods use identical model and batching configurations with no extra GPU memory overhead.Source: §6 · p. 9
  • Replacing WildGuard with Qwen3Guard improves 8 of 11 safety metrics, yielding a 4.0% average relative gain across benchmarks.The paper also reports cross-judge and label-based evidence that gains are not an artifact of the training judge.Source: Table 3 · p. 8, §6 · p. 9

7. Conclusion

SELF-REDTEAM proposes online self-play reinforcement learning in which attackers and defenders co-evolve within a game-theoretic framework. The paper concludes that this approach discovers more diverse attacks and improves robustness to adversarial inputs.

7. Conclusion

  • SELF-REDTEAM lets attacker and defender LLMs co-evolve dynamically within a game-theoretic framework.The authors describe it as an online self-play reinforcement learning framework for LLM safety.Source: §7 · p. 9
  • The framework fosters more diverse attack discovery and improves LLM robustness against adversarial inputs.The conclusion presents it as a proactive alternative to reactive safety patching via RLHF.Source: §7 · p. 9
  • The authors characterize SELF-REDTEAM as a scalable, theoretically grounded approach to building safer and capable LLMs.Source: §7 · p. 9

Impact Statement

SELF-REDTEAM aims to improve LLM safety and reliability through self-play, while the authors acknowledge possible misuse of generated attacks. They argue that vulnerability discovery supports robust defense and describe a safeguarding plan.

Impact Statement

  • SELF-REDTEAM uses self-play to improve LLM robustness to adversarial attacks by discovering vulnerabilities in a controlled setting.Source: Impact Statement · p. 9
  • The generated attack strategies could be misused against less secure LLMs.Source: Impact Statement · p. 9
  • The authors describe proactive vulnerability identification as necessary for robust defenses and report providing a safeguarding plan in Appendix F.Source: Impact Statement · p. 9

Appendices

The appendices detail SELF-REDTEAM’s training loop, data construction, and experimental setup, then present a proof that an equilibrium defender is safe against any prompt under the stated reward model.

A.1. Training Algorithm Pseudocode

  • Each training step generates attacker–defender rollouts with a frozen policy, computes role-specific rewards and normalized advantages, then updates the shared policy.The algorithm can jointly accumulate RL and optional SFT gradients before updating parameters.Source: §A.1 · p. 15

A.2. Additional Training Details

  • Training uses 4 H200-141GB GPUs, 32 examples per actor, and a cosine-annealed learning rate from 5 × 10^-7 to 5 × 10^-8 over 300 steps, with early stopping at 200.Experiments take approximately 2–4 hours on average.Source: §A.2 · p. 15

A.3. Additional Details about RL & SFT Dataset Construction

  • RL prompts are split evenly among unmodified harmful prompts, harmful attacker seeds, unmodified benign prompts, and benign attacker seeds.For SFT, the model generates a completion and then retrospective reasoning; classifier filtering removes harmful content or incorrect refusals, observed in under 3% of instances.Source: §A.3 · pp. 15–16, Figure 5 · p. 16

B. Complete Proof of Theorem 1

  • In the red-teaming game, the attacker minimizes a reward-model score while the defender maximizes it; unsafe responses receive -1, while safe responses receive a non-negative score.Source: B. Complete Proof of Theorem 1 · p. 17
  • At Nash equilibrium, the defender’s response receives a non-negative reward for any attacker prompt, which the theorem treats as safe.Source: B. Complete Proof of Theorem 1 · p. 17

B.1. Scope of Theorem 1: A Constrained Zero-Sum Interpretation

The section interprets the theorem as a safety property of equilibrium in the zero-sum core, not a convergence guarantee for the full training objective. It relates auxiliary rewards to policy constraints and reports benchmark evaluation procedures.

B.1. Scope of Theorem 1: A Constrained Zero-Sum Interpretation

  • The equilibrium safety theorem applies to the idealized zero-sum reward, not the shaped training objective, and does not guarantee convergence to equilibrium.The practical objective adds formatting and attacker-revision reward terms.Source: §B.1 · p. 17
  • Formatting and revision-faithfulness rewards act as soft penalties for violating policy constraints, while response-harmfulness and refusal rewards form the zero-sum safety core.The constraints include following prescribed formats and avoiding refusals of benign queries.Source: §B.1 · pp. 17–18
  • As the penalty weight λ approaches infinity, the regularized game recovers the constrained zero-sum game; sufficiently large finite λ yields bounded approximation error.Source: §B.1 · p. 18
  • The authors report harmlessness win rates stabilizing near 100% and consistent safety improvements across benchmarks, which they interpret as evidence that regularization is sufficient to drive training toward the safe-defender region.Source: §B.1 · p. 18

C. Evaluation

  • Evaluation uses OLMES for instruction following, Ai2 Safety Tool for safety, and lm-evaluation-harness for general capabilities.Source: C. Evaluation · p. 19

C.1. Safety Evaluation Benchmarks

This section describes safety, capability, instruction-following, and dynamic-attack benchmarks, as well as self-play ablations and comparisons with safeguarding baselines and other methods. It reports findings on safety performance, attack diversity and novelty, jailbreak-strategy coverage, and training costs, alongside the rationale for the approach.

C.1. Safety Evaluation Benchmarks

  • Safety evaluation covers HarmBench, WildGuardTest, WildJailbreak, DAN, XSTEST, and OR-Bench-Toxic, assessing harmful-prompt robustness, harmful-content detection, and over-refusal.HarmBench uses 321 vanilla prompts and 1,500 adversarial attacks; WildJailbreak adversarial partitions are used for evaluation, while its vanilla partitions seed attacker revisions. XSTEST includes 250 benign and 200 harmful prompts; OR-Bench-Toxic has 600 toxic prompts.Source: §C.1 · pp. 19–20

C.2. General Capability Evaluation Benchmarks; C.3. Evaluation Benchmarks on Instruction-Following

  • Capability evaluations on ARC-C, GPQA, MMLU, and TruthfulQA-MC1 used each benchmark’s lm-evaluation-harness defaults without sampling; AlpacaEval-2 used weighted_alpaca_eval_gpt4_turbo, temperature 0.6, and top_K 0.9.Source: §C.2 · p. 20, §C.3 · p. 21

C.4. Dynamic Evaluation

  • Dynamic evaluation samples 50 HarmBench behaviors, generates 10 X-Teaming attack plans per behavior, and executes them over three turns using Qwen2.5-32B-Instruct as attacker.PAIR and AutoDAN are also evaluated using their HarmBench implementations.Source: §C.4 · pp. 21–22

Attacker Instruction Template

  • The attacker template instructs revision of harmful or benign seed prompts into on-topic adversarial prompts, with distinct classifier-detection and defender-response criteria for winning or losing.The harmful-seed attacker aims for a classifier-detected jailbreak that elicits harmful output; the benign-seed attacker aims for a classifier-detected-benign prompt that elicits harmful output.Source: Attacker Instruction Template · p. 23

Self-Distilled SFT Prompts

  • The self-distilled SFT prompt asks a model to produce a natural, concise step-by-step thinking process that could have led to a supplied response.The appendix also reports t-SNE/DBSCAN visualization of attacks: Attacker-only forms large concentrated clusters, while Self-play attacks are more dispersed.Source: Self-Distilled SFT Prompts · p. 24, §C.5 · p. 24

D.1. Ablation Study

  • Across model configurations, self-play outperformed defender-only baselines on adversarial-harm benchmarks, auxiliary SFT recovered AlpacaEval-2 instruction-following, and Hidden CoT helped Llama avoid over-refusal.The appendix provides extended ablations for Qwen2.5-3B/7B, Llama3.1-8B, and abliterated Llama3.1-8B; the main-text ablation is Qwen2.5-14B-Instruct.Source: §D.1 · pp. 24–25

D.2. Comparing to Additional Safeguarding Baselines

  • The safeguarding comparison fine-tuned two base models with self-play and compared them against LLM-LAT and CircuitBreaker.The supplied passages identify Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct comparison tables but provide no cell values or comparative outcome.Source: §D.2 · p. 27, Table 14 · p. 27, Table 15 · p. 27

Self-play Methods Excel Across All Metrics

  • Alpaca-Eval scores were 22.04 for Llama and 35.80 for Qwen with Self-play + SFT, which showed the best overall balance.Both self-play variants reduced harmfulness on adversarial prompts while maintaining refusal accuracy and preserving performance on benign prompts (WJB AB, XSTest VB).Source: Self-play Methods Excel Across All… · p. 27

LAT Suffers from Overrefusal

  • LAT’s strong harmfulness reduction and near-perfect refusal accuracy came with poor benign handling and reduced chat ability, consistent with overrefusal.XSTest VB was 0.004 for Llama and 0.552 for Qwen; Llama Alpaca-Eval was 9.68 versus the 25.65 baseline.Source: LAT Suffers from Overrefusal · p. 27

Circuit Breaker Shows Limited Impact

  • Circuit Breaker had modest effects on Llama and virtually none on Qwen, while being less aggressive but also less effective than other methods.The passage attributes the negligible Qwen impact to architecture sensitivity; Self-play is described as more transferable across Llama and Qwen without hyperparameter adjustments.Source: Circuit Breaker Shows Limited Impact · p. 28

D.3. Discussion on efficiency and computational overhead

  • Self-play incurred no additional GPU memory overhead and a 44–48% training-time overhead, approximately matching its 50% increase in training samples.On Llama-3.1-8B, Self-play + SFT took 3h 32m versus 3h 35m for Defender-only + SFT.Source: §D.3 · p. 28

D.4. Discovery of Novel Attacks

  • Self-play improved attack diversity by 4.10% (inverse SBERT similarity: 0.8320 to 0.8661), while base-model perplexity rose by 11.38 for Self-play and 16.12 for Self-play + SFT.The trained attacker’s greater perplexity to the base model is presented as evidence of novel attacks; the reported values were 8.62 to 20.00 for Self-play and 8.62 to 24.74 for Self-play + SFT.Source: Table 16 · p. 29, §D.4 · p. 29
  • The training curve showed progressively improved defender robustness, including defense against attacks it initially failed to counter.This is presented as complementary evidence that training adapts to novel attack vectors.Source: §D.4 · p. 29

D.5. Attack Strategy Classification

  • SELF-REDTEAM’s effective strategy count was 2.4× higher than Attacker-Only’s (4.6 vs. 1.9), with no tactic exceeding 30% of attacks.Attacker-Only concentrated 86.3% of attacks on contextualizing the task; SELF-REDTEAM’s distribution was more balanced. The effect persisted for benign seeds and RL-only self-play.Source: Table 17 · p. 30, §D.5 · p. 30

D.6. Additional Qualitative Examples

  • The qualitative examples show attacker reasoning refining a harmful request and defender reasoning identifying red flags before refusing.Table 18 illustrates hidden chain-of-thought for both roles, with the defender providing a helpful refusal in its answer.Source: Table 18 · p. 31, §D.6 · p. 31

E.1. Design Rationale for Single-Model Co-Evolution

  • The single-model design aims to keep co-evolution computationally accessible while enabling a model to surface and correct its own blind spots.Both roles originate from the same initial state, reducing reliance on auxiliary modules beyond the reward model standard to RLHF.Source: §E.1 · p. 32

E.2. Analysis of Attacker Refusal Behavior

Attacker refusals can occur during Llama self-play, but the paper attributes elevated refusal rates to the Llama base model and reports continued defender safety gains. The authors also note that resulting models, particularly the attacker role, need safeguards against misuse.

E.2. Analysis of Attacker Refusal Behavior

  • At Step 200, checkpoint #5 began refusing attacker revision requests while #7, using the same prompt-sampler seed, showed no refusals.Across checkpoints, the authors report this behavior with a probability of < 10%.Source: §E.2 · p. 32
  • Llama-3.1-8B-Instruct had a 25.27% attacker refusal rate, versus < 0.11% for Qwen models; the abliterated Llama variant reduced the rate to 7.21%.The authors attribute the elevated Llama rate to its propensity for over-refusal; they report that training partially mitigates this behavior in the abliterated variant.Source: §E.2 · p. 32
  • Despite occasional Llama attacker refusals, the authors report consistent gains on downstream safety evaluations because defender robustness is the method’s primary objective.The co-evolving attacker is an auxiliary component intended to facilitate defender improvement, not a standalone objective.Source: §E.2 · p. 32

F.1. Adding External Safeguard

  • The authors say resulting models, particularly the attacker role, require additional safeguards to reduce potential misuse.They suggest external safeguards for deployment.Source: §F.1 · p. 34

Related papers

Loading 2506.07468v4…