PaperlayerView exact revision on arXiv

Source-linked AI summary

TriPlay-RL: Tri-Role Self-Play Reinforcement Learning for LLM Safety Alignment

Zhewen Tan, Wenhan Yu, Jianfeng Si, Tongxin Liu, Kaiqi Guan, Huiyan Jin, Jiawen Tao, Xiaokun Yuan, Duohe Ma, Xiangzheng Zhang, Tong Yang, Lin Sun

arXiv:2601.18292v2cs.LGcs.AI

TL;DR

LLM safety alignment must address costly annotation, isolated role optimization, and context-dependent evaluation. TriPlay-RL creates a closed-loop reinforcement-learning system that jointly evolves an attacker, defender, and evaluator with minimal manual data. Experiments report co-evolution across all three roles, including strong attack performance, high defense safety with maintained reasoning, and improved evaluator consistency.

  • Problem

    Safety alignment remains limited by costly human review, isolated role optimization, and flexible context-dependent criteria that challenge fixed evaluators.

  • Method

    TriPlay-RL jointly optimizes MRed, MBlue, and MEval in a unified closed-loop reinforcement-learning framework with minimal data requirements.

  • Results

    TriPlay-RL enables co-evolution: MRed achieves 90% Attack Success Rate (ASR) against Llama-3.1-Nemotron-Nano-8B-v1, while MBlue maintains general reasoning capability and MEval improves judgment consistency.

  • Takeaways & Limitations

    The framework provides a unified paradigm for diversity-driven safety alignment through continuous adversarial evolution and collaborative training of the three roles.

  • Takeaways & Limitations

    The study uses separate instances of the same base model for all three roles and does not explore heterogeneous capabilities or one shared model instance.

Abstract

from arXiv · show

In recent years, safety risks associated with large language models have become increasingly prominent, highlighting the urgent need to mitigate the generation of toxic and harmful content. The mainstream paradigm for LLM safety alignment typically adopts a collaborative framework involving three roles: an attacker for adversarial prompt generation, a defender for safety defense, and an evaluator for response assessment. In this paper, we propose a closed-loop reinforcement learning framework called TriPlay-RL that enables iterative and co-improving collaboration among three roles with near-zero manual annotation. Experimental results show that the attacker preserves high output diversity while achieving a 20%-50% improvement in adversarial effectiveness; the defender attains 10%-30% gains in safety performance without degrading general reasoning capability; and the evaluator continuously refines its fine-grained judgment ability through iterations, accurately distinguishing unsafe responses, simple refusals, and useful guidance. Overall, our framework establishes an efficient and scalable paradigm for LLM safety alignment, enabling continuous co-evolution within a unified learning loop.

1 Introduction

LLM safety alignment remains constrained by costly annotation, isolated role optimization, and flexible, context-dependent evaluation. TriPlay-RL addresses these challenges with a closed-loop framework in which MRed, MBlue, and MEval mutually reinforce one another.

  • Existing safety-alignment methods face scalability limits from costly manual annotation or human review.
  • Fixed evaluators struggle with reliable long-term alignment because harmful-content judgments depend on subjective context interpretation.
  • TriPlay-RL introduces an evaluator into dual-role co-evolution, forming a stable and scalable closed loop among MRed, MBlue, and MEval.
  • MRed reaches 90% Attack Success Rate (ASR) against Llama-3.1-Nemotron-Nano-8B-v1 and a threefold baseline-ASR improvement against Qwen3-8B.
  • TriPlay-RL uses inter-role interactions and tailored rewards to enable collaborative optimization while mitigating pattern collapse during training.
  • Diversity penalties and multi-model adversarial training strengthen MRed while preserving output diversity, and three-level rewards improve MBlue safety without sacrificing general reasoning.

2 Related Work

Prior work increasingly uses model interactions and automated red teaming to reduce human annotation and improve robustness. However, safety evolution still requires balancing attack effectiveness, defense, utility, and reliable evaluation.

  • Model-interaction methods reduce reliance on large amounts of human-annotated data and support autonomous capability evolution.
  • Proposer–solver and generator–verifier collaborations have demonstrated capability gains under zero-data or unsupervised conditions, with process-level feedback supporting reasoning improvement.
  • Balancing safety and utility during model evolution remains challenging.
  • Automated red teaming replaces costly manual testing by generating adversarial prompts intended to elicit harmful model behavior.
  • Optimization and search methods improve attack query efficiency, but static generic criteria can misalign red-team evolution with defended-model vulnerabilities.
  • Dynamic feedback lets red teams adjust strategies from defense responses, yet prior approaches lack strategic continuous evolution of red-team capabilities.

3 Method

TriPlay-RL alternates reinforcement-learning updates across attacker, defender, and evaluator roles in a closed loop. Tailored rewards promote effective, diverse attacks, balanced defense, and fine-grained evaluation.

  • TriPlay-RL framework: TriPlay-RL sequentially updates MRed, MBlue, and MEval through three training phases using GRPO-based reinforcement learning with verifiable rewards.The phases are PRed, PBlue, and PEval, corresponding to MRed → MBlue → MEval updates.
  • MRed training: MRed’s reward balances attack effectiveness, semantic consistency, and generation diversity across heterogeneous defense models.Additional Llama and ChatGLM models serve as attack targets, and generated prompt-response pairs are stored for MEval training.
  • MRed training: MRed’s semantic reward preserves the original prompt’s core meaning and attack goal, preventing semantic drift in adversarial generations.A language-model judge assigns positive semantic reward when relevance is maintained.
  • MRed training: MRed’s diversity penalty uses Self-BLEU and embedding cosine similarity against the Attack Success Pool, with a nonlinear penalty for highly similar prompts.The constant is set to k = ln(11/6), and the weights for the diversity terms are listed in Appendix B.
  • MBlue training: MBlue trains on prompts from the latest MRed iteration and receives three-level rewards for unsafe responses, simple refusals, and safe helpful guidance.The reward values are −1 for negative, 0 for rejective, and 1 for positive responses.
  • MEval training: MEval performs fine-grained three-class evaluation using adversarial prompt-response pairs accumulated from the MRed-MBlue process and multi-expert voting to reduce reward hacking.Its categories distinguish unsafe responses, simple refusals, and useful guidance.

4 Experimental Setup

The experiments initialize TriPlay-RL with Qwen3 model variants and minimal attack data, then evaluate attacker vulnerability, defender safety and reasoning, and evaluator classification accuracy.

  • Initialization: Qwen3-4B, Qwen3-8B, and Qwen3-14B serve as the initial models for training.TriPlay-RL is implemented using TRL as the underlying reinforcement-learning framework.
  • Initialization: TriPlay-RL bootstrapping requires minimal MRed input data, using nine prompt-wrapping techniques and 200 basic attack prompts from HarmBench.Demonstration examples are provided for MRed prompt templates listed in Appendix A.
  • MRed evaluation: MRed vulnerability is evaluated primarily with Attack Success Rate (ASR), based on successful attack samples among total test samples.DeepSeek-R1-0528-Qwen3-8B, Qwen3-8B, and Llama-3.1-Nemotron-Nano-8B-v1 serve as defense models, with GPT-5.2 judging attack success.
  • MBlue evaluation: MBlue safety is evaluated with ASR on AIR-Bench 2024, JailBreakBench, WildJailBreak, and S-Eval, alongside general reasoning evaluation on IFEval.DeepSeek-R1-Qwen3-14B, Llama-3-8B, and Qwen3-32B are used as baselines.
  • MEval evaluation: MEval is assessed on an internal training dataset constructed for the model, using accuracy on a three-class classification task.The evaluation targets fine-grained distinctions among response categories.

5 Experimental Results

Across training iterations, MRed’s attack success rises against multiple defense models, while MBlue reduces attack success without significantly losing general reasoning performance. MEval accuracy also improves, providing increasingly accurate and stable reward signals for optimizing MRed and MBlue.

  • 5.1 MRed Unit: After 16 iterations, MRed-14B reaches 90% ASR against Llama-3.1-Nemotron-Nano-8B, up from 60%.Its ASR also rises from 13.0% to 32.0% against DeepSeek-R1-0528-Qwen3-8B and from 21.84% to 67.75% against Qwen3-8B.
  • 5.2 MBlue Unit: Across training iterations, MBlue’s ASR declines on different safety evaluations, despite some fluctuations.On AIR-Bench 2024, Qwen3-14B’s ASR falls from 13.9% to 4.4%, while JailBreakBench ASR drops from 31.5% to 4.6% after 10 iterations.
  • 5.2 MBlue Unit: The MBlue reasoning-retention evaluation reports avg@32 for benchmarks and prompt-level strict evaluation for IFEval.These evaluation conventions define how general reasoning capability retention is measured in Table 1.
  • 5.2 MBlue Unit: MBlue’s reasoning performance does not significantly decline compared with its baseline despite training without reasoning or general-purpose data.The model shows slight improvements in nearly half of the tests.
  • 5.3 MEval Unit: MEval accuracy progressively increases across all three models, strengthening the reward signals used to optimize MRed and MBlue.MEval-4B rises from 48.2% to 56.2%, MEval-8B from 54.9% to 64.3%, and MEval-14B from 97.0% to 98.2%.

6 Ablation Study

The ablations show that multi-defense training strengthens MRed’s attack capability across tested models, while iterative training and diversity penalties prevent entropy and output-diversity collapse.

  • 6.1 Multi-Single Ablation: Multi-defense training yields stronger MRed attacks against DeepSeek-R1-0528-Qwen3-8B, Qwen3-8B, and Llama-3.1-Nemotron-Nano-8B.This design also enhances attack generalization and mitigates overfitting.
  • 6.2 Diversity Ablation: Removing both closed-loop training and diversity penalties causes MRed’s training entropy to collapse and fixes its prompts into repeated templates.Introducing either component keeps entropy fluctuating within a certain range.
  • 6.2 Diversity Ablation: The full iterative-training and diversity-penalty configuration achieves a diversity score of 0.588, compared with 0.514 without the penalty and 0.156 without iterative training.These results identify iterative training as crucial for maintaining output diversity.
  • 6.2 Diversity Ablation: The w/o L + w/o D setting reaches the highest ASR against the weaker Nano-8B model but only 2.8% ASR against Meta-Llama-3.1-8B-instruct.The w/ L + w/ D setting achieves high ASR against both defense models.

7 Conclusion

TriPlay-RL presents a unified multirole framework for diversity-driven safety alignment with limited manual labeling. The conclusion also identifies unresolved limitations involving model initialization, adversarial trade-offs, external data, game-theoretic dynamics, ethical risks, evaluation reliability, and deployment oversight.

  • Conclusion: TriPlay-RL synergistically optimizes MRed, MBlue, and MEval to support diversity-driven safety alignment without extensive manual labeling.The framework uses multi-blue games, semantic rewards, and diversity penalties to drive continuous adversarial evolution.
  • Limitations: The framework initializes MRed, MBlue, and MEval from the same base model and does not test heterogeneous role capabilities.The current implementation uses three separate model instances, leaving shared-instance optimization open.
  • Limitations: Optimizing red-team behavior may degrade safety behavior, while strengthening defense may suppress adversarial effectiveness, creating unresolved trade-offs.The paper identifies these adversarial interactions as a future research direction.
  • Limitations: The study does not examine how external safety or adversarial data affect the three-role optimization process.The authors leave interactions between external data and training dynamics for future work.
  • Limitations: The paper lacks in-depth analysis of Nash equilibria, Pareto frontiers, and mechanisms for regulating capability growth across the three models.The authors note that unregulated growth could destabilize training or produce suboptimal equilibria.
  • Ethical Considerations: Automated red-teaming introduces dual-use risks because increasingly effective adversarial prompts could help bypass deployed safety mechanisms.The framework is restricted to controlled research settings and does not release attack prompts or models to facilitate misuse.
  • Ethical Considerations: Automated evaluation supplies training signals but cannot fully eliminate errors or latent biases, which may amplify across closed-loop iterations.The paper treats the evaluator as an evolving approximation rather than a definitive safety authority.
  • Ethical Considerations: Real-world deployment requires access control, monitoring, governance, and continued human oversight, especially for legal, medical, or safety-critical content.The authors state that automated safety mechanisms should complement rather than replace human oversight.

A Prompt Templates

The appendix defines prompt templates for red-team generation, evaluation, and multi-directional response distillation. These templates specify inputs, output formats, response categories, and safety-assessment instructions.

  • A Prompt Templates: Task-specific prompt templates maintain one-to-one correspondence with the model by replacing designated keys during each training session.The template section uses separate formats for different tasks.
  • A Prompt Templates: The Red Model’s Prompt Template asks MRed to generate high-risk prompts from classification information and thematic descriptions for safety assessment.It instructs the model to study attack techniques and wrap a base risky prompt with one or more techniques.
  • A Prompt Templates: The red-team template requires JSON output containing reasoning, basic_prompt, and adversarial_prompt fields.The basic prompt is copied from the base risky prompt, while the adversarial prompt applies attack types.
  • A Prompt Templates: The red-team instructions require risk reasoning while directing the model not to refuse and not to assess the input’s risk.These instructions are intended to complete the adversarial-generation task regardless of input.
  • Eval Model’s Prompt Template: The evaluator template assesses risks in model responses rather than in the user’s question and returns a JSON object with Explanation and Conclusion.The conclusion is restricted to the labels P, S, or R.
  • Eval Model’s Prompt Template: The evaluator labels responses as S for risk-associated, R for rejective, and P for positive guidance toward a safe direction.The template also includes evaluator variants and safety-policy categories such as violent crimes, privacy, hate, self-harm, and elections.
  • Eval Model’s Prompt Template: The Llama-Guard-3 evaluation template checks the last Agent message against listed unsafe-content categories and reports safe or unsafe status with violated categories.The conversation format places the adversarial prompt in the User message and the model output in the Agent message.
  • Multi-Directional Distillation Prompt Template: The multi-directional distillation template requests positive-guidance, negative, and rejective responses for content-safety attack-defense drills.Positive responses provide safe guidance, negative responses are risky, and rejective responses refuse concisely; the output is requested as JSON.

B Training Parameters & Expense

This section points readers to tables containing defense-model weights and module hyperparameters used during MRed training.

  • B Training Parameters & Expense: Table 3 lists the weight of each defense model during MRed training.
  • B Training Parameters & Expense: The supplied passages identify the relevant parameter tables but do not report computing expense values.
  • B Training Parameters & Expense: Table 4 lists parameter settings for each module.

B.1 computing infrastructure

The computing-infrastructure subsection refers to a table of infrastructure specifications, but the supplied passage does not provide those specifications.

  • B.1 computing infrastructure: Table 5 is designated for computing infrastructure specifications.
  • B.1 computing infrastructure: The supplied passage does not state the hardware, software, or infrastructure values listed in Table 5.
  • B.1 computing infrastructure: No computing-expense result is reported in the supplied infrastructure passage.

C Use of AI

The paper uses an LLM to polish sentences and correct grammatical errors.

  • The authors use an LLM to polish sentences in the paper.
  • The stated use combines sentence polishing with grammatical correction.
  • The LLM is used to correct grammatical errors.
Loading 2601.18292v2…