PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Closed-Loop Curriculum for Evolving Role-Playing Agents

Zheng Zhang, Liu Liu, Qi Chai, Deheng Ye, Peilin Zhao, Mao Zheng, Hao Wang

arXiv:2609.28609v1cs.AI

TL;DR

Role-playing RL often trains on fixed scenario pools even as an improving actor’s failures shift. AdvRole alternates actor training with adversarial context rewriting, and outperforms baselines across role-playing benchmarks.

  • Problem

    Role-playing RL commonly uses fixed scenario pools, although the scenarios an improving actor struggles with change over time.

    Source: Introduction · p. 1

  • Method

    AdvRole alternates Actor training with a Rewriter that edits character profiles and dialogue contexts into actor-specific hard scenarios using a performance-gap reward.

    Source: Introduction · p. 2

  • Results

    Experiments on three role-playing benchmarks, including a new multilingual benchmark, show that AdvRole outperforms baselines.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    The authors suggest that limited scenario coverage, rather than raw model capacity, is a bottleneck for smaller open-source role-playing models.

    Source: Main Results · p. 5

    AdvRole is slightly below CogDual-RL on BLEU and ROUGE-L because its reward targets role-playing quality rather than n-gram overlap.

    Source: Main Results · p. 5

Abstract

from arXiv · show

Role-playing agents based on large language models have been widely applied in areas such as personalized assistance and social simulation. Recent RL methods typically train on a fixed scenario pool collected before learning begins. This creates a distributional bottleneck: as the agent improves, the scenarios where it performs poorly also change, while the training distribution remains static. Therefore, we propose AdvRole, an adversarial context rewriting framework that turns role-playing RL into a closed-loop curriculum. AdvRole alternates between an Actor that learns to role-play and a Rewriter that edits character profiles and dialogue contexts into actor-specific hard scenarios. The Rewriter is trained with a performance-gap reward, which favors rewrites that reduce the current Actor's score relative to the original scenario. As a result, the scenario pool evolves with the Actor and continuously targets under-mastered regions of the character-context space. Experiments on three role-playing benchmarks covering English and Chinese, as well as a new multilingual benchmark we release, show that AdvRole consistently outperforms baselines.

Introduction

Role-playing agents must stay faithful to a character and consistent with conversation, but fixed-scenario RL cannot track changing actor failures. AdvRole couples actor learning with adversarial scenario rewriting to evolve the training curriculum.

Introduction

  • Role-playing asks an agent to produce dialogue faithful to a designated character’s profile and consistent with the ongoing conversation.Applications include interactive entertainment, social simulation, and personalized assistants.Source: Introduction · p. 1
  • Prior role-playing methods progressed from prompt engineering and supervised fine-tuning to PPO or DPO on human preferences and GRPO with automatic rewards, yet RL methods largely retain fixed pools.The cited approaches include PPO or DPO using human-annotated preferences (Peng and Shang 2024; Zeng et al. 2025; Ji et al. 2025; Abdulhai et al. 2026) and GRPO with automatically computed rewards (Liu et al. 2025; Wang et al. 2025d, 2026b,a).Source: Introduction · p. 1
  • Fixed scenario pools become mismatched to learning because scenario difficulty depends on the Actor, whose remaining failures shift as it improves.Useful training data can shift from common profiles and contexts toward rarer traits, subtle conflicts, and unfamiliar dialogue states.Source: Introduction · p. 1
  • AdvRole alternates an Actor trained with a role-playing reward model and a Rewriter that uses performance gaps to create actor-specific hard scenarios.A rewrite is useful when it lowers the current Actor’s score relative to the original scenario, steering training toward under-mastered character-context regions.Source: Introduction · p. 2
  • AdvRole outperforms baselines on three English- and Chinese-language role-playing benchmarks and demonstrates effectiveness on a newly released multilingual benchmark.The multilingual benchmark is released to enable broader evaluation.Source: Introduction · p. 2

Related Work

Related work spans prompting and fine-tuning role-playing agents, label-free reinforcement learning, and LLM self-play. AdvRole distinguishes itself by evolving role-playing scenarios despite the lack of deterministic verification for its outputs.

Role-Playing Agent

Label-Free Reinforcement Learning

  • Label-free RL derives rewards without ground-truth answers or human preferences, while role-playing has no single target response and preference labels are costly.Methods use confidence, consistency across reasoning paths, output entropy, or model-generated tasks; they are mostly studied in verifiable domains (Li et al. 2025; Prabhudesai et al. 2025; Zuo et al. 2026; Zhang et al. 2026; Prasad et al. 2025; Agarwal et al. 2026; Cheng et al. 2026; Zhao et al. 2026; Zhou et al. 2026).Source: Label-Free Reinforcement Learning · p. 2

Self-Play in LLMs

  • LLM self-play co-evolves interacting roles; unlike verifier-based code and math frameworks, AdvRole applies the paradigm where outputs lack deterministic verification.Examples pair code generators with testers and math proposers with solvers (Zhao et al. 2026; Huang et al. 2026).Source: Self-Play in LLMs · p. 2

Method

AdvRole alternates Actor training, adversarial rewriting, and scenario-pool augmentation. The evolving pool is designed to expose the current Actor’s weaknesses.

Overview

  • Each AdvRole epoch trains the Actor, trains the Rewriter to expose its failures, then adds rewritten scenarios to the pool for the next epoch.The Rewriter’s reward is based on the Actor’s performance gap between original and rewritten scenarios.Source: Overview · p. 2

Actor Training

  • The Actor uses GRPO on profile-and-dialogue scenarios, with role-playing-quality rewards reduced as responses exceed length thresholds.The length-decay thresholds (τ1, τ2) are (50, 100) for English and (60, 120) for Chinese, measured in words for English and characters for Chinese.Source: Actor Training · pp. 2–3

Adversarial Rewriter Training

  • The Rewriter edits original scenarios to expose Actor weaknesses, with a reward based on the Actor’s performance gap between original and rewritten versions.It averages three Actor responses per scenario to stabilize the difficulty estimate, and receives a reward of −5 if a rewrite exceeds 150% of the original length.Source: Adversarial Rewriter Training · p. 3

Scenario Augmentation

  • For each epoch, the updated Rewriter rewrites scenarios from the initial pool, and those variants join the training pool for the next epoch.Rewriting from the original pool keeps augmentations grounded in real profiles and contexts and avoids drifting too far from the data distribution.Source: Scenario Augmentation · pp. 3–4

Experiments

AdvRole achieves the best overall scores on CharacterEval and CoSER, leads in zero-shot average performance on RAIDEN, and benefits from jointly editing profiles and dialogue contexts with a co-trained Rewriter.

Implementation Details

  • AdvRole is separately trained on English CoSER and Chinese CharacterEval, then evaluated zero-shot on Chinese RAIDEN using the CharacterEval-trained model.The CoSER train/test sets contain 17,762/200 scenarios; CharacterEval uses 2,000 training and 2,564 test scenarios.Source: Implementation Details · p. 4

Training Scheme

  • Actor and Rewriter start from the same LLM and are fine-tuned separately, alternating with scenario augmentation for three epochs.Training uses GRPO with batch size 32, learning rate 1 × 10−6, and 16 H20 GPUs; the Rewriter reward samples three Actor responses per scenario.Source: Training Scheme · p. 4

Evaluation Setup

  • CoSER scores 30-turn dialogues on four dimensions from 0–100, CharacterEval scores single responses on 12 metrics grouped into three dimensions, and RAIDEN uses six binary-scored dimensions.CoSER uses GPT-4o as dialogue partner and judge; CharacterEval uses BaichuanCharRM; RAIDEN uses GPT-4o and reports scores in [0, 1].Source: Evaluation Setup · pp. 4–5

Main Results

  • AdvRole achieves the best overall CharacterEval score with both base models and the best CoSER average, improving all four LLM-judged dimensions.The fine-tuned Qwen2.5-14B-Instruct also surpasses several larger LLMs on CharacterEval overall, while AdvRole is slightly below CogDual-RL on BLEU and ROUGE-L.Source: Table 2 · p. 5, Main Results · p. 5

Cross-Benchmark Generalization

  • On zero-shot RAIDEN, AdvRole has the best average and leads on SCK, RCB, and TA, while VeriRole leads on SBK and CM.The models are trained on CharacterEval and evaluated on RAIDEN without further tuning; the dimension pattern aligns with the methods’ different reward targets.Source: Cross-Benchmark Generalization · pp. 5–6, Table 3 · p. 6

Ablation Study

  • Editing both profiles and contexts outperforms editing either alone, and the overall score improves from no Rewriter to frozen to co-trained.N=3 yields the best overall score, although N=2 slightly surpasses it on Character Consistency; the authors use N=3 for a stable reward estimate at moderate rollout cost.Source: Table 4 · p. 6, Ablation Study · p. 6, Table 5 · p. 6

Scenario Pool Diversity

  • Intra-Pool Distance, Embedding Drift, and New Clusters all increase steadily across training epochs, indicating expansion into unexplored semantic regions.These metrics respectively capture pool spread, distance of rewrites from source scenarios, and clusters containing rewritten but no original scenarios.Source: Scenario Pool Diversity · p. 6

Lanobe Benchmark

Lanobe tests multilingual role-playing across five languages, addressing the limits of fixed scenario pools and English- or Chinese-focused benchmarks. AdvRole achieves the best overall score in every Lanobe language.

Lanobe Benchmark

  • Because a fixed training pool covers only a small part of the character-and-situation space, AdvRole adds character-context pairs that expose model failures during training.Source: Lanobe Benchmark · p. 6
  • Lanobe covers Simplified Chinese, Traditional Chinese, Japanese, Korean, and Thai, with 1,000 training and 500 test scenarios per language.Character descriptions and plot summaries from public sources are processed through a multistage LLM pipeline powered by Kimi-K2.6.Source: Lanobe Benchmark · p. 7
  • Kimi-K2.6 judges responses on Character Consistency, Situational Coherence, Linguistic Quality, and Interaction Proactivity, each scored from 0–10; the final score is their arithmetic mean.Source: Lanobe Benchmark · p. 7
  • AdvRole’s gains are clear in Chinese but smaller in Korean and Thai, suggesting that base-model language ability also affects how well targeted training scenarios translate into role-playing gains.The text says stronger language knowledge lets RL better turn targeted scenarios into gains, whereas weaker target-language ability leaves less room for the same training signal.Source: Lanobe Benchmark · p. 7
  • AdvRole achieves the highest average score across all five languages and the best overall score in each, while VeriRole leads Korean and Thai on Character Consistency.The authors attribute AdvRole’s overall results to broader improvement across all four scoring dimensions.Source: Table 7 · p. 7, Lanobe Benchmark · p. 7

Conclusion

AdvRole addresses the limitations of fixed role-playing scenario pools by using an adversarial Rewriter to create Actor-specific hard scenarios. Experiments on three benchmarks, including a newly released multilingual benchmark, show that AdvRole outperforms baselines.

  • AdvRole trains a Rewriter to edit character profiles and dialogue contexts into scenarios that expose the Actor’s weaknesses, using the performance gap between original and rewritten scenarios as reward.Source: Conclusion · p. 7
  • AdvRole outperforms baselines on three role-playing benchmarks, including a newly released multilingual benchmark.Source: Conclusion · p. 7

Related papers

Loading 2609.28609v1…