PaperlayerView exact revision on arXiv

Source-linked AI summary

KuaiRP Series Role-playing Models Technical Report

Yipeng Wang, Ziwei Zhang, Jiahui Zhang, Qi Gan, Kai Sheng

arXiv:2609.11127v1cs.AIcs.CL

TL;DR

Dedicated role-playing models must provide high-fidelity domain behavior while retaining general capabilities, but injecting world knowledge can cause catastrophic forgetting. KuaiRP addresses this with an SFT → RL → two-stage OPD pipeline with CDD, achieving state-of-the-art target-domain role-playing while retaining base-model capabilities and enabling efficient deployment.

  • Problem

    Dedicated role-playing models are needed for immersive applications, but deep world-knowledge injection can limit general capabilities and prompt quality varies across character creators.

  • Method

    KuaiRP uses an SFT → RL → OPD pipeline that distills from a domain-adapted teacher into the original base-model student, with CDD mitigating prefix-drift noise during on-policy distillation.

  • Results

    KuaiRP achieves state-of-the-art role-playing performance in target-domain scenarios while retaining the base model's tool-calling and reasoning capabilities.

  • Takeaways & Limitations

    The proposed SFT → RL → two-stage distillation route supports high-fidelity role-playing with small size and high-efficiency deployment.

  • Takeaways & Limitations

    LoRA has limited world-knowledge injection capacity because of its low-rank bottleneck, while larger learning rates introduce overfitting without significant benefits over full-parameter SFT.

Abstract

from arXiv · show

This paper introduces the complete technical solution for the KuaiRP series of role-playing models. We aim to achieve four core objectives for a dedicated role-playing model: simplified prompt engineering, highly stable output quality, built-in domain world knowledge, and high-efficiency deployment with a small parameter size. However, effectively injecting deep domain knowledge often leads to a severe catastrophic forgetting of the model's general agent capabilities. To overcome this trade-off, we propose a multi-stage training pipeline. First, we design a standardized character template and construct an SFT data pipeline based on user behavior simulation and reverse profile filtering. Next, we utilize a rule-based composite reward function during the Reinforcement Learning (RL) phase to eliminate common degradation phenomena like length expansion and repetitive generation. Finally, to recover the general capabilities compromised during SFT and RL, we propose a novel self-distillation paradigm using Two-stage On-Policy Distillation (OPD) equipped with Cumulative-Divergence Decay (CDD). By using the domain-adapted model as the teacher and the original base model as the student, we effectively balance deep domain knowledge injection with the preservation of general agent capabilities. Experimental results demonstrate that the KuaiRP models not only match the current state-of-the-art proprietary models in role-playing fidelity within our target domains, but also successfully recover general agent capabilities, maintaining extremely low deployment costs.

Appendices

KuaiRP targets dedicated role-playing models that simplify prompting, stabilize dialogue quality, internalize domain knowledge, and remain efficient to deploy. Its pipeline combines standardized character prompting, simulated-data construction, RL correction, and self-distillation to balance domain adaptation with general capabilities.

  • 1 Introduction: Dedicated role-playing models target simpler prompting, stable output quality, built-in domain knowledge, and deployment on consumer-grade hardware.The motivation frames these four objectives as requirements for immersive application scenarios.
  • Data pipeline: The pipeline combines commercial-model distillation, simulated user behavior, and reverse profile filtering to generate diverse, accurate character-following SFT data.Behavior simulation includes realistic interaction patterns and domain-knowledge queries, while filtering retains profile items reflected in responses.
  • Self-distillation: Two-stage OPD uses the domain-adapted model as teacher and the original base model as student to preserve general capabilities while transferring domain knowledge.The approach is designed to address the trade-off between specialized knowledge injection and catastrophic forgetting.
  • Self-distillation: CDD mitigates prefix drift in on-policy distillation by smoothly decaying weights according to causal divergence, avoiding forced learning from noisy teacher responses.The method is presented as a way to stabilize and thoroughly inject world knowledge while respecting autoregressive causality.
  • Results: KuaiRP achieves state-of-the-art role-playing fidelity in its target domains while maintaining general agent capabilities and low deployment costs.The reported scope includes comparisons with proprietary role-playing models and preservation of capabilities such as tool use and reasoning.
  • 2.1 Motivation: A fixed role-play template standardizes character descriptions and helps models interpret settings consistently across characters.The template supports comprehensive character specification and a unified prompt structure.

3 SFT

The SFT pipeline constructs standardized character-user dialogues using simulated roles and varied user behaviors, then filters profiles to align training inputs with demonstrated responses. Stratified turn sampling addresses long-dialogue degradation, while full-parameter SFT is selected for deeper world-knowledge injection than LoRA.

  • Data generation: The SFT pipeline fills a role library, simulates character-player interactions, and converts the resulting multi-turn dialogues into training data.A commercial model plays characters while a user model plays players.
  • Data generation: Ten user-profile styles diversify simulated conversations with behaviors such as emojis, fragmented sentences, missing punctuation, and internet slang.The profiles are intended to better cover common real-chat patterns than a single user model.
  • Data generation: Instruction injection adds uncooperative behavior, topic shifts, typos, domain-knowledge questions, tool calls, and adversarial attacks to simulated user turns.These behaviors help distill character-following ability while integrating target-domain world knowledge.
  • Data generation: Reverse profile filtering removes profile items not reflected in a dialogue, improving the correspondence between profile inputs and character responses.The RPF-based SFT model achieves better character consistency than the model without RPF.
  • Data generation: Stratified sampling across dialogue-turn groups mitigates degradation in long-turn conversations while keeping total token consumption roughly constant.The redesign followed observed performance degradation beyond ten turns and produced adequate coverage across turn-length intervals.
  • 3.2 Fine-tuning Process: LoRA learns dialogue formatting and is less prone to overfitting, but its low-rank bottleneck limits world-knowledge injection; therefore, full-parameter SFT is chosen.A larger LoRA learning rate increased knowledge injection but caused obvious overfitting without significant benefits over full-parameter SFT.

4 RL

The RL stage targets length expansion, formatting errors, and repetitive speech identified in the SFT model. A rule-based composite reward is appropriate because these intermittent failures can be evaluated with clear binary signals, and taking the minimum reward prevents shortcuts across constraints.

  • RL motivation: The SFT model exhibits length expansion, formatting errors, and repetitive speech that reduce usability and dialogue naturalness.Later turns become excessively long, markup can fail to parse, and historical content may be copied verbatim.
  • RL motivation: These degradation issues are suitable for RL because they occur intermittently and can be evaluated by rule-based binary feedback.The model has partial capabilities rather than being entirely unable to satisfy the constraints.
  • Reward design: The composite reward integrates format, length, and diversity signals for role-playing dialogue generation.The three sub-rewards are identified as rformat, rlength, and rdiversity.
  • Reward design: Taking the minimum of the three sub-rewards enforces a no-shortcut rule: the model must satisfy all hard constraints simultaneously.A high score in one dimension cannot compensate for a low score in another.

1. Format Reward (rformat)

The RL reward design uses binary constraints on formatting, length, and diversity, combining them into a composite score for policy updates.

  • 1. Format Reward (rformat): Only responses satisfying every formatting rule receive rformat = 1; any violation produces zero reward.The format reward covers action-description markup and colon usage.
  • 2. Length Reward (rlength): The length reward constrains non-action character counts to discourage both minimal responses and overly long monologues.It removes action markup before counting non-action characters.
  • 3. Diversity Reward (rdiversity): The diversity reward compares current and historical responses using character-level 2-gram Jaccard similarity between valid sentences.Sentences shorter than five characters are discarded before comparison.
  • 3. Diversity Reward (rdiversity): When either response set has no valid sentences, the diversity reward gives full credit without comparable content.Otherwise, the reward uses the maximum pairwise similarity.
  • 3. Diversity Reward (rdiversity): The diversity threshold is τ = 0.4: similarity above 0.4 receives zero, while similarity at or below 0.4 receives one.This hard threshold determines whether the response meets the novelty requirement.
  • 4.2.3 Composite Score: The composite score combines rformat, rlength, and rdiversity as three binary sub-rewards.The resulting scalar reward is used to update the policy.
  • 4.2.3 Composite Score: A positive composite reward requires simultaneous compliance with formatting, length, and novelty constraints; any failure yields zero and enters the negative sample pool.These samples participate in the policy-gradient update.
  • 4.3 Training Recipe: Table 2 summarizes the key RL training hyperparameters, while framework-component implementation details are omitted.The training recipe retains only selected hyperparameters.

5 OPD (On-Policy Distillation)

On-Policy Distillation transfers role-playing behavior and domain knowledge from a domain-adapted teacher to the original base-model student while seeking to preserve general capabilities. The paper addresses OPD’s learning-efficiency and prefix-drift problems with a two-stage scheme and Cumulative-Divergence Decay.

  • Overview of OPD Method: OPD distills teacher token-level preferences on rollouts sampled from the student, avoiding the distribution shift of offline SFT distillation.The student updates its parameters using per-token divergence or related estimators while sampled rollouts are treated as fixed.
  • Overview of OPD Method: GKD directly minimizes forward KL over student-induced states, while K1 uses a single-sample reverse-KL estimator that intervenes more gently and precisely.GKD more strongly covers the teacher distribution but has a larger impact on the student’s original distribution.
  • Encountered Problems: OPD-PG barely learns world knowledge because required domain token sequences rarely appear in the base model’s sampling distribution, whereas OPD-GKD learns it but can cause overfitting and reduced role-playing capability.These contrasting failure modes motivate combining the two OPD variants across training stages.
  • Cumulative-Divergence Decay (CDD): CDD reduces supervision weights after severe prefix drift, preserving strong signals on reasonable prefixes while exponentially suppressing signals after fatal drift.It uses local Top-K forward KL divergence and a global absolute exponential decay mechanism.
  • Two-Stage OPD Strategy: Two-stage OPD injects domain role-playing capabilities and knowledge into the student while preserving its general capabilities.Stage 1 uses OPD-PG on the complete dataset for formatting and dialogue style; Stage 2 uses GKD-OPD on world-setting samples for focused knowledge injection.

6 Discussion: Evaluation and Key Findings

The evaluation compares successive training stages with the base model across general-agent, general role-playing, domain role-playing, and domain safety and knowledge dimensions. Results are reported on a 100-point scale, with bold and underlined values marking the best and second-best results within each model group.

  • Evaluation scope: The pipeline is evaluated across general-agent capabilities, general and domain role-playing, and domain safety refusal and world knowledge.The comparison includes the original base model and stages from SFT through RL and two OPD stages, with and without CDD.
  • Evaluation protocol: All metrics use a 100-point scale with higher values better, while bold and underlined entries indicate the best and second-best results within each model group.The base models receive extra system-prompt length limits because otherwise they produce excessively long texts that sharply reduce scores.

1. SFT: Substantial Improvement in Domain and Safety Refusal Capabilities, Significant Degradation in General Agent Capabilities

SFT substantially improves domain role-playing, safety refusal, and world-knowledge mastery, but significantly degrades general agent capabilities. It also introduces output-length problems while leaving some role-playing consistency dimensions relatively stable.

  • SFT effects: Domain role-playing capabilities improve significantly after SFT, while the Length dimension declines because outputs become overly long.The result indicates that domain data injects world knowledge but also produces a length-related degradation.
  • SFT effects: General Agent capabilities show significant degradation after SFT, indicating harm from fine-tuning on role-playing data.The degradation is reported on Table 5.
  • SFT effects: Domain safety refusal capabilities and world-knowledge mastery both increase substantially after training on role-playing data.The evaluation specifically reports improved refusal capabilities for high-risk dialogues.
  • RL effects: RL preserves the degraded level of general Agent capabilities while improving Diversity, Length, and LangQuality in general and domain role-playing.The reward function is reported to reduce length expansion and repetitive speech without materially changing consistency or domain safety and knowledge performance.

3. OPD Stage 1 (PG): General Agent Capabilities Fully Recovered, Role-playing Capabilities Maintained

OPD Stage 1 fully recovers general agent capabilities to a level comparable with the base model while maintaining strong role-playing performance. The findings support online distillation as a compensation method after domain adaptation, while CDD further improves the thoroughness and stability of knowledge injection.

  • OPD Stage 1: General and domain role-playing capabilities remain strong after OPD Stage 1, while domain safety refusal remains strong but world-knowledge injection is limited.These outcomes are reported across the general, domain, and safety evaluations.
  • OPD Stage 1: General Agent capabilities fully recover after OPD Stage 1 to a level on par with the base model.The reported recovery follows the substantial degradation caused by SFT.
  • OPD Stage 1: Online distillation uses the domain-adapted model as teacher and the original base model as student to balance domain adaptation with preservation of general capabilities.The paper presents this as a low-cost general training paradigm following SFT and RL.
  • CDD effects: CDD produces superior domain character consistency and significantly smaller fluctuations than the corresponding model without CDD.The result supports cumulative-divergence weighting as a way to make distillation more thorough at positions with reasonable prefixes.

7 Benchmark and Evaluation

The benchmark combines single-turn safety and domain-knowledge tests with multi-turn agentic evaluations of character adherence and memory consistency. Scores are produced from binary judgments or checklist completion and summarized using macro averaging, alongside language-quality metrics.

  • Single-turn evaluation: Single-turn samples combine safety-related requests with character cards to evaluate high-risk refusal behavior and domain-world knowledge.The evaluation includes adult refusal, political refusal, adversarial robustness, and domain knowledge metrics.
  • Multi-turn evaluation: Multi-turn samples use prebuilt task plans and agentic user interactions to evaluate fine-grained character adherence and consistency throughout dialogue.The evaluation produces complete dialogue trajectories and final checklist status.
  • Scoring: Single-turn metrics receive binary 0/1 judgments and are displayed on a 0-or-100 scale, while overall results use macro averaging across samples.The macro score represents the model’s average score when a sample is randomly selected.
  • Multi-turn evaluation: TRACE-Character-Consistency measures completion of character-setting checklists, while TRACE-Memory-Consistency measures recall of established facts across turns.The checklists cover personality, speaking style, behavioral patterns, knowledge boundaries, and short-term memory consistency.
  • Scoring: Three additional language-quality metrics are calculated from the multi-turn evaluation data.

1. Length

The length evaluation measures assistant replies turn by turn using language-adaptive thresholds and excludes very short sentences from repetition analysis.

  • Length: Length evaluation scores turns individually, with sentences as comparison units and turns as scoring units.Each assistant turn produces one score, while sentences provide the units for sentence-level processing.
  • Length: Each assistant turn receives a length score based on language-specific word or character-count thresholds.English-dominant turns use 4–80 words; otherwise, CJK or non-space character counts use 15–150.
  • Length: Short sentences under five characters are excluded, and sentence boundaries use 。！？；!? and newline symbols.

3. Language Quality (LangQuality)

The section defines turn-level language-quality scoring and situates it within a broader evaluation of KuaiRP’s capabilities, contributions, and limitations.

  • 3. Language Quality (LangQuality): Language quality is judged turn by turn as binary fluency, grammar, wording, and semantic-completeness decisions.Turns score 0 for obvious grammatical errors, stacked typos, or incomplete semantics; otherwise they score 1.
  • 3. Language Quality (LangQuality): The final sample score for each metric averages turn-level scores and multiplies them by 100.
  • Contributions: The proposed pipeline combines user-behavior data construction, same-base-model self-distillation, and CDD to preserve general capabilities during domain adaptation.
  • Results: KuaiRP reports state-of-the-art target-domain role-playing while retaining base-model tool-calling and reasoning capabilities at small deployment size.
  • Limitations: The authors identify weaker adaptation gains on Qwen3.5-9B and limited coverage of subjective role-playing qualities such as interestingness.

A Role Play Template Structure

The role-play template organizes a character’s identity, speech, personality, relationships, behavior, capabilities, and example dialogues, with optional world and player settings.

  • A Role Play Template Structure: The template begins with an instruction to play the specified character and converse with the player, optionally using a world setting.
  • A Role Play Template Structure: Character identity includes a name, brief and detailed descriptions, and past experiences or background shaping personality.
  • A Role Play Template Structure: Speaking style and personality traits specify the character’s tone, demeanor, communication patterns, and distinctive traits.
  • A Role Play Template Structure: Optional relationship and behavioral-pattern fields describe the player connection and recurring abilities or behaviors.
  • A Role Play Template Structure: Abilities and knowledge boundaries state what the character knows or can do and what lies beyond its expertise.
  • A Role Play Template Structure: Dialogue examples demonstrate the character’s speaking style and personality across different scenarios, optionally including gestures or demeanor.
Loading 2609.11127v1…