Source-linked AI summary
ArcANE: Do Role-Playing Language Agents Stay in Character at the Right Time?
Woojung Song, Nalim Kim, Sangjun Song, Chaewon Heo, Jongwon Lim, Yohan Jo
TL;DR
Role-playing agents must reflect how a character’s values and behavior change over a narrative, but existing evaluation emphasizes point-in-time factual recall rather than psychological trajectory. ARCANE constructs phase-based Character Arcs and same-scenario probes across source-grounded and novel situations; Arc-grounded context leads every model, with the advantage widening beyond the source text.
Problem
Existing role-playing evaluation emphasizes point-in-time factual recall, while characters’ values and behavioral patterns can shift across a narrative and require responses fitting their state at each moment.
Method
ARCANE constructs psychological Character Arcs with narrative phases and probes each arc by asking the same scenario at every phase, including source-grounded, in-world, and out-of-world situations.
Results
Arc-grounded context achieves the strongest Overall score for every model, and its advantage increases from In-Scenario to In-World and Out-of-World probes.
Takeaways & Limitations
The results support evaluating whether agents track a character’s changing state, especially when retrieval cannot provide a matching source situation.
Takeaways & Limitations
The evaluation is English-only, limited to novels, and focuses on a single character’s behavior without capturing user-character or character-character interactions.
Abstract
from arXiv · showhide
Role-playing language agents (RPLAs) should play characters whose values and behavior evolve as the story progresses, not maintain a fixed persona. Existing benchmarks measure factual recall at a given chapter, not whether responses align with the character's psychological trajectory, especially in scenarios the source text never explores. We introduce ArcANE (Arc-Aware Narrative Evaluation), an automatically constructed benchmark spanning 17 novels and 80 principal characters. A Character Arc segments the narrative into phases along a psychological axis, and each probe poses the same scenario across phases, spanning both situations within the source text and situations beyond it. Across six models and six context modes, conditioning on the Character Arc tops every other context strategy on every model, and the gap is largest on scenarios outside the source text where retrieval has nothing to find. We further fine-tune open-weight models on the same data to obtain ArcANE-8B/32B, which widen the Arc advantage even more on scenarios outside the source text.
1 Introduction
ARCANE addresses whether role-playing agents track characters’ evolving psychological states rather than reproduce fixed personas or factual knowledge. It introduces Character Arcs and phase-specific probes across source-grounded and novel situations, then evaluates context strategies and trained models.
- Existing narrative-time benchmarks primarily test factual recall or closed-form decisions, leaving phase-appropriate behavioral change insufficiently measured.
- Character Arcs organize evolving psychological states into phases, while probes repeat one scenario across phases with phase-specific reference responses.
- The dataset contains 544 Character Arcs and 4,601 probes across 17 novels and 80 principal characters, including situations absent from the source text.
- Across models and context modes, Arc-grounded context performs best, with larger advantages beyond the source text; ARCANE-32B further extends this advantage.
2 Related Work
Prior RPLA benchmarks cover traits, factual knowledge, style, and consistency, but narrative-time evaluation has largely focused on recall or fixed decisions. ARCANE instead targets behavioral change along a character’s accumulating narrative events.
- RPLA research supports applications including interactive fiction, game NPCs, emotional companions, and personalized assistants.
- Users seek an immersive living character, yet agents can lose persona grounding and produce generic responses despite intact context.
- Existing benchmarks assess trait inventories, factual knowledge, persona-driven decisions, linguistic style, conversational competence, or behavioral consistency across turns.
- Simulation and event-conditioned benchmarks capture interaction variation or generic behavior chains rather than one character’s evolution through accumulating narrative events.
- Narrative-time benchmarks have remained limited mainly to closed-form decisions or factual recall.
3 ARCANE
ARCANE represents character development as phase-segmented psychological trajectories and tests whether agents produce the appropriate response at each phase. Its automatically constructed dataset combines validated arcs, cross-phase probes, and training data for contrastive fine-tuning.
- ARCANE evaluates whether an RPLA’s actions and motivations shift with a character’s evolving personality across narrative phases.
- Character Arc Construction: Each Character Arc aligns key events with psychological states in a trajectory from an initial state to a final state.
- Character Arc Construction: Arc construction generates candidate intrapersonal and relational axes, reconciles them, and validates retained axes with critics or human annotators.
- Probe Generation: Each probe repeats one scenario across phases, using one phase-specific reference per phase to distinguish trajectory tracking from fixed-persona responses.
- Probe Generation: In-Scenario, In-World, and Out-of-World probes form a difficulty gradient from source passages to situations beyond the source text.
- Dataset: ARCANE covers 17 novels, 80 characters, 544 arcs, and 4,601 probes across training, validated evaluation, and low-popularity slices.
- Training Formulation: ARCANE’s multi-phase probes also provide contrastive pairs for training, and SFT followed by DPO teaches models to reject behavior displaced from the queried narrative phase.
4 Experiments
Experiments compare six models across six context modes using phase-level and trajectory-level fidelity metrics. Arc context is consistently strongest, especially for scenarios without retrievable source passages, and the pattern extends across novels, model families, and fine-tuned ARCANE models.
- Experimental Setup: The validated evaluation slice contains five novels, 25 principal characters, 205 arcs, and 1,754 probes.
- Experimental Setup: Six models include four open-weight baselines and ARCANE-8B/32B, evaluated with Vanilla, Summary, RAG, LifeChoice, TimeCHARA, and Arc context.
- Evaluation Protocol: The evaluator scores free-form responses on a 1–100 scale using APF, RPF, RAE, and the trajectory metric PTF.
- Main Results: Arc is the top Overall mode for 29 of 30 model–novel cells and leads every model by 2.2 to 8.4 points over its strongest non-Arc mode.
- Main Results: For DeepSeek-V4-Pro, the Arc-versus-best-non-Arc Overall gap is +0.5 In-Scenario, +5.2 In-World, and +7.7 Out-of-World.
- Trajectory Results: For DeepSeek-V4-Pro, Arc’s PTF gaps are +1.1 In-Scenario, +7.0 In-World, and +8.2 Out-of-World, exceeding matched per-phase metric gaps.
- Additional Results: On two low-popularity novels, Arc remains best for all six models, with lifts of +4.1 to +15.3 over the strongest non-Arc mode.
- Additional Results: Across added role-playing models, In-World and Out-of-World lifts are non-negative; DPO raises ARCANE-32B’s combined lift to +12.5 while reducing In-Scenario lift from −2.2 to −5.6.
5 Analysis
The ablations show that Arc’s gains come from character-specific phase information rather than merely structured context or source leakage. For untrained models, compact axis-and-phase labels suffice, whereas ARCANE-32B-DPO benefits from per-phase prose during training.
- Source of the Arc lift: MIXEDARC fails to clear Vanilla and falls −4.7 Per-Phase Average and −6.5 PTF below Vanilla for ARCANE-32B-DPO.Using another character’s arc actively misleads the trained model, arguing against a generic structured-context bonus or source-leakage explanation.
- Source of the Arc lift: ARCHINT tracks Arc within ±2.6 points on every metric for DeepSeek-V4-Flash and Qwen3-32B.For these untrained models, the axis label paired with the current-phase index carries the full prompting advantage.
- Metric sensitivity: PTF Overall drops −8.8 to −23.0 points on ARCANE-32B-DPO under response-sequence perturbations, versus −1.3 to −6.3 on DeepSeek-V4-Pro.The larger degradation on the model with stronger per-phase anchoring is consistent with sensitivity to phase-order corruption.
- Training effect: ARCANE-32B-DPO beats Qwen3-32B on 1,198/1,750 validated probes, with mean Overall ∆=+9.49.A register-control test lowers Qwen’s Overall to 50.0 from 53.8 while ARCANE remains at 56.7, and the surviving gap is attributed to category-conditional canon discipline.
- Training effect: DPO beats SFT on 940/1,750 probes and shifts responses with the character arc rather than maintaining one static voice across phases.On Don Quixote’s interdependence axis, DPO drops the knightly tag in the repentant phase and turns toward family.
6 Conclusion
ARCANE evaluates whether role-playing agents track the right character state at the right narrative point. Arc-grounded context leads across models and contexts, including held-out low-popularity novels, while multi-turn dialogue remains a next step.
- Conclusion: ARCANE asks whether role-playing language agents track the right character state at the right point in a narrative.
- Conclusion: Arc-grounded context tops Overall on every model, with the gap widening from In-Scenario to Out-of-World where retrieval has nothing to find.
- Conclusion: The lift carries to two held-out low-popularity novels at +4.1 to +15.3 over the best non-Arc mode, while ARCANE-32B models extend it further.
- Future direction: Multi-turn dialogue, where the arc itself advances through interaction, is identified as the natural next step.
Limitations
ARCANE’s limitations concern scope, interaction coverage, cultural provenance, and several evaluation or construction assumptions. The benchmark is restricted to English novels and single-character behavior, while some validation and modeling choices remain bounded.
- Scope: The dataset is entirely English and limited to novels, excluding other languages, domains, and shorter narrative formats.The authors justify novels by their extended character evolution across many chapters.
- Scope: The evaluation models one character’s behavior over accumulating events, not user-character or character-character interactions.
- Cultural scope: The corpus draws on nineteenth- and early-twentieth-century novels, so trained models may reproduce period-bound social attitudes and released artifacts are intended for research only.
- Construction: The benchmark’s construction separates event omission from state misreading through parallel streams, but downstream analyses still depend on reconciliation and filtering decisions.Stream-unique candidates may be kept, flagged, or discarded according to evidence and post-processing tags.
- Validation: Only axis validity gates evaluation-set entry; other human ratings are stored as metadata rather than selection criteria.Validity uses a 2-of-3 annotator majority, while novel span and importance support later stratification.
C Datapoint information
ARCANE’s corpus uses hierarchical Character Arc and probe records to represent phase-specific character behavior and evaluate it across shared scenarios. Its training data and inference comparisons are organized around trajectory-aware context and two-stage model training.
- Corpus schema: Each Character Arc record fixes one character and psychological axis, with poles, direction, source, and an ordered phase trajectory.The arc is indexed by axis_id and may represent intrapersonal or relational change.
- Corpus schema: Each probe pairs one scenario-question with N phase-keyed reference responses, including action, speech, thought, and typicality fields.Out-of-World probes additionally encode an era and abstract axis-phase reformulations.
- Probe design: Figure 7 illustrates an Out-of-World probe asking the same scenario at every phase while phase-specific references shift along Anna Karenina’s Agency–Communion arc.
- Training data: The SFT dataset contains 45,690 rows from ten training novels and uses Arc and Summary context modes with completions pooled from two teacher models.
- Training data: DPO pairs an anchor-phase response with an adjacent-phase rejection sharing the same scenario and question, isolating developmental disposition as the contrast.The resulting set contains 14,671 preference pairs across 2,516 probes.
- Training pipeline: The two-stage pipeline uses SFT to learn ARCANE’s response conventions and DPO to distinguish target-phase behavior from temporally displaced alternatives.
G Reward-Hacking Falsification
The reward-hacking analysis tests whether ARCANE’s gains reflect substantive phase tracking rather than first-person style. Pair-trace coding, POV control, and rubric results support gains in phase-specific content, canonical specificity, negative-space fidelity, and phase-register changes, while exposing a limited diagnostic category.
- Falsification design: The analysis codes 150 probes across three model conditions to test phase distinctness, canonical specificity, negative-space fidelity, and phase-register switching.The conditions are original Qwen3-32B, POV-controlled Qwen3-32B, and ARCANE-DPO.
- Pair-trace findings: ARCANE produces substantively different phase content, whereas Qwen3-32B often paraphrases one generic response across phases.On Carrasco–Sancho, ARCANE moves from amused engagement to practical incorporation.
- Pair-trace findings: ARCANE preserves canonical specificity and negative space by routing through named characters or source-specific scenes, including silence or proxy when canon requires it.
- Pair-trace findings: ARCANE changes voice and register with phase, while Qwen3-32B maintains one composed-narrator register across phases.Darya’s examples shift from emotional collapse to late steady endurance.
- Caveat: P3 applies to only 2 probes in the stratified sample, so negative-space fidelity is reported qualitatively rather than through a broad quantitative comparison.
- POV control: 56.7 versus 53.8 and 50.0: ARCANE-32B-DPO scores higher than Qwen3-32B and POV-Qwen on the 150-probe rubric, while POV control lowers performance through canon-drop.The Dulcinea case scores 80, 40, and 22 for Qwen3-32B, ARCANE-32B-DPO, and POV-Qwen respectively.
- Analysis refinement: The discarded free-form discovery pass confounded POV style with substantive arc movement, prompting the authors to replace it with close-reading-based coding.
I Additional Novels: Full Results
Additional evaluations test ARCANE on low-popularity novels, alternate model families, and ablations that separate construction, inference, and training contributions. The reported analysis emphasizes that compact arc structure can retain the Arc benefit, while training adds further trajectory sensitivity.
- Additional novels: The low-popularity slice contains The Underdogs and East Lynne, held out from human validation as a memorization control.Their reported download rates are 469 and 1,038 per month, both below the 1,500/month threshold.
- Additional models: The model comparison extends to HER-32B, CoSER-8B, CoSER-70B, and SFT-only ARCANE-8B/32B models on the human-validated slice.
- Ablations: MIXEDARC swaps in another character’s arc, while ARCHINT compresses each arc to axis labels and the current phase index without per-phase prose.ARCHINT contexts are approximately 300 characters versus approximately 10K for full Arc JSON.
- Evaluation design: Ablation probes are sampled across 15 characters and the three probe categories, with all modes scored by the same Stage III judge.
- Ablations: The ablation interpretation partitions contributions into construction, inference, and training, with axis labels plus phase index already carrying the full Arc benefit for untrained models.The per-phase prose is described as redundant for prompting those models.
L Memorization Robustness
The memorization checks indicate that Arc grounding benefits supporting characters and low-popularity novels, where pretraining memorization should provide less help.
- Novel Popularity: Arc wins 29 of 30 model–novel cells on the validated slice and every cell on two low-popularity unvalidated novels.The validated-slice gap is smallest on Harry Potter (+3.0) and larger on Anna Karenina (+5.6) and Don Quixote (+5.1).
- Central vs. Supporting Characters: +13.1 and +15.1 are ARCANE-8B’s central- and supporting-tier lifts, respectively.The largest gains in either tier occur for the model trained on ARCANE data.
- Interpretation: The supporting-tier result concentrates Arc’s lift where memorization explains least.The character-popularity axis mirrors the source-distance axis, with Arc helping where the model needs help.
N PTF Metric Validity
The PTF trajectory metric is sensitive to response ordering while remaining related to per-phase fidelity, and complementary checks support its validity.
- Convergent-but-not-redundant Correlation: 0.51 average Pearson correlation means per-phase scoring explains roughly a quarter of PTF variance, so PTF is convergent but not redundant.Across 72 cells, Pearson ranges from 0.37 to 0.64; ARCANE-32B-DPO under Arc reaches r=0.64.
- Perturbation Tests: −8.8 Overall for shuffling responses and −23.0 for reversing them show that PTF responds to trajectory perturbations.The reversal effect is concentrated on ptf_direction (−24.7), as the rubric predicts.
- Perturbation Tests: −12.5 under block_shuffle indicates that some PTF signal depends on listing order even when reference–response pairs remain intact.The remaining approximately 10 points between block_shuffle and shuffle_response are interpreted as signal beyond positional reading and per-phase fidelity.
- Human-Anchored Validation: 87.1% of 70 samples passed the human 2-of-3 plausibility majority, while the trajectory judge passed at 95.0%.Plausibility was broadly uniform across In-Scenario, In-World, and Out-of-World categories.
O.2 Cross-judge replication on the validated slice
Cross-judge replication shows that the benchmark’s main configuration ranking is stable across independent judges, despite some movement among close competitors.
- Sample: The replication uses a stratified 300-cell sample spanning five novels, four model–mode configurations, and three probe categories.The resulting DS-Flash scores range from 15 to 96, with mean 55.6.
- Cross-Judge Agreement: α≥0.80 for Opus 4.5 and GPT-5.5 on the per-cell average indicates substantial agreement with DS-Flash.Sonnet 4.5 has α=0.60 but Spearman 0.78, preserving rank ordering despite differing absolute levels.
- System Ranking: All four judges rank ARCANE-32B-DPO under Arc first and Qwen3-32B under Vanilla last.The second- and third-place configurations reverse between judges, but the reversal does not affect the headline finding.
- Cross-Judge Agreement: r≥0.81 between DS-Flash and each of Opus 4.5 and GPT-5.5, with α≥0.80, places DS-Flash inside the independent-judge consensus.The two strongest non-DS judges correlate at r=0.92 and α=0.89.