Source-linked AI summary
Deriving Character Logic from Storyline as Codified Decision Trees
Letian Peng, Kun Zhou, Longfei Yun, Yupeng Hou, Jingbo Shang
TL;DR
Existing RP profiles are often unstructured, difficult to execute, and weakly validated, limiting situation-specific character grounding. CDT induces validated hierarchical decision trees from narrative scene-action data and consistently outperforms prior approaches, including human-written profiles. The paper positions codified, data-driven profiles as a basis for more reliable grounding while noting important scope boundaries.
Problem
Existing textual profiles are difficult to verify or execute, while prior storyline profiling can conflate behaviors across different situational contexts.
Method
CDT mines candidate question-statement rules from scene-action data, validates them against the dataset, and recursively organizes them into executable trees for situation-specific grounding.
Results
CDT consistently outperforms previous profiling approaches and even human-written profiles across the evaluated benchmarks.
Takeaways & Limitations
Codified, validated behavioral structures provide interpretable and executable profiles for grounded role-playing inference.
Takeaways & Limitations
CDT currently relies on storyline-derived scene-action pairs, excludes pre-designed persona information, and is constructed offline rather than updated continually.
Abstract
from arXiv · showhide
Role-playing (RP) agents rely on behavioral profiles to act consistently across diverse narrative contexts, yet existing profiles are largely unstructured, non-executable, and weakly validated, leading to brittle agent behavior. We propose Codified Decision Trees (CDT), a data-driven framework that induces an executable and interpretable decision structure from large-scale narrative data. CDT represents behavioral profiles as a tree of conditional rules, where internal nodes correspond to validated scene conditions and leaves encode grounded behavioral statements, enabling deterministic retrieval of context-appropriate rules at execution time. The tree is learned by iteratively inducing candidate scene-action rules, validating them against data, and refining them through hierarchical specialization, yielding profiles that support transparent inspection and principled updates. Across multiple benchmarks, CDT substantially outperforms human-written profiles and prior profile induction methods on $85$ characters across $16$ artifacts, indicating that codified and validated behavioral representations lead to more reliable agent grounding.
1 Introduction
CDT induces executable, situation-specific character profiles from storylines to ground role-playing actions. Its validated, hierarchical structure outperforms prior profiling approaches, including human-written profiles.
- Role-playing systems take scenes as input and generate actions conditioned on character-specific information.
- Plain textual profiles can conflate behaviors across situations, motivating direct induction of codified profiles from storylines.
- CDT recursively induces candidate scene-action rules, validates them against data, and specializes them into a tree of executable behavioral rules.
- At inference, scenes traverse satisfied tree conditions, and statements on visited nodes provide situation-specific grounding while unvisited alternatives are omitted.
- The enriched evaluation includes 45 Fandom characters, 40 Bandori characters, and 77,182 additional event-story conversation pairs.
- CDT substantially outperforms prior textual profiling methods and surpasses human-written profiles on action prediction.
2 Background
RP requires coherent, persona-consistent behavior, but textual profiles are difficult to verify and execute. CDT addresses this gap by inducing interpretable, executable, hierarchical rules directly from narrative data.
- RP systems aim to sustain coherent and persona-consistent behavior across evolving narrative or simulated environments.
- Plain-text profiles are hard to verify or execute and can cause inconsistency, while structured alternatives often require extensive manual definition.
- Grounding conditions agent responses on evolving world facts, social relations, and temporal cues rather than free-form text history.
- Rule mining discovers conditional regularities expressed as if–then statements that govern observable behaviors in data.
- CDT differs from flat rule extraction by recursively inducing validated, hierarchical, executable structures that encode coverage and precision.
3 Codified Decision Tree
CDT converts scene-action data into a traversable tree whose validated questions select grounded behavioral statements. Its construction combines rule mining, recursive validation, and embedding-based clustering.
- 3.1 Preliminary: Codification converts static profiles into executable functions that map scenes to character information for grounding.
- 3.2 Data Structure: CDTs traverse textual scenes through discrimination questions and incorporate grounding statements from every visited node rather than outputting one label.
- 3.3 Recursive Hypothesis-Validation: The CDT workflow begins with scene-action data and mines hypotheses formatted as a filtering question paired with a behavioral statement.
- 3.3 Recursive Hypothesis-Validation: Validation adds globally applicable statements, creates leaf nodes for highly accurate triggers, rejects inaccurate hypotheses, and recursively refines intermediate cases.
- 3.3 Recursive Hypothesis-Validation: Recursion stops when the remaining data subset or tree depth is too small or large, respectively.
- 3.4 Rule Mining: K-Means clusters concatenated scene and action embeddings so an LLM can summarize potential rules while reducing distraction from surface similarity.
4 Benchmark
The benchmarks evaluate action and dialogue-level role-playing using chronologically split narrative data, expanded artifacts, and large event-story conversations. They include fine-grained action sequences and conversational settings.
- Fine-grained Fandom Benchmark: The fine-grained Fandom benchmark breaks narrations into character-annotated action sequences and uses the preceding 10 actions as each scene.
- Fine-grained Fandom Benchmark: Fandom training and test pairs are split chronologically, training on the first half of each storyline and evaluating on the second half.
- Fine-grained Fandom Benchmark: The Fandom benchmark expands the original artifact set with Death Note and Spy × Family for broader evaluation.
- Bandori Conversational Benchmark: The Bandori conversational benchmark collects conversations from the first story of all eight bands to assess dialogue-level role-playing.
- Criterion and Statistics: Across the benchmarks, predicted next actions are compared with references using NLI scores of 100 for entailment, 50 for neutrality, and 0 for contradiction.
5 Experiment
The experiments compare CDT with prompting, fine-tuning, retrieval, textual aggregation, and human-written profiles across Fandom and Bandori benchmarks. CDT consistently achieves stronger action-prediction performance, including against human-written profiles.
- Evaluation and Baselines: CDT compares against comprehensive baselines covering methods that use known plots to ground RP in new scenes.The baselines include Vanilla prompting, fine-tuning, RICL, ETA, and human-profile variants.
- Main Results: CDT(-Lite) achieve the best scores across all 8 Fandom artifacts, surpassing Vanilla, fine-tuning, RICL, and ETA.
- Main Results: CDT(-Lite) obtain higher scores than human-written profiles for every Fandom artifact.The comparison includes both Human Profile and Codified Human Profile baselines.
- Main Results: CDT(-Lite) achieve the top performance for all 8 Bandori bands, with CDT-Lite slightly ahead of CDT in most cases.Traditional grounding methods and human-written or codified human profiles remain behind CDT(-Lite).
6 Analyses
The analyses examine validation, tree depth, clustering, statement selection, textual conversion, scaling, and goal-driven relation modeling. Results support validated hierarchical structure, increased data, and targeted specialization as useful components of CDT.
- Ablation Study: Blindly accepting LLM hypotheses is consistently worse than explicit statement validation in CDT-Lite.
- Ablation Study: RP performance generally improves with deeper CDT traversal, followed by gradual saturation.A maximum depth of dmax = 1 produces only root-level rules.
- Ablation Study: Removing clustering or instruction-following action embeddings consistently lowers performance relative to CDT-Lite.The analysis identifies semantic grouping and instruction-following action representations as crucial for discovering reusable triggers.
- Ablation Study: Usability Rank generally yields the strongest top-k results, prioritizing empirically applicable rules for grounding.Depth Rank favors specificity, while Accuracy Rank orders statements by validated accuracy.
- CDT-to-Wiki: Verbalized and Wikified CDT remain stronger than human-written profiles despite removing runtime traversal and sacrificing fine-grained situational control.
- Scaling Analysis: CDT performance steadily improves as more training data are used across both benchmarks.On Fandom, 64 pairs already surpass human-written profile baselines; Bandori gains continue from 128 to 1024 pairs.
- Relation Modeling: Goal-driven CDT consistently outperforms base CDT on target subsets across all nine character pairs.The relation-specific CDT filters actions following a designated related character and is used when that character supplies the latest context action.
7 Conclusion and Future Work
The paper presents Codified Decision Trees as a data-driven framework for executable, situation-specific character profiles and reports consistent gains over prior methods and human-written profiles.
- CDT induces executable, situation-specific character profiles from storylines.
- CDT consistently outperforms previous profiling methods, including human-written profiles.
- Future work extends CDT to joint multi-character modeling, continual updates, and multimodal behavior modeling.
Limitations
CDT is limited to storyline-derived scene-action pairs and offline construction, excluding pre-designed persona information and evolving narratives.
- CDT currently relies only on storyline-derived scene-action pairs and excludes canonical traits or author intent.
- CDT is constructed offline and remains static rather than updating as new storylines unfold.
- The paper identifies incremental or continual CDT updates as an important direction for future work.
A Statistics Information
The experiments use diverse Fandom and Bandori benchmarks and report compact, shallow induced trees with consistent statement granularity across domains.
- The benchmarks cover diverse narrative lengths, cast sizes, and balanced representation across Bandori’s eight bands.
- Induced CDTs remain shallow and compact despite variation in data volume, with consistent statement lengths across artifacts and benchmarks.
- The main experiments use fixed CDT hyperparameters, including maximum depth 4 and minimum node-growing data of 16.
- Baseline implementations use distinct training and retrieval configurations for Fine-tuning, RICL, ETA, and human-profile methods.
C More Variant Studies
Variant studies examine ablations, RP backbones, codifiers, and hyperparameters, finding strong performance across model sizes and relative robustness to configuration choices.
- Ablation and Variant Studies: The ablation and variant studies reach similar conclusions about component contributions, statement selection, and verbalization.
- RP Model Variants: CDT-Lite delivers large gains over Vanilla and Human Profile across 1B and 3B RP models on both Fandom and Bandori.
- Codifier Variants: GPT-4.1 generally performs best as codifier, while GPT-4.1-mini and Qwen3-Coder remain competitive on both benchmarks.
- Benchmark and Tree Statistics: The experiments summarize benchmark and induced-tree statistics in Tables 5 and 6.
- Hyperparameter Setup: CDT performance is relatively stable across varied hyperparameter configurations, supporting reduced sensitivity to precise tuning.
D Evaluation Generality Validation
CDT is evaluated across matching-based and open-ended role-playing settings, with results indicating consistent gains across behavioral dimensions and out-of-domain data. The matching score is also supported as a reliable automatic evaluator through agreement with human and LLM judgments.
- Figure 4 compares grounding methods using matching scores, while Figure 5 compares open-ended RP through human and LLM judgment.
- CDT shows systematic gains across intent, emotion, content, stance, and causality, with especially strong improvements on intent and causality.These gains are reported across both evaluated benchmarks.
- 89.00% matching-score consistency on “contradicted” and 90.50% on “matched” and “mismatched” support the evaluator’s reliability.The reported consistency confirms both NLI accuracy and aggregated matching score as dependable proxies for RP quality.
- CDT is tested on 200 novel Fandom starting scenes with 10-turn interactions to assess generalization beyond training storylines.Responses are evaluated against the Human Profile baseline by an LLM judge and human judgment.
- CDT consistently outperforms Vanilla prompting and Human Profiles across all eight Bandori bands on wild event stories.The CDTs are trained only on the first half of the main band stories and tested on heterogeneous event-story conversations.
E Cost and Efficiency
CDT efficiency is improved by retaining only important triggers and by replacing expensive LLM validation with distilled discriminators. These choices reduce tree size and inference cost while preserving performance close to CDT-Lite, with Boosted CDT trading some accuracy for additional speed.
- Boosted CDT retains eight important triggers before validation, reducing tree nodes by an order of magnitude while achieving performance close to CDT-Lite.The reduction omits some fine-grained details.
- Validation dominates CDT construction because each hypothesized trigger is tested against all scene–action pairs.This motivates replacing expensive LLM validators with distilled discriminators.
- Distilled discrimination makes CDT practical to train at scale without significant performance degradation.
- CDT traversal requires at most approximately the number of tree nodes in Table 12 and has negligible runtime relative to final RP response generation.Boosted CDT provides additional speedups by shrinking the tree, at the cost of lower accuracy.
- Figure 6 illustrates converting clustered scene–action evidence into reusable IF–THEN hypotheses, while Figure 7 illustrates traversal and guidance collection.The traversal example merges supported THEN statements into response-generation guidance.
- The wikification pipeline linearizes the CDT before prompts propose chapters and fill their contents.