Source-linked AI summary
Guess What I am Thinking: A Benchmark for Inner Thought Reasoning of Role-Playing Language Agents
Rui Xu, MingYu Wang, XinTao Wang, Dakuan Lu, Xiaoyu Tan, Wei Chu, Yinghui Xu
TL;DR
Inner thought processes in role-playing language agents remain unexplored, despite their relevance to character understanding and role-playing. The paper introduces ROLETHINK to evaluate thought generation and proposes MIRROR, which recalls memories, predicts reactions, and synthesizes motivations. Experiments report that MIRROR improves thought generation and that thinking before acting improves downstream task performance.
Problem
Role-playing language agents’ inner thought processes remain unexplored, although generating character thoughts can help analyze motivations and may improve role-playing capabilities.
Source: §1 · pp. 1–2Method
ROLETHINK evaluates generated thoughts against original monologues or synthesized character analyses, while MIRROR retrieves memories, predicts related entities’ reactions, and synthesizes character motivations.
Source: §1 · p. 2Results
MIRROR-enhanced models consistently score higher on ROLETHINK, and enabling agents to think before acting improves performance across downstream tasks.
Source: §1 · p. 2Takeaways & Limitations
Experiments indicate that generating inner thoughts can provide deeper character understanding and analysis while improving role-playing performance across multiple downstream tasks.
Source: §7 · pp. 8–9ROLETHINK draws on a single literary source, so effectiveness across genres, cultural backgrounds, and writing styles remains unexplored; evaluation also assumes references represent characters’ true thoughts.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Recent advances in LLM-based role-playing language agents (RPLAs) have attracted broad attention in various applications. While chain-of-thought reasoning has shown importance in many tasks for LLMs, the internal thinking processes of RPLAs remain unexplored. Understanding characters' inner thoughts is crucial for developing advanced RPLAs. In this paper, we introduce ROLETHINK, a novel benchmark constructed from literature for evaluating character thought generation. We propose the task of inner thought reasoning, which includes two sets: the gold set that compares generated thoughts with original character monologues, and the silver set that uses expert synthesized character analyses as references. To address this challenge, we propose MIRROR, a chain-of-thought approach that generates character thoughts by retrieving memories, predicting character reactions, and synthesizing motivations. Through extensive experiments, we demonstrate the importance of inner thought reasoning for RPLAs, and MIRROR consistently outperforms existing methods. Resources are available at https://github.com/airaer1998/RPA_Thought.
1 Introduction
The paper addresses the unexplored inner thought processes of role-playing language agents, which matter for understanding character motivations and role-playing capabilities. It introduces ROLETHINK and MIRROR, and reports that reasoning before acting improves performance across downstream tasks.
1 Introduction
- Inner thought processes remain unexplored in RPLAs, despite their relevance to analyzing character motivations and role-playing capabilities.The paper also notes limited model capabilities on complex decision-making and psychological reasoning tasks.Source: §1 · pp. 1–2
- ROLETHINK evaluates generated character thoughts against original novel monologues in its Gold Set and synthesized expert analyses in its Silver Set.The benchmark uses automatic metrics and human assessment.Source: §1 · p. 2
- MIRROR retrieves relevant memories, predicts reactions of related characters and environments, and synthesizes these into character motivations.It is proposed as a chain-of-thought approach to thought generation.Source: §1 · p. 2
- Enabling role-playing agents to think before acting improves performance across various downstream tasks.MIRROR-enhanced models also consistently achieve higher scores on ROLETHINK.Source: §1 · p. 2
2 Related Work
Prior work studies character traits, relationships, behavior, and role-playing, but leaves internal psychological processes less directly addressed. This paper positions explicit thought generation as a way to model characters’ memory recall, perspective-taking, and decisions.
2.1 Character Understanding
- Character-understanding research covers traits, relationships, behavioral patterns, and profile summarization, while existing benchmarks mainly assess identification or question answering.The paper frames its focus on inner thoughts as extending beyond these surface-level assessments.Source: §2.1 · p. 2
2.2 Role Playing Language Model
- Role-playing methods improve dialogue consistency through fine-tuning, memory retrieval, and evaluation, but do not address underlying psychological processes.Prior studies report limited consistency in inner reasoning and psychological fidelity.Source: §2.2 · pp. 2–3
- The paper models characters’ recall of relevant memories, consideration of others’ perspectives, and reasoning about decisions.It presents this as a framework for understanding and reproducing character behaviors.Source: §2.2 · p. 3
3 ROLETHINK Benchmark
ROLETHINK asks models to generate character thoughts from a profile and scenario, using two reference sets that differ in source. The benchmark draws on A Song of Ice and Fire and evaluates generations with automatic and human assessments.
3.1 Task Formulation
- Given a character profile and scenario, inner thought reasoning generates the thoughts that led to the character’s behavior.The Gold Set compares generations with original novel thoughts; the Silver Set compares them with fan-community and literary-expert analyses.Source: §3.1 · p. 3
3.2 Gold Set
- The Gold Set masks original character thoughts in novel passages and asks models to generate the missing content.GPT-4o identifies thought segments in point-of-view chapters, which are then manually filtered; the examples include masked thoughts and references.Source: §3.2 · p. 3
3.3 Silver Set
- The Silver Set asks models to generate plausible thoughts at scenario points using references synthesized from character analyses.Fan-site analyses are processed by GPT-4o to summarize motivations and locate thought points, then manually filtered.Source: §3.3 · pp. 3–4
3.4 Dataset
- ROLETHINK uses A Song of Ice and Fire and contains 405 Gold examples from 22 characters and 211 Silver examples from 18 characters.Average scenario and reference lengths are 5,036 and 90 tokens for Gold, and 7,801 and 301 tokens for Silver.Source: §3.4 · p. 4
3.5 Evaluation
- Evaluation combines BLEU and ROUGE-L, model-based assessment, and human ratings.Because Gold references can be accurate but incomplete, the authors add NLI entailment probabilities and GPT-4o coverage scores on a 1–5 scale; five crowdsourced annotators provide human ratings.Source: §3.5 · p. 4
4 MIRROR
MIRROR addresses the challenges of finding relevant evidence in long-context memory and reasoning about related entities’ reactions. It combines memory recall, Theory of Mind thinking, and reflection and summarization to produce character thoughts.
4 MIRROR
- MIRROR has three stages: memory recall, Theory of Mind thinking, and reflection and summarization.The approach targets relevant evidence retrieval and perspective-based reaction reasoning.Source: §4 · p. 4
4.1 Memory Recall
- MIRROR first recalls events related to a scenario, then retrieves memory chunks relevant to each event.It retrieves the highest-scoring chunks using cosine similarity between event and chunk embeddings; each scenario triggers an average of 2.6 related events.Source: §4.1 · p. 4
4.2 Theory of Mind Thinking
- Theory of Mind thinking predicts potential objects in a scenario and analyzes each object’s possible response.Objects include characters, groups, and environments; the method models their perspectives to guide contextually appropriate decisions.Source: §4.2 · pp. 4–5, Figure 2 · p. 5
4.3 Reflection & Summarization
- Reflection filters irrelevant content from retrieved memories and reaction analyses before summarization produces the final thought process.The two stages are intended to maintain coherence and relevance and align the thought with the character’s personality and motivations.Source: §4.3 · p. 5
5 Experiment Settings
The experiments compare character-profile construction methods and base language models, then test thought-generation methods on three downstream role-playing benchmarks.
5.1 Character Profiling
- Character profiles are compared using zero-shot, retrieval-based, long-context, and MIRROR approaches.Zero-shot uses a name and brief introduction; retrieval-based adds three relevant 1k-token memory chunks, while long-context uses prior character POV data averaging 85k tokens.Source: §5.1 · p. 5
5.2 Base Language Models
- The experiments use multiple LLMs, with Claude3.5, Gemini2, and Qwen2.5-14B-1M evaluated for long-context approaches.Other methods also use GPT-4o, GPT-4o1, Qwen2.5-72B, Llama-3.3-70B, and DeepSeek-R1.Source: §5.2 · pp. 5–6
5.3 Downstream RPLA Tasks
- Downstream evaluation covers LifeChoice, CROSS-MR, and RoleEval, using multiple-choice accuracy across all characters.The benchmarks assess decision reproduction, behavior motivations, and basic role-playing abilities, respectively; they provide complete character memories.Source: §5.3 · p. 6
6 Experiment Results
Experiments find that character thoughts improve downstream role-playing performance, while MIRROR performs best among thought-generation approaches. Results also show that thought quality, length, and component choice matter, though models still miss psychological complexity.
6.1 Can LLMs generate high-quality character thought data?
- MIRROR-based methods achieve the best ROLETHINK performance, followed by long-context profiling, while Gold scores are generally below Silver scores.The authors attribute the Gold–Silver difference to the greater difficulty of reproducing original descriptions than generating plausible thoughts.Source: §6.1 · pp. 6–7
- Selective memory retrieval with structured reasoning gives MIRROR the best performance among profiling approaches, while zero-shot performs worst and retrieval-based profiling can miss relevant information.Long-context profiling processes more character information but can suffer attention dispersion.Source: §6.1 · p. 7
- GPT-4o1 and Claude3.5 show balanced performance, while long-text-capable models do especially well in long-context settings and reasoning-focused models do not necessarily excel.The authors note that thought generation relies more on scenario-relevant memories than complex reasoning.Source: §6.1 · p. 7
- Experts found model thoughts more logically connected but less emotionally complex than references, with stronger expression of explicit than implicit emotions.They also report consistent character voice and vocabulary, and stronger narrative coherence in MIRROR-generated thoughts; models still struggle to capture full psychological complexity.Source: §6.1 · p. 7
6.2 Does character thought data improve role-playing performance?
- Character thought data improves all downstream tasks, with larger gains on decision-making and motivation analysis than on knowledge recall and tone consistency.MIRROR-generated high-quality thoughts show larger improvements than simple zero-shot thoughts, and thought processes provide explanation chains for model behavior.Source: §6.2 · pp. 7–8, Table 3 · p. 7
- Longer thoughts generally improve performance, but gains plateau after 300 tokens; decision-making benefits more than knowledge tasks, and MIRROR remains strong at 200 tokens.Other methods require 400+ tokens to achieve similar performance.Source: §6.2 · p. 8, Figure 3 · p. 8
- Removing memory retrieval causes the largest ablation drops, including -0.8 on Gold, -1.0 on Silver, -9.1 on LifeChoice, and -7.3 on CROSS-MR.Theory of Mind removal causes Gold and Silver drops of -0.5 and -0.8; removing summary produces a -0.2 drop across tasks.Source: §6.2 · p. 8, Table 4 · p. 8
7 Conclusion
The paper presents a first systematic study of inner-thought generation for role-playing language agents, introducing ROLETHINK and MIRROR. Its experiments report improved thought generation and downstream performance, while noting that complex psychological processes remain challenging.
7 Conclusion
- MIRROR combines memory retrieval, role reasoning, and observation reflection, and the experiments report significant improvements in character thought generation.The benchmark is constructed from A Song of Ice and Fire and includes direct monologues and expert-analyzed character behaviors.Source: §7 · p. 8
- Experiments with various LLMs indicate that generating complex psychological processes remains challenging.Source: §7 · p. 8
- High-quality thought generation can provide deeper character understanding and analysis while improving RPLAs’ performance across multiple downstream tasks.Source: §7 · p. 9
Limitations
The benchmark uses a single literary source, and its results may not generalize to characters from other genres, cultural backgrounds, or writing styles. Reference-based evaluation also assumes that monologues or expert analyses represent characters’ true thoughts, although they may not capture psychological complexity.
Limitations
- ROLETHINK uses only A Song of Ice and Fire, leaving its effectiveness for other genres, cultural backgrounds, and writing styles unexplored.Source: Limitations · p. 9
- Reference-based evaluation assumes monologues or expert analyses accurately represent characters’ true thought processes.Expert analyses can be subjective, and original monologues may not fully capture character psychology.Source: Limitations · p. 9
- Original monologues may not fully capture the complexity of character psychology, while expert analyses can be subjective.Source: Limitations · p. 9
Ethics Statement
The research uses literature experts to provide character analyses and evaluations for the Silver task, with compensation and informed consent. It also acknowledges risks from biased or inappropriate generated thoughts and possible misuse for analyzing real people.
Ethics Statement
- Experts providing Silver-task analyses and evaluations were compensated above local minimum-wage standards and gave informed consent.Source: Ethics Statement · p. 9
- The authors identify risks of biased or inappropriate generated thoughts reflecting source-material content or language-model biases.Source: Ethics Statement · p. 9
- The authors warn that character-psychology analysis could be misused to manipulate or deceive and say the research should not be used to analyze real people.Source: Ethics Statement · p. 9
A.1 Dataset Statistics
The appendix reports the sizes of the Gold and Silver sets, permissions for Silver-set data, and prompt workflows for the two sets and MIRROR.
A.1 Dataset Statistics
- The Gold Set contains 405 data points from 22 characters, and the Silver Set contains 211 data points from 18 characters.Source: §A.1 · p. 12
A.2 Data Usage and Permissions
- Silver-set material was collected from three fan websites with explicit permissions and in accordance with their terms and data policies.Content creators were informed and agreed to inclusion, and collection and use were stated to comply with academic fair-use guidelines.Source: §A.2 · p. 12
B.1 Prompts for Gold Set
- Gold-set prompts identify key characters in POV chapters, locate coherent thought segments, and generate thoughts from profile and scenario context.Source: §B.1 · p. 12
B.2 Prompts for Silver Set
- Silver-set prompts extract behaviors and motivations from fan analyses, locate where thoughts might occur in chapter content, then generate character-perspective thoughts.Source: §B.2 · p. 12
B.3 Prompts for MIRROR
- MIRROR prompts retrieve character memories, analyze reactions of other characters or environments, then filter and organize information into a coherent thought process.Source: §B.3 · p. 12
C.1 Data Filtering Criteria
C.1 filters extracted examples for clear, complete, character-consistent thought processes, while C.2 evaluates generated thoughts against original monologues or established character motivations. The filtering removed approximately 63.4% of automatically extracted data.
C.1 Data Filtering Criteria
- Filtering excludes unclear motivations, thoughts located after actions, mixed-character thoughts, and Gold segments containing narration, dialogue, or physical reactions.Retained examples have documented motivations, clear decision-making context, consistency with character profiles, and complete pure internal monologues.Source: §C.1 · p. 13
- Approximately 63.4% of automatically extracted data was removed to retain high-quality examples for evaluation.Source: §C.1 · p. 13
C.2 Evaluation Guidelines
- Gold Set evaluators compare generated thoughts with original-text references, assessing reference coverage, additional reasoning quality, and character voice.Source: §C.2 · p. 13
- Silver Set evaluators assess alignment with established motivations and knowledge state, including reasoning depth and contextual consistency.Human annotators and LLM evaluators use the same structured prompts and scoring criteria.Source: §C.2 · p. 13