Source-linked AI summary
InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews
Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, Jiangjie Chen, Cheng Li, Yanghua Xiao
TL;DR
The paper addresses the limited evaluation of whether role-playing agents reproduce characters’ personalities rather than only their knowledge or language. It introduces INCHARACTER, which interviews RPAs with psychological-scale-derived questions and assesses the responses. Experiments across 32 characters and 14 scales validate the framework, while state-of-the-art RPAs successfully portray many character personality traits.
Problem
Existing RPA evaluations mainly examine characters’ knowledge, experience, and linguistic patterns, leaving personality fidelity and underlying mindsets underexplored.
Method
INCHARACTER evaluates RPA personalities through interviews that elicit behavioral, cognitive, and emotional patterns, followed by option conversion or expert rating.
Results
Experiments across 32 characters and 14 psychological scales validate INCHARACTER, and state-of-the-art RPAs successfully portray many personality traits of their characters.
Takeaways & Limitations
Psychological interviews provide a framework for measuring RPA personality fidelity against human-perceived character personalities.
Takeaways & Limitations
Measured results rely on interviewer LLMs whose errors or biases may compromise accuracy and underestimate RPA personality fidelity.
Abstract
from arXiv · showhide
Role-playing agents (RPAs), powered by large language models, have emerged as a flourishing field of applications. However, a key challenge lies in assessing whether RPAs accurately reproduce the personas of target characters, namely their character fidelity. Existing methods mainly focus on the knowledge and linguistic patterns of characters. This paper, instead, introduces a novel perspective to evaluate the personality fidelity of RPAs with psychological scales. Overcoming drawbacks of previous self-report assessments on RPAs, we propose InCharacter, namely Interviewing Character agents for personality tests. Experiments include various types of RPAs and LLMs, covering 32 distinct characters on 14 widely used psychological scales. The results validate the effectiveness of InCharacter in measuring RPA personalities. Then, with InCharacter, we show that state-of-the-art RPAs exhibit personalities highly aligned with the human-perceived personalities of the characters, achieving an accuracy up to 80.7%.
1 Introduction
The paper frames personality fidelity as an underexplored dimension of role-playing-agent evaluation and proposes INCHARACTER, an interview-based psychological assessment framework. Experiments across 32 characters and 14 scales support its effectiveness and show improved alignment with character personalities.
- Motivation: Personality fidelity evaluates whether RPAs reproduce the personalities of their target characters, complementing assessments of knowledge, experience, and linguistic patterns.Prior approaches require character-specific datasets and overlook thoughts and underlying mindsets.
- Motivation: Self-report scales can conflict with role-playing instructions, produce responses inconsistent with actual behavior, and be affected by instruction misunderstandings or training-data biases.These limitations motivate an interview-based procedure designed to elicit thoughts through conversation.
- Method: INCHARACTER replaces direct self-reporting with interviews that elicit RPAs’ behavioral, cognitive, and emotional patterns before personality assessment.Its assessment phase uses option conversion or expert rating based on interview results.
- Experiments: Experiments cover various RPAs and LLMs, evaluating 32 characters across 14 psychological scales, including BFI, 16P, and DTDD.Character labels come from the Personality Database and human annotators familiar with the characters.
- Resources: The authors release a benchmark and 18,304 interview dialogues to support future research on RPA personality evaluation and model fine-tuning.The benchmark is presented as the first for RPA personality evaluation.
2 Preliminaries
The paper presents psychological scales as structured instruments for measuring personality and contrasts prior self-report testing with the interview-based INCHARACTER framework. INCHARACTER elicits behavioral, cognitive, and emotional patterns before assessing personality.
- Role-playing agents: Role-playing agents are interactive AI systems that simulate assigned personas, including fictional characters, celebrities, and other roles.Their development has been supported by LLM abilities such as instruction following, reasoning, and empathy.
- Psychological scales: A Likert-rated psychological scale consists of items, dimensions, response options, and a scoring scheme that aggregates coded responses into personality scores.Items correspond positively or negatively to dimensions, and scoring can use averages or sums.
- Self-report assessment: Self-report scales ask participants to select options or ratings directly for questionnaire items.This closed-ended procedure produces a response array that is processed by the scale’s scoring scheme.
- Interview-based assessment: INCHARACTER uses an interview-based framework in which open-ended questions elicit behavioral, cognitive, and emotional patterns reflecting RPAs’ underlying mindsets.Personality is then measured from the interview results through option conversion or expert rating.
3 INCHARACTER
INCHARACTER transforms psychological-scale items into structured open-ended interviews for RPAs, then quantitatively assesses personality from their responses. Its assessment stage offers option conversion and expert rating while isolating questions and anonymizing character names to reduce interference and leakage.
- Interview stage: INCHARACTER converts each scale item into an open-ended interview question and manually checks the resulting question list.The question list contains one question for each scale item.
- Interview stage: Each question is posed to the RPA in an isolated context, and the corresponding response is recorded to avoid cross-question context effects.The procedure interviews an RPA representing a character across the structured question list.
- Assessment stage: The assessment phase computes personality scores for each dimension using either option conversion or expert rating.Both methods use LLMs to analyze interview results rather than requiring direct self-report selections.
- Option conversion: Option conversion maps each open-ended response to a scale option before applying the scale’s scoring scheme.This bridges open-ended interviews and closed-ended psychological-scale scoring.
- Expert rating: Expert rating directly evaluates each personality dimension from all corresponding question-response pairs without intermediate item ratings.The approach follows structured clinical interviews in which clinicians derive scores from observed responses.
- Assessment controls: Character names are anonymized in assessment prompts to reduce interviewer-LLM data leakage from memorized personality types.The safeguard is applied to expert rating and dimension-descriptive option conversion.
4 Experimental Setup
The experiments test whether LLMs can assess RPA personalities from interview responses and compare measured RPA personalities with human-labeled character personalities. The setup spans multiple models, RPA sources, characters, scales, and alignment metrics.
- Interviewer evaluation: Interviewer LLMs perform option conversion, dimension-descriptive option conversion, and expert-rating tasks from RPA interview results.The study compares GPT-4, GPT-3.5, and Gemini with human judgments using correlations and accuracy.
- Interviewer evaluation: GPT-4 produces only 4% wrong cases in expert rating, primarily when RPAs provide contradictory responses.Direct option conversion is less accurate, while dimension-descriptive options substantially improve that task.
- Interviewer evaluation: Interviewer LLMs achieve acceptable consistency with human interviewers when assessing personalities through expert rating or dimension-descriptive option conversion.This supports using those methods to simulate human interviewer assessments.
- RPAs and characters: The RPA evaluation uses 32 widely known characters drawn from ChatHaruhi and RoleLLM character data.The characters include figures from works such as Harry Potter, The Big Bang Theory, and Genshin Impact.
- Scales and labels: Personality evaluation covers 14 psychological scales, including BFI and 16P, with additional results reported in the appendix.Character personality labels combine Personality Database annotations with labels from human annotators familiar with the characters.
- Metrics: Measured alignment compares RPA personalities with human-annotated character personalities using rescaled MAE and dimension-level or full-scale accuracy.Marginal character dimensions are ignored because of their ambiguity.
5 Experimental Results
Experiments compare interview-based and self-report personality assessments across RPA types, foundation models, character data, and 14 psychological scales. INCHARACTER generally produces aligned, robust, and distinctive personality measurements, while interviewer errors remain a limitation.
- INCHARACTER with ER generally achieves better measured-alignment metrics than d-OC, while d-OC surpasses OC.
- Robustness, Consistency and Distinctiveness of RPA Personalities: StdScore remains below 6% across three runs, indicating robust measurements and stable RPA personalities.
- Self-report v.s. Interview-based Methods: Interview-based measurements are more aligned with characters than self-report and yield more consistent, distinctive personalities across questions.
- Comprehensive Personality Tests on 14 Scales: 78.9% average AccDim shows alignment between state-of-the-art RPAs and target characters across 14 psychological scales.
- More Results and Analyses: Scale questions are crucial for eliciting personality-indicative responses, but adapting questions to characters improves measurement accuracy when concepts exceed their knowledge scope.
- Character Data for RPAs: Description-only RPAs achieve measured-alignment metrics close to the combined description-and-memory setup, highlighting descriptions’ role in shaping personality.
- Foundation Models for RPAs: GPT-3.5 and GPT-4 RPAs achieve the best personality fidelity, whereas incremental fine-tuning of already capable open-source models brings limited improvement.
- Close-source RPAs: Character.ai RPAs barely reproduce target personalities and significantly underperform GPT-3.5 using a similar framework.
6 Related Work
Related work develops RPAs through character data, prompting, retrieval, and specialized models, while evaluation has largely emphasized general conversational capabilities and character knowledge or language.
- Existing RPAs use raw scripts or dialogues, character descriptions, and retrieved character experiences to simulate personas.
- Prior work targets either character-specific RPAs for established fictional characters or foundation models designed for role-playing.
- Evaluation research mainly examines character-independent capabilities such as conversational ability and human-likeness.
7 Conclusion
This study investigates personality fidelity in role-playing agents and introduces INCHARACTER to measure it across varied agents, characters, and psychological scales. The results validate the framework and show that state-of-the-art agents portray many target-character personality traits.
- INCHARACTER measures whether role-playing agents reproduce the personalities of their intended characters across 32 characters and 14 psychological scales.
- The experiments validate INCHARACTER’s effectiveness for measuring role-playing-agent personalities.
- State-of-the-art role-playing agents successfully portray many personality traits of their target characters.
Limitations
The study identifies two main limitations: interviewer-LLM measurement may introduce errors or bias, and static character labels may not capture personality changes over time.
- Interviewer LLMs may introduce errors or biases that compromise measurement accuracy and underestimate role-playing-agent personality fidelity.
- Static personality labels may add evaluation noise because human and fictional-character personalities can change over time.
- The study does not examine progressive changes in role-playing-agent personalities, leaving personality dynamics for future research.
Ethical Statement
The ethical statement describes human-annotation practices, privacy commitments, and risks from interviewer-LLM bias and assessing darker personality traits. The included notation and scale descriptions define the evaluation instruments used in the study.
- University-student annotators were compensated substantially above local minimum-wage standards for collecting personality labels.
- Interviewer LLMs may introduce bias into personality assessments because of biases in their training data.
- Assessing dark personality traits with the DTDD scale raises concerns about harmful-content generation and motivates safeguards.
- The evaluation covers 14 personality scales, including the Dark Triad of Personality scale.
E Implementation Details
The implementation uses ChatHaruhi-Suzumiya and LangChain-based agents, handles self-report and interviewer-LLM outputs with specific processing rules, and fine-tunes Mistral-2 7B with LoRA. GPT-4 shows high agreement with humans on emotion-recognition ratings.
- The role-playing-agent implementation uses ChatHaruhi-Suzumiya and LangChain with foundation-LLM temperature set to 0.7.
- Self-report responses in which agents refuse to participate are categorized as Neutral.
- Interviewer LLM outputs are requested in JSON, regenerated when parsing or batch results fail, and processed with temperatures of 0 and 0.2.
- Mistral-2 7B is fine-tuned on ChatHaruhi-English-62K using LoRA for three epochs with batch size 16 and learning rate 5e−5.
- GPT-4 rates 82% of emotion-recognition cases as highly consistent with human ratings, while 4% are notably different.
F.2 Self-report v.s. Interview-based Methods
INCHARACTER elicits personality evidence through interviews, producing measurements that better align with target characters than self-report methods. Its interview conversations also support character-agent dataset creation.
- INCHARACTER elicits RPA thoughts and behaviors, whereas self-report directly prompts choices that may be biased by foundation-model pretraining.This difference yields more distinct personality measurements than self-report.
- SR-CoT improves only limitedly over SR because RPAs implicitly perform option conversion, while INCHARACTER can use d-OC or ER.Existing LLMs generally perform better on ER and d-OC than OC.
- GPT-3.5-based RPAs using ChatHaruhi and RoleLLM character data achieve high personality fidelity, demonstrating the quality of those character datasets.The comparison covers ChatHaruhi, RoleLLM, and their union, evaluated with INCHARACTER.
- Character.ai RPAs exhibit low personality fidelity compared with GPT-3.5-based RPAs using character descriptions from ChatHaruhi and RoleLLM.Character.ai uses a close-source model, whereas the comparison RPAs use GPT-3.5.
F.4 Foundation Models for RPAs
Foundation-model capabilities shape RPA personality fidelity and common interaction problems. Across 14 scales, state-of-the-art RPAs show substantial alignment, while question design and generic demonstrations affect measurement quality.
- Foundation Models for RPAs: Personality fidelity largely depends on LLM language capacity: LLaMa-2-Chat 13B performs competitively for English characters but unsatisfactorily for Chinese characters.The results distinguish characters with English or Chinese data.
- Foundation Models for RPAs: GPT-3.5 and GPT-4 seldom produce multilingual, non-immersive, or repetitive responses, while LLaMa-2-Chat 13B has the most unexpected multilingual generation.The multilingual issue is associated primarily with inadequacy in Chinese and handling Chinese-based character data.
- Questionnaires of Different Topics: Questions from a personality scale better elicit responses indicative of corresponding traits than general instructions or questions from other scales.Other-scale questions still reflect personalities from some perspectives but relate less to intended dimensions.
- Question Adaptation: 60.7% of potentially mismatched 16P questions were adapted for character contexts, improving alignment between measured personalities and character labels.The adaptation changes concepts such as “books and movies” into character-appropriate equivalents.
- Enhancing Self-report Methods with In-context Learning: In-context learning decreased self-report performance when unified examples were used across characters.The paper attributes this to interference and bias from examples containing responses from other personas.
- Self-report versus Interview-based Methods: INCHARACTER measurements better align with character personalities than self-report baselines in radar-chart comparisons.The comparison uses INCHARACTER with ERbatch and GPT-4 versus SR.
H.2 Compliant Responses from character.ai
Character.ai RPAs often respond compliantly rather than reproducing target-character personalities. Their measured agreement pattern and examples contrast with more character-consistent responses from ChatHaruhi RPAs.
- Compliant Responses from character.ai: Character.ai responses tend to comply with and please users instead of reproducing target-character personalities.The paper illustrates this pattern with Zhong Li and contrasts it with a more character-consistent ChatHaruhi response.
- Zhong Li Example: A Zhong Li ChatHaruhi RPA frames worry as unproductive and recommends calm problem-solving, unlike the character.ai response that repeatedly endorses worrying.The paper identifies the ChatHaruhi response as closer to Zhong Li’s personality.
- Compliant Responses from character.ai: 65.8% probability of (strong) agreement versus 17.1% disagreement indicates a strongly compliance-skewed character.ai response distribution on the BFI.Responses were converted into BFI options with GPT-4.
- Example Responses: Interview examples show RPAs producing personality-indicative answers rather than merely selecting “agree” or “disagree” options.Examples span BFI, 16P, and additional personality scales for Zhong Li and Hermione Granger.