Capturing Minds, Not Just Words: Enhancing Role-Playing Language Models with Personality-Indicative Data

• Existing RPLMs reproduce character knowledge and tone but struggle with character minds, including in personality assessment and decision simulation.Source: §1 · p. 1
• The paper proposes personality-indicative data to help RPLMs better capture characters’ minds.Source: §1 · p. 1
• ROLEPERSONALITY uses questions from 14 psychological scales and includes single-turn and multi-turn dialogues.Source: §1 · p. 1
• Fine-tuned RPLMs show improved capabilities in both personality-related and general role-playing evaluations.Source: §1 · pp. 1–2

2.1 Role-Playing Language Models

• Experience extraction draws dialogue and other information from original media such as novels and television shows (Li et al., 2023; Yuan et al., 2024).Source: §2.1 · p. 2
• Dialogue synthesis uses LLM generation or human annotation, with topics drawn from literature, general instructions, or personality-test scenarios (Shao et al., 2023; Wang et al., 2024b; Wang et al., 2024a).Source: §2.1 · p. 2

2.2 Construction of Role-Playing Agents

• Parametric learning fine-tunes a base model on role-playing data, while non-parametric approaches engineer character-level resources, including profiles and query-retrieved long-term memory (Shao et al., 2023; Yu et al., 2024; Zhou et al., 2023; Wang et al., 2024b; Li et al., 2023).Source: §2.2 · p. 2

3.1 Dataset Construction

• ROLEPERSONALITY elicits character responses to open-ended psychological-scale questions using RPAs built from an RPLM, character descriptions, and memory (Li et al., 2023).Questions adapted from scales such as the Big Five Inventory and 16Personalities are intended to elicit character mindsets and behaviors.Source: §3.1 · p. 2
• Generated responses are filtered against human-annotated personality labels, and questions judged unsuitable for a character’s background are excluded.Source: §3.1 · p. 2
• Table 1’s three sub-datasets are designed to assess the effects of scale screening and adding multi-round data.Source: Table 1 · p. 2
• The selected scale subset Part reflects character personality, while Full contains the entire set of selected scales (Wang et al., 2024a).Source: §3.1 · p. 2
• Multi-turn data use questions from different dimensions of the same scale, while Single contains only single-turn data.The stated purpose of multi-turn dialogue is to maintain conversational consistency and enhance contextual understanding.Source: §3.1 · pp. 2–3

3.2 Dataset Statistics

• The dataset has three sub-datasets generated through interviews with 16 ChatHaruhi characters and 30 English RoleLLM characters.Source: §3.2 · p. 3

4.1 Settings

• Mistral-7Bv0.2-Chat was LoRA-tuned for three epochs at rank 8 on roughly 20,000 samples per dataset, then evaluated on personality fidelity, motivation recognition, and general role-playing.Source: §4.1 · p. 3

4.4.1 Direct Scoring; 4.4.2 Dimensional Scoring

• Our models lead in most role-playing dimensions except personality, while Part+Single has the best win-rate and our models trail only gpt-3.5-turbo despite lower Rouge-L scores.Table 3 reports the general role-playing benchmarks.Source: Table 3 · p. 4, §4.4.1 · p. 4, §4.4.2 · p. 4

4.2 Personality Fidelity (PF)

• Fine-tuned models improve overall personality fidelity, while models trained on the other two datasets perform worse than untrained Mistral, possibly because those datasets emphasize character knowledge over personality traits.Source: §4.2 · p. 4

4.3 Motivation Recognition (MR)

• ROLEPERSONALITY fine-tunes significantly outperform other models on motivation recognition, indicating stronger recognition of characters’ motivations.Source: §4.3 · p. 4

• The authors conclude that personality-indicative data helps capture complex character mindsets and significantly enhances role-playing agent performance.Source: §5 · p. 5
• ROLEPERSONALITY addresses traditional datasets’ focus on character knowledge and linguistic habits by capturing character personalities.Source: §5 · p. 5
• The authors say these advances pave the way for role-playing models that simulate complex character behaviors and lead to more immersive user experiences.Source: §5 · p. 5

• LLM-built fine-tuning data may carry training-data biases or inaccuracies, interview data lacks compliance mechanisms, and automated evaluation omits human assessment of portrayal nuances.Source: Limitations · p. 5

• The authors state that they are aware of and honor the ACL Code of Ethics.Source: Ethics Statement · p. 5
• LLM-generated data may carry biases or inaccuracies, while missing compliance mechanisms can make interview data inconsistent and less authentic.Source: Ethics Statement · p. 5
• Automated evaluation may miss subtle portrayal nuances, and overfitting selected scale traits may limit generalizability across role-playing datasets.Source: Ethics Statement · p. 5

B Psychological Scales

• The Big Five Inventory (BFI) covers Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism, summarized as OCEAN.The passage describes these as curiosity, organization, sociability, empathy, and negative feelings, respectively.Source: B Psychological Scales · p. 7

Eysenck Personality Questionnaire (Revised)

• The Revised Eysenck Personality Questionnaire (EPQ-R) assesses Extraversion, Neuroticism, Psychoticism, and socially desirable responding through its Lying Scale.The passage distinguishes Psychoticism as a personality characteristic rather than a serious mental health condition.Source: Eysenck Personality Questionnaire… · p. 7
• The Dark Triad Dirty Dozen (DTDD) is a 12-item measure of narcissism, Machiavellianism, and psychopathy, contrasted with the BFI and EPQ-R’s “Light” traits.Source: Eysenck Personality Questionnaire… · p. 7
• The 16Personalities type code adapts five BFI-related spectrums rather than prioritizing Jungian concepts or cognitive functions.Source: Eysenck Personality Questionnaire… · p. 7

Experiences in Close Relationships (Revised)

• Other scales cover sex-role characteristics, 41 vocational-interest dimensions, cultural influence on identity, attachment, self-efficacy, optimism, money attitudes, and emotional intelligence.The BSRI groups participants by median scores on masculine and feminine characteristics; CABIN introduces the eight-dimension SETPOINT model, also adaptable to Holland’s six-dimension RIASEC framework.Source: Eysenck Personality Questionnaire… · pp. 7–8, Experiences in Close Relationships… · p. 8

Wong and Law Emotional Intelligence Scale

• The Wong and Law Emotional Intelligence Scale (WLEIS) divides emotional intelligence into self- and others’ emotion appraisal, use of emotion, and regulation of emotion.Source: Wong and Law Emotional Intelligence… · pp. 8–9
• The Empathy Scale distinguishes cognitive empathy, or understanding another person’s thoughts and feelings, from emotional empathy, or vicariously experiencing their emotions.Source: Wong and Law Emotional Intelligence… · p. 9

C Character Selection

• The selected characters span animations, films, television, and other works, with 30 RoleLLM characters and 16 ChatHaruhi characters in the training set.Source: C Character Selection · p. 9
• Character selection aimed for broad source and personality diversity, and the section presents personality information for selected characters.Source: C Character Selection · p. 9, Table 6 · p. 9

D Multi-turn Dialogue Consistency

• Five interaction rounds each give the model one scale-based question, allowing consistency to be assessed across the conversation.Source: D Multi-turn Dialogue Consistency · p. 9
• Consistency is quantified by averaging the standard deviations across personality dimensions, with lower scores indicating greater multi-turn consistency.Source: D Multi-turn Dialogue Consistency · p. 9, Table 7 · p. 9
• The model fine-tuned with ROLEPERSONALITY exhibits higher personality consistency during multi-turn dialogue.Source: D Multi-turn Dialogue Consistency · p. 9, Table 7 · p. 9

E Evaluation Prompt

• Win-rate and dimensional scoring are assessed directly using a large language model.Source: E Evaluation Prompt · p. 10
• The prompts used for these evaluations are listed in Table 8.Source: E Evaluation Prompt · p. 10, Table 8 · p. 11
• Table 8 is identified as a collection of evaluation prompts.Source: Table 8 · p. 11