PsyPlay: Personality-Infused Role-Playing Conversational Agents

1 Introduction

• Existing role-playing agents often mimic character speaking styles and backgrounds while overlooking more fundamental personality traits.The paper argues that personality traits capture differences in thinking, feeling, and behavior, and may help make interactions more realistic and engaging.Source: §1 · pp. 1–2
• PsyPlay creates personalized roles, extracts topics from Human Stress Prediction, and prompts agents to discuss those topics in character.Its three stages are Role Card Creation, Topic Extraction, and Dialogue Generation.Source: §1 · p. 2
• GPT-3.5 portrays positive personality roles more successfully than negative ones; stronger traits improve portrayal, while more turns create expression opportunities but increase partner-misdirection risk.The authors suggest alignment with positive values may explain the positive-role advantage.Source: §1 · p. 2
• PsyPlay-Bench contains 4,745 groups of dialogues judged successful through back-testing and is proposed for personality-role-play instruction and personality detection evaluation.The corpus is intended to support research on personalized role-playing and dialogue personality detection.Source: §1 · p. 2

2.1 Personalized Dialogue Systems

• Personalized dialogue systems encode user profiles or personas to improve response specificity, fluency, or consistency, but have not sufficiently explored individual personality traits.Examples include persona-aware seq2seq, memory-augmented, and pre-training-based approaches.Source: §2.1 · p. 2

2.2 Role-Playing with LLMs

• LLM role-playing commonly prompts characters from fiction or media, yet names or descriptions may fail to convey intended personality traits.The paper also notes that self-assessment tests can reflect training data and value alignment rather than personality manifested in dialogue.Source: §2.2 · pp. 2–3

3 Approach

• PsyPlay’s approach has three stages: Role Card Creation, Dialogue Topic Extraction, and Dialogue Generation.The stages produce roles and topics before prompting agents to converse around the selected topic.Source: §3 · p. 3

3.1 Role Card Creation

• Roles are deeply customizable, with role attributes separated from personality attributes and personality traits infused both explicitly and through tailored experiences.The trait set uses Big Five labels drawn from WASSA 2022; traits are paired with descriptors and intensity levels to form personality descriptions and experiences.Source: §3.1 · pp. 3–4

3.2 Topic Extraction

• Dialogue topics are extracted from the Human Stress Prediction dataset and selected from psychologically healthy scenarios spanning real-world issues.The topic prompt excludes restrictive age and gender information that could conflict with role attributes.Source: §3.2 · p. 4

3.3 Dialogue Generation

• AutoGen controls two-agent dialogue generation around a selected topic, using role cards and explicit Big Five traits alongside personality-shaped experiences.The prompt is revised for subsequent turns, asking agents to respond to a partner’s message and avoid addressing them by name too often.Source: §3.3 · pp. 4–5

3.4 Personality Back-Testing

• Personality back-testing identifies traits expressed in dialogue and compares them with the predefined traits, rather than relying on an agent’s questionnaire self-assessment.The authors use an LLM evaluator to assess personality expression from dialogue data.Source: §3.4 · p. 5

4.1 Collections

• The personality trait set uses WASSA 2022 labels mapped to three intensity levels; the collections also include 307 personality combinations, 132 role cards, and 161 topics.The traits are sampled from real-world individuals to avoid potential conflicts between personality dimensions.Source: §4.1 · p. 5

4.2 PsyPlay-Bench

• 8,750 raw multi-turn dialogues were generated from randomly selected role cards and topics, then divided into evaluation, test, and clean sets.The evaluation set contains 200 samples, the test set 550, and the clean set 4,745.Source: §4.2 · pp. 5–6
• PsyPlay-Bench-Clean retains 4,745 samples from the remaining 8,000 after dialogues that failed automatic personality back-testing were removed.Source: §4.2 · p. 6

5.1 GPT-3.5 for Back-Testing

• 88.01% total agreement was observed between GPT-3.5 back-testing and human evaluation, supporting its use for automated personality back-testing in subsequent experiments.Agreeableness, Conscientiousness, and Neuroticism each show consistency above 90.00%, while Extraversion and Openness are lower.Source: §5.1 · p. 6

5.2 LLMs for PsyPlay

• Negative-role success increased from 61.57 to 78.69 with Higgs-Llama-3 70B, while positive-role success declined from 90.71 to 77.68 compared with GPT-3.5.The authors attribute this contrast to positive-value alignment during RLHF contributing to positive-role portrayal while compromising negative-role portrayal.Source: §5.2 · p. 6
• 80.31% overall success rate was achieved by PsyPlay with GPT-3.5 on PsyPlay-Bench-Test.The reported rates are higher for Agreeableness, Conscientiousness, and Neuroticism than for Extraversion and Openness; positive roles also outperform negative ones.Source: §5.2 · p. 6, Table 4 · p. 7

5.3 Ablation Study

• Removing explicit traits decreased negative-role success by 2.55%, while removing role experiences decreased positive-role success by 2.63%.The authors find both prompt components contribute to character-portrayal success.Source: Table 5 · p. 7, §5.3 · p. 7

5.2 LLMs for PsyPlay

• Evaluations with Gemma-2-27b-it and Llama-3.1-405b-instruct indicate that a more powerful backbone further improves PsyPlay performance.Source: §5.2 · p. 7

6.1 Effect of Levels

• “Very” and “extremely” personality levels yield higher portrayal success rates than “a bit.”The study compares success rates across the three levels on PsyPlay-Bench-Test.Source: Figure 3 · p. 7, §6.1 · p. 7

6.2 Effect of Turns

• More dialogue turns improve positive-personality portrayal but can cause negative-personality characters to deviate from their predefined traits.Additional turns offer positive characters more opportunity to express opinions, while negative characters are more susceptible to partner influence.Source: §6.2 · p. 7, Figure 4 · p. 8

6.3 Diversity Analysis

• Back-testing found mixtures of traits: agreeableness roles also reflected conscientiousness, and openness roles also influenced conscientiousness.The analysis randomly sampled 50 roles, with 10 roles per predefined personality trait.Source: §6.3 · p. 8, Figure 5 · p. 8

7.1 GPT-3.5 Prefers Positive Roles

• A character with neurotic traits remains stable and caring in a sensitive discussion, which the authors link to GPT-3.5’s over-alignment on positive values.The authors say the model may generate successful personal experiences and allow negative characters to express positive views on mental-health topics.Source: §7.1 · p. 8

7.2 Dual Implications of Increasing Turns

• More turns let Larry show warmth and thoughtfulness, as he shifts from discussing a problem to partners’ emotional and mental health.The example concerns a character with a bit of agreeableness.Source: §7.2 · p. 8
• After interacting with a highly conscientious partner, Edward shifts from nervousness to emotional stability over additional turns.Edward is described as having a bit of neuroticism.Source: §7.2 · p. 8

• PsyPlay integrates personality-infused role-playing into LLMs through fine-grained personality shaping and explicit personality introduction.Source: §8 · p. 8
• PsyPlay’s automatic personality back-test indicates that generated agents accurately incorporate their desired traits.Source: §8 · p. 8
• GPT-3.5 portrays positive roles more successfully than negative roles, a discrepancy the authors attribute potentially to alignment with positive values.The authors also introduce PsyPlay-Bench for further research on role-playing conversational agents.Source: §8 · p. 8

• PsyPlay relies on LLM capabilities and may be influenced by the model’s alignment.Source: Limitations · p. 9
• The study is limited to two-character dialogues; multi-party dialogue remains future work.Source: Limitations · p. 9
• The fourth dialogue-generation prompt rule may potentially disrupt dialogue fluidity.Source: Limitations · p. 9

Ethics Statement

• The study uses publicly sourced WASSA 2022 and Human Stress Prediction data without sensitive information, and derived research or applications are permitted only for research purposes.Source: Ethics Statement · p. 9

A Appendix: Prompts

• The topic-extraction prompt asks for one general sentence, without explicit gender or age restrictions, focused on one main aspect.Source: A Appendix: Prompts · p. 10

B.1 Role Cards

• The appendix provides examples of generated role cards.Source: §B.1 · p. 11

B.2 Dialogue Topics

• Topic extraction uses few-shot prompting, with the complete prompt and example topics included in the appendix.Source: §B.2 · p. 11

B.3 Personality-Infused Dialogues

• The appendix provides generated-dialogue examples alongside their role attributes and topics.Source: §B.3 · p. 11

C.2 Statistics of the Dataset

• PsyPlay-Bench has three subsets: Eval, Test, and Clean; the appendix points to tables detailing role settings and dialogue statistics.Source: §C.2 · p. 11

D Appendix: Personality Descriptors

• The appendix lists personality descriptors adapted from Safdari et al. (2023).Source: D Appendix: Personality Descriptors · p. 11

E Appendix: Human Evaluation

• Human evaluators judged whether dialogue reflected each role’s Big Five traits using language, behavior, and expressed emotions; evaluator agreement was 0.9054, with Kappa 0.7998.Source: E Appendix: Human Evaluation · pp. 11–12

F Appendix: PsyPlay with Various LLMs

• The additional-model evaluation used Gemma-2-27b-it and Llama-3.1-405b-instruct, with overall PsyPlay-bench-test results reported in Table 10.Source: F Appendix: PsyPlay with Various LLMs · p. 12, Table 10 · p. 12
• The newly released Llama-3.1 model with 405B parameters achieved the highest overall success rate, 85.77%, indicating that a more powerful backbone further improves PsyPlay performance.Across GPT-3.5 and the two introduced LLMs, success rates were highest for Conscientiousness (CON) and lowest for Openness (OPN).Source: F Appendix: PsyPlay with Various LLMs · p. 12