PaperlayerView exact revision on arXiv

Source-linked AI summary

PersonaGym: Evaluating Persona Agents and LLMs

Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik Narasimhan, Vishvak Murahari

arXiv:2407.18416v5cs.CLcs.AIcs.LG

TL;DR

Persona-agent evaluation lacks robust, targeted large-scale testing of arbitrary personas in relevant environments and across multiple dimensions. PersonaGym dynamically selects environments and questions, and PersonaScore provides a decision-theory-grounded automatic metric. Evaluations across 10 LLMs, 200 personas, and 10,000 questions reveal performance gaps and show that model size does not necessarily correlate with persona-agent capability.

  • Problem

    Prior large-scale persona-agent evaluations were static, constrained to predetermined personas, lacked persona-relevant environments, and were often unidimensional.

    Source: §1 · pp. 1–2

  • Method

    PersonaGym dynamically evaluates persona agents in relevant environments, while PersonaScore measures persona adherence across diverse tasks and settings.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Across 10 LLMs and 200 personas, model size does not necessarily correlate with persona-agent capabilities, and performance gaps remain between leading and less capable models.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The findings support targeted research into persona-agent performance and careful study of state-of-the-art models’ capabilities before deployment.

    Source: §1 · p. 2, §7 · p. 9

    The 200 benchmark personas do not equally represent all socio-demographic groups.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Persona agents, which are LLM agents conditioned to act according to an assigned persona, enable contextually rich and user aligned interactions across domains like education and healthcare. However, evaluating how faithfully these agents adhere to their personas remains a significant challenge, particularly in free-form settings that demand consistency across diverse, persona-relevant environments. We introduce PersonaGym, the first dynamic evaluation framework for persona agents, and PersonaScore, a human-aligned automatic metric grounded in decision theory that enables comprehensive large-scale evaluation. Our evaluation of 10 leading LLMs across 200 personas and 10,000 questions reveals significant advancement opportunities. For example, GPT-4.1 had the exact same PersonaScore as LLaMA-3-8b despite being a more recent and advanced closed source model. Importantly, increased model size and complexity do not necessarily enhance persona agent capabilities, underscoring the need for algorithmic and architectural innovation toward faithful, performant persona agents.

1 Introduction

PersonaGym and PersonaScore address the lack of robust, targeted evaluation for persona agents with dynamic, persona-relevant testing and a human-aligned metric. Evaluations expose weaknesses in leading models and show that greater model size or capacity does not guarantee stronger persona-agent capabilities.

1 Introduction

  • Existing large-scale evaluations were static, limited to predetermined personas, lacked persona-relevant environments, and assessed agents on only one dimension.The authors also note concerns about data contamination with new state-of-the-art LLMs.Source: §1 · pp. 1–2
  • PersonaGym dynamically evaluates arbitrary persona agents in targeted environments, while PersonaScore measures persona adherence across diverse tasks and settings.The paper describes PersonaScore as the first automatic metric aligned with human judgment for this purpose.Source: §1 · p. 2
  • PersonaGym selects persona-relevant environments, generates task-specific questions, and evaluates responses against expert-curated rubrics.Exemplar responses for each rubric score calibrate multiple evaluators, whose judgments are ensembled.Source: §1 · p. 2
  • The framework separates evaluator models from evaluated agents and supports model swapping to reduce single-model bias and enable fairer evaluation.A static benchmark of 200 personas and 10,000 questions complements custom evaluations.Source: §1 · p. 2
  • Across 200 personas and 10,000 questions, Claude 3.5 Sonnet and GPT-4.5 do not outperform GPT-3.5 as strongly as on other tasks and domains.The evaluation covered ten leading open- and closed-source LLMs.Source: §1 · p. 2
  • Model size or capacity does not reliably indicate persona-agent capability; Claude 3 Haiku, for example, resists generating persona-agent responses.The authors call for further study of persona-agent ability in state-of-the-art models before deployment.Source: §1 · p. 2

2 Evaluation Tasks

PersonaGym places agents in relevant environments and organizes five evaluation tasks around three branches of decision theory. These tasks test optimal action selection, adherence to persona norms, and explanations of purported actions.

2.1 Normative Evaluation

  • Expected Action tests whether an agent selects an action meeting persona-specific optimality criteria and maximizing expected utility.This task operationalizes normative decision theory’s criteria for optimal decisions by perfectly rational actors.Source: §2.1 · p. 3

2.2 Prescriptive Evaluation

  • Linguistic Habits, Persona Consistency, and Toxicity Control assess persona-appropriate communication, fidelity under direct questioning, and responsible responses to provocative prompts.Toxicity Control assigns higher scores to appropriate responses and lower scores to toxic ones.Source: §2.2 · p. 3

2.3 Descriptive Evaluation

  • Action Justification asks agents to explain purported actions, assessing whether their post-hoc explanations fit their persona characteristics.This task examines actual decision processes and reasoning mechanisms through agents’ explanations.Source: §2.3 · pp. 3–4

2 Evaluation Tasks

  • PersonaGym evaluates agents in relevant environments across five tasks mapped to decision-theory branches; Table 1 frames comparison around arbitrary personas, persona-specific tasks, and multidimensional open-ended evaluation.The framework defines environments as external settings or conditions in which agents operate.Source: §2 · pp. 2–3, Figure 2 · p. 3, Table 1 · p. 3

3 PersonaGym

PersonaGym selects persona-relevant environments and generates task-specific questions, then evaluates agent responses with task-specific rubrics and ensembled LLM evaluators. Its formulation tailors rubric examples to each persona and question, while its method specifies the evaluation pipeline.

3.1 Formulation

  • An environment selector chooses persona-relevant settings, and a question generator creates task-specific questions for those settings.The formulation maps the persona and environment set to selected environments, then uses selected environments, persona, and task to generate questions.Source: §3.1 · p. 4
  • For each task and question, PersonaGym augments task descriptions and scoring guidelines with persona- and question-specific examples for every possible score.The example generator uses the persona, question, and rubric outline to create examples that become part of the final rubric.Source: §3.1 · p. 4
  • Evaluators receive the persona, question, response, and rubric when scoring task responses.The formulation defines evaluator-generated score matrices for each task and combines them into a final score matrix for the persona agent.Source: §3.1 · p. 4

3.2 Method

  • PersonaGym selects environments from 150 options and generates 10 questions per selected environment for each evaluation task.The agent responds to the generated questions while assuming its assigned persona through a system prompt.Source: §3.2 · pp. 4–5
  • Two LLM evaluators score each response from 1–5 using a comprehensive rubric, and their scores are averaged.The authors report validating LLM evaluation through human evaluation and using ensembling to reduce potential variance.Source: §3.2 · p. 5

4 Experiments

Across PersonaGym's tasks, models showed uneven performance, with no single model excelling consistently; the cross-evaluation found no significant score differences across question generators and evaluators. The experiments also examined persona and environment diversity.

4.1 Experimental Settings

  • Experiments used GPT-4O to select persona-relevant environments and generate questions, while GPT-4O and LLAMA-3-70B evaluated responses.The study generated 200 personas with GPT-4O; beyond 200, limited diversity led to overlapping attributes.Source: §4.1 · p. 5

4.2 Main Results

  • 2.10 (Action Justification) and 1.34 (Persona Consistency) were the widest model-performance spreads; Expected Action (0.56), Linguistic Habits (1.22), and Toxicity Control (0.78) had lower spreads.Table 2 reports results for 10 LLMs across 200 personas and 10 questions per task, totaling 10K questions.Source: §4.2 · pp. 5–6, Table 2 · p. 6
  • LLAMA-3-8B outperformed LLAMA-3.3-70B, while LLaMA 2 improved across tasks from 13B to 70B, with an average increase of 0.414.The results therefore do not show a consistent relationship between model size or capacity and PersonaGym performance.Source: §4.2 · p. 6
  • Linguistic Habits was the most challenging task: all models except GPT-4.1, GPT-4.5, and DEEPSEEK-V3 scored below 4.The section identifies associating personas with appropriate jargon and speech styles as a significant difficulty.Source: §4.2 · p. 6
  • CLAUDE 3 HAIKU refused role-play 8.5 times as often as LLAMA-3-8B and 2.6 times as often as all other benchmarked models combined.The authors connect its resistance to underperformance in Action Justification and Persona Consistency.Source: §4.2 · p. 6, Figure 4 · p. 7

4.3 PersonaGym is robust to model bias

  • Across the sampled personas and environments, PersonaScore showed no significant differences across question generators and evaluators; evaluators did not score their own models’ responses.The cross-evaluation sampled 25 of 200 personas and used 1,250 questions per generator.Source: §4.3 · pp. 6–7, Figure 5 · p. 7

4.4 Environments and Personas Distribution

  • The experiments span varied settings, including social events, recreational activities, and gatherings, and persona attributes such as professional roles, locations, and interests.Examples include weddings, hiking trails, teachers, and environmental activists.Source: §4.4 · p. 7

5 Human Evaluation

A human study compared PersonaScore with expert ratings across 1,500 responses and found strong overall correlations, alongside both matching and discrepant individual cases.

Human experimental settings

  • Five expert annotators evaluated 1,500 responses from three models across a subsample of 100 personas using the evaluators’ rubric.The annotators had university-level English proficiency and substantial world knowledge of different personas and their expected characteristics.Source: §5 · pp. 7–8

Human evaluation results

  • 75.1% (Spearman) and 62.73% (Kendall-Tau) were the average PersonaScore correlations with human evaluations across three models.The task-level Spearman peak was 84.8% for Expected Action with LLAMA-2-70B; the peak Kendall-Tau was 77.2% for Linguistic Habits with LLAMA-2-70B and LLAMA-2-13B.Source: §5 · p. 8

Model-human agreement case

  • For an Australian environmental lawyer persona, LLAMA-2-13B received 4.5 from both evaluation methods, while GPT-3.5 and LLAMA-2-70B each scored 4.0.The paper attributes LLAMA-2-13B’s result in part to references to Indigenous peoples and Australian colloquialisms fitting the persona.Source: §5 · p. 8

Model-human disagreement case

  • For a London writer persona, PersonaGym scores were higher than human ratings, and the responses lacked the sophisticated language expected of a writer describing artwork.Only LLAMA-2-70B incorporated British vernacular; the discrepancy points to an opportunity to improve PersonaGym’s assessment of expected linguistic characteristics.Source: §5 · p. 8

6 Related Work

Related work covers both methods for developing role-playing LLMs and emerging frameworks for evaluating their fidelity. The cited benchmarks differ in language, data, and evaluation approach.

Role-Play in LLMs

Role-Play Evaluation

  • Evaluation frameworks include RoleBench’s GPT-generated QA pairs from 100 character profiles (Wang et al., 2024a) and psychological-scale and Likert evaluations of character fidelity (Wang et al., 2024b).CharacterEval contains 1,785 Chinese multi-interaction dialogues (Tu et al., 2024), while RoleEval has 6,000 bilingual multiple-choice questions across 300 personas (Shen et al., 2023).Source: §6 · p. 8

7 Conclusion

PersonaGym evaluates persona agents across five tasks with persona-specific questions in contextually relevant environments, and introduces PersonaScore as a quantitative measure of role-playing proficiency. Its evaluation finds that model size does not necessarily predict persona-agent capability, while strong correlations with human evaluations support the framework’s effectiveness.

7 Conclusion

  • PersonaGym assesses persona agents across five tasks using persona-specific questions in contextually relevant environments, and PersonaScore quantifies role-playing proficiency.Source: §7 · pp. 8–9
  • Model size does not necessarily correlate with persona-agent capabilities, and the evaluation finds performance gaps between leading and less capable models.The evaluation covers 10 LLMs across 200 personas.Source: §7 · p. 9
  • Strong Spearman and Kendall-Tau correlations with human evaluations validate PersonaGym’s effectiveness.Source: §7 · p. 9

Limitations

The benchmark’s 200 personas do not represent all socio-demographic groups equally.

Limitations

  • The 200 benchmark personas do not provide equal representation of all socio-demographic groups.Source: Limitations · p. 9

Ethics Statement

The authors identify risks associated with persona-based LLM research, including misuse, privacy and intellectual-property concerns, stereotyping, and anthropomorphization. They urge responsible use and reject harmful applications.

Ethics Statement

  • PersonaGym could be misused to generate harmful content targeting specific groups, particularly through its Toxicity Control task.Source: Ethics Statement · p. 9
  • Generated personas may resemble real people or copyrighted characters, raising privacy and intellectual-property concerns.Source: Ethics Statement · p. 9
  • Persona creation risks reinforcing demographic stereotypes, while anthropomorphization may lead users to misinterpret model responses across contexts.The authors urge researchers to remain mindful of these implications and reject harmful applications.Source: Ethics Statement · p. 9

A Prompts

The appendix documents prompts and examples for selecting persona-relevant environments, generating task questions, instantiating persona agents, and scoring their responses. It also describes a distribution of static environments used to select relevant environments for each agent.

A.1 Dynamic Environment Selection

  • Environment selection prompts ask an LLM to choose persona-relevant options from an environment list and return the selections as a Python list.Source: §A.1 · p. 12

A.2 Question Generation

  • Question prompts condition generation on the persona, selected environments, task, and task-specific quality criteria; toxicity questions elicit challenging responses to test decorum.The template requests exactly 10 challenging multi-step questions.Source: §A.2 · p. 12

A.3 Persona Agent Response Generation

  • The persona is instantiated through the LLM’s system prompt, which instructs it to adopt the persona’s identity and answer in accordance with it.Source: §A.3 · p. 12

A.4 Score Examples Generation

  • An LLM generates persona- and question-specific examples for each score from 1 to 5, ranging from inconsistent actions to an optimal expected action.Source: §A.4 · pp. 12–14

A.5 Ensemble Evaluation

  • Two LLM evaluators receive a grading rubric and rate responses by matching them to the closest score example, with a justification.Source: §A.5 · pp. 15–16
  • A worked example pairs an aspiring writer from Mexico City with a library-session question and a response about choosing fiction and taking notes.Source: §A.5 · p. 16

B Environments

  • Figure 6 shows the distribution of static environments from which relevant environments are selected for each persona agent.Source: Figure 6 · p. 17

C Qualitative Examples

The examples show both agreement and disagreement between PersonaGym and human scores. The appendices also list personas, formulation notation, and correlation-testing information.

C Qualitative Examples

D Personas

  • The persona examples include varied ages, occupations, locations, interests, and identities; the word cloud notes frequent locations and diverse occupations.Source: D Personas · p. 20, Figure 7 · p. 21

E Formulation Notation

  • Table 4 is identified as the full list of formulation notation and definitions.Source: Table 4 · p. 22

F Significance Testing

  • The significance-testing appendix reports correlation scores with 95% confidence intervals, using an asterisk for p < 0.05.Source: F Significance Testing · p. 23

F.1 Significance Testing Details

The appendix describes bootstrap significance testing for correlations between PersonaScore and human judgments, using Fisher’s z-transformation and confidence intervals to assess significance.

F.1 Significance Testing Details

  • The test draws 10,000 bootstrap samples with replacement from paired human-model scores and applies Fisher’s z-transformation to correlation coefficients.Source: §F.1 · p. 23
  • The 95% confidence interval uses the 2.5th and 97.5th percentiles of bootstrap z-values, transformed back to the correlation scale.Source: §F.1 · p. 23
  • Correlations are significant at p < 0.05 when the 95% confidence interval excludes zero; the appendix describes the approach as combining bootstrap robustness with Fisher’s transformation.Source: §F.1 · p. 23
  • Tables 5 and 6 report Spearman rank and Kendall’s tau correlations, respectively, between PersonaScore and human judgments.Source: Table 5 · p. 24, Table 6 · p. 24
Loading 2407.18416v5…