PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentPersonaBench: Benchmarking Persona-Driven User Simulation

Jintao Huang, Yifan Wang, Hongyu Shen, Yi Daniel Lu, Shirley Huang, Minsik Oh, Yewen Wang, Muhammad Ahmed Mohsin, Zhen Xu, Yilan Fan, Zichen Yuan, Ahsan Bilal, Zibu Wei, Sankalp Jajee, Henry Gagnier, Saksham Kapoor, Jicheng Wang, Qianfeng Wen, Yixuan He, Steven Dillmann, Jiashu He, Yucheng Lu, Linqiang Guo, Danyang Zhang, Shi Bo, Raunak Mondal, Haixiang Tang, Weihang Xiao, Allen Nie, Jing Tang, Yueying Li, Yifan Simon Liu, Jianheng Hou, Dianzhuo Wang, Qianyu Zhu, Zhixu Silvia Tao, Zhejian Peng, Zihan Wang, Ishan Gupta, Jinxuan Fan, Wanting Jiang, Shushu Liang, Chenxi Qiu, Yijun Wang, Xiaomin Li, Yuexing Hao

arXiv:2610.04379v1cs.AI

TL;DR

Existing evaluations of persona-driven systems focus largely on conversational style or self-reports, not whether latent traits govern actions. AgentPersonaBench tests unprompted behavior across four interaction surfaces using traits embedded in full profiles. Leading models reach up to 84.7% full-pass adherence, while cross-surface fragility and multi-attribute co-exposure remain challenges.

  • Problem

    Existing evaluations focus primarily on conversational styling or self-reports rather than whether an unmentioned latent trait governs an agent’s observable decisions.

    Source: §1 · p. 2

  • Method

    APB embeds a target trait in a complete persona profile and tests unprompted, observable behavior across survey, chat, web, and app tasks.

    Source: §1 · p. 2

  • Results

    Up to 84.7% full-pass adherence is achieved under unprompted conditions, while cross-surface fragility and multi-attribute co-exposure remain open challenges.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Persona fidelity cannot be inferred from dialogue or aggregate competence; high adherence on one surface can coexist with failure on another.

    Source: §6 · p. 14

    Full-pass and mean normalized credit comparisons do not establish an additional cost of combining traits; that requires matched single- and multi-attribute versions of the same scenario.

    Source: §4.4 · p. 11

Abstract

from arXiv · show

We introduce AgentPersonaBench (APB), a benchmark evaluating whether persona conditioning faithfully steers downstream agent behavior. While language models are increasingly deployed for persona-driven user simulation, existing benchmarks primarily evaluate conversational styling or self-reports rather than authentic behavioral fidelity. APB evaluates latent persona adherence one trait at a time, embedding each target trait within a complete synthetic profile without explicitly naming the trait or disclosing the test. Ground-truth adherence is verified strictly from observable actions across four interaction surfaces of increasing realism: survey, chat, web (interactive web environments), and app (desktop software environments). APB comprises 2,460 tasks spanning 867 traits, verified through automated audits and expert review. Our evaluation of 20 frontier model arms demonstrates that high-fidelity user simulation is already attainable: leading models achieve up to 84.7% full-pass adherence under unprompted conditions. At the same time, APB identifies clear behavioral boundaries: adherence drops across interaction modalities (only 37.9-64.3% pass all four surfaces), multi-attribute demands degrade retention, and competing model families exhibit pronounced behavioral divergence.

1 Introduction

AgentPersonaBench tests whether latent persona traits steer observable decisions, rather than merely shaping conversational style. It embeds traits in complete profiles and evaluates unprompted behavior across four interaction surfaces, finding strong peak adherence alongside persistent challenges.

1 Introduction

  • APB comprises 2,460 tasks spanning 867 traits, embedding each target trait in a complete profile sampled from a one-million-persona corpus.The many realistic background attributes make the target trait compete for attention.Source: §1 · p. 2
  • Tasks span survey, chat, web, and app surfaces, with the target trait unnamed, the evaluation undisclosed, and ground truth hidden from the agent.Audits remove guessing shortcuts, position bias, and superficial prompt leaks before admission.Source: §1 · p. 2
  • Up to 84.7% full-pass adherence is achieved under unprompted conditions, while cross-surface fragility and multi-attribute co-exposure remain open challenges.The evaluation covers 20 model arms across eight providers; reported leaders include gemini-3-8-flash at 84.7%, opus-5-5 at 82.9%, and gpt-6-astra at 82.1%.Source: §1 · p. 2
  • APB measures adherence through unprompted, verified actions rather than conversational self-reports or explicit instructions.Its protocol enforces leak prevention and audit verification.Source: §1 · p. 2

2 Related Work

Prior evaluations cover conversational role-play, aggregate demographic simulation, explicit instruction following, and functional agent competence. APB instead tests whether an unmentioned latent trait governs an individual's concrete actions.

2 Related Work

  • Conversational role-play benchmarks assess voice, frame retention, and biographical recall, while synthetic cohorts assess aggregate statistical alignment rather than individual behavioral fidelity.An eloquent vegan persona selecting meat can pass text-based role-play rubrics while failing in practice.Source: §2 · pp. 2–3
  • Instruction-following, alignment, prompt-robustness, and steerability benchmarks explicitly state target behavior, whereas APB tests whether an unnamed trait guides decisions.Prior work includes instruction following (Ouyang et al., 2022; Zhou et al., 2023; Jiang et al., 2024b), preference optimization (Christiano et al., 2017; Rafailov et al., 2023; Bai et al., 2022b), system-prompt robustness (Qin et al., 2024a; Mu et al., 2025), and tone or viewpoint steerability (Dong et al., 2023; Chen et al., 2025).Source: §2 · p. 3
  • Web, GUI, tool-use, and assistance benchmarks measure functional task execution, while APB treats execution as a baseline control and measures persona-consistent action.The cited benchmark areas include web browsers (Yao et al., 2022; Zhou et al., 2024a; Koh et al., 2024; De Chezelles et al., 2024), operating systems and native GUIs (Xie et al., 2024; Rawles et al., 2024; Trivedi et al., 2024), tool use, and general assistance (Mialon et al., 2023).Source: §2 · p. 3

3 Benchmark Design

APB tests whether an implicitly carried persona trait governs observable decisions, embedding that trait in a complete profile and testing it across four interaction surfaces. Its construction pipeline fixes the trait, builds a matching everyday task, and verifies adherence from resulting artifacts.

3.1.1 Design Principles

  • The benchmark isolates individual traits: single-attribute tasks score one trait, while multi-attribute tasks score co-occurring traits in one artifact.This distinguishes whether a trait persists alone from whether several traits are sustained jointly.Source: §3.1.1 · p. 3
  • Adherence is measured through unprompted actions across survey, chat, web, and app, with the target trait and evaluation hidden from the agent.The recorded evidence includes option selections, dialogue turns, web traces, and application state files.Source: §3.1.1 · p. 3, §3.1 · p. 3

3.1.2 Benchmark Overview

  • APB’s 2,460 tasks span 867 traits, with suite composition varying sharply by trait family and most tasks on the two live surfaces.Live surfaces carry 1,393 of 2,460 tasks; Interests has 1,404 tasks versus six for Professional, so the overview uses proportions as well as counts.Source: §3.1.2 · pp. 3–4, Figure 2 · p. 4

3.2.1 The Construction Pipeline

  • Each task samples a whole persona carrying a target trait, builds an everyday errand that exposes its behavioral consequence, and checks the resulting artifact.The same persona, instruction, and decision contract are preserved across surfaces; roughly four-fifths of checks use deterministic predicates, with a standardized LLM judge for open-ended text.Source: §3.2.1 · p. 5

3.2.2 Working Through an Example

  • The worked examples contrast a single vegan-diet task with a coding-style task that co-exposes three independent habits across the four surfaces.The coding artifact is scored against three rule-based checks, yielding an integer score from zero to three.Source: §3.2.2 · pp. 5–6
  • For the vegan task, adherence is inferred from meal choices and verified from selections, orders, or transaction state rather than an explicit trait prompt.The chat dialogue uses an LLM judge; the survey, web, and app examples verify observable choices or artifacts.Source: §3.2.2 · pp. 5–6

3.3 Verification

  • Tasks undergo automated audits and expert review to reject leakage, shortcut pass strategies, invalid checks, and tasks whose traits are not independent.Automated checks run on every submission; human reviewers assess qualitative validity that automation cannot establish.Source: §3.3 · pp. 6–7

4 Results and Analysis

Across the leaderboard, persona adherence varies by surface, trait domain, attribute load, and model; matched evaluations show that passing on one surface does not guarantee passing on all four. Model differences remain visible across vendors and repeats, though task-level outcomes vary.

4.1 Benchmark Leaderboard

  • Chat is the lowest-scoring surface for 15 of 20 arms, while app is lowest for four of six OpenAI arms; GPT-6 Luna falls from 74.4% single-attribute adherence to 45.7% under multi-trait exposure.Chat includes off-persona conversational pressure; app interactions introduce grounding errors through coordinate clicks and visual perception.Source: §4.1 · p. 9

4.2 Domain-Level Adherence

  • At least 71.4% median pass rates persist across surfaces for health lifestyle, values, and risk attitudes, while food, media, and sports adherence drops sharply on chat.Median food, media, and sports adherence falls from 81.2%, 85.1%, and 66.7% on survey to 49.3%, 49.0%, and 34.4% on chat, respectively.Source: §4.2 · p. 10

4.3 Cross-Surface Consistency

  • Astra passes 80.0–89.3% of scenarios on individual surfaces but 64.3% on all four; GPT-5.6 Luna falls from 85.7% on survey to 37.9% all-four adherence.Figure 6a compares matched survey-minus-interactive full-pass rates; its intervals show that Astra’s 5.0-point survey–app gap crosses zero, unlike several other arms’ gaps.Source: Figure 6 · p. 10, §4.3 · p. 10
  • Matched chat gaps vary by model: Astra’s survey–chat difference is −0.7 points, while GPT-6 Luna’s survey advantage is 12.9 points and GPT-5.6 Luna’s is 15.7 points.For GPT-6 Luna and GPT-5.6 Luna, the reported 95% intervals exclude zero; for Astra, the interval includes zero.Source: §4.3 · p. 10, Figure 6 · p. 10
  • Astra’s survey and chat averages conceal 27 scenarios (19.3%) that change outcome; app has the largest conditional failure rate for every arm, and many failures are limited to one surface.Among scenarios failing on at least one surface, 43.9–62.0% fail on exactly one; the ribbons in Figure 7 track matched outcomes, not a temporal sequence.Source: §4.3 · pp. 10–11, Figure 7 · p. 11

4.4 Joint Trait Adherence

  • On 1,172 shared multi-attribute tasks, Astra passes every check on 81.1% and retains some checks on 16.6%; 71.2–87.8% of failed tasks across six arms retain at least one check.GPT-5.6 Luna has a 42.0% full-pass rate and another 41.3% of tasks with partial credit; mean normalized credit and full-pass rate measure different outcomes.Source: Figure 6 · p. 10, §4.4 · p. 11

4.5 Model Differences Within and Across Families

  • Across 1,086 shared chat/web tasks, Astra has the highest full-pass rate at 84.7%, followed by Opus-5.5 at 84.0% and Gemini-3.8 at 83.3%; 77.8% of tasks have mixed model outcomes.Model failure overlap changes by interface, and the authors caution that selected-run complementarity partly reflects run-to-run variation.Source: §4.5 · pp. 12–13

4.6 Run-to-Run Stability

  • Across three repeats, surface-level pass-rate standard deviations are at most 2.0 points, while GPT-6 Sol leads GPT-6 Luna in all twelve surface–repeat comparisons.Task-level repeat agreement ranges from 74.3% to 92.0%; chat agreement is lower (κ 0.46 and 0.47) than web and app (κ 0.63–0.75).Source: §4.6 · p. 13, Table 3 · p. 13

5 Discussion

Persona fidelity is distinct from general task competence: an agent can complete an interface errand while choosing content that conflicts with the user profile. The discussion also connects cross-surface fragility and multi-trait attentional competition to model-specific behavioral blind spots.

5.1 Decoupling Competence from Persona Fidelity

  • Agents can complete multi-step interface operations yet choose items that violate latent lifestyle or dietary constraints, showing that functional execution alone does not establish persona compliance.In one example, top-tier models complete a device setup but select public social apps contrary to the persona’s skepticism, while another arm chooses local encrypted notes and a weather utility.Source: §5.1 · pp. 13–14

5.2 Cross-Surface Fragility and Attentional Load

  • High survey or dialogue compliance can drop in interactive web and desktop settings, while joint exposure to multiple traits strains lighter models and model families show distinct behavioral blind spots.The discussion suggests heterogeneous routing and surface-specialized ensembles rather than relying on one model, within the scope of these evaluations.Source: §5.2 · p. 14

6 Conclusion

The conclusion presents APB as an action-based evaluation of latent persona adherence across tasks, traits, and interaction surfaces. The supplied appendix overview and related-work passage describe additional materials and a comparison of benchmark paradigms.

6 Conclusion

  • APB evaluates latent persona adherence through unprompted, verified downstream behavior across 2,460 tasks, four surfaces, and 867 traits.The conclusion states that adherence cannot be inferred from dialogue or aggregate competence, and identifies cross-surface failure, multi-attribute strain, and behavioral complementarity.Source: §6 · p. 14

Appendix Overview and Table of Contents

  • The appendix includes detailed adherence breakdowns, reproducible analysis and audit protocols, benchmark limitations, future directions, responsible-use material, and ethical considerations.Source: Appendix Overview and Table of Contents · p. 20

B Extended Related Work

  • The extended related-work section says prior human-simulation research generally centers on conversational character consistency or synthetic population sampling, rather than unprompted background-guided decisions.It also notes that Table 4 summarizes structural differences between APB and representative benchmark paradigms.Source: B Extended Related Work · p. 22

B.1 Persona Agents, Role-Play, and User Simulation

Prior work evaluates conversational persona consistency, synthetic population fidelity, and explicit instruction following, but the supplied passages identify a gap in testing whether latent background identity guides individual unprompted actions. APB frames that problem as implicit behavioral steering.

B.1 Persona Agents, Role-Play, and User Simulation

  • Role-play benchmarks assess character consistency in dialogue, while social-science simulations examine synthetic respondents and aggregate population fidelity; neither tradition tests whether background identity guides unprompted individual decisions.APB instead evaluates individual decisions against known, deterministic ground truth without naming the target trait in the task prompt.Source: §B.1 · pp. 22–23

B.2 Instruction Following, Steerability, and System-Prompt Adherence

  • Explicit instruction-following and steerability benchmarks differ from latent persona adherence, where an agent must infer when a profile trait governs an action that does not mention it.APB isolates this implicit behavioral steering rather than testing directly prompted behavior.Source: §B.2 · p. 23

B.3 Agentic Environments, Computer Use, and Competence Benchmarks

APB repurposes interactive agent environments to measure persona adherence rather than functional competence, and reports adherence by category and interaction surface.

B.3 Agentic Environments, Computer Use, and Competence Benchmarks

  • APB makes persona-conforming choices among functionally valid paths the central challenge, while keeping tool execution from being the principal barrier.This reverses the goal of competence benchmarks that test whether an agent can complete tasks such as configuring a server or buying a product.Source: §B.3 · p. 23
  • A two-stage verification framework combines automated contract audits with expert review for naturalness and unambiguous ground truth.Fine-grained traces also separate interface execution failures from persona violations.Source: §B.3 · p. 23

C Adherence by Category and Surface

  • Table 5 breaks category adherence out by surface; its cells pool single- and multi-attribute tasks, and its final block reports the median across 19 arms.The accompanying description defines n as the smallest scored-trial count over arms in a category; a multi-attribute task counts in every category covered by its checks.Source: C Adherence by Category and Surface · p. 24, Table 5 · p. 24

D Reproducible Analysis Protocol

The reproducible analysis uses official APB release artifacts and recorded transcripts, scoring verified behavioral checks and using matched cohorts for modality comparisons.

D Reproducible Analysis Protocol

  • Analyses use recorded runtime artifacts from the official APB release without additional model generations or changes to original trajectories.Leaderboard and category tables use full-release evaluations, while diagnostics use matched cohorts to isolate modalities and architectures.Source: D Reproducible Analysis Protocol · p. 26
  • Tasks report full pass only when all K_t behavioral checks hold, alongside normalized credit s_t/K_t, based strictly on verifier findings.Verifiers use rule-based assertions or rubric-guided evaluations, with no post-hoc credit inference.Source: D Reproducible Analysis Protocol · p. 26
  • The four-surface single-attribute cohort matches 140 scenarios across 44 personas, and uncertainty uses 10,000 persona-clustered bootstrap resamples.The multi-attribute matched cohort contains 1,172 tasks under identical trait co-exposure conditions.Source: D Reproducible Analysis Protocol · p. 26
  • Adherence rates condition on valid completed executions, while an error-inclusive audit assigns zero credit to incomplete attempts to test ranking sensitivity.The audit reports that relative rankings and performance separations remain stable.Source: D Reproducible Analysis Protocol · p. 26

D.1 Evaluation Coverage and Standardized Environments

The evaluation covers standardized model arms on frozen APB environments, with participation and judging adapted to available interaction capabilities.

D.1 Evaluation Coverage and Standardized Environments

  • The evaluation tests 20 model arms across eight families on 2,460 tasks: 502 survey, 565 chat, 707 web, and 686 app.Each versioned arm records its checkpoint, decoding parameters, and interaction wrapper; scores use frozen release environments.Source: §D.1 · p. 26
  • Models without native vision or desktop GUI capability participate on text surfaces, with app marked N/A when no validated run exists.N/A also denotes any interface without a validated execution run.Source: §D.1 · p. 26
  • Open-ended dialogue checks use a standardized frozen gpt-5.6-luna judge with rubric-grounded scoring prompts.The protocol uses this judge to support comparability across vendors.Source: §D.1 · p. 26

D.2 Robustness to Execution and Verifier Choice

Robustness checks find that model separation persists across verifier types and that rankings remain stable when incomplete executions are penalized.

D.2 Robustness to Execution and Verifier Choice

  • Assigning zero credit to incomplete attempts leaves relative model rankings and tier separations intact across arms.For gpt-5-6-sol chat, full pass changes from 68.2% over 487 scored tasks to 59.2% over 561 recorded attempts; for grok-4-6 app, it changes from 69.9% over 662 scored attempts to 67.5% over 686 recorded attempts.Source: §D.2 · p. 26
  • GPT-6 model arms differ by 17.7 points on rule-only tasks and 11.4 points on judge-only tasks; GPT-5.6 arms differ by 13.0 and 19.4 points, respectively.The reported gaps are full-pass differences on shared scored tasks within each evaluator subset.Source: §D.2 · p. 26
  • The chat/web failure-overlap analysis fixes personas, pinned traits, and the rubric evaluator across 1,086 common tasks from 18 arms.The qualitative case was inspected against the action trace, environment state, and persona specification to illustrate interface competence alongside latent-constraint violations.Source: §D.2 · p. 26

D.3 Repeat-Run Protocol and Empirical Stability

Repeated evaluations show stable outcomes across runs, while additional diagnostics compare cross-surface failures and model complementarity on matched tasks.

D.4 Additional Four-Surface and Cross-Vendor Diagnostics

  • Figure 9 compares model complementarity on 1,086 common chat/web tasks and shows how many models pass matched tasks on each surface.Complementarity is the percentage of one model’s failed tasks passed by another; each row uses its own failure denominator.Source: Figure 9 · p. 27

D.3 Repeat-Run Protocol and Empirical Stability

  • Across surfaces, repeated evaluations vary by at most 2.0 points, and leaderboard rankings remain stable across the audited runs.The study repeats each task three times for gpt-6-sol and gpt-6-luna under identical prompts, environments, and decoding parameters.Source: §D.3 · pp. 26–27
  • Chat repeat agreement is κ≈0.46, while web and app agreement is κ≥0.63, indicating greater repeat consistency in deterministic interfaces.The estimates pool task-level repeat pairs within each surface.Source: §D.3 · p. 27

D.4 Additional Four-Surface and Cross-Vendor Diagnostics

  • The cross-vendor diagnostics define conditional surface failure and pairwise complementarity, measuring failure after a survey pass and failures resolved by another model.These descriptive measures use matched single-attribute quartets and do not fit parametric assumptions.Source: §D.4 · p. 27

E Limitations

APB prioritizes verifiable adherence to declared traits over modeling unconstrained human behavior, while outlining extensions for realism, prediction, and longer-horizon evaluation. Its use is bounded by synthetic diagnostic scope, safeguards, and explicit ethical concerns.

E Limitations

  • APB measures adherence to declared constraints rather than real-world human behavioral distributions, which require complementary validation against human-user telemetry.The paper treats reliable adherence as a prerequisite before evaluating whether agents emulate nuanced human cohort variation.Source: E Limitations · pp. 27–28
  • Single-attribute tasks support clear attribution, while 1,172 multi-attribute tasks test simultaneous constraints; 78.5% of checks are deterministic predicates.Open-ended dialogue uses rubric-anchored LLM judges, and sensitivity audits find model separations robust on deterministic checks alone.Source: E Limitations · p. 28
  • The 20-arm leaderboard uses one evaluation pass, while repeated audits cover representative arms and report variation of σ≤2.0 points.The paper reports 11–19 point tier separations persisting across repeats, deterministic subsets, and error-inclusive analyses.Source: E Limitations · p. 28

F Future Directions and Validation Priorities

  • Future validation includes comparing simulated trajectories with real-user telemetry and testing individual predictions against historical actions and non-persona baselines.The paper also proposes counterbalanced factorial interventions to measure interactions between orthogonal traits.Source: F Future Directions and Validation… · p. 28
  • Because APB uses bounded single-session episodes, future work could test persona persistence over days or weeks and collective dynamics in multi-agent sandboxes.The paper identifies persona drift, forgetting, and context collapse as risks of sustained deployment.Source: F Future Directions and Validation… · p. 28

G Responsible Use and Artifact Release

  • APB personas are diagnostic instruments, not statistical samples of people; prohibited uses include impersonation, surveillance or political dossiers, deceptive persuasion, and protected-group discrimination.The task definitions and persona schemas use CC-BY 4.0, while wrappers and execution harnesses use Apache 2.0; practitioners should record the persona backbone and cross-validate model families.Source: G Responsible Use and Artifact Release · p. 29

H Ethical Considerations

  • APB makes no claim about how real people should behave; the paper calls for clear artificial-identity disclosure and identifies deceptive anthropomorphism and impersonation risks.Risk mitigations include modular attribute sampling, benign scenarios, synthetic profiles, and isolated interactive sandboxes.Source: H Ethical Considerations · p. 29
Loading 2610.04379v1…