Source-linked AI summary
A Three-Tier Persona Vector for Controllable User Simulation in Agentic Evaluation
Rahul Khedar, Eshita, Sneha Teja Sree Reddy Thondapu, Mayank Malhotra, Arup Kumar Das, Jitesh Chandra Mishra, Arun Menon, Avinash Karn, Mouli V
TL;DR
User simulation for tool-augmented LLM evaluation lacks realistic, controllable variation when it relies on flat roles or uncontrolled sampling. The paper introduces a fully specified three-tier persona vector with an orthogonal query-complexity overlay and evaluates it across 64,698 conversations. Persona differences produce a 15.8 percentage-point spread in goal achievement, while scenario-reactive states and auditable trait rules support varied, inspectable behavior.
Problem
Existing user simulation approaches provide insufficiently diverse or controllable inputs because flat roles yield similar conversations and uncontrolled sampling makes persona effects difficult to attribute.
Method
The paper specifies a 23-dimensional three-tier persona vector with measurable demographics, noisy behavioral traits, scenario-reactive emotional states, and an independent four-level query-complexity overlay.
Results
15.8 percentage points separate persona goal-achievement rates across 64,698 conversations, with scenario-reactive states producing different behavior across scenarios and domain-specific axes showing persona sensitivity.
Takeaways & Limitations
Structured, continuous, and auditable persona properties provide practitioners with controllable user populations for agent stress-testing across scenarios.
Takeaways & Limitations
Emotional states are fixed for each session, and human evaluation of user-turn realism has not yet disentangled persona-driven difficulty from simulator artifacts.
Abstract
from arXiv · showhide
Evaluating tool-augmented LLM agents requires diverse, realistic user inputs yet most evaluation frameworks use flat role descriptions ("you are an angry customer") that produce near-identical conversations regardless of the underlying scenario. In this paper, we propose a three-tier persona vector with 23 operationalized dimensions: 6 categorical demographics (jurisdiction, age, channel, device, language proficiency, time availability), 12 continuous behavioral traits (patience, assertiveness, digital literacy, etc.) sampled with Gaussian noise around curated profile base vectors, and 5 continuous emotional states (frustration, anxiety, trust, confidence, stress) that shift in response to scenario context. Orthogonal to the persona, a 4-level query-complexity overlay controls utterance phrasing from direct to deliberately vague. We evaluate the persona model inside a synthetic data generation pipeline across 64,698 multi-turn conversations spanning 8 named profiles and 3 production corpora. Key findings: (i) a 15.8 percentage-point spread in agent goal-achievement across personas confirms trait vectors produce measurably different user behavior; (ii) the same persona behaves differently across scenarios due to scenario-reactive emotional state shifts, validating the scenario-reactive design; (iii) domain-specific projects show persona sensitivity on booking-flow compliance (~15-20 percentage points gap between tier-aware and pressure-test personas), demonstrating the model faithfully reproduces real-world difficulty distributions; (iv) seven rule-described trait correlations produce auditable co-occurrence patterns without requiring learned covariance matrices. The persona model is fully specified for reproduction.
1 Introduction
The paper addresses limited realism and controllability in user simulation by replacing flat roles with a structured, measurable persona vector. It evaluates this design across large multi-turn corpora and reports distinct persona-driven behavior.
- Flat role descriptions produce qualitatively similar conversations because they do not constrain vocabulary, assertiveness, domain knowledge, or emotional trajectory.
- Scripted bots are reproducible but require re-authoring for new scenarios, while unconstrained sampling creates variation that is difficult to attribute to persona rather than sampling noise.
- The proposed persona vector p ∈R23 separates categorical demographics, noisy continuous behavioral traits, and scenario-reactive emotional states.
- Continuous values are bucketed before prompt injection, while raw values remain measurable for downstream analysis.
- The complexity overlay independently varies query phrasing from simple to vague, supporting factorial designs and multi-turn clarification.
- 15.8 percentage points separate goal-achievement outcomes across 8 personas, and evaluation covers 64K samples from 3 production corpora.
2 Related Work
Related work spans rule-based, neural, persona-conditioned, and prompt-based user simulation. The paper positions its contribution as session-level, structured control over measurable behavioral variation rather than a new generation mechanism.
- Agenda-based simulators update user goals with hand-written rules, whereas neural simulators generate responses end-to-end without explicit behavioral control.
- PersonaChat and related work condition individual responses on persona descriptions, while this paper generates entire multi-turn sessions from a persona vector.
- Existing evaluation systems provide trajectories or social personas but generally do not decompose user behavior into continuous, measurable trait dimensions.
- Its contribution is a structured specification that makes persona variation measurable and attributable, rather than a new control mechanism.
- The paper describes its target gap as flat descriptions without measurable dimensions, learned embeddings without interpretability, and scripts without generalization.
3 The Three-Tier Persona Model
The model represents users with three persona tiers—categorical demographics, noisy behavioral traits, and scenario-reactive emotional states—then adds an independent query-complexity overlay before assembling the simulator prompt. Trait correlations are encoded through profiles and explicit rules, prioritizing auditability over full covariance modeling.
- Three-tier architecture: The persona vector comprises three tiers: categorical demographics, continuous behavioral traits, and scenario-reactive emotional states.The architecture combines six demographic attributes, twelve noisy traits, and five scenario-shifted states.
- Tier 1: Demographics: Six demographic attributes are independently sampled categorically and converted into prompt-guidance strings before simulation.The attributes are jurisdiction, age bracket, channel, device type, language proficiency, and time availability.
- Tier 2: Behavioral traits: Twelve behavioral traits are sampled around hand-curated profile vectors with Gaussian noise, then bucketed into low, medium, or high prompt guidance.The default noise level σ = 0.08 balances recognizable personas against duplicate-looking samples.
- Tier 3: Emotional state: Five emotional states are initialized from profile-specific ranges and shifted by scenario-dependent deltas, making one persona behave differently across scenarios.For example, disputes increase frustration by +0.25 and decrease trust by −0.20.
- Trait correlations: Trait correlations are produced by profile design and seven threshold-based rules rather than a learned covariance matrix.This trades expressiveness for auditability and requires new correlations to be explicitly authored.
- Query-complexity overlay: An independent four-level query-complexity overlay controls phrasing, with the vague tier forbidding domain vocabulary to force multi-turn clarification.The final prompt concatenates demographic, trait, emotional, correlation, and complexity guidance.
4 Experiments
The evaluation spans 64,698 conversations across three corpora and eight personas, showing that persona traits, emotional context, domain, query complexity, and noise materially affect simulated-user behavior and agent outcomes.
- Experimental setup: 64,698 conversations across three corpora use eight named persona profiles and an eight-axis LLM judge.The corpora include mixed-project, CRM training, and CRM golden-evaluation samples; judge axes include goal achievement, tool use, hallucination, reasoning, communication, consistency, and error handling.
- Persona effects: 15.8 percentage points separate goal-achievement rates for budget-conscious users at 51.2% and tier-1 users at 67.0%.Tier-aware personas cluster higher, while budget-conscious, error-prone, and ambiguous pressure-test personas cluster lower.
- Scenario-reactive emotion: 23% more pushback turns and 1.4 additional turns occur for power_user in dispute scenarios than checkout scenarios.Dispute deltas of +0.25 frustration and −0.20 trust accompany a 0.8-point communication-quality decrease because the user is harder to satisfy.
- Domain-specific sensitivity: Approximately 23 percentage points separate tier-aware and pressure-test personas on CRM booking-flow compliance.Tier-aware personas achieve 63–68% compliance, while pressure-test personas achieve 45–52%; Curious personas fall between the groups.
- Query complexity: The vague query tier lowers goal achievement by 15–18 percentage points across personas, while persona gaps remain roughly 11–18 percentage points across complexity tiers.The reported additivity supports treating persona and query complexity as orthogonal controls.
- Noise calibration: At σ = 0.05, 12.4% of openings are near-duplicates; at σ = 0.15, the bucket flip rate reaches 31.8%.The default σ = 0.08 yields a 14.7% bucket-flip rate and 3.1% duplicate rate, balancing variety and persona coherence.
5 Combinatorial Analysis
The persona architecture yields a very large combinatorial space by combining demographic, behavioral, emotional, and query-complexity configurations, while profile constraints reduce the reachable subset.
- Configuration space: The demographic tier provides 11,520 combinations from six categorical dimensions.The stated calculation is 6 × 6 × 5 × 4 × 4 × 4.
- Configuration space: The behavioral tier provides 531,441 configurations by mapping 12 traits to three buckets each.This is represented as 3^12.
- Configuration space: The emotional tier provides 243 configurations by mapping five states to three buckets each, alongside four complexity levels.The emotional count is represented as 3^5.
- Configuration space: Approximately 5.95 × 10^12 prompt-distinct configurations are supported before profile constraints and correlation rules reduce the reachable space.The design therefore supports millions of unique persona configurations without repetition in practice.
6 Limitations and Future Work
The model has several stated scope and design limitations: rule-described correlations trade expressiveness for auditability, emotional states remain fixed within sessions, bucketing loses information, cultural validity is untested, and judge scores do not isolate simulator artifacts.
- Correlation modeling: Rule-described trait correlations improve auditability but omit emergent correlations unless explicitly authored.A learned covariance matrix could model more realistic co-occurrence, but would reduce interpretability.
- Emotional dynamics: Session-level emotional states do not model frustration escalation after agent failures.Intra-session emotional trajectories are identified as future work.
- Bucketing: Three-bucket discretization loses distinctions between users near bucket thresholds.Finer bucketing or direct continuous injection is under investigation.
- Cultural validity: English-centric, Western-normed prompt guidance limits cultural validity across jurisdictions.The paper notes that jurisdiction-specific payment context does not adjust interaction norms and calls for cross-cultural validation.
- Evaluation attribution: Agent-score effects cannot be cleanly attributed to persona because user-simulator artifacts were not evaluated by humans.Human evaluation of user-turn realism is needed to disentangle persona-driven difficulty from simulation artifacts.
7 Conclusion
The paper concludes that a fully specified three-tier persona vector can provide controllable, measurable user simulation for agentic evaluation. Across 64,698 conversations, it produced differentiated difficulty while offering an interpretable alternative to flat descriptions and learned black boxes.
- Conclusion: The complete specification covers distributions, noise parameters, bucketing rules, and correlation rules.These specification elements support reproduction of the persona model.
- Conclusion: 23 operationalized dimensions separate identity, behavioral traits with noise, and scenario-reactive emotional context.The structure supports factorial experiments and measurable behavioral attribution.
- Conclusion: 15.8 percentage points spread in goal achievement appeared across 64,698 evaluated conversations.The conclusion also reports faithful domain-specific difficulty distributions and orthogonal interaction with query complexity.
- Conclusion: Structured, continuous, measurable, auditable properties give practitioners control to stress-test agents on relevant user populations.This positions the model as a middle ground between flat descriptions and learned black boxes.