PaperlayerView exact revision on arXiv

Source-linked AI summary

PersonaForge: Realistic Multi-Turn User Simulation for Agentic Systems

Hanglong Lv, Dawei Zhu, Lei Li, Bowen Ye, Huaqiu Liu, Yifan Song, Bofei Gao, Weimin Xiong, Jinhao Dong, Chenhong He, Lingpeng Kong, Qi Liu, Tong Yang, Fuli Luo

arXiv:2608.28378v1cs.CL

TL;DR

Agentic training and evaluation often assume complete single-turn queries, despite real users progressively disclosing intent and correcting systems. PersonaForge simulates these interactions through structured personas, controlled behavior, and authentic seed queries, and its training data improves benchmark performance across evaluated models.

  • Problem

    75.9% of 16K real-world interactions are multi-turn, exposing a gap between incomplete, user-steered agentic workflows and complete-query assumptions in training and evaluation.

  • Method

    PersonaForge combines a 4-dimensional persona space, SOUL behavioral control, Reverse Deep Construction from real seed queries, and live interaction with deployed agents.

  • Results

    +4.1% composite score on Qwen3.5-27B and +15.7% on MiMo-V2-Flash demonstrate improvements across all four evaluated dimensions after PersonaForge training.

  • Takeaways & Limitations

    PersonaForge and PersonaForge-Bench provide a foundation for training and evaluating agents under realistic multi-turn user interaction.

  • Takeaways & Limitations

    MBTI is used for controllability rather than psychometric validity and is not claimed to capture real personality structure.

Abstract

from arXiv · show

Large language models are increasingly used as agentic workflow executors, yet existing training data and benchmarks largely assume informationally complete, single-turn queries. Our analysis of 16K real-world sessions shows that 75.9% of interactions are multi-turn, revealing a substantial gap between how users interact with agents and how such systems are trained and evaluated. We introduce \textbf{PersonaForge}, a user simulation framework for synthesizing realistic multi-turn user--agent interactions. PersonaForge combines a four-dimensional persona space, SOUL-driven behavioral control calibrated to real-user statistics, and Reverse Deep Construction grounded in authentic seed queries. Using PersonaForge, we construct a 6.3K-record training dataset and \textbf{PersonaForge-Bench}, a manually annotated 138-task benchmark spanning over 20 professional domains with four-dimensional scoring. Experiments on Qwen3.5-27B show that PersonaForge training improves the composite score by +4.1%, with gains across all four dimensions and the largest improvements in Task Completion (+6.0%) and Response Quality (+6.8%). Further analyses show that PersonaForge-trained agents use fewer turns and tool calls, suggesting improved interaction efficiency, while ablations confirm the contribution of SOUL components and adaptive simulation. Together, PersonaForge and PersonaForge-Bench establish a foundation for training and evaluating agents under realistic multi-turn user interaction.

1 Introduction

Agentic systems increasingly execute multi-step workflows, but training and evaluation often model users as providing complete single-turn queries. PersonaForge addresses this mismatch by simulating realistic, incomplete, evolving, and corrective user interactions.

  • Existing tool-use synthesis commonly generates fixed-schema trajectories from fully specified queries, missing workflows with incomplete and evolving user intent.
  • 75.9% of 16K real-world interactions are multi-turn, with a median of 10 user turns and corrections requested in 38.7% of multi-turn sessions.These interactions include evolving requirements, direction changes, intermediate-output inspection, and error correction.
  • PersonaForge combines a 4-dimensional persona space, SOUL-driven behavioral control, and Reverse Deep Construction from authentic seed queries.The persona dimensions are occupation, personality, technical proficiency, and knowledge background.
  • 6.3K quality-filtered conversations are produced for training, while PersonaForge-Bench evaluates realistic user-agent interaction.
  • +4.1% on Qwen3.5-27B and +15.7% on MiMo-V2-Flash are the composite-score gains from PersonaForge training across all four dimensions.The largest gains occur on task completion and response quality.

2 Multi-Turn Interactions in the Wild

Analysis of real agentic sessions shows heterogeneous tasks, incrementally disclosed intent, execution-coupled interaction, and corrective feedback. These patterns establish why realistic simulation must reproduce user steering across multi-turn workflows.

  • Real agentic use is heterogeneous and situated: Task volume spans seven categories, including data analysis, system/DevOps, coding, file operations, IM/bot interactions, writing, and information retrieval.The categories imply different user backgrounds and communication norms.
  • Intent is disclosed over time: 75.9% of sessions involve at least two user turns, with a median of 10 turns, contrasting with a deployed assistant’s 100% single-turn training corpus.
  • User turns are coupled to tool execution: Multi-turn sessions average 99.3 tool calls versus 9.6 in single-turn sessions, alongside more intermediate content and user turns.User messages respond to files, command outputs, retrieved information, partial calculations, and prior assistant decisions.
  • User feedback is corrective: 38.7% of multi-turn sessions contain explicit correction signals, compared with 19.4× the single-turn rate; 36.0% of corrections follow tool execution.Correction targets include context loss, wrong approach, format errors, ignored instructions, and hallucination.
  • Realistic simulation must model users and tasks jointly, reveal intent over turns, and react to intermediate outputs and assistant errors.

3 PersonaForge

PersonaForge converts authentic seed queries into trainable multi-turn user-agent trajectories by constructing coherent user profiles, controlling turn-level behavior, and filtering generated conversations for quality.

  • PersonaForge decomposes synthesis into profile construction, conversation generation, and quality control.The stages align user identity with the task, control intent revelation and reactions, and prevent simulator failures from entering training data.
  • Profile Construction: A 4-dimensional persona space links occupation, personality, technical proficiency, and knowledge background to authentic seed-query needs.Profiles are not sampled independently from tasks; the seed query anchors the user’s task need.
  • Profile Construction: SOUL represents identity, personality-behavior mappings, communication style, and connected memories for consistent user simulation.Connected memory preserves session-specific facts and was the single most impactful individual component in ablation analysis.
  • Conversation Generation: Information asymmetry lets the simulator access hidden state and actual agent execution while the target system sees accumulated messages and tool outputs.This supports gradual intent disclosure, short follow-ups, realistic inconsistencies, and corrective messages.
  • Conversation Generation: Fixed behavioral rules enforce brevity, one question per turn, active verification, persona consistency, and natural termination.

4 PersonaForge-Bench

PersonaForge-Bench is designed around realistic, sustained user-agent interaction rather than one-shot execution. Its task organization supports fine-grained domain reporting and reproducible aggregation.

  • The benchmark’s design targets the interaction properties identified in real sessions: heterogeneous domains, progressive intent disclosure, and corrective feedback coupled to tool execution.
  • Each benchmark task carries a fine-grained domain tag, aggregated into eight reporting buckets through a released task-to-tag-to-bucket mapping.The keyword priority lexicon and full mapping are also released.
  • Tasks derive from realistic user scenarios and require sustained multi-turn interaction instead of one-shot task execution.The benchmark evaluates user-LLM interaction trajectories.

5 Experiments

PersonaForge is evaluated through controlled experiments, ablations, efficiency analyses, robustness checks, and transfer to held-out tasks. Training improves performance across models and dimensions, with especially large gains for MiMo-V2-Flash and evidence of more efficient interaction.

  • Setup: The evaluation compares PersonaForge-trained Qwen3.5-27B and MiMo-V2-Flash models with base and proprietary systems on 138 PersonaForge-Bench tasks, using three trials per task.Evaluation averages scores across trials and tasks and additionally tests transfer on held-out CLAW-EVAL tasks.
  • Main Results: +4.1% composite gain on Qwen3.5-27B accompanies Task Completion (+6.0%) and Response Quality (+6.8%) improvements, with all four dimensions improving.The base model reaches 76.2% composite and 88.1% Tool Appropriateness before PersonaForge training.
  • Main Results: +15.7% absolute composite gain on MiMo-V2-Flash raises performance from 60.4% to 76.1%, including Task Completion (+22.0%) and Tool Appropriateness (+10.1%).The reported composite improvement is 26.0% relative, and the effect is larger than on Qwen3.5-27B.
  • Ablation Studies: Removing connected memory reduces Qwen3.5-27B composite performance from 80.3% to 78.9%, while removing flaws and traits lowers it further to 78.2%.The largest connected-memory drop is Task Completion (−3.5%); a small Tool Appropriateness inversion is within the reported noise band.
  • Ablation Studies: PersonaForge outperforms fixed real-user replay by +4.7% composite and +8.5% Task Completion, indicating value from adaptive simulation beyond replayed human transcripts.The replay baseline preserves authentic utterances but cannot adapt to assistant responses.
  • Robustness: Additional analyses report robustness to scoring protocol, judge choice, response length, formatting bias, training method, and transfer to an independent benchmark.The reported improvement persists within length quartiles and after regression adjustment for response length.
  • Analysis: MiMo-V2-Flash uses 20.7% fewer turns, 9.2% fewer tool calls, and 54.2% fewer web_fetch calls after PersonaForge SFT, while 25-turn-limit tasks fall from 73 to 30.Task Completion simultaneously increases by +22.0%.
  • Analysis: Held-out CLAW-EVAL performance improves from 59.0% to 69.1% (+10.1%) for MiMo-V2-Flash, while Qwen3.5-27B improves from 68.9% to 70.3% (+1.4%).Pass@3 and Pass3 also increase for MiMo-V2-Flash.

6 Related Work

Prior work advances agentic tool use and interactive evaluation, but commonly evaluates fixed tasks rather than realistic users who progressively steer agents. Persona modeling diversifies synthetic behavior, while existing approaches also raise concerns about systematic bias.

  • Agentic LLMs: ReAct, Toolformer, and AutoGen develop reasoning, tool use, and conversational multi-agent workflows for agentic systems.These systems motivate the broader study of agents operating through tools and external environments.
  • Agentic LLMs: Agent benchmarks such as AgentBench and SWE-bench evaluate executable, interactive tasks, but most use fixed tasks rather than progressively steering users.The paper positions PersonaForge as addressing this user-interaction setting.
  • Persona modeling and user simulation: PersonaHub and related work use persona-based generation to diversify synthetic behavior, while LLM-generated personas can introduce systematic biases.This motivates careful persona design and controllability.

7 Conclusion

PersonaForge synthesizes realistic multi-turn interactions, while PersonaForge-Bench evaluates agents under progressive user steering. Across architectures and scales, training improves performance and transfers to held-out agentic tasks.

  • Conclusion: PersonaForge synthesizes realistic multi-turn user interactions, and PersonaForge-Bench evaluates agentic systems under progressive user steering.Together they provide complementary data-synthesis and evaluation resources.
  • Conclusion: Training on PersonaForge data improves models across dense and MoE architectures and across model scales, with gains transferring beyond PersonaForge-Bench.The conclusion frames realistic user simulation as a practical approach for studying and improving agentic interaction.
  • Conclusion: The experiments provide insight into managing incomplete intent, correction, and tool-intensive workflows.These interaction properties are central to the paper’s realistic user-steering setting.

Limitations

The paper bounds its claims through explicit persona-design caveats, fictional high-stakes scenarios, heuristic interaction analysis, and implementation choices affecting coherence and coverage. These constraints shape how the framework and reported statistics should be interpreted.

  • Persona design choices: MBTI is used for controllable behavioral diversity rather than as a psychometrically valid model of real personality structure.The paper notes that Big Five or other trait models could substitute with similar diversity effects.
  • Ethical considerations: Medical and legal benchmark scenarios are fictional composites, and model outputs are not intended as professional advice.The analyzed interaction logs were collected with opt-in consent and anonymized.
  • Analysis methodology: Descriptive statistics rely on deterministic regular-expression matching after normalization and keyword-based task assignment.Correction and failure-mode counts depend on predefined expression families and lexicons.
  • Persona coherence: Coherent persona dimensions produce consistent trajectories, whereas mismatched dimensions create contradictory simulated behavior.The paper illustrates this contrast through Figure 6 and a non-technical user paired with expert jargon.
  • Analysis methodology: Correction sessions remain binary at the session level even when multiple failure families match the same correction.The histogram counts all matched families, but the session-level correction rate does not.
  • Simulation setting: The target agent cannot see the simulator’s hidden SOUL prompt, connected memories, or task scenario, only accumulated user messages and tool outputs.This information asymmetry is a deliberate simulation design choice.
  • Profile construction: Reverse Deep Construction begins with real seed queries and infers persona profiles and scenarios, while forward sampling can produce plausible but disjoint memories.The paper reports using the reverse path for all synthetic training data.
  • Profile construction: Connected memory is reported as the single most impactful SOUL component for downstream task completion.This finding is consistent with the paper’s comparison of reverse-constructed and forward-sampled profiles.

B.5 Data Quality Analysis

PersonaForge’s quality-control pipeline filters incoherent personas, insufficiently complex seed queries, role reversal, and degenerate sessions. Its benchmark uses progressive disclosure, contradictions, domain-specific rubrics, and multiple scoring dimensions to evaluate realistic interaction.

  • Data quality control: 12% of persona combinations and 8% of seed queries are removed before generation for coherence violations and insufficient complexity.Runtime role-reversal detection flags 7% of sessions, while post-processing removes 4% for infinite loops or repetitive exchanges.
  • Data quality control: A 1,500-character role-reversal threshold reflects real-user behavior, with 99.2% of follow-up messages below the cutoff.Raising the threshold to 2,000 characters lowers flagged sessions to 3% but admits borderline reversals; lowering it to 1,000 raises flags to 12% and may discard legitimate code-heavy messages.
  • Benchmark design: PersonaForge-Bench contains 138 Chinese and English tasks, with the same multi-turn protocol and four scoring dimensions applied across both language subsets.The benchmark spans established construction axes including live user simulation, persona diversity, progressive disclosure, contradictions, corrective behavior, human verification, and per-task rubrics.
  • Benchmark design: PersonaForge-Bench tasks reveal information progressively and embed contradictions, requiring assistants to ask clarifying questions and reconcile conflicting details.Each task includes domain-specific rubrics, reference calculations with tolerance windows, and step-by-step decision chains.
  • Benchmark design: The benchmark applies four equally weighted dimensions: Interaction Efficiency, Tool Appropriateness, Task Completion, and Response Quality.Efficiency rewards fewer turns and tool calls at comparable quality, while Response Quality assesses clarity, accuracy, and directness.

E Judge Reliability and Robustness Analysis

Multiple audits indicate that the evaluation gains are reliable across judges, human scoring, response lengths, and formatting conditions. The SFT advantage remains after controlling for response length and structural presentation, supporting an interaction-quality improvement beyond verbosity or formatting.

  • Judge reliability: Pearson r = 0.94 between LLM-judge and human scores, with both protocols agreeing on improvement magnitude.The human audit covered 60 conversations, all four dimensions, both models, and a range of score levels.
  • Judge reliability: Pearson r = 0.84 across 828 scored traces shows strong agreement between the primary judge and Gemini 3.1 Pro.Both judges estimated the PersonaForge gain at nearly identical levels.
  • Length robustness: +24.8% is the largest SFT composite advantage in the shortest response-length quartile, and the advantage remains positive in every quartile.This pattern contradicts the hypothesis that gains arise only from longer responses.
  • Length robustness: +29.6% is the SFT advantage in the clean shortest-quartile subset after excluding zero-output and turn-ceiling traces.The subset compares 45 Base and 39 SFT traces, with scores of 48.1% and 77.7%, respectively.
  • Regression robustness: βSFT = +0.087 (p = 0.001) after controlling for response length, while length and SFT identity together explain 33% of variance.The raw composite improvement is +0.157, with the SFT effect accounting for most of it.
  • Formatting robustness: SFT gains remain positive and similar across structure-stratified subgroups, indicating Markdown tables and headings do not explain the improvement.The “No Headings” subgroup is excluded because it contains only 4 SFT traces.

F.1 Data Recipe and Transfer Analysis

Matched-scale controls and independent transfer tests examine whether PersonaForge’s gains arise from its data recipe and generalize beyond its benchmark. The gains are strongest on MiMo-V2-Flash, while Qwen3.5-27B shows significant improvements mainly in Task Completion and Response Quality.

  • Data recipe: −2.7% composite [CI −6.0, +0.7] results from APIGen-MT-5k, while ToolACE yields −5.5% [−9.4, −1.6] on matched-scale Qwen3.5-27B training.These controls suggest that comparable data quantity alone does not reproduce PersonaForge’s gains.
  • Transfer: 63.0% to 65.3% is the BFCL v3 multi-turn improvement for a LoRA model trained on PersonaForge data, with gains in every failure category.BFCL v3 contains 800 independent rule-scored tasks without an LLM judge.
  • Per-domain analysis: Task Completion and Response Quality show the largest absolute gains across most domains.The paper associates this pattern with converting tool-execution-heavy trajectories into actionable answers.
  • Per-language analysis: The aggregate language result is driven primarily by Chinese tasks: MiMo-V2-Flash improves +17.4% on Chinese tasks, while the 28-task English subset is directionally consistent but not significant.On Qwen3.5-27B, neither language split reaches significance.
  • Model results: +15.7% is the MiMo-V2-Flash overall gain, with improvements significant across all four dimensions under both judge families.The conclusions replicate under Gemini 3.1 Pro, and per-domain tables report all four dimensions plus Overall.
  • Model results: +4.1% is the Qwen3.5-27B composite gain; Task Completion improves +6.0% and Response Quality +6.8%.Tool Appropriateness and Interaction Efficiency fall within noise for this model.

G Training–Evaluation Data Separation

PersonaForge-Bench is independently curated from real consultation scenarios and separated from the training corpus. Exact, fingerprint, substring, and semantic-similarity checks found no meaningful training–evaluation overlap.

  • Independent construction: The 138 benchmark tasks and PersonaForge training corpus are constructed through independent pipelines with no shared data.Benchmark seeds come from real internal consultation scenarios and are manually revised with rubrics, contradictions, and progressive disclosure.
  • Overlap analysis: Zero exact-string, first-150-character fingerprint, and substring matches were found between 6,333 candidate training records and 138 benchmark prompts.These checks were designed to detect direct prompt overlap before near-duplicate removal.
  • Semantic similarity: Mean TF-IDF character n-gram cosine similarity was 0.010 across all 138 × 6,333 training–benchmark pairs, confirming distributional distinctness.Two benchmark queries had elevated similarity of at least 0.80 to candidate training queries, requiring additional review beyond the mean statistic.

H Ablation Details

The ablations show that SOUL components jointly shape realistic simulated-user behavior, with connected memory especially important for progressive disclosure and downstream task completion. Blind evaluations further indicate that the full system approaches real-user realism, while reverse-constructed scenarios preserve seed intent after quality control; however, attribution cannot be refined beyond a first-order component-to-behavior-to-outcome account.

  • SOUL ablations: Connected memory is the only single-component ablation producing multiple significant behavioral effects, including shifts in progressive disclosure, lexical diversity, message length, and topic switching.Its removal yields four significant metric effects, whereas removing personality, flaws, or behavior parameters individually produces small effects on most metrics.
  • SOUL ablations: The Minimal condition causes the strongest degradation, with |d| = 0.34–0.53 across metrics and five significant effects.Removing personality, human flaws, and behavior parameters together degrades behavior more than removing any one component.
  • Progressive Disclosure: 0.80 is the full SOUL mean InfoGrowth ratio, compared with 0.54 for Minimal and 0.58 after memory removal.The results associate connected memory and the full persona specification with progressive revelation of requirements across turns.
  • Blind realism: 51.6% overall discrimination in the blind realism study was indistinguishable from chance, while the full system was judged human 72% of the time versus 75% for real sessions.SOUL-ablated and forward-sampled conditions were judged human less often, at 66% and 62%, respectively.
  • Intent faithfulness: 96.0% of reverse-constructed scenarios preserved seed intent by majority vote, and no evaluation channel found a drifted construction across 600 judgments.The audit sampled the quality-controlled corpus used for training; strict unanimity still classified 70.5% as preserved.
  • Downstream impact: −3.5% Task Completion follows memory removal, while removing the redundant trait ensemble together costs −6.0% downstream.The reported attribution assigns memory to what is disclosed and when, and the trait ensemble to how disclosure unfolds.
  • Attribution boundary: Persona variables explain under 10% of annotation variance, so the component-to-behavior-to-downstream chain remains only a first-order attribution.MBTI is used as controllable scaffolding without a psychometric claim.
Loading 2608.28378v1…