PaperlayerView exact revision on arXiv

Source-linked AI summary

Intent Speaks Louder: Controllable User Simulation Beyond Response Imitation

Bo Wang, Ruixing Zhang, Yunqi Liu, Yang Zhang, Liangzhe Han, Tongyu Zhu, Leilei Sun

arXiv:2608.09420v1cs.CL

TL;DR

User simulators must choose among multiple plausible next-turn intents, but response imitation alone does not specify which local interaction action to realize. UserIDA makes intent an explicit per-turn directive and, across evaluations, improves intent adherence and controllability while preserving response quality, reaching 86.6% intent accuracy and exceeding the strongest dedicated simulator by 24.3 percentage points.

  • Problem

    Per-turn user intent is underspecified by profiles and global goals, limiting controllable simulation despite multiple valid next-turn actions.

  • Method

    UserIDA combines a six-way intent interface, directive-conditioned supervised fine-tuning, and intent-calibrated group-relative policy optimization.

  • Results

    86.6% intent accuracy exceeds the strongest dedicated simulator by 24.3 percentage points while preserving semantic and stylistic fidelity across evaluated settings.

  • Takeaways & Limitations

    Local interaction intent is a complementary control dimension beyond response imitation for user simulation.

Abstract

from arXiv · show

User simulators are widely used as scalable environments for training and evaluating interactive assistants. Generating the next user turn is inherently one-to-many: the same profile and dialogue context may support multiple plausible continuations with different local interaction intents. A fluent response may therefore advance the dialogue through an inappropriate intent, such as acceptance rather than repair. Our key insight is that controllable user simulation should separate which local interaction intent the next user turn should realize from how that intent is expressed in language. We introduce UserIDA (User Intent-Directive Alignment), which exposes interaction intent as an explicit per-turn directive. UserIDA defines a six-way intent interface, learns directive-conditioned generation through supervised fine-tuning, and uses intent-calibrated policy optimization during group-based reinforcement learning. The reward preserves composite response quality while ensuring that intent-violating candidates rank below compliant alternatives in mixed groups. On LMSYS-USP, UserIDA achieves 86.6\% intent accuracy, outperforming the strongest dedicated user-simulator baseline by 24.3 percentage points while improving semantic and stylistic similarity. In within-context interventions, it realizes at least four of the six target intents in 91.7\% of evaluated dialogue states, compared with 22.9\% for the strongest external baseline. These results establish per-turn intent control as a complementary dimension to response fidelity in user simulation.

Introduction

UserIDA frames per-turn interaction intent as the missing control variable in user simulation, separating what the next turn should do from how it is phrased. It implements this control through a six-way directive interface and directive-conditioned training, improving intent accuracy and controlled interaction success while preserving semantic and stylistic fidelity.

  • Motivation: Interactive assistants require scalable, repeatable environments that reproduce users’ multi-turn behaviors, including clarification, correction, constraint refinement, and completion signals.These behaviors arise because users reveal and revise requests across turns.
  • Motivation: Per-turn interaction intent is missing from standard next-user imitation, leaving contexts with multiple valid continuations but no signal selecting among them.Profiles and conversation-level goals underspecify the next action, whereas target-turn paraphrases overconstrain linguistic realization.
  • Method: UserIDA defines six directives—Initiate, Amend, Supply, Repair, SetRegister, and GroundAccept—to specify how the user changes or closes the local interaction state.The framework presents interaction intent as an explicit per-turn directive beyond response imitation.
  • Method: UserIDA combines directive-conditioned supervised fine-tuning with intent-calibrated policy optimization that ranks compliant candidates above intent-violating alternatives without collapsing quality discrimination.The contribution formalizes a surface-underspecified intent interface and preserves distinctions among compliant responses.
  • Results: 86.6% turn-level intent accuracy exceeds the strongest dedicated simulator by 24.3 percentage points, while controlled-prefix all-turn intent success rises from 13% to 58%.The reported improvements preserve semantic and stylistic fidelity across the evaluated real human–assistant conversations.

Related Work

Prior user simulation has progressed from agenda- and role-play-based generation to trained models and reinforcement learning, enriching identity and long-horizon behavior. Related work commonly optimizes similarity or alignment objectives and evaluates simulators with automatic metrics, LLM judges, and human evaluation.

  • From prompted role-play to trained user simulators: User simulation evolved from agendas, goals, and dialogue acts to prompted role-play and supervised training of profile-conditioned or dedicated user models.This progression is described across classical task-oriented systems, LLM role-play users, and trained user models.
  • From prompted role-play to trained user simulators: Reinforcement learning methods target goal adherence, strategic behavior, or human-likeness, enriching user identity and long-horizon behavior.The cited reinforcement-learning approaches extend the field beyond role-play and supervised user modeling.
  • Learning Objectives and Simulator Evaluation: Most trained simulators optimize next-turn likelihood or reference similarity, with alternatives targeting profile consistency, latent-state alignment, goal alignment, or response indistinguishability.These objectives span both likelihood-based training and recent alignment-oriented alternatives.
  • Learning Objectives and Simulator Evaluation: Simulator evaluation commonly combines automatic similarity metrics and LLM judges with human evaluation, alongside benchmarks including SimulatorArena, ChatBench, and MirrorBench.The passage also mentions recent Sim2Real evaluation efforts, but the supplied text truncates that discussion.

Problem Formulation

The formulation represents each next user turn as a context-dependent realization of an explicit local interaction intent. It separates intent control from response realization while distinguishing retrospective verification, prospective prediction, and externally supplied directives.

  • Dialogue state and user information: A dialogue state c_t contains all preceding user–assistant exchanges through the latest assistant response, with optional implicit user information p.The conversation is defined as d = {(u_1, a_1), . . . , (u_T, a_T )}, and c_t excludes the upcoming user turn u_t.
  • Controlled intent realization: The interaction intent z_t ∈ Z is a categorical local state transition realized by the next user turn u_t.The complete user policy is factorized around this explicit intent variable.
  • Controlled intent realization: Standard imitation models the marginal distribution, whereas the conditional realizer π_θ(u_t | c_t, p, z_t) controls the intended category while generating context-appropriate content, wording, and style.The explicit intent variable addresses the lack of control over which plausible transition is generated.
  • Directive sources and scope: Retrospective verification assigns canonical intents to observed or generated turns, while controlled generation receives z_t directly from an evaluator or scripted policy.A separate prospective predictor h_ϕ(c_t, p) may propose an intent without observing u_t.

Method

UserIDA separates local interaction intent from user-turn wording through a six-intent interface and a three-stage pipeline. It combines directive-conditioned supervised fine-tuning with intent-calibrated group-relative policy optimization to prioritize intent-compliant candidates without discarding response-quality criteria.

  • Pipeline: UserIDA comprises canonical intent annotation, intent supervised fine-tuning, and intent-calibrated policy optimization.These stages form the method’s end-to-end pipeline.
  • Intent interface: The six-label interface captures turn-local changes to dialogue state without specifying target payload or wording.The labels are intended as a primary control interface rather than a complete multidimensional dialogue-act ontology.
  • Directive-conditioned generation: Conditioning on the directive maps the same dialogue state to distinct classes of valid user behavior rather than merely producing user-like language.The renderer supplies dialogue context, optional implicit user information, and the target directive in a fixed chat template, while training uses target user-turn tokens.
  • Intent-calibrated optimization: Intent-SFT addresses directive following, while policy optimization separately calibrates candidate rewards when response quality and intent compliance diverge.Composite quality uses semantic similarity, style similarity, and an AI-likeness penalty; intent compliance is imposed through relative calibration.
  • Intent-calibrated optimization: For mixed groups, calibration guarantees every intent-violating candidate ranks below every compliant alternative by a fixed margin m.All-compliant and all-violating groups retain the original quality reward because no relative intent preference can be established without both candidate types.

Experiments

Experiments show that directive-conditioned training, especially with intent calibration, substantially improves controllable user simulation while preserving semantic, stylistic, and contextual quality. UserIDA achieves strong aggregate and minority-intent control, including in within-context interventions.

  • Main results: 81.98% intent accuracy for Intent-SFT versus 62.28% for USP shows that directive-conditioned learning is substantially more effective than prompting alone.Adding directives improves matched prompted baselines by 6.01–26.16 percentage points, but prompting alone remains insufficient.
  • Main results: UserIDA reaches 86.62% intent accuracy and 0.864 macroF1, exceeding the strongest general-model baseline by 9.28 points and USP by 24.34 points.It also obtains the best point estimates for SimCSE, StyleCSE, Contextual Validity, and User Authenticity.
  • Per-intent results: 73.4% Supply accuracy for UserIDA versus 32.8% for USP with directives demonstrates that gains extend beyond the majority intents.UserIDA achieves the highest accuracy on all six directives, although Initiate and Amend comprise 81.4% of labeled turns and Supply comprises 1.9%.
  • Ablation: 86.6% intent accuracy and 0.59 SimCSE for full UserIDA improve on supervised training, while recovering 5.9 points of intent accuracy over quality-only RL.Quality-only GRPO raises SimCSE to 0.61 but lowers intent accuracy to 80.7%, demonstrating the quality–intent mismatch.
  • Within-context control: 91.67% of evaluated contexts achieve at least four correct interventions for UserIDA, versus 22.92% for the strongest external baseline.UserIDA also exceeds that baseline by 34.03 percentage points in Intent Accuracy and increases realized-intent coverage from 4.29 to 4.71 while maintaining comparable Contextual Validity.
  • Within-context control: Changing only the directive from one fixed dialogue state produces distinct continuations for Initiate, Amend, Repair, GroundAccept, Supply, and SetRegister.These examples show that directives control the local dialogue transition without prescribing a single surface response.

Conclusion … C.3 Intent Distribution

UserIDA separates each next user turn’s local interaction intent from its linguistic realization through explicit directives and intent-calibrated optimization. The paper defines the taxonomy and annotation protocol, formalizes group-relative policy optimization, and documents corpus construction and intent-distribution handling.

  • Conclusion: UserIDA combines a six-way intent interface, Intent-SFT, and intent-calibrated group-relative optimization to improve intent adherence and compositional controllability while preserving response qualities.The evaluation spans turn-level generation, controlled multi-turn evaluation, and within-context intervention.
  • A.1 Design Principles: Directives are turn-local, surface-underspecified, operational, and open-domain, describing the next turn’s effect on the current interaction state without revealing its payload.These principles support consistent application by human annotators and a frozen verifier.
  • A.2 Full Directive Definitions: The main training and evaluation interface uses six canonical directives, while Ambiguous is retained only as an annotation fallback.The taxonomy is presented as the full canonical interaction-intent directive taxonomy.
  • A.3 Canonical Annotation Prompt: The classifier identifies one primary local interaction action from the dialogue state, using profile and dialogue context to reconstruct the active task and the target turn’s relation to the preceding assistant response.The canonical prompt returns a structured JSON object containing confidence, payload, secondary attributes, and rationale.
  • B.1 Group-Relative Advantages: Intent-Calibrated Relative Reward assigns rewards to sampled candidates and optimizes them with a GRPO-based group-relative update inherited from PPO’s clipped surrogate.A frozen behavior policy samples K candidates, and rewards are standardized within each group without requiring a separate value model.
  • B.2 Ordering Preservation under Group Normalization: Group normalization preserves pairwise ordering, so an intent-violating candidate cannot outrank an intent-compliant alternative, although negative normalized advantage depends on the full group reward distribution.Identical rewards produce zero normalized advantages and no directional policy-gradient signal.
  • B.3 Clipped Policy Objective: The policy objective applies sequence-level advantages across tokens, normalizes by response length, clips updates, and penalizes divergence from a frozen reference policy.Intent-Calibrated Relative Reward determines within-group ordering, which the objective converts into a stable update.
  • C.1 Source Corpus and Turn Expansion: The dataset expands conversation-level LMSYS-USP partitions into turn-level examples whose inputs contain the implicit profile and dialogue prefix ending at the preceding assistant turn.Each eligible user turn serves as the generation target, with conversation-level splitting preserved.

C.4 Human Validation of Intent Annotation and Prediction

Human experts audited intent annotations on 1,315 observed test-set turns and evaluated both retrospective verification and prospective prediction against expert reference labels. Retrospective errors clustered at adjacent taxonomy boundaries, while prospective disagreements may reflect alternative valid next intents rather than invalid behavior.

  • Annotation audit: 1,315 observed test-set user turns were independently labeled by three human experts using the canonical intent taxonomy.The audit used intent-stratified sampling and anonymized turns without access to automatic labels.
  • Retrospective verification: 10.32% was the Amend→Initiate confusion rate, the most frequent retrospective-verifier error among adjacent taxonomy boundaries.Other frequent confusions were Repair→Amend at 9.09% and Initiate→Amend at 7.81%.
  • Prospective prediction: Prospective accuracy measures agreement with the observed next intent, not whether every disagreement is interactionally invalid.The prospective predictor lacks the target turn, and multiple next intents may be plausible; its high recall but lower precision on Supply and SetRegister reflects this ambiguity.

C.5 Human Intent Evaluation on Generated User Turns · D Implementation and Evaluation Details

Human expert evaluation shows that UserIDA’s intent-control gains persist beyond the automatic verifier. It consistently outperforms UserIDA w/o RL and USP under balanced and naturally reweighted intent accuracy.

  • C.5 Human Intent Evaluation on Generated User Turns: 90.83% Intent Accuracy under balanced human labels places UserIDA above UserIDA w/o RL at 84.17% and USP at 49.17%.The balanced metric gives each intent equal representation in the audit set.
  • C.5 Human Intent Evaluation on Generated User Turns: 7.18-point advantage over UserIDA w/o RL and 32.23point advantage over USP remains after reweighting to the natural intent distribution.The reweighted metric combines human per-intent accuracies according to the test set’s natural intent distribution.
  • C.5 Human Intent Evaluation on Generated User Turns: All four measurements produce the same system ordering: UserIDA > UserIDA w/o RL > USP.This consistency reduces concern that UserIDA only optimizes for training-time verifier idiosyncrasies.

D.1 Training Configuration … E.2 Per-Directive Accuracy

The paper specifies training and decoding configurations, then evaluates user simulation with intent, fidelity, interaction-quality, and controlled-prefix trajectory metrics. Per-directive accuracy reports both class-balanced and frequency-weighted performance across six canonical directives.

  • D.1 Training Configuration: Training configuration is reported in Table 12.
  • D.2 Rollout and Decoding Configuration: Rollout and evaluation decoding settings are reported in Table 13.
  • E.1 Metric Definitions: Intent Accuracy measures exact agreement between target directives and predictions from the frozen canonical verifier.
  • E.1 Metric Definitions: The six canonical directives are Initiate, Amend, Repair, Supply, GroundAccept, and SetRegister, with Macro-F1 averaging classwise precision and recall equally.Ambiguous is retained for audit accounting but excluded from the macro average.
  • E.1 Metric Definitions: SimCSE and StyleCSE measure semantic agreement and user-side linguistic-style similarity against the observed next-user turn.Neither metric requires exact lexical overlap or directly verifies requested intent realization.
  • E.1 Metric Definitions: A model-blind judge scores reference-free turn quality using directive realization, dialogue coherence, information appropriateness, profile consistency, and user authenticity and economy.Contextual Validity and User Authenticity are reported on a 0–100 scale.
  • E.1 Metric Definitions: Controlled-prefix trajectory evaluation isolates simulator consistency across dialogue states from downstream error propagation.Step Accuracy, Mean Trajectory Accuracy, and All-Turn Success characterize checkpoint- and trajectory-level directive realization, while multi-turn SimCSE and StyleCSE average over evaluated checkpoint turns.
  • E.2 Per-Directive Accuracy: Table 14 reports exact directive accuracy for each target intent, including equally weighted Macro Avg. and empirically frequency-weighted Weighted Acc.The two summaries expose rare-class failures while retaining performance on the natural test mixture.

E.3 Counterfactual Directive Intervention · Target: GROUND_ACCEPT · Target: SET_REGISTER

The counterfactual intervention evaluates whether simulators can realize different directives from identical dialogue states, using frozen, balanced, affordance-screened contexts. The examples show UserIDA producing directive-specific repairs and register-controlled continuations where baselines diverge or collapse.

  • E.3 Counterfactual Directive Intervention: The intervention freezes held-out dialogue states and balances them across six domains, four dialogue-depth buckets, and two profile-salience levels.Only states where every canonical directive is contextually possible are retained.
  • E.3 Counterfactual Directive Intervention: Directive Accuracy compares each requested directive with the frozen verifier’s prediction for the generated next-user turn.The evaluation defines C as 48 contexts and Z as six canonical directives.
  • E.3 Counterfactual Directive Intervention: Macro-F1 averages the six directive-wise F1 scores, while Ambiguous remains a verifier output rather than a seventh target class.This preserves six target directives despite allowing ambiguous predictions.
  • E.3 Counterfactual Directive Intervention: Realized-action coverage distinguishes producing several imperfect actions from collapsing all requested directives into one behavior.Its maximum is six.
  • E.3 Counterfactual Directive Intervention: Table 15 compares USP w/ Dir., UserIDA w/o RL, and UserIDA under identical visible dialogue states, changing only the requested next-user directive.The shared state concerns a detail-oriented creative writer whose assistant proposed a three-act outline for The Elementalists.
  • Target: GROUND_ACCEPT: For GROUND_ACCEPT, USP repeats expansion, Intent-SFT rejects the prior response, and UserIDA requests a better-fitting revision.The three outputs therefore express different continuations of the same proposed outline.
  • Target: SET_REGISTER: For SET_REGISTER, USP gives abstract definitions, Intent-SFT invents a different escape premise, and UserIDA specifies a time-travel story with letters and a World War One ancestor meeting.The UserIDA continuation explicitly preserves the requested storytelling setup and register.

E.4 Recursive Closed-Loop Pilot

The recursive closed-loop pilot evaluates directive-controlled user simulation as generated turns interact with the same assistant and trajectories diverge naturally. Across 24 frozen scenarios, UserIDA improves over USP in directive accuracy, goal success, and session score.

  • Closed-loop protocol: Each generated user turn is sent to the same Qwen3.5-9B assistant, whose appended response informs the next user turn.A model-independent controller observes the dialogue and private scenario state before selecting one of six canonical directives.
  • Evaluation setup: 24 scenarios balanced across six domains and three difficulty levels yield 72 recursively generated sessions and 365 user turns.Unlike fixed-prefix evaluation, trajectories diverge after the first generated turn; this divergence is the behavior of interest.
  • Results: +31.85 percentage points in directive accuracy is UserIDA’s improvement over USP (95% CI [14.17, 40.14]).Directive Accuracy compares the controller directive with the frozen verifier prediction for every generated user turn.

E.5 Cross-Judge Groupwise Evaluation · F Prompt Templates and Model Interfaces

The paper evaluates anonymous user-simulator candidates both independently and groupwise, using intent-aware, reference-free judgments that separate interaction validity from wording. Its prompt templates preserve simulator context while exposing local intent directives and explicitly penalizing directive violations, leakage, and role-inappropriate behavior.

  • E.5 Cross-Judge Groupwise Evaluation: Five anonymous candidates are compared jointly under the same profile, dialogue state, and target intent, with observed turns and model identities hidden.The candidates come from LLaMA-3-8B Base, UserLM, USP, UserIDA w/o RL, and UserIDA.
  • E.5 Cross-Judge Groupwise Evaluation: Both judges produce the identical ordering: LLaMA < UserLM < USP < UserIDA w/o RL < UserIDA.The comparison uses mean per-candidate overall scores rather than scores derived from rankings.
  • E.5 Cross-Judge Groupwise Evaluation: 0.86, 1.25, 2.25, 2.76, and 2.87 are the Qwen-based judge’s mean scores, while the LLaMA-based judge assigns 1.61, 1.87, 2.06, 2.21, and 2.25.Because absolute calibrations differ, the analysis interprets cross-judge rank consistency rather than directly comparing score magnitudes.
  • F.1 Intent-SFT and Controlled Inference Prompt: Intent-SFT preserves the original system prompt, profile, and dialogue history, then inserts a turn-local system message immediately before the next user turn.During training the final user block contains the supervised target; during controlled inference it remains open for generation, and the directive specifies the interaction action rather than target wording.
  • F.1 Intent-SFT and Controlled Inference Prompt: The controlled prompt treats intent as simulation metadata that must not be copied into the user message and defines labels such as INITIATE and AMEND.The supplied template states that the next turn may be controlled by a user-intent signal and provides intent-label definitions.
  • F.2 Reference-Free Turn-Level Rubric Prompt: The turn-level rubric separately scores directive realization, dialogue-state coherence, information appropriateness, profile consistency, and natural, economical user-side expression.Explicit rules cap scores for wrong directives, assistant-role behavior, control leakage, truncation, and empty output; length alone is never rewarded.
  • F.3 Controlled-Prefix Session-Level Rubric Prompt: The session-level evaluator checks each candidate against its own visible state before assessing coherent user role, interaction policy, and goal trajectory across checkpoints.The controlled-prefix protocol prevents later generated turns from being treated as inputs to subsequent checkpoints.
  • F.4 Comparative Groupwise Preference Prompt: The comparative evaluator ranks anonymous candidates under a shared context and target directive, rewarding intent realization, coherence, profile consistency, naturalness, and appropriate detail while penalizing control leakage.Length alone is not rewarded, and a fluent candidate with the wrong interaction directive receives a low intent-fit score; candidate order is independently randomized twice.
Loading 2608.09420v1…