PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards User Simulators That Stay On Task And Still Sound Natural

Mustafa Omer Gul, Philippe Laban, Tanya Goyal, Jennifer Neville

arXiv:2610.31512v1cs.CL

TL;DR

Current LLM systems degrade in prolonged interactions, and existing user simulators trade off task-intent adherence against natural behavior. This paper uses multi-turn RL with a dynamically retrained discriminator to optimize both, and UserLM-2.0 produces valid conversations more than three times as often as the strongest prior simulator. A substantial gap from real conversations remains.

  • Problem

    Current LLM systems degrade over prolonged interaction, while prompted and trained user simulators trade off following complex intents against natural conversational behavior.

    Source: §1 · p. 2

  • Method

    The paper post-trains an SFT user simulator with multi-turn RL rewards for task fidelity and behavioral naturalness, using a dynamically retrained discriminator for naturalness.

    Source: §1 · p. 2

  • Results

    46.2% of task intents yield valid UserLM-2.0 conversations, versus 14.1% for the strongest prior user simulator, SFTUser-14B.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The validity metric measures the proportion of conversations usable for downstream purposes by requiring both task-fidelity and naturalness criteria.

    Source: §3.4 · p. 6

    More than 80% of human reference conversations are valid, over 30% above UserLM-2.0, leaving a substantial gap from real interactions.

    Source: §5 · p. 10

Abstract

from arXiv · show

Users increasingly rely on AI systems for extended, multi-turn tasks, yet evidence indicates that LLM-based system performance degrades over prolonged interaction. To evaluate and improve AI assistants in such settings, we need user simulators that faithfully follow complex user intents while exhibiting realistic conversational behavior. Existing approaches manifest a trade-off: prompted assistants can follow complex user intents but lack behavioral fidelity, while specialized user simulators better approximate conversational behavior but struggle to remain faithful to complex intents. In this paper, we propose a reinforcement learning (RL) recipe that jointly rewards task intent adherence and behavioral naturalness. A crucial contribution of the method is operationalizing naturalness with a lightweight discriminator trained in an alternating, adversarial loop with the user simulator. We release the resulting model, UserLM-2.0, which produces valid simulated conversations (i.e., meeting both task-faithfulness and naturalness criteria) 46.2% of the time, over three times the rate of the strongest prior user simulator (14.1%).

1 INTRODUCTION

Extended, multi-turn tasks expose performance degradation in current LLM systems, making evaluations that reflect real user interaction important. This paper proposes an RL recipe that jointly optimizes task fidelity and behavioral naturalness, yielding more valid conversations than the strongest prior user simulator.

1 INTRODUCTION

  • Evaluating assistants in sustained interactions requires user simulators that take a high-level task intent and generate plausible utterances through iterative interaction.Users increasingly expect assistants to support complex, multi-turn tasks rather than isolated queries, while prior work reports performance degradation in these settings.Source: §1 · p. 2
  • User simulators must convey task requirements without omission or contradiction while matching real users’ message phrasing and interaction across turns.Omitting requirements can make benchmarking unreliable, while failing to capture real interactive behavior may limit transfer to real interactions.Source: §1 · p. 2
  • Prompted assistants can stay on task but lack behavioral naturalness, while SFT-trained simulators often behave realistically but fail to enact all specified requirements.The paper illustrates the contrast with a prompted assistant’s unusually polished, exhaustive messages and an SFT simulator that ends prematurely.Source: §1 · p. 2
  • The RL recipe jointly rewards task fidelity and behavioral naturalness using task-fidelity judges and a dynamically retrained discriminator for naturalness.The discriminator is continually retrained on real and generated conversations to keep its reward responsive as the simulator changes.Source: §1 · p. 2
  • 46.2% of task intents yield valid UserLM-2.0 conversations, versus 14.1% for the strongest prior user simulator, SFTUser-14B.Validity requires quality and termination criteria, at least 80% intent coverage, and passing the naturalness criterion.Source: §1 · p. 2

2 METHOD

The method starts from an SFT user simulator and post-trains it with multi-turn RL to balance task fidelity and behavioral naturalness. Its reward combines coverage and naturalness with quality and termination checks, while periodically retraining the discriminator to address reward hacking.

2.1 PROBLEM DEFINITION

  • The simulator is initialized with an SFT model trained on user-assistant interactions, then post-trained with multi-turn RL to improve task fidelity without sacrificing naturalness.The reward balances both goals because optimizing one alone can produce behaviors such as putting all information in the first utterance.Source: §2.1 · p. 3

2.2 REWARD DESIGN FOR REINFORCEMENT LEARNING

  • The naturalness reward uses a lightweight discriminator trained to distinguish real from simulated user utterances, with the task intent as input; the binary reward is 1 above a 0.5 score threshold.The authors favor trained discriminators over LLM judges because naturalness includes implicit dimensions such as surface form, conversational pace, and follow-up coherence.Source: §2.2 · p. 3
  • Coverage measures the proportion of task-intent sub-goals conveyed, but can be reward-hacked by paraphrasing them; an abstract input intent and fuller reward intent discourage this behavior.The simulator receives the more abstract intent c, while the coverage judge uses the fully specified intent c*; this is intended to encourage more diverse behavior.Source: §2.2 · pp. 3–4
  • Quality is a binary prerequisite that penalizes intent repetition, repeated user turns, and role switching; termination rewards ending only after all sub-goals are met.The overall reward is quality multiplied by the sum of coverage, naturalness, and termination rewards, so a zero quality score makes the overall reward zero.Source: §2.2 · p. 4

2.3 GAN-STYLE TRAINING WITH DYNAMIC DISCRIMINATORS

  • Periodic discriminator retraining addresses reward hacking observed with static discriminators, which simulators can exploit by lengthening utterances and reducing conversation turns.The authors report that static-discriminator training reached maximum naturalness reward within 75 steps.Source: §2.3 · p. 4

3 EXPERIMENTAL SETUP

The experiments use processed WildChat interactions, train a multi-turn user simulator and an alternating naturalness discriminator, and compare against prompted assistants and specialized simulators. Evaluation combines task-fidelity and naturalness measures into a conversation-validity metric.

3.1 DATASET CONSTRUCTION

  • WildChat-4.8M’s non-toxic subset was reduced from 3,199,860 to 199,249 multi-turn interactions, with held-out data allocated to RL and discriminator training and the remainder to SFT.The study also uses Qwen3-30B-A3B-Instruct-2507 to extract intents and prompted GPT models to extract subgoals.Source: §3.1 · p. 4
  • The balanced test set contains 2,000 intents spanning one to seven subgoals, with four conversations sampled per intent using Qwen3.5-9B as assistant.The test set is constructed by balancing the number of intents per subgoal count.Source: §3.1 · p. 5

3.2 TRAINING DETAILS

  • UserLM-2.0 uses Llama3.1-8B-Instruct, SFT initialization, and RL conversations capped at five user turns with Qwen3.5-9B as assistant.The RL configuration sets λ = 0.2 for termination reward, batch size 64, group size 8, and KL hyperparameter β = 0.001.Source: §3.2 · p. 5
  • A balanced ModernBERT-Large discriminator is retrained every 25 RL steps, using half recent-step conversations and half sampled from the remaining training history.For each intent, training pairs one simulated and one real conversation; the discriminator trains for five epochs and selects the checkpoint with best validation accuracy.Source: §3.2 · p. 5

3.3 BASELINES

  • Comparisons include prompted assistant LLMs, specialized fine-tuned simulators, and held-out reference users.Fine-tuned baselines include USP, UserLM-8B, UserLM-2.0-SFT, SFTUser-14B, and HumanLM-Opinion; prompted models span small and large open- and closed-source systems.Source: §3.3 · p. 5

3.4 EVALUATION METRICS

  • Evaluation uses 2,000 test intents, four conversations per intent, three task-fidelity metrics, three naturalness metrics, and an aggregate conversation-validity measure.Validity requires Quality = 1, Terminated = 1, Coverage of at least 0.8, and passing the naturalness criterion; estimated cost is reported per 100 valid conversations.Source: §3.4 · pp. 5–6
  • Task fidelity reuses Terminated, Quality, and Coverage; discriminator training uses user turns because adding assistant turns made conversations perfectly distinguishable by style.Coverage uses GPT-5.6-Luna to extract additional per-turn intent information.Source: §3.4 · pp. 5–6
  • Naturalness passes when at least one of the simulator-specific, general, or Pangram scores reaches 0.5.The simulator-specific discriminator is tailored to each simulator, while the general discriminator is trained on data from multiple simulators and is described as less strict.Source: §3.4 · p. 6

4 RESULTS AND ANALYSIS

UserLM-2.0 leads the reported comparisons in overall validity, combining strong task coverage with naturalness. Across harder intents and behavioral analyses, it remains more reliable than baselines, while reward alternatives show that validity depends on both task fidelity and naturalness.

4.1 MAIN RESULTS

  • 46.2% validity is 3.28 times the best previously released user simulator, SFTUser-14B, at 14.1%.Table 1 summarizes evaluation results across prompted assistants, finetuned models, and human references.Source: §4.1 · pp. 6–7, Table 1 · p. 7
  • UserLM-2.0 has the highest simulator coverage (88.0%) and naturalness pass rate (71.8%); the best prompted assistant, GPT-5.6-Sol, passes naturalness at 10.2%.Prompted assistants tend toward high coverage but lower naturalness, while finetuned simulators tend toward higher naturalness but lower coverage.Source: §4.1 · p. 6, Table 1 · p. 7

4.2 STRESS TESTING UNDER INCREASING TASK COMPLEXITY

  • At 5 subgoals, UserLM-2.0 validity is 36.6% versus 11.3% for UserLM-2.0-SFT; at 6 and 7 subgoals, it is 23.6% and 12.6%, versus below 1% for every prior simulator.Validity generally declines as task complexity increases, but UserLM-2.0 leads at every tested level.Source: §4.2 · p. 7, Table 2 · p. 8

4.3 BEHAVIORAL ANALYSIS OF SIMULATORS

  • UserLM-2.0 matches human repetition rates (39.0% revisitation; 33.3% repetition-only turns) and has 4.3% empty turns, but uses 59 words per goal versus 34 for humans.Table 3 reports repetition and verbosity metrics; its darker shading indicates closer agreement with humans.Source: Table 3 · p. 8, §4.3 · pp. 8–9
  • On conversations of at most six turns, UserLM-2.0’s goal-disclosure dynamics most resemble human references, with less front-loading and repetition and 90% end-of-conversation coverage.Figure 2 encodes cumulative coverage by turn, with newly covered subgoals in bold and revisited subgoals in gray.Source: Figure 2 · p. 9, §4.3 · p. 9

4.4 REWARD DESIGN ALTERNATIVES

  • Dynamic discriminators outperform the fixed-discriminator and LLM-judge naturalness alternatives; removing either naturalness or coverage rewards lowers validity.Without the naturalness reward, discriminator pass rate is 42.0% versus 71.8% for the full model; without coverage, validity is 19.0% versus 46.2%.Source: Table 4 · p. 10, §4.4 · p. 10

5 DISCUSSION & IMPLICATIONS

The discussion identifies a remaining gap to real-user conversations and outlines limits involving interaction settings, discriminator design, privacy, and assistant diversity. It also argues that simulator evaluation should combine corpus-level and simulation-level tests.

The remaining gap between UserLM-2.0 and real user conversations

  • More than 80% of human reference conversations are valid, over 30% above UserLM-2.0, despite imperfect metrics; the authors see this as a substantial remaining gap.They note the gap is roughly as large as the gain from the prior best simulator to UserLM-2.0, from 14.1% to 46.2%.Source: §5 · p. 10

Poorly chosen naturalness rewards can be worse than none at all

  • LLM-judge and fixed-discriminator naturalness rewards can make simulations less natural, while dynamically retraining the discriminator helps prevent reward hacking.A fixed discriminator can be fooled by convergence to a narrow range of word- and turn-counts.Source: §5 · pp. 10–11

UserLM-2.0 is trained and evaluated only on chatbot interactions

  • UserLM-2.0 is trained and evaluated on chatbot interactions from WildChat-4.8M, not agentic interactions involving tools or environments.The authors identify diverse agentic user-AI interaction data as a challenge for extending simulators to those settings.Source: §5 · p. 11

Alternative designs for the discriminator

  • The naturalness discriminator sees user utterances and task intent, but not assistant utterances; alternative conditioning choices could define distinct rewards.The authors suggest exploring turn- or conversation-level discriminators and combining formulations.Source: §5 · p. 11

User simulators for real-world use-cases must take privacy into account

  • Training on user data risks regurgitating private information, so simulator development should pair with privacy-aware training or naturalistic synthetic data.UserLM-2.0 is initialized through supervised finetuning on WildChat-4.8M.Source: §5 · p. 11

User simulators should be robust to a variety of assistants

  • Training and evaluation used Qwen3.5-9B as the assistant, leaving robustness to other assistant behaviors as a next step.The authors specifically include both state-of-the-art and poorly or inconsistently performing assistants as relevant settings.Source: §5 · p. 11

User simulators should be evaluated holistically

  • Corpus-level behavioral alignment does not guarantee usable individual simulations, so evaluation should include both corpus- and simulation-level tests.Their validity metric combines task fidelity and naturalness, though that combination can vary by domain.Source: §5 · pp. 11–12

6 RELATED WORK

Prior user simulation spans rule-based and neural methods, prompted LLMs, and finetuned dialogue models, with a recurring tension between task fidelity and naturalness. Evaluation approaches vary, while this work applies adversarial discriminator training to user simulation.

User Simulation

  • Prompted LLM assistants can enact specified task intents but often align poorly with human behavior, while finetuned dialogue models are naturalistic but brittle and weak in task fidelity.The paper positions RL over an SFT initialization as a way to optimize both properties.Source: §6 · p. 12

Evaluation for User Simulators

  • User-simulator evaluation lacks consensus and includes behavioral-statistic comparisons, assistant-performance approximation, and human judgments, which can be costly.This work uses AI-detection models for generic naturalness and corpus-trained discriminators for dataset-specific performance.Source: §6 · p. 12

GAN-Style Training for Language Models

  • Earlier adversarial language-model work includes dialogue generation, but GAN-style training has largely been abandoned for LLMs; this paper applies it to user simulation.The authors contrast trained discriminators with prompted LLM judges, which they report harm naturalness.Source: §6 · p. 12

7 CONCLUSION

UserLM-2.0 uses reinforcement learning to jointly optimize task fidelity and naturalness, with naturalness operationalized through a periodically retrained lightweight discriminator. It produces valid conversations more often than the next-best simulator, with the advantage widening at the highest task complexity.

7 CONCLUSION

  • 46.2% of UserLM-2.0 conversations are valid, over three times the next-best simulator; at highest task complexity, validity is 12.6% versus 0.8%.The discriminator is periodically retrained on real and model-generated conversations.Source: §7 · p. 12

A DATASET CONSTRUCTION

The dataset pipeline filters WildChat conversations, constructs user-separated splits and balanced test intents, and extracts generic and fully specified intents from user turns. The generic-intent prompt uses conversation text formatted according to few-shot examples.

A.1 DATASET FILTERING

  • The WildChat filtering removes non-English, heavily templated, single-user-turn, toxic, and apparent prompt-injection conversations.The process also excludes repeated opening turns and selected recurring opening n-grams.Source: §A.1 · p. 18

A.2 DATASET SPLITS

  • Users are split by country into train, validation, and test sets at 90/5/5; the 2,000-example test set spans intents with 1–7 sub-goals.The test set is subsampled from the remaining 9,608 conversations and balanced across sub-goal counts, though it contains fewer 1- and 7-sub-goal intents.Source: §A.2 · p. 18

A.3 INTENT EXTRACTION

  • Generic intents are extracted from user turns with Qwen3-30B-A3B-Instruct-2507, while fully specified intents are derived by extracting and then genericizing sub-goals.RL data extraction uses GPT-5.3 and test extraction uses GPT-5.6-Terra; Figure 3 shows the generic-intent prompt template and its conversation placeholder.Source: §A.3 · pp. 18–19, Figure 3 · p. 19

B.1 SFT TRAINING DETAILS

The SFT setup adapts the model for user-turn generation and trains on turns conditioned on prior conversation. Intent extraction uses a two-step prompt sequence, from detailed conversation-specific goals to genericized subgoals.

B.1 SFT TRAINING DETAILS

  • The model adds an <|endconversation|> token initialized with mean embeddings and changes the chat template to use a user role header for generation.The token embedding and language-model head embedding are initialized to the mean embedding value.Source: §B.1 · pp. 19, 21
  • The extraction prompts first produce numbered, conversation-specific goals, then genericize them using the generic intent and detailed goals.The first prompt substitutes the conversation’s user turns; the second inserts the generic intent and detailed intent verbatim.Source: Figure 4 · p. 20, Figure 5 · p. 21
  • SFT uses one user turn per datapoint, conditioned on the prior templated conversation, and selects the checkpoint with the best validation loss.Training runs for one epoch with batch size 1024, learning rate 2e-5, weight decay 1e-4, and AdamW; validation loss is measured every 100 steps.Source: §B.1 · p. 21

B.2 RL TRAINING DETAILS

RL training builds on the SFT model, using simulated conversations and judge-based rewards for coverage, quality, and termination. Checkpoint selection favors naturalness above a minimum coverage threshold.

B.2 RL TRAINING DETAILS

  • RL uses GRPO on UserLM-2.0-SFT for 156 steps, with batch size 64, group size 8, and one on-policy update per rollout.Training uses learning rate 1e-6, KL loss weight 0.001, and validation every 25 steps.Source: §B.2 · p. 21
  • The rollout loop caps conversations at six user turns and five assistant turns, and treats a conversation that continues after the sixth user turn as unterminated.User turns use temperature 1.0; assistant turns use temperature 0.7, top-p 0.8, and top-k 20.Source: §B.2 · p. 21
  • Coverage judging counts addressed numbered subgoals against the fully specified intent, while quality judging checks the generic intent and user turns.A rule-based check for verbatim repetition runs before the quality judge; termination reward is 0.2, and the discriminator is retrained every 25 steps.Source: §B.2 · pp. 21–22
  • The quality-prompt examples are manually labeled by error mode or validity, and the selected prompt achieves 85% accuracy with no errors on valid conversations.Coverage-prompt selection uses MAE against subgoal counts, with the best prompt scoring 0.773.Source: §B.2 · pp. 22–23
  • Checkpoint selection chooses the lowest-JSD model above 85% validation coverage; when no checkpoint clears that threshold in the No Coverage experiment, it chooses among the two highest-coverage checkpoints.JSD compares utterance-length and turn-count histograms with Human Reference conversations.Source: §B.2 · p. 23

B.3 DISCRIMINATOR TRAINING DETAILS

The discriminator is trained on balanced simulated and real conversations, using only user turns. The appendices also detail judge prompts, baseline inference, per-turn coverage extraction, and naturalness metrics.

B.3 DISCRIMINATOR TRAINING DETAILS

  • The 395M ModernBERT-Large discriminator sees one simulated and one real conversation per task intent, using only user turns because assistant styles made conversations perfectly discriminable.This construction keeps the discriminator dataset balanced.Source: §B.3 · p. 23

B.4 LLM JUDGE PROMPT CONSTRUCTION

  • LLM-judge prompts use Qwen3.5-9B with greedy sampling, while few-shot selection targets Likert MAE and accuracy at identifying real conversations.The selected Likert prompt has MAE 1.202, and the selected winrate prompt has 71% accuracy at picking the real conversation.Source: §B.4 · p. 23

C.1 FINETUNED BASELINE INFERENCE PROMPTS

  • USP and HumanLM-Opinion prompts are adapted to include the task intent and <|endconversation|> tag while retaining their existing system prompts.Inference hyperparameters match those used for RL validation.Source: §C.1 · p. 23

C.2 ASSISTANT BASELINE INFERENCE PROMPTS

  • Prompted assistant simulators use one prompt for the first user turn and another for every subsequent turn.GPT-5.6-Luna and GPT-5.6-Sol allow up to 2048 tokens per user turn to account for reasoning tokens.Source: §C.2 · p. 24, Figure 8 · p. 24

C.3 COVERAGE PROMPT WITH PER-TURN SUBGOAL EXTRACTION

  • Main-evaluation coverage uses GPT-5.6-Luna to extract covered subgoals per user turn, enabling pacing analyses and finer-grained metrics.The prompt is shown in Figure 14.Source: §C.3 · p. 24

C.4 NATURALNESS METRICS

  • Naturalness-Specific and Naturalness-General discriminators share training hyperparameters and backbone, but use different sampling and balancing procedures.Naturalness-General samples three simulated conversations per intent and upsamples ground-truth interactions; model reporting uses the best of three runs by validation macro-averaged accuracy.Source: §C.4 · p. 25

C.5 BEHAVIORAL METRIC DEFINITIONS

The behavioral metrics quantify repeated, empty, and verbose user turns, with distinct aggregation rules for Words/goal. The stress test reports naturalness components by task-subgoal count, alongside quality and termination results.

C.5 BEHAVIORAL METRIC DEFINITIONS

  • Revisitation, repetition-only, and empty measure previously mentioned goals, turns containing only previously covered goals, and turns covering no subgoal, respectively.Turn-level annotations count each goal at most once per turn and exclude assistant turns.Source: §C.5 · p. 25
  • Words/goal is total user-turn words divided by distinct subgoals, reported as the median across eligible conversations; other metrics average within intents and then equally across intents.Words/goal instead pools eligible conversation-level ratios across intents before taking the median.Source: §C.5 · pp. 25–26
  • Figure 2 analyzes intents with Human Reference conversations of at most six turns, plotting cumulative coverage, new subgoals, and revisitation by turn.Each turn’s averages pool partial conversations leading up to that turn.Source: §C.5 · p. 26

D.1 TASK COMPLEXITY STRESS-TEST

  • Table 5 reports Naturalness-Specific, Naturalness-General, and Pangram component means across one to seven task subgoals; Table 6 reports quality and termination rate.Table 5 values are rounded to whole percentages, with darker shading indicating higher unrounded scores within each metric’s scale.Source: §D.1 · p. 27, Table 5 · p. 27

D.2 TRAINING DYNAMICS

The training-dynamics analysis finds that UserLM-2.0 fools earlier discriminators while newly retrained discriminators detect earlier simulated rollouts. Dynamic discriminators also help maintain alignment with human output-length and turn-count distributions.

D.2 TRAINING DYNAMICS

  • Figure 16’s rows represent validation steps, columns represent the evaluating discriminator, and each cell averages Pdisc(y = real) for those rollouts.The analysis evaluates validation rollouts with discriminators trained during RL.Source: §D.2 · pp. 27–28, Figure 16 · p. 30
  • UserLM-2.0 keeps fooling earlier discriminators, while retrained discriminators detect earlier rollouts and remain robust to reward hacking.Each retrained discriminator uses the most recent 25 steps and simulated conversations from earlier in training.Source: §D.2 · p. 28, Figure 16 · p. 30
  • UserLM-2.0 and No Coverage sustain low Jensen-Shannon Divergence (JSD) against human length and turn-count histograms, while alternative reward variants increasingly diverge.The LLM judge variants ultimately reach up to 400 words per turn on average; lower JSD is better.Source: §D.2 · pp. 28–29, Figure 15 · p. 29
Loading 2610.31512v1…