PaperlayerView exact revision on arXiv

Source-linked AI summary

Training User Simulators with Latent Intrinsic Feedback

Tarek Naous, Craig Boutilier, Wei Xu, Ofer Meshi

arXiv:2610.13617v1cs.CL

TL;DR

Multi-turn assistant training needs user-value feedback, but existing user simulators focus on utterance generation and rely on external judges to estimate it. This paper trains simulators with latent user-evaluation states; these improve realism and feedback alignment, provide effective RL rewards, and cut RL computation by more than half.

  • Problem

    User simulators have mainly generated realistic utterances, leaving direct first-person assessment of assistant responses to external judges that may misalign with users and add training overhead.

    Source: §1 · p. 1

  • Method

    The paper trains user simulators on real interaction logs with latent evaluation states that represent users’ first-person thoughts, reactions, feedback summaries, and scalar ratings of assistant responses.

    Source: §1 · pp. 1–2, §3.1 · p. 3

  • Results

    Trained simulators improve realism and feedback alignment over baselines, provide effective rewards for multi-turn RL, and reduce RL-run computation by more than half.

    Source: §1 · p. 2, §7 · p. 11

  • Takeaways & Limitations

    The authors conclude that these simulators offer value for multi-turn assistant training and dynamic agent evaluation beyond traditional single-turn static setups.

    Source: §7 · p. 11

    Correlation analysis is inapplicable for these baselines.

    Source: §4.1 · p. 6

Abstract

from arXiv · show

User simulators are gaining attention as environments to train LLM assistants that can better cope with the nuances of multi-turn conversations with real users. Prior work on user simulation has emphasized generating realistic user utterances, leaving unexplored their potential as a source of direct feedback on the assistant's responses. Such evaluation signals may not be explicitly communicated in language, but can instead reside in the latent reasoning of users. In this work, we introduce user simulators with intrinsic feedback, namely, models equipped with latent evaluation states: internal thoughts, reactions, and scalar ratings that evaluate the assistant from a first-person user perspective at each conversation turn. We train LLM-based simulators using interaction logs between real users and assistants and demonstrate that intrinsic feedback improves simulation realism and critiquing behaviors versus prompted simulator baselines, while also providing turn-level evaluations that existing fine-tuned simulators lack. Further, evaluations generated by our trained simulators correlate better with human judgments than those of prompted simulators of the same size. Lastly, our experiments reveal that simulators with intrinsic feedback provide effective reward signals for multi-turn RL training, matching setups that rely on external judge LLMs for rewards while cutting RL compute by more than half.

1. Introduction

The introduction argues that multi-turn assistants still mishandle user goals, while existing simulators mainly generate utterances and leave user-value assessment to external judges. It proposes simulators that model users’ latent evaluations, reporting improved realism and feedback alignment as well as more compute-efficient assistant RL training.

1. Introduction

  • Multi-turn assistant failures include incorrect assumptions about earlier details and losing track of user goals, while existing simulators are primarily utterance generators that rely on external judges for reward estimates.The introduction identifies possible judge misalignment with first-person user assessments and added computational overhead.Source: §1 · p. 1
  • The proposed simulators model both what users say and how they evaluate assistant responses through latent evaluation states.These states include internal thoughts, reactions, feedback summaries, and scalar ratings at each turn.Source: §1 · pp. 1–2
  • Trained simulators improve simulation realism and critique behavior over prompted baselines, with ratings correlating more strongly with human judgments on rated conversations.The comparison also includes fine-tuned simulators without latent states for realism.Source: §1 · p. 2
  • Intrinsic-feedback simulators provide effective RL rewards and reduce computation for an RL training run by more than half, while supporting better assistant training than prompted simulator baselines.The authors also report that these simulators produce competitive assistant LLMs.Source: §1 · p. 2

2. Background on Training User Simulators

The background describes user simulation as modeling a user’s next turn within an assistant conversation, increasingly using real interaction logs rather than prompting alone. It reviews supervised fine-tuning, intent or persona conditioning, and latent states as approaches to realism and control.

2. Background on Training User Simulators

  • User simulators are increasingly trained on real interactions; the conversation setting models a next user turn conditioned on history and an assistant response conditioned on the user turn.Conversation trajectories are decomposed into the alternating user-turn and assistant-response probabilities.Source: §2 · p. 2
  • Supervised fine-tuning on interaction logs maximizes the likelihood of human user utterances, producing realistic utterances but making specific user-intent control difficult.The cited approach by Naous et al. (2026) conditions on high-level user intent; profile- and persona-conditioned approaches are also described (Wang et al., 2025; Wei et al., 2026).Source: §2 · pp. 2–3
  • Intent conditioning can anchor tasks but misses turn-level user dynamics; latent-state simulators instead generate an intermediate state before each user utterance.Prior work reports robustness to assistants unlike those seen in training (Tennenholtz et al., 2026).Source: §2 · p. 3

3. User Simulators with Latent Evaluation States

The section introduces latent evaluation states (LESs) to model both realistic user turns and per-turn evaluations of assistant responses from a user perspective. It describes the data preparation and fine-tuning setups used to train these simulators.

3. User Simulators with Latent Evaluation States

  • LES simulators are trained to generate realistic user turns and per-turn assistant evaluations that match human judgments from a user perspective.Source: §3 · p. 3

3.1. Problem Description

  • An LES contains multiple sub-states assessing an assistant response from the simulated user’s perspective to guide the next user turn.Source: §3.1 · p. 3
  • The five components include user-observable reasoning, a 1–5 satisfaction rating, instruction-following and reaction labels, and brief feedback for critique or praise.Internal_Thought is restricted to constraints users can observe; Feedback is null for Neutral reactions.Source: §3.1 · pp. 3–4

3.2. Training Details

  • After filtering WildChat-1M for toxicity, language, repeated prompt templates, and WildBench overlap, the training data contains 135,901 conversations.Source: §3.2 · p. 4
  • Gemini3.1-Pro generated 238,636 LESs; 14.15% were Critique, 1.26% Praise, and 84.59% Neutral.Source: §3.2 · p. 4
  • The authors fine-tuned Gemma3 4B, 12B, and 27B models with standard SFT or LES generation, with and without intent conditioning, using 129,897 training conversations and a 4,000-conversation held-out set.The held-out set was sampled to include conversations with critiques and those without them.Source: §3.2 · pp. 4–5

4. Evaluating the Feedback Behavior of User Simulators

The evaluation compares simulator feedback behavior and realism against prompted and fine-tuned baselines, then examines latent-state components, intent conditioning, and agreement with human ratings. LES simulators improve key feedback and realism measures, with trade-offs across state designs and stronger human-rating correlations than prompted baselines.

4.1. Evaluation Setup

  • The evaluation uses 4,000 held-out WildChat conversations to measure critique behavior, rating correlation, realism, and perplexity against prompted and SFT baselines.SFT models generate turns but not ratings, so their rating-correlation analysis is inapplicable.Source: §4.1 · p. 5

4.2. Main Results

  • Gemma3 LES simulators outperform same-size prompted Gemma3 baselines on F1 and rating correlation, while SFT slightly leads on F1 and the largest LES model approaches prompted Gemini3.1-Pro.Rating-correlation improvements scale from 4B to 27B; the SFT F1 edge is mainly associated with lower critique recall for LES models.Source: §4.2 · p. 6
  • Across model sizes, LES simulators achieve better realism than prompted baselines and improve over SFT simulators in most cases.The reported gains reflect greater statistical alignment with human turns and lower perplexity.Source: §4.2 · p. 6

4.3. Training Ablations

  • Rating-only latent states yield the strongest feedback behavior but lowest realism; adding language-based states improves realism, and all sub-states give the best balance.The all-sub-state model ranks second on F1 and third on rating correlation.Source: §4.3 · p. 7
  • Training LES simulators with user intent consistently improves their feedback behavior compared with training without intent conditioning.The reported comparison attributes value to intent in steering simulators toward the particular task intent.Source: §4.3 · p. 7

4.4. Correlation with real human ratings

  • On SimulatorArena, LES simulators’ turn-level ratings correlate more strongly with human judgments than prompted baselines across all model sizes.The analysis averages per-turn simulator ratings over each conversation and compares them with human conversation-level ratings.Source: §4.4 · p. 8

5. Training LLM Assistants with Simulators via Multi-Turn RL

The paper tests user simulators and reward sources in multi-turn RL training of an assistant, evaluating checklist satisfaction on WildBenchV2. LES simulators produce the highest mean satisfaction, while intrinsic simulator rewards perform on par with or slightly better than external-judge rewards and require less computation.

5.1. Multi-turn RL Experimental Setup

  • The experiment compares prompted and LES simulators and intrinsic and external-judge turn-level rewards in multi-turn GRPO training of Qwen2.5-1.5B-Instruct.Rollouts use user intents and WildChat conversations; assistants are evaluated on WildBenchV2 checklist satisfaction using a Gemini3.1-Pro judge.Source: §5 · p. 8, §5.1 · pp. 8–9, §5.2 · p. 9

5.2. Results

  • RL-trained assistants using Gemma3-4B-LES achieve higher mean checklist satisfaction than those trained with Gemma3-4B-IT or Gemma3-4B-SFT.All RL-trained setups improve over the initial assistant checkpoint.Source: §5.2 · pp. 9–10
  • Intrinsic rewards produce assistants on par with or slightly better than external-judge rewards; the best overall performance uses intrinsic rewards from Gemma3-4B-LES.The differences between intrinsic and judge rewards are not large.Source: §5.2 · p. 10
  • 2.45 × 10^19 FLOPs for Gemma3-4B intrinsic rewards versus 5.48 × 10^19 FLOPs with the Qwen2.5-14B-Instruct judge (2.24× more computation).The paper reports that a smaller Qwen2.5-3B-Instruct judge led to minimal improvements in initial experiments.Source: §5.2 · p. 10

6. Related Work

Prior work developed prompted, fine-tuned, and latent-state user simulators; this work adds per-turn assistant evaluations and uses them as direct rewards for multi-turn RL.

User Simulation

  • Prompted LLM simulators offer controllability but have been reported to lack realism and drift from real user behavior.Earlier systems also included rule-based and neural simulators for domain-specific task-oriented dialogue.Source: §6 · p. 10

Trained User Simulators

  • Fine-tuning interaction logs can improve realism over prompting but reduce controllability; prior latent-state methods use reasoning traces to support consistency, goal adherence, and policy adaptation.This paper extends that direction by training simulators to produce per-turn first-person evaluations, not only user utterances.Source: §6 · p. 10
  • Simulators trained with latent evaluation states offer better feedback behavior than prompted baselines while also achieving greater realism.The paper distinguishes this contribution from prior simulator training focused on matching real users’ language and behavior distributions.Source: §6 · pp. 10–11

Multi-Turn RL Training of Assistants with Simulators

  • Prior multi-turn RL approaches use external LLM judges for rewards; this work obtains rewards directly from simulator latent evaluations, improving computational efficiency.The authors report a somewhat better RL training signal and eliminate external-judge calls.Source: §6 · p. 11

7. Conclusion

The paper introduces user simulators with latent evaluation states to assess assistants during interaction. The authors report improved feedback alignment and realism, and value for RL training with lower computational cost.

7. Conclusion

  • Latent evaluation states extend user simulators beyond turn generation to evaluate LLM assistants in interactive settings.The paper’s conclusion identifies these simulators as a component for dynamic agent evaluation and multi-turn assistant training.Source: §7 · p. 11
  • Training with latent evaluation states produces simulators with better feedback alignment and greater realism than baseline approaches.Source: §7 · p. 11
  • Latent-evaluation-state simulators provide value for RL-training assistants, with high performance at lower computational cost.Source: §7 · p. 11

A. Dataset: Details

The dataset analysis describes user-turn ratings and reaction types, highlighting differences between critique and neutral turns and the kinds of issues users raise.

A. Dataset: Details

  • Most user-critique turns have ratings from 1 to 3, while neutral-reaction turns mostly have positive ratings from 3 to 5.Source: A. Dataset: Details · p. 15
  • User critiques commonly flag missing content when instructions are not followed, failures in assistant-generated code, factual errors, and formatting issues.Source: A. Dataset: Details · p. 15
  • The dataset analysis stratifies annotated WildChat turns by reaction type and summarizes categories of feedback real users give assistants.Source: A. Dataset: Details · p. 15

B. Human Alignment Score

The human alignment score aggregates text metrics comparing simulator-generated turns with human turns. The section specifies measures of lexical, syntactic, and information-related properties.

B. Human Alignment Score

  • Mean word count per turn, type-token ratio, and hapax legomena ratio capture verbosity and aspects of vocabulary diversity or rarity.Source: B. Human Alignment Score · p. 16
  • Stop-word ratio and mean sentence length measure functional-word use and average sentence length.Source: B. Human Alignment Score · p. 16
  • The metric set includes punctuation density, part-of-speech distribution entropy, and the noun-to-verb ratio.Punctuation density is punctuation marks per character, scaled by 100; the noun-to-verb ratio compares noun and proper-noun counts with verb and auxiliary-verb counts.Source: B. Human Alignment Score · p. 16
  • Word-level Shannon entropy and compression ratio measure word-use variability and text information density or redundancy.The corpus compression ratio averages the per-turn ratios.Source: B. Human Alignment Score · p. 16
  • The human alignment score is 1.0 minus the mean absolute error of ten relative metric ratios against the human reference value of 1.0.Source: B. Human Alignment Score · p. 16

C.1. Additional Results for Gemma-based Simulators

Gemma-based LES simulators correlate more strongly with human ratings than prompted simulators at every scale, significantly at 4B and 12B. Their critiques are also more cautious, with F1 drops driven mainly by lower recall.

C.1. Additional Results for Gemma-based Simulators

  • Drops in critique-behavior F1 for LES simulators versus SFT simulators are driven mainly by lower recall, indicating greater caution about initiating critiques.The passage attributes the F1 decline primarily to recall rather than precision.Source: §C.1 · p. 17
  • The paired-bootstrap analysis compares each LES simulator’s Spearman correlation against a prompted simulator of the same size.It reports 95% confidence intervals and two-sided p-values.Source: §C.1 · p. 17
  • LES-based simulators lead prompted simulators in human-rating correlations at every scale, significantly at 4B and 12B.The paired comparison reports differences in Spearman correlation with 95% bootstrap intervals and two-sided p-values.Source: Table 7 · p. 18

C.2. Results of Training Qwen-based Simulators

Across Qwen2.5 scales, LES simulators achieve the lowest WildChat perplexity and generally improve realism and rating correlations over baselines. The additional RL results show that larger external judges outperform smaller ones, while the plotted setups compare intrinsic and judge-based rewards.

C.2. Results of Training Qwen-based Simulators

  • LES-based Qwen2.5 simulators achieve the lowest perplexity at every scale and generally better realism and rating correlations than baselines on held-out WildChat.The same results show lower recall and higher precision than SFT simulators without latent states, consistent with more cautious critiquing.Source: §C.2 · p. 19
  • LES-based Qwen2.5 simulators correlate better with human judgments than prompted baselines, significantly at the 3B and 7B scales.Table 10 reports pairwise differences in Spearman correlation with bootstrap intervals and Holm-corrected tests.Source: §C.2 · p. 19, Table 10 · p. 20

C.3. Additional Results for RL Training of Assistants

  • Assistants trained with the Qwen2.5-14B-Instruct judge achieve significantly better WildBenchv2 performance than those trained with the Qwen2.5-3B-Instruct judge.The passage identifies the simulator-plus-judge setup’s computational overhead relative to using intrinsic rewards from a smaller trained simulator.Source: §C.3 · p. 21
  • Figure 3 compares mean reward per training step for intrinsic simulator rewards and external Qwen2.5-14B-Instruct judge rewards, with Gemma3-4B-IT and Gemma3-4B-LES simulators.The curves are smoothed using a time-weighted exponential moving average.Source: Figure 3 · p. 22

D. Implementation Details

The implementation details describe simulator and assistant training setups, while the prompts cover latent-state generation, intent conditioning, user role-play, and critique judging.

D. Implementation Details

  • Gemma3-12B and Gemma3-27B simulators were full-parameter fine-tuned on 8 H100 GPUs, requiring about 88 and 208 GPU hours, respectively.Source: D. Implementation Details · p. 22
  • Assistant training used multi-turn GRPO rollouts of up to three turns, with seven GPUs for training and one serving the simulator.Each sample produced eight candidate rollouts; the effective batch comprised 56 trajectories, with a 5e−6 learning rate and KL coefficient 0.001.Source: D. Implementation Details · p. 22
  • Figure 3 plots mean reward per step for all intrinsic-reward and judge-reward training setups, and reports that reward increased during training.Source: D. Implementation Details · p. 22

E. Prompts

  • Generic user intents are generated as simulator system prompts before the first user turn in each WildChat conversation.Source: E. Prompts · p. 23
  • The prompted-simulator template asks instruction-tuned LLMs to role-play users by generating realistic responses and satisfaction ratings.Source: E. Prompts · p. 23
  • The critique judge classifies simulator responses on held-out WildChat and explicitly includes implicit critiques such as re-prompting and elliptical corrections.Gemini3.1-Pro is used as the judge model.Source: E. Prompts · pp. 23, 28
  • The latent-state prompt asks for first-person reasoning based on observable response constraints and outputs instruction-following, rating, reaction, and feedback fields.The rating is 1–5; reaction labels include Neutral, Critique, and Praise, while feedback is a short summary for critiques or praise.Source: E. Prompts · p. 24
Loading 2610.13617v1…