Flipping the Dialogue: Training and Evaluating User Language Models

1 INTRODUCTION

• Prompted assistant LMs generate cooperative, structured user turns that fail to capture users’ gradual, minimal, and sometimes confusing communication.This can overestimate assistant performance in interactive evaluation.Source: §1 · p. 2
• User LMs are post-trained to simulate users and can be conditioned on high-level intent to steer conversations toward target tasks.The approach is designed to preserve realistic language choices while directing the conversation.Source: §1 · p. 2
• In a coding example, indirect but task-relevant UserLM-8b turns cause GPT-4o to fail where direct GPT-4o-simulated turns let the assistant succeed.The example illustrates why realistic user behavior can produce a more demanding evaluation.Source: §1 · p. 2
• User LMs generate more diverse turns, decompose intent across turns, terminate dialogues more appropriately, and adhere more robustly to user roles than existing methods.The paper evaluates these properties through six multi-turn and simulation-robustness evaluations.Source: §1 · p. 2

2.1 PROBLEM DESCRIPTION

• User LMs model three user functions: initiating a conversation from an intent, following up based on assistant responses, and ending the conversation when the interaction is complete.The training objective models user utterances conditioned on intent initially and on intent plus conversation state thereafter.Source: §2.1 · p. 2
• The method flips each real conversation into K+1 training samples, conditioning next-user-utterance generation on the intent and conversation state.Figure 2 summarizes this transformation using in-the-wild human-assistant conversations.Source: §2.1 · p. 2, Figure 2 · p. 3
• High-level intents capture overall conversation objectives without explicit details, balancing task steerability against diverse and realistic language generation.Fully specified intents risk making the simulator merely parrot the intended content, while no intent reduces usability.Source: §2.1 · p. 3

2.2 EXPERIMENTAL DETAILS

• Training uses deduplicated WildChat conversations, GPT-4o-generated generic intents, and full-parameter fine-tuning of Llama3-8b-Base and Llama3.2-1b-Base.The resulting models are called UserLM-8b and UserLM-1b.Source: §2.2 · pp. 3–4

2.3 INITIAL ANALYSES

• The perplexity metric measures how well models predict human user language, with lower PPL indicating less surprisal and better distributional alignment.Evaluation uses held-out WildChat conversations and the out-of-domain PRISM dataset.Source: §2.3 · p. 4
• UserLM-8b achieves the lowest perplexity by a wide margin, often 60-70% lower than all baselines, across WildChat and the out-of-domain PRISM dataset.Intent conditioning improves PPL for all models, while PRISM remains the more challenging test set.Source: §2.3 · p. 4, Table 1 · p. 4
• Intent conditioning during training produces the largest PPL drops and makes User LMs more sensitive to intent at test time.Models trained without intent can still benefit from test-time intent, but training with intent yields greater gains.Source: §2.3 · pp. 4–5, Figure 3 · p. 5
• User LMs trained from base checkpoints achieve lower PPL than those trained from instruction-tuned checkpoints, suggesting base models are closer to natural user-language distributions.The paper attributes this hypothesis to instruction-tuning data being semantically distant from user behavior.Source: §2.3 · p. 5, Figure 3 · p. 5

3.3 RESULTS & ANALYSES

• UserLM-8B achieves 94.55% unique 1-grams versus 94.01% for real users and 74.42% for GPT-4o, while its 2.69% intent overlap approaches humans’ 1.68%.User LMs also achieve a 63.54 dialogue-termination F1, compared with 3–15 for prompted assistants.Source: §3.3 · p. 6
• User LMs produce more natural-looking utterances, scoring 77–81% human-written likelihood versus 0–3% for prompted assistants and 90.2% for real users.These results indicate that prompting an assistant does not substantially shift its generation distribution toward user-like text.Source: §3.3 · p. 7
• Trained User LMs maintain the user role in 91–98% of ambiguous conversations, whereas assistant-prompted simulators revert to assistant behavior in 20–60%.User LMs also follow the original intent in 93–97% of cases, while prompted assistants more readily accept conversational diversions.Source: §3.3 · p. 7
• Scaling improves User LMs consistently: UserLM-8b outperforms UserLM-1b on all metrics, unlike the inconsistent gains observed for prompted assistant models.The comparison spans Llama, GPT-4o, and UserLM model families at varied scales.Source: §3.3 · p. 7

4.1 EXPERIMENTAL SETTING

• The study evaluates 65 math and Python-programming task intents, using 10 simulations per intent with GPT-4o as the fixed assistant.UserLM-8b receives generation guardrails in this extrinsic evaluation to reduce noise from its smaller model size.Source: §4.1 · p. 8

4.2 SIMULATION RESULTS & ANALYSIS

• UserLM-8b covers 76–86% of original-intent information, repeats required details more, and uniquely injects additional demands such as test cases, naming conventions, and implementation constraints.GPT-based simulators reveal intent more monotonically, rarely add demands, and produce more homogeneous simulations.Source: §4.2 · p. 8
• UserLM-8b varies conversations from 2.1 to 6.7 turns, compared with 3.7 to 5.7 turns for GPT-based simulators.This wider range reflects different simulated interaction paces.Source: §4.2 · p. 8
• UserLM-8b achieves higher lexical diversity by varying language and style across simulations, whereas GPT-based simulators retain greater similarity to the original intent.The evaluation also measures intent coverage, information diversity, pace diversity, lexical diversity, and assistant performance.Source: §4.2 · p. 8, §4.1 · p. 8
• Assistant task performance is around 17% lower in conversations with UserLM-8b than with GPT-based simulators.The authors attribute this challenge to UserLM-8b’s greater diversity in information selection, conversational pace, and lexical choices.Source: §4.2 · p. 9

• Purpose-built User LMs are needed because stronger assistant LMs are not necessarily better user simulators.The 1b assistant initialization achieved lower perplexity than the 8b initialization, and GPT-4o-mini surpassed GPT-4o on some intrinsic evaluations.Source: §5 · p. 9
• Personalized User LMs remain a future direction because demographic and domain-specific users may express requests differently.Large-scale data collection for specific settings is challenging, so general User LMs could provide a foundation for later fine-tuning.Source: §5 · p. 9
• User LMs support broader applications including user modeling, user-preference judging, and synthetic data generation.Their varied simulations make them suitable for generating data to fine-tune assistants and improve robustness.Source: §5 · p. 9
• User simulation can scale experiments and expose broad system flaws, but experts remain necessary for technical domains and specific user populations.The proposed scope combines efficient automated evaluation with expert studies of user nuances.Source: §5 · p. 9
• Scaling model size and training data is expected to produce more realistic User LMs and more robust assistants.The released 1b and 8b models already outperform prompted assistants despite limited post-training data.Source: §5 · pp. 9–10

• User simulation progressed from rule-based agendas to neural dialogue systems and, more recently, prompted assistant LMs.Prompted assistants are commonly instructed to generate the next user turn from dialogue history.Source: §6 · p. 10
• Prompted assistant LMs often diverge from human behavior, motivating methods that better reproduce conversational user behavior.Prior findings identify low alignment with real interactions and problems such as overly cooperative simulated turns.Source: §6 · p. 10
• Existing fine-tuning work largely targets task-oriented datasets and synthetic training data, whereas this work introduces general-purpose User LMs guided by defined user intent.The paper reports better alignment with real human behavior than prompting assistant LMs to role-play users.Source: §6 · p. 10
• Simulator evaluation has used lexical, stylistic, diversity, human-annotation, and assistant-task-correlation measures, while this study adds fine-grained multi-turn and robustness evaluations.The study also tests simulators on coding and math tasks with assistants.Source: §6 · p. 10

• The paper introduces approaches to train and evaluate User LMs using user utterances from a large corpus of user-assistant interactions.It also demonstrates their use in simulated conversations with assistants.Source: §7 · p. 10
• More realistic User LM simulations degrade current assistant performance, revealing limitations that simpler prompted simulations can miss.The conclusion presents this as a practical application of User LMs for interactive assistant evaluation.Source: §7 · p. 10

• UserLM-8b may be more prone to toxic language because it was trained from a base checkpoint without conventional safety and alignment post-training.The model is intended for research simulation and similar beneficial uses.Source: ETHICS STATEMENT · p. 11
• UserLM-8b generates realistic user utterances that current state-of-the-art detectors do not easily detect as AI-generated.The authors encourage further work on detectors distinguishing UserLM outputs from human writing.Source: ETHICS STATEMENT · p. 11

A ADDITIONAL EXPERIMENTAL DETAILS

• WildChat deduplication removes near-duplicate prompt templates, producing 384,336 conversations and only 4.19% train-test 7-gram overlap.The filtering and user-based splitting procedure is intended to minimize data leakage.Source: A ADDITIONAL EXPERIMENTAL DETAILS · p. 16
• Our User LMs achieve the lowest cumulative n-gram overlap with generic intents, aligning their utterances more closely with real human behavior.The analysis averages overlap across turns in PRISM conversations.Source: Figure 5 · p. 17
• The dialogue-termination evaluation reports per-turn precision, recall, and F1 for models exceeding an average F1 threshold.The figure focuses on models with average F1 above 10.Source: Figure 6 · p. 17

B.1 PER-TURN ANALYSES

• User LMs outperform instruction-tuned and USP-8b baselines in token-level PPL on human PRISM utterances, especially at the first turn.The result indicates stronger modeling of how real users initiate conversations.Source: §B.1 · p. 17, Figure 4 · p. 17
• User models have the lowest cumulative 1-gram overlap with conditioned intent, aligning more closely with humans than GPT-4o and instruction-tuned Llama models.The comparison measures copying and intent decomposition across turns.Source: §B.1 · p. 17
• GPT-4o rarely terminates dialogues and achieves near-zero F1, whereas UserLM-1b and UserLM-8b balance precision and recall more like humans.Prompted assistants can have high precision but very low recall because they continue conversations instead of ending them.Source: §B.1 · p. 17

B.2 STARTING CHECKPOINT ADDITIONAL ANALYSIS

• User LMs trained from base checkpoints generally outperform those trained from instruction-tuned checkpoints on intent decomposition, dialogue termination, and human-like style.The checkpoint difference is smaller for role and intent adherence.Source: §B.2 · p. 18, Table 4 · p. 18

C.1 GENERATION CONFIGURATION FOR USERLM-8B

• UserLM-8b simulations use generation guardrails that filter first tokens, prohibit termination, constrain length, and remove verbatim repetitions.These controls were introduced because unfiltered outputs from the small model were noisy or repetitive.Source: §C.1 · p. 19

C.2 ADDITIONAL EXAMPLES OF SIMULATED CONVERSATIONS

• UserLM-8b simulations are more diverse and produce greater pace and lexical diversity, creating a more challenging environment that lowers assistant performance.The comparison uses ten simulations per task instruction and contrasts UserLM-8b with GPT-4o simulation.Source: §C.2 · p. 19

C.3 DEFINITION OF SIMULATION EVALUATION METRICS

• The evaluation aggregates eight metrics across five simulator aspects: intent coverage, information diversity, pace diversity, lexical diversity, and downstream behavior.These metrics are designed to characterize multi-turn user simulation rather than assistant quality alone.Source: §C.3 · p. 19
• Intent coverage measures the fraction of instruction shards revealed at least once during a simulated conversation.Sharded instructions separate required and non-required information for analyzing what simulators reveal.Source: §C.3 · p. 20
• Repeat Required measures whether a conversation repeats at least one required shard across turns, highlighting repetition of important information.A score of 1 indicates that at least one required shard is repeated two or more times.Source: §C.3 · p. 20
• Skip Non-Required measures whether the simulator omits at least one non-required shard during the conversation.Non-required shards include secondary information such as trivial base cases or clarifying examples.Source: §C.3 · p. 20
• Additional Demands captures information introduced by the simulator beyond the original shards, including examples, naming conventions, and implementation constraints.Novel information is identified before computing the metric.Source: §C.3 · p. 21
• Turn variance, turn range, and lexical diversity are computed across ten simulations per instruction to measure variation in conversation length and wording.Lexical diversity uses pairwise unigram overlap across simulations sharing an instruction.Source: §C.3 · p. 21
• Assistant task success records whether GPT-4o produces a solution passing held-out coding tests or reaches the correct mathematical solution, measuring simulator effects on downstream performance.This metric evaluates the impact of the user simulator on the assistant rather than the simulator directly.Source: §C.3 · p. 21, Table 5 · p. 22

C.4 SIMULATION RESULTS PER-TASK

• The evaluation aggregates simulator behavior across coding and math tasks, with Table 5 reporting intent coverage, information diversity, pace, lexical diversity, and assistant performance.Math tasks cannot support Skip Non-Required because all shards are required.Source: §C.4 · p. 22
• Math and coding tasks show similar overall trends, but math simulations have nearly complete shard coverage, little turn variance, and a larger UserLM-8b assistant-performance gap.Coding simulations exhibit somewhat greater lexical and turn diversity across all simulators.Source: §C.4 · p. 22

D PROMPTS

• Generic user intents are generated by prompting GPT-4o to summarize each full conversation's high-level intent without specific details.The prompt includes three manually written examples.Source: D PROMPTS · p. 23, Figure 8 · p. 24
• Assistant-based simulators receive the generic intent and conversation history, with prompts encouraging intent decomposition, occasional typos, and nonverbose user turns.The prompting approach follows prior user-simulation work and generates first and subsequent turns.Source: D PROMPTS · pp. 23, 25, Figure 9 · p. 25
• The intent-adherence judge classifies whether a simulator maintains its original request or follows an assistant suggestion that diverts from it.GPT-4o makes the classification from the conversation history.Source: D PROMPTS · p. 23, Figure 10 · p. 25