PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning User Representations to Improve Simulation Fidelity

Shuhaib Mehri, Philippe Laban, Michel Galley, Dilek Tür, Jennifer Neville, Jianfeng Gao

arXiv:2610.22518v1cs.CL

TL;DR

User simulators can produce generic behavior, while representation generation has received comparatively little attention despite its role in determining which user behaviors a simulator captures. The paper trains UserRepGen-4B with hindsight distillation, using feedback from simulated conversations to improve representations. It reports more faithful simulation than GPT-5.5 and benefits for unseen tasks, population diversity, and distinguishing users across conversations.

  • Problem

    Current simulators struggle to reproduce diverse real-user behavior, and representation generation is an underexplored component of simulation fidelity.

    Source: §1 · p. 2, §3 · p. 3

  • Method

    The paper trains UserRepGen-4B with hindsight distillation, using feedback from comparisons of original and simulated conversations to improve representations.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    UserRepGen-4B enables more faithful simulation than GPT-5.5, and learned representations improve fidelity for unseen tasks, preserve population-level diversity, and better distinguish users across conversations.

    Source: §1 · pp. 2–3, §8 · p. 10

  • Takeaways & Limitations

    Simulation fidelity depends not only on generating realistic behavior but also on capturing relevant information in user representations.

    Source: §8 · p. 10

    The datasets do not ensure representative user-population coverage, and generalization to other languages and populations is unconfirmed.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

User simulators enable interactive training and evaluation of AI assistants at scale. Yet current simulators struggle to reproduce the diverse behaviors of real users, leading to assistants that generalize poorly to underrepresented users. We identify user representations as an underexplored bottleneck in user simulation. User representations are natural-language descriptions that condition simulators to reproduce a particular user's behavior. A representation's quality becomes apparent in hindsight, through the behavior it induces in a simulator. To evaluate and optimize user representations, we formulate a conversation reconstruction task: given an original conversation, we generate a representation, condition a simulator on it, simulate a conversation, and then measure fidelity by comparing user behavior in the original and simulated conversations. This comparison also yields hindsight feedback, which identifies where the representation falls short and how it can be improved. We introduce hindsight distillation, an online training procedure that distills feedback-conditioned improvements into the representation generator, enabling it to generate improved representations directly from the original conversation. We present UserRepGen-4B, a representation generator trained with hindsight distillation. Across 700 conversations spanning seven datasets, hindsight feedback helps improve user representations at test time, and UserRepGen-4B generates representations that enable more faithful simulation than those generated by GPT-5.5. Beyond conversation reconstruction, learned representations improve simulation fidelity for unseen tasks, better preserve the population-level diversity of real users, and better distinguish users across conversations.

1 INTRODUCTION

The paper identifies user representations as an underexplored bottleneck in simulation: missing behavioral information can lead simulators to produce generic rather than user-specific behavior. It introduces conversation reconstruction and hindsight distillation to train representations for greater simulation fidelity, and presents UserRepGen-4B with broader benefits beyond reconstruction.

1 INTRODUCTION

  • User representations are an underexplored bottleneck because missing information can lead simulators to fill gaps with assumptions and generic behavior.Existing work primarily optimizes simulators, while representation generation has typically received comparatively little attention and used ad hoc prompting.Source: §1 · p. 2
  • Conversation reconstruction evaluates a representation by conditioning a simulator on it, comparing simulated and original user behavior, and using the comparison as hindsight feedback.Collecting feedback requires expensive multi-turn simulations; hindsight distillation uses feedback-improved representations as supervised targets so the generator can produce them directly from the original conversation.Source: §1 · p. 2
  • UserRepGen-4B generates representations that enable more faithful simulation than proprietary models such as GPT-5.5, with improvements also reported for unseen tasks, population-level diversity, and distinguishing users across conversations.The generator is trained with hindsight distillation and evaluated across 700 conversations spanning seven datasets.Source: §1 · pp. 2–3
  • The paper identifies structured representations, hindsight distillation, and analysis of learned representations as contributions alongside UserRepGen-4B.The representations are intended to improve fidelity for unseen tasks, preserve population-level diversity, and distinguish users across conversations.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work studies user simulators, user representations, and learning from privileged information. This paper focuses on learning representations for fidelity and uses simulator behavior in multi-turn conversations as privileged information for improving the representation generator.

User simulation

  • User simulation research has moved from prompting general-purpose LLMs toward specialized simulators, while recent studies document behavioral homogenization and mismatch with real user populations.The paper addresses this problem by conditioning simulators on learned user representations.Source: §2 · p. 3

User representation

  • Prior user representations include personas, goals, demographics, interview transcripts, and learned vectors; recent work also explores underspecification, scaling, population diversity, and behavior-distribution matching.This paper instead learns representations intended to capture user-specific information needed for faithful simulation.Source: §2 · p. 3

Learning from privileged information

  • Where prior privileged-information methods use environmental feedback or multi-turn rollouts to improve models or agents, this work uses simulator behavior induced by a representation to improve the representation generator.Source: §2 · p. 3

3 TASK DESCRIPTION

The paper frames user simulation as generating a representation from a real conversation and conditioning a simulator on it. It treats representation generation as a learning problem, introduces a task-and-behavior representation, and evaluates representations through conversation reconstruction.

3 TASK DESCRIPTION

  • A representation generator maps a real user–assistant conversation to a natural-language representation, which conditions an LLM-based simulator to produce a simulated conversation.Source: §3 · p. 3
  • Representation generation has received comparatively little attention, and missing behavioral information can prompt generic rather than user-specific simulator behavior.Prior work has largely focused on optimizing the simulator, motivating optimization of the representation generator for downstream fidelity.Source: §3 · pp. 3–4
  • The structured representation r = (t, b) separates the conversation-specific task from behavior patterns that persist across conversations.This enables combining different behavior descriptions with a fixed task, or simulating different tasks with fixed behavior.Source: §3 · p. 4
  • Conversation reconstruction measures whether a representation enables a simulator to preserve user behavior observed in the original conversation.A fixed simulator, assistant, and judge form each environment; the judge scores fidelity across dimensions using interaction context rather than requiring exact reconstruction.Source: §3 · p. 4
  • The training objective is to generate representations that maximize fidelity across fixed simulators.Source: §3 · p. 4

4 HINDSIGHT FEEDBACK

The paper treats a representation’s quality as visible in hindsight through the behavior a simulator reproduces or misses. It collects feedback by comparing simulated and original behavior, then conditions the generator on that feedback to improve representations iteratively.

4 HINDSIGHT FEEDBACK

  • A representation’s strengths and omissions become apparent through the behaviors a simulator reproduces faithfully or fails to reproduce.The paper describes these observations in natural language as hindsight feedback.Source: §4 · p. 4
  • Hindsight feedback is collected across an ensemble of simulation environments built on different underlying models, whose judges compare simulated and original user behavior.The passages introduce the environments and feedback collection, though the displayed formalization is incomplete.Source: §4 · p. 4
  • The generator improves a representation by conditioning on the conversation, current representation, and collected feedback, repeating the process for multiple iterations.Each iteration may reveal previously unseen failures or show that earlier improvements were insufficient; the final representation is denoted r(I).Source: §4 · pp. 4–5

5 HINDSIGHT DISTILLATION

Hindsight distillation turns feedback-conditioned improvements to user representations into training targets for the representation generator. The trained generator can then produce improved representations directly from the conversation, without feedback at inference.

5 HINDSIGHT DISTILLATION

  • Obtaining hindsight feedback requires simulating and evaluating a multi-turn conversation, which distillation avoids at inference.Feedback-conditioned improvements are transferred into the generator during training.Source: §5 · p. 5
  • The generator is trained to predict feedback-improved representations from conversation C using a negative log-likelihood objective.This distills feedback-conditioned improvements into Gθ(· | C), so it can generate improved representations without hindsight feedback.Source: §5 · p. 5
  • Hindsight feedback improves an initial user representation through I rounds, producing r(I) as the training target.The procedure samples an initial representation for conversation C, then uses the improved representation as its target.Source: Figure 2 · p. 5, §5 · p. 5

6 LEARNING USER REPRESENTATIONS

Across the evaluated settings, hindsight feedback improves simulation fidelity, and UserRepGen-4B outperforms GPT-5.5. The experiments compare feedback-improvement and training approaches, then analyze how training changes the representations.

6.1 EXPERIMENTAL SETUP

  • The 3,200-conversation training set spans five real-world and two simulated datasets; evaluation uses 700 conversations and a different model to test generalization to unseen simulation environments.Evaluation samples 100 conversations per dataset.Source: §6.1 · p. 5

6.2 TEST-TIME IMPROVEMENT WITH HINDSIGHT FEEDBACK

  • After one feedback round, best-of-1 fidelity rises from 3.41 to 4.93 and best-of-3 from 4.42 to 6.14; gains mostly occur in the first two rounds and plateau after six.Across iterations, best-of-1 reaches 6.10 after ten rounds, while best-of-3 peaks at 7.07 after nine.Source: Figure 3 · p. 6, §6.2 · p. 6

6.3 HINDSIGHT DISTILLATION

  • OSFT scores 4.95 best-of-1 and 6.02 best-of-3; OPSD scores 4.46 and 5.60, while SFT scores 4.64 and 5.68.OSFT explicitly supervises on an improved representation and performs best among the compared approaches; SFT generates its targets once from the base policy, rather than continually from the latest policy.Source: §6.3 · pp. 6–7
  • GRPO reaches 3.57 best-of-1 and 4.59 best-of-3; its scalar rewards provide limited improvement guidance, and median within-group reward variance is 0.029.Identical rewards yield zero advantages and no gradient signal; each reward also requires an expensive multi-turn simulation.Source: §6.3 · p. 7
  • UserRepGen-4B surpasses GPT-5.5 by 1.05 best-of-1 and 1.12 best-of-3 points.Table 1 reports best-of-1 and best-of-3 simulation fidelity across user representation generators.Source: Table 1 · p. 7, §6.3 · p. 7

6.4 HOW DOES TRAINING CHANGE USER REPRESENTATIONS?

  • Representation lengths stay in the same general range, with slight growth in behavior descriptions, while lower ROUGE-L precision indicates fidelity gains are not simple copying.Table 2 reports average word counts for task, behavior, and complete representations, plus ROUGE-L precision against the source conversation.Source: §6.4 · p. 7, Table 2 · p. 7
  • After training, task descriptions more consistently capture later constraints, shifts in direction, and multiple objectives, while behavior descriptions become more specific and actionable.They also use more negative and contrastive instructions and capture previously overlooked patterns in requests, communication, context, and engagement across turns.Source: §6.4 · p. 7

7 BROADER UTILITY OF LEARNED REPRESENTATIONS

The section tests whether learned representations help simulate users beyond reconstructing their source conversations. They improve fidelity on unseen tasks, preserve population-level behavioral diversity, and distinguish users across conversations.

7.1 SIMULATING USERS IN NEW SCENARIOS

  • The unseen-task evaluation uses 147 held-out ThoughtTrace users with at least two conversations and compares same-task conditions with different behavior descriptions.The simulator produces three rollouts per condition, with best-of-3 fidelity reported.Source: §7.1 · p. 8
  • Adding a behavior description improves fidelity even when it comes from another user, offering concrete guidance that reduces generic or assistant-like simulation.The paper says this can make simulations more representative of real users generally, without identifying the particular user being simulated.Source: §7.1 · p. 8
  • For unseen tasks, same-user behavior descriptions outperform different-user descriptions for every trained generator, though gains are modest and below the target-conversation upper bound.The evaluation uses held-out ThoughtTrace users and compares behavior descriptions from another conversation by the same user with descriptions from a different user.Source: §7.1 · p. 8, Figure 4 · p. 8

7.2 PRESERVING BEHAVIORAL DIVERSITY AT THE POPULATION LEVEL

  • Learned representations generally lower both JS divergence and Self-BLEU, and adding behavior descriptions lowers both metrics for every generator.Lower JS divergence indicates closer alignment between original and simulated behavior distributions; lower Self-BLEU indicates less lexical homogeneity.Source: §7.2 · p. 9, Table 3 · p. 9

7.3 DISTINGUISHING USERS ACROSS CONVERSATIONS

  • 51.02% OSFT and 50.34% OPSD accuracy exceed the full-conversation baseline by 6.80% and 6.12%, respectively, in distinguishing users across conversations.GRPO and SFT narrow the gap but remain below the full-conversation baseline; base-model representations score 38.78%.Source: Table 4 · p. 9, §7.3 · p. 9

8 CONCLUSION

The conclusion frames user representations as a bottleneck because simulators can reproduce only the behaviors captured in them. It presents hindsight-distilled UserRepGen-4B and reports benefits across reconstruction and broader simulation evaluations.

8 CONCLUSION

  • UserRepGen-4B is optimized for downstream simulation fidelity using hindsight distillation.The paper identifies user representations as an underexplored bottleneck in simulation.Source: §8 · p. 9
  • When relevant behaviors are missing from a representation, the simulator fills gaps with assumptions, often producing generic rather than user-specific behavior.Source: §8 · p. 9
  • Learned representations improve fidelity for individual-user simulation, unseen tasks, population-level diversity, and distinguishing users across conversations.The conclusion emphasizes that simulation fidelity depends on capturing relevant information in representations, not only on generating realistic behavior.Source: §8 · p. 10

AI USE STATEMENT

The statement describes generative AI assistance in implementation, research support, literature analysis, and writing, while specifying tasks it did not perform and author review.

AI USE STATEMENT

  • Generative AI tools supported code and method implementation, research feedback, qualitative analysis, result interpretation, literature search, and writing.Source: AI USE STATEMENT · p. 10
  • The authors state that AI tools did not develop conceptual frameworks, propose or refine hypotheses, or directly clean and reformat datasets.Source: AI USE STATEMENT · p. 10
  • The authors say AI-assisted work was reviewed and verified, and they take responsibility for the final paper content and artifacts.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The ethics statement describes privacy and mischaracterization risks from user representations and cautions against impersonation or replacing real users. It also notes limits in population and language coverage and possible judge bias.

ETHICS STATEMENT

  • Representations inferred from one conversation may retain sensitive information or make unsupported assumptions about an individual.The statement also identifies privacy loss and mischaracterization as risks of more faithful simulation.Source: ETHICS STATEMENT · p. 10
  • The authors say simulations should not impersonate individuals, be presented as their interactions, or replace real users.Source: ETHICS STATEMENT · p. 10
  • The English-language data sources do not ensure representative population coverage or establish generalization to other languages and populations; judge evaluations may still reflect model biases and inconsistencies.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The statement identifies where the model and reproducibility materials will be available. The algorithm trains a representation generator to produce improved representations from conversations without needing feedback at inference time.

REPRODUCIBILITY STATEMENT

  • UserRepGen-4B will be publicly available; dataset sources, preprocessing criteria, and training and evaluation setup are described in Section 6.1.Prompts for representation generation, simulation, and evaluation are provided in Appendix C.Source: REPRODUCIBILITY STATEMENT · p. 10

A HINDSIGHT DISTILLATION ALGORITHM

  • Hindsight distillation trains the generator to predict a refined representation from the conversation alone.During training, it samples a conversation and initial representation, simulates interactions in multiple environments, collects hindsight feedback, and generates a refined representation before updating the generator.Source: A HINDSIGHT DISTILLATION ALGORITHM · p. 16
  • The algorithm repeats refinement rounds and updates the generator with gradient descent on a loss to predict the final refined representation from the input conversation.Each round rolls out simulations for the environments, generates feedback, collects it, and samples the next representation.Source: A HINDSIGHT DISTILLATION ALGORITHM · p. 16

B HUMAN VALIDATION OF LLM JUDGE

This section validates the LLM judge’s fidelity rankings against blinded human preferences and describes the prompts used to represent, refine, simulate, and assess user behavior. It also specifies how judges compare behavior and lists training hyperparameters for GRPO, OPSD, SFT, and OSFT.

B HUMAN VALIDATION OF LLM JUDGE

  • The validation set pairs 100 original conversations with Qwen3.5-4B and UserRepGen-4B simulations, then compares randomized, blinded human preferences against unequal judge scores.GPT-5.4 mini served as both the user simulator and assistant; human raters saw neither the source nor the scores.Source: B HUMAN VALIDATION OF LLM JUDGE · p. 16
  • 88% of the judge’s rankings agree with human labels, rising from 80.5% for a one-point score gap to 97.4% for gaps of at least three points.Table 5 defines the score gap as the absolute difference between the two simulations’ overall fidelity scores.Source: B HUMAN VALIDATION OF LLM JUDGE · p. 16, Table 5 · p. 16
  • The results support the judge as a fidelity measure, with population-diversity and cross-conversation user-distinction evaluations providing complementary evidence from downstream tasks.Those downstream evaluations do not rely on the fidelity judge.Source: B HUMAN VALIDATION OF LLM JUDGE · p. 16

C PROMPTS

  • The generation prompt separates a topic-specific task description from a general user-behavior profile inferred from stable conversational tendencies.The profile covers request phrasing, turn-by-turn goal decomposition, feedback, context use, communication style, dialog acts, and engagement depth, while avoiding conversation-specific content.Source: C PROMPTS · p. 17
  • The improvement prompt uses judge feedback to make behavior descriptions operational and preserve real-user interaction patterns rather than optimize for efficient task completion.It calls for retaining repetition, incremental follow-ups, verbose constraints, and inefficient patterns, and specifying turn forms, interaction arcs, and role distinctions when relevant.Source: C PROMPTS · p. 17
  • The judge prompt scores user-behavior fidelity across communication style, engagement, requests, responses, and context, while instructing judges to focus on behavior rather than topic.It also asks judges to account for different assistant behavior when assessing the user’s underlying behavioral tendencies.Source: C PROMPTS · pp. 18–19
  • The rubric compares users’ interaction structure, requests, engagement with assistant outputs, and context while focusing on behavior rather than topic and penalizing mismatches in the full turn arc.Judges are instructed not to infer hidden assistant content or reward generic simulator behavior.Source: C PROMPTS · p. 20

D TRAINING HYPERPARAMETERS

  • Table 6 reports training hyperparameters for GRPO, OPSD, SFT, and OSFT.Source: Table 6 · p. 21

E.1 PAIRWISE ANALYSIS WITH GPT-5.4 MINI

A pairwise analysis compares representations from UserRepGen-4B and Qwen3.5-4B across task and behavior criteria, with examples illustrating their differences. The appendix also describes how original and simulated user behaviors are assessed for population-level diversity.

E.1 PAIRWISE ANALYSIS WITH GPT-5.4 MINI

  • UserRepGen-4B task descriptions better captured later-turn constraints in 63.6% of comparisons, goal shifts in 61.0%, and multiple objectives in 58.3%.It was selected as less repetitive in only 33.1% of comparisons, so Qwen3.5-4B usually produced less repetitive task descriptions; Table 7 reports the comparison percentages.Source: §E.1 · p. 22, Table 7 · p. 23
  • GPT-5.4 mini selected UserRepGen-4B over Qwen3.5-4B in comparisons for contrastive instructions (91.6%), engagement patterns (90.0%), communication style (88.1%), and specificity and actionability (87.7%).Source: §E.1 · p. 23, Table 7 · p. 23

E.2 USER REPRESENTATION EXAMPLES

  • The examples contrast broad characterizations with descriptions of observable interaction patterns, such as short factual questions, little upfront context, and pivoting rather than giving feedback.The authors describe the trained model's task descriptions as capturing the full request sequence rather than summarizing the conversation at a high level.Source: §E.2 · pp. 23–24

F MEASURING THE GAP BETWEEN DISTRIBUTIONS OF ORIGINAL AND SIMULATED USER BEHAVIORS

  • The population-level diversity analysis measures the distributional gap between original and simulated user behaviors and describes behaviors along six dimensions.The descriptions cover requests, responses, context, communication style, and dialog acts under DAMSL and SGD taxonomies, while excluding task-specific details.Source: F MEASURING THE GAP BETWEEN… · p. 24
Loading 2610.22518v1…