PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning User Simulators with Turing Rewards

Yingshan Susan Wang, Cedegao E. Zhang, Linlu Qiu, Zexue He, Pengyuan Li, Alex Pentland, Roger P. Levy, Yoon Kim

arXiv:2606.19336v2cs.CL

TL;DR

User simulators are difficult to train because existing methods match specific ground-truth responses rather than indistinguishability. Turing-RL uses a discriminative Turing reward, and across chat and Reddit it consistently outperforms response-similarity and log-probability baselines on LLM and human evaluations.

  • Problem

    Existing user-simulation methods derive training signals from matching a specific ground-truth response, despite the difficulty of simulating individual users.

  • Method

    Turing-RL trains LLM user simulators with reinforcement learning using an LLM-judge Turing reward conditioned on the user’s history.

  • Results

    Across chat and Reddit, Turing-RL consistently outperforms response-similarity and log-probability baselines on both LLM and human evaluation metrics.

  • Takeaways & Limitations

    Content matching and human-likeness can diverge, suggesting discriminative signals are better suited than matching signals for user simulation.

  • Takeaways & Limitations

    The study evaluates only open-ended chat and forum discussion, leaving generalization to other interaction types and frontier-sized models untested.

Abstract

from arXiv · show

Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, research in the social sciences, and more. Existing approaches generally do so by training a large language model (LLM) to match a single ground truth response, either by maximizing the log probability or by using a similarity reward. We instead propose Turing-RL: a Turing-Test-based reinforcement learning approach for training user simulator models. Turing-RL uses a discriminative Turing reward with an LLM judge to score how indistinguishable a generated response is from the real user's given the user's history, and the user simulator LLM learns to produce responses indistinguishable from what the user could have said with such rewards. Across two different domains--conversational chat and Reddit forum discussion--we find that Turing-RL consistently outperforms baseline methods on both LLM and human evaluation metrics. Our study suggests that optimizing for indistinguishability, rather than response matching, is effective for learning user simulators.

1 Introduction

User simulators could support interactive-system training, evaluation, and social-world modeling, but simulating individuals is difficult because personal opinions and preferences are not captured by demographics or group labels. Turing-RL addresses this challenge by training LLM simulators with an LLM-judged discriminative Turing reward and evaluates them across dialogue and Reddit settings.

  • User simulators could support social-world models, training environments, and testbeds for interactive systems by having LLMs play the user rather than the assistant.
  • Individual simulation is difficult because people with identical demographics can hold sharply different opinions, while group-level labels cannot recover individual preferences.
  • Turing-RL trains LLM user simulators with reinforcement learning on a discriminative Turing reward from an LLM judge conditioned on the user’s history.
  • The method is evaluated in multi-turn dialogue and Reddit forum discussions against response-similarity rewards and log-probability maximization with chain-of-thought.
  • Turing-RL consistently outperforms the baseline training signals across the two evaluated settings.

2 Learning User Simulators

This section formulates user simulation as generating a response from the current session context and a user representation, typically combining behavior history with an induced persona. Turing-RL trains the simulator using an LLM judge’s human-indistinguishability reward, with score capping and GRPO after supervised fine-tuning.

  • Problem formulation: User simulation generates a plausible response y from the current session context x and user representation u.The user representation may encode information about the simulated user beyond the current interaction context.
  • User representation: The default user representation combines a fixed, target-disjoint behavior history h with an induced persona ρ summarizing stable traits.The history block is reused across prediction targets, while persona induction uses an auxiliary language model.
  • Turing reward: An LLM judge compares a real-user response y⋆ with a sampled model response y and rates which is human-written on a 1–7 Likert scale.The judgment considers the local context, the user’s motive, and stylistic fit.
  • Turing reward: The judge score is capped and normalized to [0, 1], with the cap at 5 intended to mitigate reward hacking.Responses judged substantially more human than actual human responses are treated as undesirable because they may indicate reward hacking.
  • Optimization: Training uses Group Relative Policy Optimization after supervised fine-tuning on a disjoint subset of the training data.The optimization follows the initial SFT phase.

3 Experimental Setup

The experiments evaluate user simulators in multi-turn chat and Reddit discussions using held-out users and target interactions. Comparisons use a common Qwen3-8B-based training setup, with LLM-judge and human Turing-test evaluation.

  • Datasets: Experiments cover multi-turn chat and Reddit forum discussion, reserving prior interactions as behavior history and remaining interactions as prediction targets.Users are split into training and evaluation sets.
  • Datasets: 1,288 PRISM users with at least 6 conversations are selected, with 128 users and 880 target user-response turns held out for evaluation.PRISM contains multi-turn conversations between humans and LLM assistants spanning 1,500 participants from 75 countries.
  • Datasets: 1,282 ConvoKit users across 14 subreddits are selected, while r/tifu and r/worldnews provide 102 held-out users and 267 evaluation examples.Each example contains the user’s last comment and its ancestor comment chain; held-out users do not overlap with training.
  • Models and baselines: Qwen3-8B is the base model, and RL methods share the same SFT checkpoint, histories, personas, and GRPO training for fair comparison.Reported comparators also include the SFT-init checkpoint, base Qwen3-8B, Qwen3.5-397B-A17B, and OpenAI GPT- .
  • Evaluation: Evaluation combines LLM judging of Turing distinguishability, response similarity, and context and user specificity with a binary-choice human Turing test.The human test recruits 360 Prolific participants and compares SFT-Init, Sim-RL, and Turing-RL.

4 Results

Across Chat and Reddit, Turing-RL produces the most human-like simulations, outperforming alternative training methods while preserving response similarity and improving contextual grounding. Human evaluations support these trends, though Reddit judgments are substantially harder and should be interpreted cautiously.

  • Turing judge scores: Turing-RL outperforms all other models on Turing judge scores in both Reddit and Chat, with a particularly large margin over the next-best model on Chat.Scores use a 1–7 Likert scale, where higher values indicate more human-like responses.
  • Limitations: User simulation remains difficult: GPT-5 and Qwen3.5-397B produce verbose, hedged, assistant-like responses, and similarity-reward training does not improve Turing scores over SFT-Init.Reddit is harder for humans to judge than Chat, with a Chat/Reddit mean reaction-time ratio per target question of 1.43 and a Reddit/Chat mean reaction-time-per-word ratio of 1.48.
  • Content alignment: Among trained models, Turing-RL and Sim-RL perform comparably on ground-truth similarity and both improve over SFT-Init.This indicates that optimizing indistinguishability does not sacrifice content alignment.
  • Contextual grounding: Turing-RL and Sim-RL outperform SFT-Init and Logprob-RL on response specificity across both domains.Response specificity measures grounding in the interaction context and compatibility with the target user.
  • Human evaluation: On Chat, Turing-RL has the highest human-evaluation win rate (WR = .57), significantly outperforming SFT-Init (p = 0.044) and Sim-RL (p = 0.022).On Reddit, Turing-RL significantly improves on SFT-Init (p = 0.0095), while its difference from Sim-RL is not statistically significant.

5 Ablations and Analysis

Ablations show that Turing-RL’s human-likeness is largely robust to user representation, although specificity depends more on domain and history. Qualitative examples further show Turing-RL producing plausible human-like reactions rather than assistant-like or overly context-anchored responses.

  • User representation ablation: The ablation compares history-only, persona-only, and combined history-and-persona user representations.The three conditions are u = h, u = ρ, and u = (h, ρ).
  • User representation ablation: Turing scores remain largely robust across representations, while specificity is stable on Chat but favors history-inclusive inputs on Reddit.Persona alone achieves comparable Turing scores but not comparable similarity scores, suggesting it captures stylistic and behavioral patterns without reproducing exact content.
  • User representation ablation: Larger persona-inducing models do not reliably produce better user representations, and combining history with persona does not necessarily improve performance.The paper leaves systematic investigation of better user representations for future work.
  • Qualitative analysis: Qualitative examples show Turing-RL generating plausible human-like reactions, whereas GPT-5 and Qwen3.5-397B are overly verbose and assistant-like on Reddit.On Chat, Turing-RL and SFT-Init ask plausible follow-ups, although SFT-Init’s question is partly answered in context; Sim-RL is more content-aligned.

6 Discussion

The discussion finds that Turing rewards improve indistinguishability without reducing content similarity, making discriminative signals better suited than matching signals for user simulation. It also highlights impersonation risks and future work on personalization, alignment, and human-like reasoning processes.

  • Findings: Content matching and human-likeness come apart: Turing rewards improve indistinguishability without lowering content similarity, unlike similarity rewards.Similarity rewards improve coverage of ground-truth content but do not necessarily make responses harder to distinguish from real users.
  • Risks: Indistinguishable simulators conditioned on specific users’ prior behavior could fabricate attributed messages or scale fraud and social-engineering attacks.The study limits training and evaluation to public or consented research data and intends the simulators for studying interactive systems.
  • Future work: Future work could examine whether user simulators help assistants become more personalized and better aligned with users’ goals and intentions.Suggested settings include multi-agent systems and cognitive architectures.
  • Future work: Whether models’ reasoning processes align with humans remains open, motivating comparisons between generated reasoning traces and humans’ verbalized thought traces.This question extends beyond producing outputs indistinguishable from human-written text.

7 Related Work

Related work spans LLM-based user simulation, persona-conditioned user representation, and broader human behavior prediction. Existing human-behavior prediction approaches have largely focused on constrained, single-turn settings.

  • LLM-based user simulation: LLMs have been used to evaluate dialogue systems, train conversational agents via self-play, and replicate human subject studies.
  • LLM-based user simulation: Assistant-tuned LLMs can be structurally misaligned with the user role, while other work addresses goal drift in task-oriented settings.
  • Persona and user representation: Persona-conditioned generation represents characters using natural-language descriptions of personality, demographics, and traits.
  • Human behavior prediction: LLMs have been evaluated as proxies for human participants, and foundation models have been fine-tuned on psychology experiments to predict trial-level responses.
  • Human behavior prediction: These human-behavior prediction approaches have largely focused on constrained, single-turn settings such as survey responses or experimental trials.

8 Conclusion

The paper proposes training LLM-based user simulators with a discriminative Turing reward that favors responses indistinguishable from real users given prior history. Across two domains, this approach produces more human-like responses than log-probability maximization and response-similarity rewards under LLM- and human-based Turing evaluations.

  • 8 Conclusion: The proposed discriminative Turing reward scores generated responses by their indistinguishability from the real user’s response, conditioned on prior user history.This reward trains LLM-based user simulators using a Turing-Test-based signal.
  • 8 Conclusion: Across two substantially different domains, the discriminative Turing reward consistently produces more human-like responses than log-probability maximization and response-similarity rewards.The comparison is made using both LLM- and human-based Turing evaluations.
  • 8 Conclusion: The approach is evaluated with both LLM-based and human-based Turing evaluations.These evaluations assess how human-like the generated user-simulator responses are.

Limitations · A Dataset Details

The study’s limitations include testing Turing-RL across only open-ended chat and forum discussion, using Qwen3-8B as the base model, and leaving broader interaction settings for future evaluation. Dataset examples separate fixed user representations from context and target responses to prevent target leakage.

  • Limitations: The evaluation covers two structurally different domains: open-ended chat and forum discussion.The paper notes that additional interaction types remain to be systematically tested.
  • Limitations: Task-oriented dialogue, negotiation, and collaborative problemsolving are identified as untested settings for generalization.These interaction types are proposed as extensions for systematically evaluating the Turing reward.
  • Limitations: The experiments use Qwen3-8B as the base model.The passage states that this setup is sufficient to demonstrate the effectiveness of the Turing reward, but the provided text ends before further qualification.
  • A Dataset Details: Each training example is represented as a tuple (u, x, y).The user representation is u = (h, ρ), while x is the current context and y denotes the ground truth response.
  • A Dataset Details: The user representation u = (h, ρ) is fixed per user and disjoint from both the current context x and ground truth response y⋆.This construction separates user-specific information from the example’s context and target.
  • A Dataset Details: Behavioral history h is selected once per user from a deterministic seed, and persona ρ is induced from h alone.These procedures define the user representation before separating it from the current context and target.
  • A Dataset Details: The separation of h and ρ from x and y⋆ guarantees that the user representation does not leak target information.The passage explicitly attributes this guarantee to the disjointness of the representation from the current context and ground truth response.

A.1 Dataset Preprocessing … C.1 SFT Training Details

The paper preprocesses disjoint chat and Reddit user splits, induces structured personas from reserved histories, and formats simulation prompts around history, context, reasoning, and responses. SFT traces are generated from ground-truth responses using the same user information.

  • A.1 Dataset Preprocessing: PRISM retains 1,288 users with at least six conversations, holding out 128 users and splitting the remaining 1,160 into 696 GRPO and 464 SFT users.For each user, 2–4 conversations are reserved as history, with the remainder serving as targets.
  • A.1 Dataset Preprocessing: ConvoKit reconstructs Reddit threads, holds out r/tifu and r/worldnews, and uses disjoint 40/60 user splits for SFT and GRPO after discarding users with insufficient threads.Each target is the user’s final comment, paired with the original post and ancestor reply chain.
  • A.2 Persona Induction: The default user representation combines behavioral history with a GPT-5.4 nano-induced persona containing values, verbal quirks, expression style, length prior, and background.The persona uses temperature 0.2, a 2,048-token maximum, and is reused across the user’s examples.
  • A.2 Persona Induction: Persona induction requires concise first-person JSON notes, avoids reproducing exact history wording, and records concrete behavioral fields while using `unknown` when evidence is weak.The schema contains values, verbal_quirks, expression_style, length_prior, and background.
  • B.1 Sampling Parameters: Held-out generations use dataset- and model-specific sampling parameters; Qwen3-8B uses temperature 0.4 for Reddit, 0.6 for Chat, and one generation per target.These settings are listed as held-out generation sampling parameters.
  • B.2 Prompt Formatting: Each simulation example uses a system message with instructions, task description, and optional persona, followed by a user message containing history and current context.The model produces reasoning in <reasoning> tags and prefixes the simulated response with [HUMAN]:.
  • B.2 Prompt Formatting: The simulation prompt explicitly asks the model to predict what [HUMAN] says as if it were that person, with PRISM history organized in numbered conversation tags.The PRISM layout includes turns up to the prediction point.
  • C.1 SFT Training Details: SFT examples pair ground-truth responses with generated reasoning traces produced by Qwen3-8B from reserved history, induced persona, current context, and target response.The trace prompt focuses on the local target, intent, and stance behind the response.

C.2 GRPO Training Details … D.3 Balancing

The paper trains GRPO user simulators from disjoint SFT data using four sampled candidates, group-relative advantages, LoRA adapters, and controlled length penalties. Human evaluation uses filtered, balanced forced-choice Turing tests across Reddit and chat datasets.

  • C.2 GRPO Training Details: C.2 GRPO Training Details: GRPO runs initialize from SFT checkpoints, use a disjoint GRPO user split, sample four candidates, compute group-relative advantages, and optimize with LoRA adapters.All GRPO training is veRL-based and uses approximately 1680 GPU hours across SFT and GRPO.
  • C.2 GRPO Training Details: C.2 GRPO Training Details: The simulator policy is optimized against the SFT reference with β = 1 × 10−3 and PPO clipping range ε = 0.2.The four candidates are sampled from the old policy, with reasoning tokens receiving the same group-normalized advantage as their response.
  • C.3 Length Penalty: C.3 Length Penalty: Generated responses outside an acceptable length-ratio band receive linearly increasing penalties, with larger weight assigned to short responses.The band is defined relative to ground-truth length, and dataset-specific parameters are based on SFT-generated responses.
  • D Human Evaluation Details: D Human Evaluation Details: Participants view a target user’s history and choose which of 10 randomized response pairs contains the real user’s response.Each pair contains one ground-truth response and one model-generated response; participants are paid $6, with total costs approximately $2880.
  • D.1 Design: D.1 Design: Evaluation compares SFT-Init, Sim-RL, and Turing-RL on ConvoKit and PRISM using 100 target users divided into 10 groups of 10.Six annotators per dataset–model–group cell produce 60 participants and 600 binary judgments for each dataset–model pair.
  • D.2 Comprehension Filtering: D.2 Comprehension Filtering: Participants answer history-based comprehension checks, and those scoring below 75% accuracy are excluded.The checks include identifying the target user’s most-discussed topic or most recent message.
  • D.3 Balancing: D.3 Balancing: After removing incomplete and comprehension-failing submissions, the study retains the first 6 valid completions per dataset, model, and user group.This produces exactly 60 participants and 600 votes for every dataset and model.

D.4 Metric … E Judge Prompts

The evaluation defines model win rate and target-level statistical tests, while also measuring displayed-text length and judgment reaction time. Three judge prompts assess distinguishability, similarity, and user-specific contextual grounding.

  • D.4 Metric: Model win rate is the fraction of judgments where annotators choose a model-generated response over ground truth, averaged across 100 targets.Win rate is averaged over approximately six annotators per target, with 95% confidence intervals computed as mean ± 1.96 × SEM.
  • D.5 Testing for Statistical Significance: Paired model differences are computed as target-level win-rate differences over 100 heldout targets per dataset.Bootstrap intervals resample the 100 targets with replacement 10^6 times and report the 2.5–97.5 percentile interval.
  • D.5 Testing for Statistical Significance: A paired permutation test evaluates whether target-level differences are equally likely to favor either model under the null hypothesis of equal performance.For each nonzero difference, the procedure reassigned its sign and compared recomputed mean differences across assignments.
  • D.6 Word Counts: Displayed user-history and target-context word counts are summarized for each target, excluding response options and interface text.These counts quantify the text shown during evaluation.
  • D.7 Reaction Time: 72.83s and 50.95s are the aggregated Reddit and Chat mean reaction times, respectively, yielding a Reddit/Chat mean RT ratio of 1.43.Reaction time runs from judgment-page rendering to clicking Next and includes comprehension checks, context reading, optional history reopening, and response selection.
  • D.7 Reaction Time: 0.185s and 0.125s are the aggregated Reddit and Chat mean reaction times per word, respectively, yielding a Reddit/Chat mean RT/word ratio of 1.48.The normalization uses total displayed history and context word counts for each target.
  • E Judge Prompts: The Turing distinguishability judge scores how indistinguishable generated responses are from real-user responses.The paper presents this judge in Figure 16.
  • E Judge Prompts: The similarity and specificity judges assess ground-truth content overlap and contextual compatibility with the target user, respectively.The similarity and specificity judges are presented in Figures 17 and 18.

F Training Dynamics · G More Qualitative Examples

The section presents Turing-RL’s training dynamics and input ablations, alongside qualitative examples from Reddit and Chat and the judge criteria used to assess distinguishability. It also reports a persona-induction ablation and emphasizes context-sensitive evaluation of targets, goals, and communication style.

  • F Training Dynamics: Figure 19 shows GRPO training dynamics for all three rewards and input-ablation runs for Turing-RL.
  • G More Qualitative Examples: Figure 20 presents Reddit and Chat examples comparing ground truth with GPT-5, Qwen3.5-397B, Qwen3-8B Base, SFT-Init, Logprob-RL, Sim-RL, and Turing-RL.
  • G More Qualitative Examples: Persona induction has limited effects relative to confidence intervals, with Opus performing best on Reddit and GPT-5.4 nano performing best on Chat.The ablation compares history-only u = h with history-and-persona u = (h, ρ) using personas induced by GPT-5.4 nano, Qwen3-8B with thinking enabled, and Opus 4.8.
  • G More Qualitative Examples: The judge identifies the exact context addressed and checks whether the response understands the latest turn, preserves speaker perspective, and targets the correct participant.
  • G More Qualitative Examples: Human-goal scoring evaluates whether a response preserves a plausible, context-sensitive local move rather than substituting a generic assistant-like task.The judge considers jokes, asides, anecdotes, corrections, quote-replies, agreements, disagreements, questions, brief reactions, and plausible topic pivots.
  • G More Qualitative Examples: Communication-style scoring compares wording, tone, humor, length, grammar, punctuation, specificity, and effort with the human’s history and local framing.Generic fluency is not rewarded unless it matches the human’s style, and artifact-like writing is heavily penalized.
  • G More Qualitative Examples: The judge penalizes source copying, wrong targets or speaker roles, unsupported adversarial reframing, and assistant-like responses when these make a candidate implausibly human.
  • G More Qualitative Examples: Final distinguishability ratings use a 1–7 scale derived from the score gap, where 4 means the judge cannot tell or considers both responses equally likely.
Loading 2606.19336v2…