Source-linked AI summary
HumanLM: Simulating Users with State Alignment Beats Response Imitation
Shirley Wu, Evelyn Choi, Arpandeep Khatua, Zhanghan Wang, Joy He-Yueya, Tharindu Cyril Weerasooriya, Wei Wei, Diyi Yang, Jure Leskovec, James Zou
TL;DR
Existing user simulators often imitate surface language without capturing the underlying states that shape real users’ responses. HumanLM trains simulators to align psychologically grounded latent states with ground-truth responses before synthesizing outputs, and it outperforms alternatives across HUMANUAL while achieving the highest similarity in a real-time study.
Problem
Existing user simulators primarily imitate surface-level language, limiting their ability to capture underlying states such as beliefs, emotions, values, and stances.
Method
HumanLM uses reinforcement learning to align generated latent states across psychologically grounded dimensions, then synthesizes responses conditioned on those states.
Results
16.3% average relative improvement in alignment scores was achieved across HUMANUAL datasets, while HumanLM also had the highest similarity in a real-time study with 111 participants.
Takeaways & Limitations
HumanLM produces user-simulator responses that are highly aligned with real users and human-like across benchmark and real-time evaluations.
Takeaways & Limitations
The framework assumes some state dimensions may be only weakly expressed in responses despite being present in the underlying generation process.
Abstract
from arXiv · showhide
Large Language Models (LLMs) are increasingly used to simulate how specific users respond to a given context, enabling more user-centric applications that rely on user feedback. However, existing user simulators mostly imitate surface-level patterns and language styles, which fail to reflect the underlying states of real users (e.g., beliefs and emotions). To address these limitations, we propose a novel training framework, HumanLM, which builds user simulators that accurately reflect real users. Our key insight is that, in addition to generating responses, the model should generate natural-language latent states that align with ground-truth responses through reinforcement learning. These latent states correspond to a set of psychologically grounded state dimensions that drive how real users respond. HumanLM further synthesizes these aligned latent states into responses that accurately represent real users. For extensive evaluation, we develop Humanual, a comprehensive benchmark for simulating real users based on public data. Humanual consists of six large-scale datasets with 26k users and 216k responses in total, spanning diverse tasks such as generating user responses to daily life issues, political blogs, and chat sessions with LLM assistants. Across datasets, HumanLM significantly outperforms alternative approaches, achieving an average relative improvement of 16.3% in alignment scores from an LLM judge. In a real-time simulation study with 111 participants, HumanLM achieves the highest similarity to real user responses and competitive human-likeness scores.
1. Introduction
HumanLM addresses the limits of surface-level user simulation by aligning psychologically grounded latent states with real user responses, then synthesizing responses from those states. On HUMANUAL and in a real-time study, it achieves stronger alignment and similarity than alternative approaches.
- Motivation: Existing simulators imitate surface-level language instead of higher-level user states, producing responses that may not reflect real user behavior.Relevant states include stance, emotion, and values that influence responses and matter for applications intended to generalize to real users.
- Approach: HumanLM aligns multiple user-state dimensions, generates latent states in reasoning traces, and synthesizes responses conditioned on those states.The framework uses reinforcement learning to maximize alignment between latent states and ground-truth responses.
- Evaluation: HUMANUAL contains six datasets from public sources spanning 26k users, 216k responses, and diverse real-user contexts.The benchmark includes domains such as life issues, blog feedback, book reviews, and conversations with LLM assistants.
- Results: 16.3% average relative improvement in alignment scores is reported for HumanLM across HUMANUAL datasets.The result is reported across datasets against approaches using prompting, supervised fine-tuning, and reinforcement learning.
- Results: 41.4% win rate on overall similarity was achieved by HumanLM, with 55.9% of participants rating its responses mostly similar or nearly identical to their own versus 45.0% for the best baseline.In the same real-time study, 76.6% of HumanLM responses were rated above quite natural.
2. Problem Formulation
The problem formulation represents each example as a persona, context, and ground-truth response, then models responses through the latent states they express. The objective is to generate responses whose latent-state sets match those of the ground truth.
- Data representation: Each training example consists of a user persona p, an input context x, and the user’s ground-truth response y.Contexts may be single-turn or multi-turn, while personas can be built from profiles, identifiers, or summarized user histories.
- Latent states: For each context x, the formulation defines a finite latent state space S(x) containing high-level attributes expressed or reflected by responses.Examples include heartbreak for wildfire victims, irritation about untimely rescue, and providing claims with evidence.
- Objective: A mapping M assigns each response the latent states it expresses, and the goal is to generate a response whose states match those of the ground truth.This formulation treats a response as a collection of latent states rather than requiring exact wording.
- Objective: The objective penalizes generated responses for missing ground-truth latent states or adding redundant states outside the ground-truth response.The formulation uses an indicator function to represent these state-set discrepancies.
3. Training Aligned User Simulators
HumanLM trains simulators to generate aligned latent states along psychologically grounded dimensions before synthesizing responses. This design addresses the ambiguity of post-hoc response alignment and uses comparative rewards to guide reinforcement learning.
- Motivation: Exact response imitation is limited because user responses are nondeterministic and different phrasings can express the same stance.Surface-level imitation can also miss distinctions such as sarcasm versus direct criticism.
- Challenge: Post-hoc alignment scores create combinatorial ambiguity because aggregate rewards do not reveal which individual latent states were correct.Responses matching one state while mismatching others can receive similar rewards, confusing training about which states to improve.
- Key idea: HumanLM explicitly generates latent states and treats responses as outcomes conditioned on those states rather than inferring states from responses.This reframes alignment as generating states that lead to aligned synthesized responses.
- State dimensions: The latent-state space assumes states are distinct only when their similarity is at most a granularity threshold τ.The formulation defines distinctness using a similarity function over latent states.
- State dimensions: The state dimensions include belief, goal, emotion, value, stance, and communication, covering cognitive, normative, affective, and linguistic aspects.Communication concerns how information is structured, such as directness, evidence use, answering, or questioning.
- Training: Comparative LLM-judge scores for batches of latent-state rollouts provide reinforcement-learning rewards with reduced scoring bias.The judge evaluates generated states under one dimension by comparing them together rather than assigning isolated scores.
- Response synthesis: HumanLM synthesizes responses from reasoning traces that analyze how latent states should influence organization, emphasis, and concision.Response alignment scores are also computed for the synthesized responses.
4. Benchmark and Experiment Setup
HUMANUAL evaluates user simulators across six diverse datasets built from real or publicly available user data, using response and state alignment measures alongside multiple baselines.
- Benchmark: HUMANUAL includes six datasets spanning opinions, politics, news, chat, email, and book reviews from publicly available sources.The benchmark covers diverse user behaviors and contexts, including reactions to news, interactive assistant conversations, and preferences expressed through reviews.
- Benchmark: Humanual-Chat models interactive behavior over 5–10-turn conversations, including follow-ups, goal changes, and clarification turns.The dataset is adapted from WildChat and is designed to simulate user interactions with LLM assistants.
- Experimental setup: Official splits keep test contexts unseen by assigning chat turns chronologically within conversations and arranging other contexts chronologically.For non-chat datasets, original posts, news, or blogs are divided by timestamp; user profiles are summarized from up to 20 earliest training responses except for chat.
- Experimental setup: Evaluation uses LLM-judged response alignment and six-dimensional state alignment, while comparing HUMANLM with base, supervised, thought-augmented, and reinforcement-learning baselines.Training uses gpt-5-mini as the judge, whereas testing uses claude-4.5-haiku; the benchmark reports response alignment scores in Table 1 and state alignment scores in Figure 4.
- Benchmark: The datasets capture real-user responses across varied contexts, including Reddit life issues, Medium political blogs, YouTube news videos, and Amazon books.These sources provide different behavioral targets, such as moral standards, political stances, reactions to news, and satisfaction with book content.
5. Results on Benchmark
Across benchmark evaluations, HUMANLM consistently outperforms alternative simulators in response and latent-state alignment. Additional analyses attribute this performance to broader state exploration and identify goal and stance as important dimensions for Humanual-Opinion responses.
- Benchmark results: Simulating real-world user responses remains challenging, with Qwen3-8b averaging around 10% across datasets.The results also report improvements from enabling reasoning or learning from high-quality reasoning traces on some datasets.
- Benchmark results: SFT-based approaches perform worst overall, often mimicking tone while producing overly long responses and opposite opinions.This pattern is presented as evidence that response imitation does not reliably capture higher-level user states.
- Benchmark results: 3.94 (SFT-think) →4.78 (+0.84) (GRPO-think) on Humanual-Email illustrates that directly optimizing alignment scores can improve over SFT, sometimes marginally.Standard GRPO approaches outperform SFT by some margins during testing, although gains vary across datasets.
- Benchmark results: 16.3% average relative improvement in alignment scores makes HUMANLM the best-performing method across datasets, with the highest scores on 80% of latent states.The reported relative improvements are 38% over base-think and 17% over GRPO-think.
- Benchmark results: 7.5% higher embedding similarity than Qwen3-8b-think shows gains on a metric that was not used as HUMANLM’s reward.This provides an additional comparison between generated responses and ground-truth responses.
6. Real-time User Simulation
In a real-time study, HUMANLM produced responses participants judged most similar to their own, significantly outperforming both baselines and also receiving strong humanlikeness ratings.
- HUMANLM achieved the highest overall-similarity average of 6.5 and a 41.4% win rate, versus 30.6% for Qwen3-8b-think and 27.9% for GRPO-think.
- HUMANLM’s similarity improvements were statistically significant against Qwen3-8b-think (p = 0.0279 < 0.05) and GRPO-think (p = 0.00284 < 0.01).
- Qualitative analysis: Participants reported that HUMANLM better matched their stance, key considerations, and nuanced emotional intensity.
- Humanlikeness scores: Participants described HUMANLM as less redundant, clearer, more casual, more honest, and smoother than the comparison models.
7. Related Work
Prior work simulates general users, profiled individuals, or groups, but existing methods and benchmarks often miss deeper user states, diverse contexts, and rich response information.
- User modeling and simulation: User simulation research spans broad users, specific users described by profiles, and groups or societies of users.
- User modeling and simulation: Prompting cannot adapt model parameters to user data, SFT emphasizes surface language, and prior RL work rewards persona consistency rather than deeper state alignment.
- User simulation benchmarks and evaluation: Prevailing benchmarks commonly use LLM-assistant chats or survey questions, limiting context diversity.
- User simulation benchmarks and evaluation: Synthetic personas may not reflect real users, whereas HUMANUAL provides a diverse and comprehensive testbed.
- User simulation benchmarks and evaluation: Survey-like evaluations reduce responses to discrete choices or probability distributions, omitting fine-grained user thoughts.
- Applications of user simulators: User simulators support behavioral analysis, synthetic-data generation, collaborative-LLM rewards, and evaluation of LLMs or recommender systems.
8. Conclusion
The paper proposes HUMANLM and HUMANUAL to improve the alignment, scale, and realism of user simulation for future applications.
- HUMANLM reinforces learning along psychologically grounded state dimensions to reflect real user states.
- HUMANUAL contains 66k real-world contexts and 26k worldwide user responses, supporting comprehensive user-simulation evaluation.
- Across HUMANUAL and a real-time user study, HUMANLM generates high-quality, well-aligned, and human-like responses.
- Future work can explore user-simulator diversity and multidomain training.
Impact Statement
The paper frames better-aligned user simulators as part of human-centric AI and documents privacy protections for benchmark and participant data.
- The work seeks more representative and better-aligned user simulators so systems trained and tested with them can better generalize to real-world deployments.
- Public benchmark data were de-identified, and user-study participants consented to release their written text while being instructed to avoid personally identifiable information.
A. HUMANUAL Details
Humanual is presented as a benchmark with dataset statistics and publicly sourced user-response data. Its processing includes user profiles, temporal splits, standardized sample structure, and comparisons using shared baselines.
- Dataset construction: Humanual includes dataset statistics and draws its datasets from public sources such as YouTube, Amazon Reviews, Reddit, Medium, and user–LLM conversations.The supplied dataset description names Humanual-News, Humanual-Book, Humanual-Opinion, Humanual-Politics, and Humanual-Chat, while the table title identifies dataset statistics.
- User profiles: Users are retained using dataset-specific response-count thresholds, with users appearing only in validation or test removed before persona generation.Profiles are generated from each retained user’s earliest 20 responses using claude-4.5-haiku.
- Evaluation split: Contexts are split chronologically into 90% training, 2% validation, and 8% test so test contexts are unseen during training.The split is performed by post timestamps.
- Data format: Each sample contains a user profile, an input context with preceding thread responses when applicable, and the ground-truth response, plus timestamp and identifier metadata.The context uses role labels and may represent single- or multi-turn interactions.
- Baselines: Baselines use Qwen3-8b and the same processed datasets, including direct generation, reasoning-enabled generation, supervised fine-tuning, UserLM, and GRPO variants.GRPO optimizes response alignment directly without explicit latent states.
C. HUMANLM Training Details
HUMANLM trains responses and psychologically grounded state generations with alignment-based rewards, using GRPO and LLM-judge scores. The evaluation reports embedding similarity and state-alignment scores across Humanual datasets and state dimensions.
- Training procedure: HUMANLM generates either a hierarchy state or a response from a user profile, post context, and hierarchy-specific system prompt.The listed hierarchy dimensions include stance, emotion, belief, value, goal, and communication.
- Training procedure: GRPO rewards responses with response-alignment scores and hierarchy generations with their corresponding state-specific scores.This separates the reward assigned to response generation from rewards assigned to state generation.
- Implementation: Training uses vLLM with sampling temperature 0.8, while evaluation uses temperature 0.4 and a no-repeat 4-gram constraint.The maximum response length is 1024 tokens.
- Evaluation: The evaluation includes embedding similarity scores on HUMANUAL and state-alignment score tables for Humanual-News, Humanual-Book, Humanual-Opinion, Humanual-Politics, Humanual-Chat, and Humanual-Email.The supplied passages identify the reported tables but do not provide their numerical entries.
D.3. More Training Dynamics Results
The training-dynamics analysis compares checkpoint response alignment with alignment on individual state dimensions. The supplied material also details the LLM-judge scoring procedure used to produce alignment rewards.
- Training dynamics: Each training-dynamics dot is a checkpoint saved every 25 steps during Humanual-Opinion training, with state alignment on the x-axis and response alignment on the y-axis.The state dimensions shown are goal, emotion, and communication, and the comparison is between HUMANLM and GRPO-think.
- LLM-judge scoring: The judge extracts 1–3 key points from the ground-truth response for the evaluated item and scores how well generated content matches them.The procedure applies both to response alignment and to state-specific alignment.
- LLM-judge scoring: The scoring rubric assigns 1.0 to perfect reflection, ranges from 0.7 to 0.9 for mostly reflected content, and 0.0 to missed, contradicted, or reversed content.Intermediate ranges distinguish partial, vague, and very weak reflection.
- LLM-judge scoring: The coverage score averages key-point match values, while a penalty accounts for unsupported, irrelevant, speculative, or conflicting extra content.The final score is computed as max(0, min(1, C - P)).
- Response-specific checks: For response evaluation, extra length does not increase the score, and directly copying prior context without quoting it is treated as off-task behavior.Longer responses are acceptable only when consistent and nonredundant.