Source-linked AI summary
Investigating Assistant Bias in LLM User Simulators Using a Role Vector
Daeheon Jeong, Yoonjoo Lee, Eugene Choi, Sinie van der Ben, Juho Kim
TL;DR
LLM user simulators can overestimate autonomous-agent performance because assistant bias makes them cooperate and pursue goals instead of reproducing real-user disengagement. The paper extracts a user-role vector by contrasting model activations for user and assistant reflections on the same dialogues, then analyzes and steers along that direction. The vector elicits user-like behavior and is associated with simulation realism, but steering can exaggerate behavior and obscure individual user profiles.
Problem
LLM user simulators are unreliable proxies for real users because assistant bias favors cooperative, goal-oriented behavior over disengagement, compromising evaluation validity.
Method
The paper extracts a user-role vector by subtracting mean assistant-reflection activations from mean user-reflection activations on the same dialogues.
Results
The user direction is identifiable, elicits user-like behaviors, and is associated with simulation realism, while steering can exaggerate user behavior and obscure individual profiles.
Takeaways & Limitations
Assistant bias is a measurable representational property, and user behavior can be analyzed directionally through user-role activation.
Takeaways & Limitations
The evaluation uses proxy realism measures rather than direct human judgment, and the cross-model checks do not establish the same causal pattern across model families or scales.
Abstract
from arXiv · showhide
LLM-based user simulators are increasingly used to evaluate autonomous agents at scale, in place of costly human evaluations. Despite this promise, these simulators exhibit "assistant bias," a tendency to cooperate and pursue task goals. They rarely reproduce the frustration or disengagement that real users exhibit, compromising evaluation validity. Prior work outlines that this bias is baked in during model training, which role-playing prompts fail to override. We analyze this bias from model activations, extracting a user role vector by contrasting how the model represents user versus assistant perspectives on the same dialogue. We observe two findings: (i) the user direction is identifiable in activations, elicits user-like behaviors, and captures characteristics distinct from assistant traits; and (ii) although user-role activation associates with simulation realism and steering strengthens it, it can exaggerate user behaviors and override individual user profiles. Together, our findings provide a representation-level analysis of LLM user simulators, confirming that assistant bias is structurally identifiable and that user behavior can be directionally analyzed.
1 Introduction
LLM user simulators often over-cooperate and pursue goals, so they can overestimate autonomous-agent performance relative to real users. This paper extracts and analyzes a user-role direction in model activations, finding that it elicits user-like behavior and improves realism while risking exaggerated behavior and obscured user profiles.
- Motivation: User simulators can overestimate agent performance because they continue interactions that real users might abandon.Their cooperative, goal-oriented behavior contrasts with real-user disengagement caused by confusion, lost trust, or assistant failure.
- Approach: The user role vector is extracted by subtracting mean assistant-reflection activations from mean user-reflection activations on the same dialogues.Role-specific reflections are designed to cancel irrelevant variation while isolating user-role characteristics.
- Representation analysis: Steering along the user role vector produces shorter, more informal, less information-dense outputs and increases disengagement after agent errors.These activation changes affect both communication style and behavioral response.
- Representation analysis: The user direction is negatively aligned with assistant-like traits including helpfulness and cooperativeness.This geometric relationship distinguishes user-role characteristics from assistant-oriented traits.
- Multi-turn evaluation: User-role steering improves writing-style similarity to real-user messages but exaggerates instructed interaction behavior by 17.9 percentage points on average.Uniform steering may also obscure individual user profiles, making calibration difficult.
- Multi-turn evaluation: User-role activation positively correlates with user-likeness, while its weakening over multi-turn simulations provides a diagnostic signal for simulation realism.The results support directional analysis of assistant bias but indicate that steering strength is difficult to calibrate.
2 Extracting the User Role Vector
The user role vector is extracted by contrasting role-specific reflections on the same dialogues, filtering unreliable reflections, and differencing averaged activations. The pipeline uses balanced LMSYS-Chat-1M dialogues and multiple prompt variants to isolate role-related activation patterns.
- Vector Computation: The user role vector is computed by subtracting mean assistant-reflection activations from mean user-reflection activations across paired dialogues.The activations are measured at a target layer from the first reflection token.
- Data Source: The extraction set samples 2–50-turn dialogues from LMSYS-Chat-1M with valid role labels and non-empty messages.
- Data Source: Topic balancing categorizes dialogues into 24 conversation types and samples 30 dialogues per category, yielding 720 dialogues.
- Role-Specific Reflection: For each dialogue, the model generates reflections from both the user and assistant perspectives on the same conversation.This role contrast is intended to reduce topic, task, and wording variation unrelated to role.
- Vector Computation: Activations are averaged across three role-prompt variants to reduce sensitivity to a particular instruction template.
- Reflection Validation: An LLM-as-judge labels reflections by role fidelity, and reflections labeled “not represented” are discarded.Low-fidelity reflections would add noise to the difference-in-means estimate.
- Steering at Inference: During inference, the normalized direction is added to hidden states at a selected layer across prompt and response tokens.The intervention strength is controlled by α, with α = 0 leaving the model unchanged.
3 What Does the User Role Vector Represent?
The user role vector is analyzed through changes in simulator communication, disengagement, and geometric alignment with assistant-like traits. Steering makes outputs more user-like and increases disengagement, but stronger steering can exaggerate behavior and reduce timing accuracy.
- Overview: The user role vector is evaluated through communication style, disengagement behavior, and its geometric relationship to assistant-like traits.These lenses cover stylistic output, interaction responses to errors, and alignment with assistant-trait directions.
- Communication Style: User-role steering increases user-likeness, producing shorter, more informal, and less information-dense messages.Mean user-likeness rises from 1.62 at α = 0.1 to 2.01 at α = 0.3 across layers.
- Communication Style: Steering at layers 11–13 lifts the mean user-likeness score by roughly 2.0 points and surpasses the user-prompted baseline.The layer-wise effect is concentrated in a narrow mid-layer band and is consistent across two other models.
- Interaction Behavior: User-role steering increases disengagement from 48% unsteered to 67% at α = 0.3, while exact-match accuracy peaks at only 5.9% at α = 0.2.The asymmetry indicates that steering primarily changes whether the simulator disengages rather than when.
- Interaction Behavior: At α = 0.3, predictions cluster prematurely near the first turns, so stronger steering overshoots realistic disengagement behavior.Turn 1 accounts for approximately 15% of dialogues, despite the ground-truth final turn being turn 8.
- Assistant-Trait Geometry: The user role vector points opposite most assistant-like trait directions, including helpfulness, cooperativeness, accommodating, rational, and cooperative traits.Absolute similarities are modest; the consistent negative alignment is more informative than any individual magnitude.
4 What Does the User Role Vector Reveal About Simulation?
User-role activation is associated with greater simulation realism and steering can move simulators toward real-user behavior, but fixed-strength steering may exaggerate behaviors and suppress individual profiles.
- Activation and realism: Steering increases user-role activation and produces shorter, more informal, less information-dense outputs while increasing disengagement after agent errors.The evaluation measures communication style and behavioral response as complementary dimensions of simulation.
- Activation and realism: Zero-shot steering increases overall similarity from 2.61 unsteered to 2.73 at α = 0.3, driven mainly by writing-style similarity rising from 2.10 to 2.33.Interaction-style similarity remains 3.13 at both endpoints.
- Behavioral response: At α = 0.3, doubt and mistake rates exceed the human reference, showing that strong steering can overshoot confused-student behavior.The behavioral analysis covers doubt, misunderstanding, mistake, and clarification questions.
- Individual profiles: With profile prompts, stronger steering reduces overall similarity and weakens expression of assigned writing- and interaction-style features.The unsteered baseline ranks highest most often, while α = 0.3 ranks lowest most often for both profile categories.
- Activation and realism: Intrinsic user-role activation positively correlates with SimulatorArena similarity, especially writing-style similarity, across 3,150 simulated conversations.Overall similarity correlations are Pearson’s r = 0.426 and Spearman’s ρ = 0.451; writing-style correlations are r = 0.518 and ρ = 0.553.
- Individual profiles: User-role activation declines as turns accumulate, while demographic profiles initially raise average activation but decline more sharply across turns.Linguistic and full profiles average −1.80 and −1.78 versus −1.69 for the goal-only baseline; demographic profiles average −1.65.
5 Related Work
Prior research finds that user simulators default to cooperative assistant behavior, while activation studies show that roles, personas, traits, styles, emotions, and toxic behaviors can correspond to linear directions in model representations.
- Assistant bias: User simulators often produce polite, explanation-heavy, goal-aligned behavior beyond the human turns they simulate.This assistant-role default motivates studying representational directions associated with user behavior.
- Representation engineering: Representation studies report linear activation directions for roles, personas, stable traits, personality dimensions, stylistic traits, emotions, and toxic behaviors.These directions can capture or anticipate the behaviors they represent.
6 Conclusion
The paper analyzes assistant bias through a user-role vector extracted by contrasting user and assistant perspectives on the same dialogue. It finds that the direction is identifiable, influences user-like behavior, improves realism, and can exaggerate behavior or obscure individual profiles.
- Conclusion: The user-role vector is a direction in activation space extracted by contrasting role-specific representations of the same dialogue.The paper treats assistant bias as a measurable representational property that can be analyzed directionally.
- Conclusion: The direction influences user-like behavior and improves behavioral realism, while steering can exaggerate user behavior and obscure individual user profiles.Tracking activation also shows its association with simulation realism, supporting its use as a diagnostic signal.
Limitations
The evaluation is limited by model coverage, proxy-based realism measures, a narrow benchmark setting, and fixed-strength steering. The extraction pipeline uses filtered and sampled LMSYS dialogues with role-specific reflection screening.
- Scope: The analysis focuses primarily on Qwen 3.5 9B, so the causal pattern is not established across other model families or scales.Larger and more diverse models are needed to test whether the observed layer localization persists.
- Measurement: Realism is evaluated with proxy measures and LLM-as-judge scores rather than direct human judgment.Targeted human studies could sharpen measurement of interaction realism.
- Scope: The benchmark centers on math tutoring and sampled WildChat dialogues, covering only a slice of user-simulation use cases.Coding-agent and customer-support workflows are proposed for testing transfer beyond the current setting.
- Steering: Fixed activation strength can exaggerate instructed behaviors to unrealistic levels, motivating adaptive or optimization-based steering.The limitation concerns adjustment by task, layer, conversation state, or uncertainty rather than a single coefficient per interaction.
- Extraction pipeline: The extraction set filters LMSYS-Chat-1M dialogues by turn count, role labels, and non-empty content before sampling 100,000 dialogues.The accepted pool contains 982,581 of one million source dialogues, with 17,419 dropped.
- Extraction pipeline: Role-specific reflections are generated for each dialogue and retained only when GPT 5 Mini judges the assigned perspective strongly or weakly represented.Paired user-minus-assistant activations are then averaged and normalized for the extracted direction.
B.1.1 Task Goal and Request Generation
The experiment steers Qwen 3.5 9B while generating request messages conditioned on diverse task goals, then evaluates communication style and task-goal overlap across layers and strengths.
- The simulator generates request messages from diverse task goals under steered and unsteered conditions, using a user-like request prompt for the baseline.The steering sweep covers layers 10–14 and strengths α ∈ {0.1, 0.2, 0.3, 0.4}, with 100 task goals per condition.
- Above α = 0.3, the simulator produces extremely short messages that largely repeat the task goal.
- Stronger steering improves user-likeness but also increases reuse of task-goal text.Table 10 reports communication-style scores alongside ROUGE-L precision against the task goal.
B.1.3 Task-Goal Generalization
The steering effect generalizes from generated task goals to human-authored MultiWOZ and TREC health-information goals. User-likeness scores increase monotonically across both external datasets as steering strength rises.
- Task-Goal Generalization: The experiments use 100 sampled MultiWOZ goals and all 50 topics from each of the 2021 and 2022 TREC Health sets.The recorded first utterance and dialogue are withheld for MultiWOZ, while gold questions are withheld for TREC.
- Task-Goal Generalization: User-likeness scores increase monotonically with steering strength on both MultiWOZ and TREC task goals.Each source uses Qwen 3.5 9B at layer 11 across α ∈ {0, 0.1, 0.2, 0.3}.
- Task-Goal Generalization: On MultiWOZ 2.0, the overall score rises from 1.47 to 2.85, with information pacing increasing from 1.30 to 3.08.Information pacing shows the largest change, indicating less disclosure of the full multi-turn goal in the opening message.
- Task-Goal Generalization: On TREC Health, the overall score increases from 2.55 to 3.53, while brevity and information pacing rise to 4.17 and 4.13.Informality remains the lowest dimension, increasing from 1.54 to 2.30.
- Task-Goal Generalization: A GPT 5 Mini judge scores brevity, informality, and information pacing on 1–5 scales, with higher scores indicating more user-like style.
B.1.5 LLM-as-Judge Validation
The LLM judge distinguishes real WildChat messages from reconstructed messages and agrees substantially with human ratings. Real messages score higher across all communication-style dimensions, especially information pacing.
- Real-versus-Reconstructed Validation: The validation compares 1,000 sampled WildChat conversations by hiding each final user message and reconstructing it from prior dialogue; two samples are excluded.
- Real-versus-Reconstructed Validation: Figures 10 and 11 show that real messages score higher overall and on every dimension, with the largest gap in information pacing.
- Real-versus-Reconstructed Validation: On 998 paired WildChat samples, real messages score 4.07 versus 2.94 for synthetic messages, a mean gap of 1.13.Real messages win 63.2% of pairwise comparisons, tie in 24.9%, and lose in 11.8%.
- Real-versus-Reconstructed Validation: Real messages exceed reconstructed messages on brevity (+0.91), informality (+0.90), and information pacing (+1.56).The largest separation is in information pacing.
- Human-Rating Agreement: Human and judge scores correlate strongly overall at ρ = 0.800 across 1,200 human ratings.Dimension-level correlations are 0.798 for brevity, 0.439 for informality, and 0.754 for information pacing.
- Human-Rating Agreement: Human scores rise from 2.23 without steering to 3.77 at α = 0.3, while judge scores rise from 1.63 to 3.39.Human raters show substantial overall agreement, with Kendall’s W = 0.700.
B.1.7 Activation Correlation with Communication Style
The study projects teacher-forced request hidden states onto a normalized user-minus-assistant direction and averages across response tokens. User-role activation is positively associated with communication-style scores, including after length control.
- Activation Measurement and Correlation: The experiment uses 100 task goals and scores requests for brevity, informality, and information pacing with GPT 5 Mini.
- Activation Measurement and Correlation: The activation measure projects layer-11 hidden states onto the normalized user-minus-assistant direction and averages projections across response tokens.
- Activation Measurement and Correlation: User-role activation is positively correlated with overall communication-style score, with Pearson’s r = .260 and Spearman’s ρ = .313.The strongest rank association is for brevity, at ρ = .322.
- Steering-Layer Comparison: Figure 13 reports brevity scores across steering layers and strengths, with scores peaking at layers 11–13.
- Activation Measurement and Correlation: After controlling for response length, the overall association strengthens to r = .292 and ρ = .337.Response length is associated with communication-style scores but only weakly with user-role activation in this setup.
B.1.8 Layer-Wise Steering Effects
User-like communication responds most strongly to steering in layers 11–13, where stronger coefficients improve scores without saturation. Information pacing shifts most, while later layers provide little benefit.
- 1.51 is the empirical unsteered baseline for communication-style scoring across layers.
- All three communication dimensions respond strongly in layers 11–13, while layers 19 and beyond provide little or no benefit from stronger steering.
- Within layers 11–13, α = 0.3 outperforms smaller coefficients by 0.45–0.57 points without saturation.
- Information pacing shows the largest absolute shift, peaking near 4.7, whereas informality reaches the top of the scale earliest and saturates.
- Stronger steering is useful only within the mid-layer window because α = 0.3 often matches or underperforms α = 0.1 from layer 16 onward.
B.1.9 Cross-Model Steering Effects
Cross-model steering effects are strongest in architecture-specific early-to-mid-layer regions and increase user-likeness monotonically with signed steering. However, stronger steering can produce degenerate outputs and does not reliably improve disengagement timing.
- Cross-model layer effects: Granite gains appear around layer 12 and layers 19–23, whereas Llama gains are strongest mostly in layers 1–10.
- Limitations: At α = 0.3, Granite and Llama sometimes produce degenerate outputs, so cross-model layer-localization evidence is qualitative and reliable strength remains model-dependent.
- Cross-model layer effects: User-role directions form architecture-specific layer blocks: Qwen has a broad layers 6–20 cluster, Granite transitions around layers 12–16, and Llama stabilizes later.
- Bidirectional steering: User-likeness rises monotonically from 1.05 at α = −0.3 to 3.42 at α = 0.3, with the unsteered baseline at 1.53.
- Disengagement prediction: Llama disengagement rises from 15.3% without steering to 52.7% at α = 0.3, while exact match peaks at α = 0.1 and average distance increases from 3.86 to 5.81 turns.
C.1.3 Cross-Model Results
Across open-weight models, user-role steering improves writing-style and overall similarity after appropriate evaluation handling. The gain is accompanied by weaker expression of assigned profiles and model-specific evaluation constraints.
- Evaluation handling: Llama requires explicit termination detection because repeated responses otherwise make every raw conversation reach the 15-turn limit.
- Evaluation handling: The termination detector shortens at least 97.7% of dialogues, reducing mean length to 5.03–6.05 user turns.
- Cross-model similarity: After termination detection, Llama’s writing-style similarity increases from 2.18 to 2.46 and overall similarity from 2.61 to 2.79 as α rises to 0.3.
- Evaluation stability: Across-run standard deviation is at most 0.023, below the zero-shot Qwen gain of 0.116 from α = 0 to α = 0.3.
- Profile preservation: Higher steering weakens profile-specific attribute expression broadly, reflected by increasing mean ranks across profile conditions.
E.5 User-Role Activation Analysis
The evaluation prompts vary whether simulated users receive demographic, linguistic, or full profiles alongside the task goal and conversation history. They instruct the model to generate only the next user message while remaining consistent with the dialogue context.
- Prompt instructions: The prompts direct the model to reproduce the ground-truth trajectory, respond consistently with context, and write only the next user message.
- Prompt variants: The demographics condition adds a user-demographics field to the goal and conversation history.
- Prompt variants: The full-profile condition supplies both demographic and linguistic attributes with the goal and conversation history.
- Prompt variants: The goal-only condition supplies the user goal and conversation history without demographic or linguistic profile information.
- Prompt variants: The linguistic-profile condition adds a user linguistic style field while retaining the goal and conversation history.