Source-linked AI summary
Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces
Jiawei Chen, Ruoxi Xu, Boxi Cao, Ruotong Pan, Yunfei Zhang, Yifei Hu, Yong Du, Tingting Gao, Yaojie Lu, Yingfei Sun, Xianpei Han, Le Sun, Xiangyu Wu, Hongyu Lin
TL;DR
Existing user-simulation benchmarks provide limited evidence because they often isolate scenarios, narrow action spaces, or rely on synthetic data. The paper introduces OmniBehavior, a real-world benchmark integrating long-horizon, cross-scenario, heterogeneous traces, and finds that LLMs struggle with realistic behavior while converging toward a positive average person.
Problem
Existing benchmarks are often limited to isolated scenarios, narrow action spaces, or synthetic data rather than holistic authentic behavior.
Method
The paper constructs OmniBehavior from authentic Kuaishou logs, integrating long-horizon, cross-scenario, and heterogeneous behavior traces for user-conditioned simulation evaluation.
Results
LLMs show substantial limitations in realistic long-horizon, cross-scenario simulation, with performance not necessarily improving when context extends beyond 32K or to 128K tokens.
Takeaways & Limitations
LLM simulators exhibit a positivity-and-average bias that overestimates engagement, homogenizes users, and suppresses negative and long-tail behaviors.
Takeaways & Limitations
The evaluation does not yet encompass all state-of-the-art models and memory-management algorithms.
Abstract
from arXiv · showhide
The emergence of Large Language Models (LLMs) has illuminated the potential for a general-purpose user simulator. However, existing benchmarks remain constrained to isolated scenarios, narrow action spaces, or synthetic data, failing to capture the holistic nature of authentic human behavior. To bridge this gap, we introduce OmniBehavior, the first user simulation benchmark constructed entirely from real-world data, integrating long-horizon, cross-scenario, and heterogeneous behavioral patterns into a unified framework. Based on this benchmark, we first provide empirical evidence that previous datasets with isolated scenarios suffer from tunnel vision, whereas real-world decision-making relies on long-term, cross-scenario causal chains. Extensive evaluations of state-of-the-art LLMs reveal that current models struggle to accurately simulate these complex behaviors, with performance plateauing even as context windows expand. Crucially, a systematic comparison between simulated and authentic behaviors uncovers a fundamental structural bias: LLMs tend to converge toward a positive average person, exhibiting hyper-activity, persona homogenization, and a utopian bias. This results in the loss of individual differences and long-tail behaviors, highlighting critical directions for future high-fidelity simulation research.
1 Introduction
OmniBehavior addresses the limits of isolated, narrow, or synthetic user-simulation benchmarks with a real-world benchmark that integrates long-horizon, cross-scenario, heterogeneous behavior. Evaluations show that LLMs struggle with realistic simulation and exhibit a positivity-and-average bias that obscures individual and long-tail behavior.
- Motivation: Existing benchmarks focus on isolated scenarios and narrow action spaces, overlooking the interconnected, heterogeneous continuum of authentic user behavior.Real behaviors range from browsing and clicking to searching, commenting, consulting, and purchasing across scenarios.
- Behavioral analysis: Integrating new scenarios expands interest coverage by approximately 20-30%, while over 80% of conversion paths span multiple scenarios and extend over days.These findings indicate that fragmented single-scenario snapshots miss important structure in authentic decision-making.
- Evaluation: The best-performing evaluated LLM, Claude-4.5-Opus, achieves an overall score of 44.55, and most models’ binary behavior-prediction F1 scores do not exceed 40%.Extending context beyond 32K or to 128K tokens does not necessarily improve performance.
- Structural bias: LLM simulators display a positivity-and-average bias, overestimating engagement while homogenizing personas and suppressing negative and long-tail interactions.This structural bias limits faithful modeling of individual differences and realistic human behavior.
- Benchmark: OmniBehavior is introduced as a real-world benchmark unifying long-horizon, cross-scenario, and heterogeneous behavioral traces.It is constructed from authentic interaction logs and designed to evaluate LLM-based user simulators in realistic settings.
2 OmniBehavior Benchmark
OmniBehavior is built from end-to-end Kuaishou logs to represent diverse, chronological user behavior across five scenarios. Its construction combines multi-stage cleaning, representative sampling, anonymization, and long-horizon trajectory aggregation.
- Scope and data collection: The benchmark unifies major Kuaishou scenarios and complete three-month interaction histories, requiring models to represent cross-scenario links and ultra-long sequences.Unlike fragmented datasets, the construction preserves interconnected activities and chronological behavior.
- Taxonomy: OmniBehavior covers 5 major scenarios and 22 distinct user actions to model holistic user decision-making.The action space spans activities across video browsing, live streaming, advertising, and e-commerce-related interactions.
- Construction pipeline: The end-to-end construction pipeline includes data collection, cleaning, sampling, and anonymization to improve quality, representativeness, and privacy compliance.These stages are designed to produce high-fidelity sequences reflecting real-world decision-making.
- Data cleaning: Behavior-level cleaning removes spurious interactions using 99.9th-percentile duration thresholds of 879s for videos and 9,601s for live streams.The procedure removes a median of 91 noisy records per user; text-level cleaning is also applied.
- Data sampling: Clustering-based sampling selects diverse representative users using demographic, activity, interest, and scenario-preference features.The feature vector spans four axes to support coverage of varied behavioral patterns.
- Dataset scale and anonymization: The released benchmark contains 200 users across 5 scenarios, with an average sequence length of 8,143 actions.Sensitive entities are replaced with semantic placeholders, harmful content is pruned, and processed data undergoes manual verification.
3 Authentic User Behavior Analysis
Authentic user behavior is better captured by combining scenarios and tracing interactions over long horizons. Analyses of OmniBehavior show that real preferences evolve more smoothly than synthetic trajectories, while decisions commonly depend on multi-day, cross-scenario causal chains.
- Multi-scenario user modeling: Multi-scenario profiles provide richer contextual signals and better capture users’ stable characteristics than single-scenario profiles.Single-scenario profiles are often fragmented and biased, whereas multi-scenario data captures broader user interests.
- Multi-scenario user modeling: Information coverage increases consistently as scenario diversity grows, confirming a statistical advantage for multi-scenario modeling.The analysis measures cumulative growth in unique interest categories and keywords across users.
- Long-horizon cross-scenario causal chains: Over 60% of decisions use cues from more than 3 days earlier, and 81.8% of causal chains span multiple scenarios.These distributions indicate that user decisions extend beyond short-session and single-scenario assumptions.
- Long-horizon cross-scenario causal chains: A representative 12-day chain links searches, cross-scenario interactions, and a livestream before culminating in a purchase.The case study illustrates how interests accumulate across diverse interactions over time.
- Authentic versus synthetic preference evolution: Authentic users show smoother, more stochastic interest changes than synthetic users, with average drift rates of 0.6311 versus 0.1698.Daily drift is measured as 1 − Jaccard between category sets on consecutive days.
4 LLM Evaluation
OmniBehavior evaluates diverse LLMs on real-world, long-horizon user behavior prediction and finds that high-fidelity simulation remains difficult. Longer context windows and memory management provide no consistent or substantial solution.
- 4.1 Experiment Settings: OmniBehavior evaluates LLMs across binary, continuous, and textual user behaviors using F1-score, NMAE, and textual-behavior metrics.The benchmark includes diverse closed-source and open-source model baselines and 6,000 balanced behavior prediction tasks.
- 4.2 Overall Results: 44.55 overall score is achieved by Claude-Opus-4.5, while most models cluster between 32 and 41.The results indicate that general instruction tuning remains insufficient for stochasticity, long-tail distributions, and cross-scenario causal dependencies.
- 4.2 Overall Results: 41.46 overall score places open-source GLM-4.7 second, ahead of Claude-Sonnet-4.5 at 40.49 and GPT-5.2 at 39.07.DeepSeek-V3 also exceeds Claude-Opus-4.5 in E-commerce binary behavior simulation, 33.31 versus 29.98.
- 4.3 Evaluation of Long Context Modeling: Longer context windows from 16K to 128K tokens do not consistently improve performance on users with histories exceeding 128K tokens.The evaluation suggests that current LLMs struggle to leverage extended interaction histories for long-context reasoning.
- 4.3 Evaluation of Long Context Modeling: History summarization and RAG-based retrieval offer only modest gains in memory management.RAG can ignore causal dependencies, while summarization may discard critical behavioral details.
5 The Structural Bias of LLM Simulator
The study identifies a positivity-and-average structural bias in LLM simulators by comparing simulated and authentic behavior distributions, emotions, and cross-user differences. Simulators overproduce engagement and positive language while reducing behavioral heterogeneity.
- Structural Bias: LLM simulators exhibit a positivity-and-average tendency across behavior probabilities, emotional distributions, and cross-user behavioral differences.The comparison is designed to probe the causes of the simulation gap between generated and authentic behavior.
- Hyper-activity Bias: 40–60% overestimation of user engagement is observed for Qwen3-235B and Gemini-3-Flash, while real positive behavior rates remain below 10%.This hyper-activity bias reflects poor modeling of implicit rejection and limits applications such as user churn warning.
- Utopian Tendency: LLM-generated customer-service utterances cluster around neutral or positive sentiment, whereas real users often express strong negative emotions.Claude-Sonnet-4.5 outputs are also more polite than real users’ language.
- Personality Homogenization: Real users show much larger inter-user than intra-user behavioral variation, while LLM-generated users reduce this distinction.The analysis uses 19-dimensional action-rate vectors to assess within-user consistency and cross-user heterogeneity.
6 Related Works
Prior human-simulation research uses controlled or synthetic settings and benchmarks that often isolate scenarios. These designs miss cross-scenario dependencies and broader behavioral heterogeneity.
- Human Simulation Benchmarks: Cross-scenario behavioral distances are used to compare real users with LLM-simulated users.Figure 10 frames the comparison in terms of intra-user and inter-user behavioral distances.
- Existing Simulation Approaches: Earlier LLM agents primarily simulate cognition and interaction in controlled, single-task environments or synthetic sandboxes.Such approaches often fail to capture stochasticity and long-tail distributions of human behavior.
- Human Simulation Benchmarks: Log-based benchmarks have progressed from static interaction matrices to sequential logs and interactive environments, but usually remain scenario-specific.Role-playing benchmarks instead emphasize persona consistency, memory, and related capabilities.
7 Discussion
OmniBehavior shows that realistic user simulation requires long-term, cross-scenario behavior modeling, while current LLMs retain substantial capability gaps and positivity-and-average biases. The evaluation is also bounded by incomplete coverage of evolving models and memory methods.
- Discussion and Conclusion: OmniBehavior’s statistical analysis finds long-term and cross-scenario dependencies in human behavior, making single-scenario and purely synthetic settings insufficient for realistic application.The benchmark is intended to support modeling of long-horizon, cross-scenario, and heterogeneous behavior.
- Discussion and Conclusion: Current LLMs show a substantial capability gap in real-world behavior modeling regardless of context length.The identified positivity-and-average tendency homogenizes users and overestimates engagement, limiting simulator reliability in real-world scenarios.
- Limitations: The evaluated models and memory-management algorithms do not yet encompass all state-of-the-art approaches.The authors plan a public leaderboard for community-contributed submissions and ongoing updates.
LLM Usage Statement
LLMs were used only for limited manuscript editing and coding assistance, with all suggestions reviewed and approved by the authors. They did not contribute to formulating the research questions.
- LLMs assisted with grammar correction, spelling checks, and coding during experimental implementation.
- The authors reviewed, revised, tested, and approved all generated text and code suggestions.
- LLMs did not contribute to the formulation of the research questions.
Ethics Statement
OmniBehavior uses behavior data sampled and processed under applicable legal, platform, and privacy-protection requirements. Released data underwent quality control, representative sampling, and privacy-preserving cleaning.
- The benchmark uses only data permitted for research and benchmark construction.
- Raw data underwent multiple stages of quality control, representative sampling, and privacy-preserving cleaning before release.
- The data-processing process addresses personally identifiable and sensitive information.
A Data Statistics
OmniBehavior contains behavior sequences spanning short traces to histories exceeding 100k steps, alongside diverse user demographics, ages, and interests. These statistics reflect long-horizon behavior and substantial population heterogeneity.
- User action sequences span multiple orders of magnitude and can exceed 100k steps.
- The benchmark includes relatively short traces as well as extremely long user histories.
- Sampled users span diverse gender and age groups.
- Users also cover a broad spectrum of content interests, preserving heterogeneity in preferences.
B Benchmark Schema
OmniBehavior defines a unified benchmark across representative Kuaishou scenarios, using heterogeneous contextual signals to generate diverse user actions. Its consistent evaluation protocol supports realistic assessment of complex behavior modeling across product surfaces.
- Benchmark Scope: The benchmark covers Video Browsing, Live Streaming, E-commerce, and Advertisement scenarios on Kuaishou.Customer-service next-utterance prediction is treated as part of E-commerce.
- Context and Actions: Simulator inputs combine timestamps, content titles, item attributes, historical statistics, and available OCR or ASR text.
- Context and Actions: The LLM-based simulator generates actions whose spaces vary by scenario, including behaviors such as liking and collecting.
- Evaluation Framework: A unified formulation captures diverse interaction patterns while maintaining a consistent evaluation protocol.
- Evaluation Framework: The benchmark provides a structured environment for assessing LLM-based simulators on complex user behaviors.
C.2 Memory Management Results
Memory strategies trade off long-horizon signal aggregation, sequential fidelity, and fine-grained behavioral detail. Summary performs best overall, while existing approaches remain insufficient for realistic user-behavior modeling.
- Strategy comparison: RAG can underperform truncation in E-commerce, scoring 16.14 versus 19.22.The authors suggest retrieval noise or disrupted sequential dependencies may explain this result.
- Limitations: Existing memory strategies remain insufficient because truncation lacks long-term reasoning, RAG ignores causal structure, and summarization loses fine-grained details.The paper calls for structure-aware mechanisms modeling temporal order, cross-scenario dependencies, and causal relationships.
- Strategy comparison: Table 2 compares truncation, RAG, and Summary using Qwen3-235B across binary behavior-prediction scenarios.The evaluated behaviors include actions such as like, collect, and share.