Source-linked AI summary
UserArena: Benchmarking Interactive User Simulation for Long-Horizon Shopping Agents
Jujia Zhao, Jiakai Tang, Jing Yu, Wen Chen, Jian Wu, Han Zhu, Fei Sun, Suzan Verberne, Zhaochun Ren
TL;DR
Existing resources rarely provide long-horizon, fine-grained real-user trajectories or support trajectory-level evaluation against production A/B outcomes. UserArena combines these trajectories with an interactive environment and five-level evaluation, while OPTD distills trajectory-level intent into teacher targets on student-generated rollouts. OPTD achieves the strongest overall fidelity under UserArena’s protocol and most closely reproduces the production policy effect in offline simulation.
Problem
Existing resources rarely provide long-horizon, fine-grained real-user trajectories or test whether simulated policy comparisons agree with production A/B outcomes, limiting assessment of behavioral fidelity and simulation validity.
Source: §1 · pp. 1–2Method
UserArena pairs real-user shopping trajectories with an interactive environment and five-level evaluation, while OPTD uses trajectory-level intent to guide dense teacher supervision on student-generated rollouts.
Source: §1 · p. 2, §3.2 · p. 5Results
OPTD achieves the strongest overall fidelity under UserArena’s evaluation protocol and most closely reproduces the production policy effect in the controlled comparison.
Source: §6 · p. 9Takeaways & Limitations
UserArena supports evaluation from environment compliance and local decisions through sequential behavior, personalization, and outcomes, including offline comparison with controlled production A/B results.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Agentic recommendation pursues shopper goals through sequences of context-dependent decisions. Developing and evaluating such policies requires faithful recommendation world models that capture both the interactive shopping environment and users' behavioral responses over long interactions. Existing resources rarely provide long-horizon, fine-grained real-user trajectories or support trajectory-level evaluation and offline validation against production A/B results. We introduce UserArena, a real-world e-commerce interactive benchmark containing 64,729 session trajectories from 11,010 users, spanning 566,102 items and 13 action types. UserArena pairs these trajectories with an interactive environment and supports offline validation against outcomes from a controlled production A/B test, enabling five-level evaluation from output validity and local decisions to sequential behavior, personalization, and outcomes. To improve simulator fidelity over these long-horizon interactions, we further propose On-Policy Trajectory Distillation (OPTD), which uses trajectory-level intent to provide dense teacher targets on student-generated rollouts. Experiments show that OPTD achieves the strongest overall fidelity among the evaluated simulators under UserArena's evaluation protocol and most closely reproduces the production policy effect in offline simulation.
1 INTRODUCTION
Agentic recommendation depends on world models that simulate both shopping environments and user responses, but existing resources rarely capture long-horizon real-user trajectories or validate simulated policy comparisons against production outcomes. UserArena pairs such trajectories with an interactive benchmark and five-level evaluation, while OPTD uses trajectory-level intent to guide student-generated rollouts.
1 INTRODUCTION
- Existing datasets rarely align real-user profiles, visual observations, product context, and actions across long-horizon repeated sessions, while next-action agreement misses trajectory-level behavioral tendencies.Benchmarks also rarely test whether simulated policy comparisons agree with production A/B outcomes, limiting assessment of behavioral fidelity and real-world validity.Source: §1 · pp. 1–2
- UserArena combines real-user trajectories, an interactive shopping environment, production A/B validation, and five-level evaluation from output validity to outcomes.The benchmark covers 11,010 users, 566,102 items, 64,729 session trajectories, and 13 action types.Source: §1 · p. 2
- OPTD distills teacher guidance on current-policy responses at real states, using student-generated prefixes and trajectory-level intent to guide coherent long-horizon behavior.The paper identifies behavioral miscalibration, sparse supervision, and exposure bias as challenges to long-rollout simulation.Source: §1 · p. 2
- The paper benchmarks text- and GUI-based simulators for effectiveness and efficiency and characterizes behavioral composition and interaction horizons.These are stated as part of the paper’s threefold contributions alongside UserArena and OPTD.Source: §1 · p. 2
2 THE USERARENA BENCHMARK
UserArena combines production-grounded shopping interactions with fine-grained real-user trajectories and an evaluation protocol spanning local decisions, long-horizon behavior, personalization, and outcomes. Its protocol uses replay and free-running rollout to assess distinct levels of simulator fidelity.
2.1 DATA AND INTERACTION DESIGN
- Existing benchmarks do not combine real step-level trajectories, visual context, a full shopping action space, long-horizon cross-session behavior, and production A/B validation.The passage contrasts datasets that often omit visual context with interactive platforms whose user-dynamics fidelity is insufficiently established.Source: §2.1 · p. 3
- UserArena reconstructs production feed and item-detail pages with 13 action types and aligns each recorded decision with user profile, visual observation, product context, and action.It contains 11,010 users, 64,729 sessions, 577,670 actions, 566,102 products, and six user attributes.Source: §2.1 · p. 3
- 59,378 sessions from 5,768 users form cross-session trajectories averaging 10.29 sessions and 86.36 actions per user, while the benchmark supports offline comparison with controlled production A/B outcomes.The benchmark addresses requirements for fine-grained long-horizon trajectories and policy comparisons against production outcomes.Source: §2.1 · pp. 2–3
2.2 EVALUATION PROTOCOL
- The five evaluation levels progress from parseable outputs (L0), through local action agreement (L1), sequential dynamics (L2), and personalized population behavior (L3), to aggregate outcomes (L4).Replay evaluates next-action fidelity from recorded states; rollout evaluates free-running sessions and subsequent states.Source: §2.2 · pp. 3–4
- Figure 2 contrasts supervision: SFT learns from recorded responses, GRPO assigns response-level scalar rewards, and OPTD gives full-vocabulary teacher targets at each token of student rollouts.The figure highlights differences in where and at what granularity each method supplies supervision.Source: Figure 2 · p. 4
3 ON-POLICY TRAJECTORY DISTILLATION
OPTD trains a student using teacher guidance on its own responses, conditioned on intent extracted from complete trajectories. A per-token trust region limits intent-driven changes, while the training objective combines distillation with grounding in recorded responses.
3 ON-POLICY TRAJECTORY DISTILLATION
- OPTD provides dense token-level teacher supervision on student-generated responses, using complete-trajectory intent as privileged context while limiting divergence from the student’s observable state.This on-policy setup addresses the mismatch between teacher-forced ground-truth prefixes and student-generated prefixes at inference.Source: §3 · p. 4, §3.1 · p. 5
3.1 TASK SETUP
- The student policy generates a complete response from a sampled real state, and the teacher scores tokens on the student’s prefixes rather than only recorded prefixes.This one-step on-policy rollout exposes training to self-generated prefixes.Source: §3.1 · pp. 4–5
3.2 TRAJECTORY-LEVEL INTENT SUPERVISION
- Trajectory-level intent is extracted as typed anchors for preferences, concerns, information needs, comparison criteria, and conditional decision tendencies, without specifying the recorded next action.The student does not directly receive the intent; it influences learning through teacher token distributions.Source: §3.2 · p. 5
3.3 TRUST-REGION REGULARIZATION
- At each token, the trust region selects the largest interpolation toward intent-conditioned teacher guidance that remains within a divergence budget from the unconditioned reference.The constraint bounds how much training-only intent can alter the teacher distribution.Source: §3.3 · pp. 5–6
3.4 TRAINING OBJECTIVE
- The training objective combines distillation on student responses with a weighted grounding loss on recorded responses.The generalized Jensen–Shannon divergence matches the regularized target at each generated token; the grounding term uses a teacher-forced loss on the recorded response.Source: §3.4 · p. 6
4 EXPERIMENTS
Experiments compare text- and GUI-based simulators on single-session and cross-session USERARENA trajectories, then validate simulated policy effects against production A/B outcomes. OPTD leads broadly on fidelity and most closely reproduces the production effect; component ablations and human evaluation probe its behavior.
4.1 EXPERIMENTAL SETUP
- The comparison covers text-based and GUI-based simulators using Qwen3.5-2B, and evaluates both single-session and cross-session trajectories.Cross-session evaluation follows chronologically ordered sessions from the same user.Source: §4.1 · p. 6, §4.2 · p. 7
4.2 MAIN RESULTS
- OPTD ranks first on all eight single-session L1–L4 metrics and first on five, second on two, in cross-session evaluation.Its cross-session results include the strongest sequential and personalized population fidelity.Source: §4.2 · p. 7, Table 2 · p. 7
4.3 PRODUCTION A/B VALIDATION
- OPTD most closely reproduces the online policy effect, reaching an effect ratio of 0.713× in offline validation.The validation compares the same ranking-policy pair in simulation and a controlled production A/B test.Source: §4.3 · p. 8, Table 3 · p. 8
4.4 IN-DEPTH ANALYSES
- Removing any OPTD component reduces at least one selected metric, while removing SFT produces the largest degradation across the selected metrics.The ablations separately remove on-policy responses, intent, trust-region interpolation, grounding, or SFT initialization.Source: §4.4 · p. 8
- In source-blinded human evaluation, OPTD has the lowest Human win rate and highest Model win rate among the four simulators.Three annotators judged anonymized, order-randomized real–simulated trajectory pairs for each user context.Source: §4.4 · p. 8
5 RELATED WORK
Related work spans richer behavioral datasets, interactive recommendation benchmarks, and increasingly expressive user simulators. UserArena addresses gaps in long-horizon, step-aligned real-user evidence and production-effect validation, while OPTD targets sparse supervision and exposure bias.
Interactive recommendation benchmarks
- Recommendation resources progress from ratings and reviews to multimodal sessions, sequential interactions, and interactive agent-oriented evaluation.Examples include MovieLens, Amazon Reviews, Qilin, RL4RS, RecBench+, AgentRecBench, OPeRA, and OmniBehavior.Source: §5 · pp. 8–9
- Existing resources rarely record what real users see and do throughout long-horizon sessions or test whether simulated policy comparisons reproduce production A/B effects.UserArena provides step-aligned real-world shopping trajectories and offline validation against a controlled production A/B test.Source: §5 · p. 9
User simulator development
- User simulators evolved from configurable dynamics to log-driven models and LLM-based systems, but sparse rewards and exposure bias remain optimization challenges.OPTD addresses both with dense distributional supervision on student-sampled responses and trajectory-level intent.Source: §5 · p. 9
6 CONCLUSION
The paper introduces USERARENA and a five-level simulator-evaluation protocol, and proposes OPTD for long-horizon simulation. OPTD achieves the strongest overall fidelity under the protocol and most closely reproduces the production policy effect in the controlled comparison.
6 CONCLUSION
- USERARENA pairs a production-grounded shopping environment with fine-grained, long-horizon real-user trajectories and evaluates environment compliance through outcomes.The protocol also covers local decisions, sequential behavior, and personalized population fidelity.Source: §6 · p. 9
- OPTD uses trajectory-level intent to guide dense teacher supervision on student-generated rollouts and achieves the strongest overall fidelity under USERARENA's protocol.It also most closely reproduces the production policy effect in the controlled comparison.Source: §6 · p. 9
- Future work will develop a continuously updated USERARENA leaderboard and investigate simulators that preserve evolving preferences and behavioral consistency across sessions.Source: §6 · p. 9
AI USE STATEMENT
The authors used generative AI for manuscript editing and Chinese interface and item-title translation, with manual review against the original materials.
AI USE STATEMENT
- Generative AI assisted with clarity edits and Chinese text translation or shortening; authors manually checked the outputs and report no changes to underlying data or results.Source: AI USE STATEMENT · p. 10
REPRODUCIBILITY STATEMENT
The paper identifies where reproducibility information is documented across the main text and appendices.
REPRODUCIBILITY STATEMENT
- Benchmark construction and evaluation protocol are described in Section 2, and the complete OPTD formulation in Section 3.The experimental setup is in Section 4.Source: REPRODUCIBILITY STATEMENT · p. 10
- Appendix A provides dataset statistics and metric definitions; Appendix B documents data processing, baselines, implementation details, and prompts.Additional analyses appear in Appendix C.Source: REPRODUCIBILITY STATEMENT · p. 10
ETHICS AND DATA STATEMENT
The data are de-identified, research-authorized, and released under restrictions; the interaction-horizon comparison focuses on real-user trajectories.
ETHICS AND DATA STATEMENT
- Sessions were cleaned and de-identified, with direct identifiers removed and production identifiers replaced by non-linkable sequential identifiers.Use was authorized under the platform’s privacy policy and internal data-governance procedures.Source: ETHICS AND DATA STATEMENT · p. 10
- Released interface images are offline reconstructions, not screenshots captured from real users; the dataset is restricted to non-commercial research under a data use agreement.The agreement prohibits reidentification, external personal-data linkage, and unauthorized redistribution.Source: ETHICS AND DATA STATEMENT · p. 10
A.1 COMPARISON OF INTERACTION HORIZONS
- Table 4 compares interaction horizons for real-user shopping benchmarks with a full action space, reporting USERARENA’s cross-session subset.The supplied passage gives the comparison’s scope but not its cell values.Source: Table 4 · p. 14
A.2 DATASET AND BEHAVIOR STATISTICS
USERARENA captures skewed action frequencies alongside repeated-session histories, diverse items, and multi-step behavioral paths. Its metric appendix defines compliance and fidelity measures across local, sequential, personalized, and outcome behavior.
A.2 DATASET AND BEHAVIOR STATISTICS
- 38.34% of actions are feed-page downward scrolls, while add-to-cart and purchase account for 3.90% and 1.02%, motivating macro-averaged and distributional evaluation.The 73,717 item clicks span 41 categories; apparel represents 54.90%.Source: §A.2 · p. 14
- Cross-session users average 10.29 sessions and 86.36 actions, while 92.0% of clicked items appear once and transitions include repeated feed navigation and item exploration.The dataset also captures successive-session gaps from minutes to over 12 hours, with 57.8% revisiting a category from the preceding session.Source: §A.2 · pp. 14–15, Figure 5 · p. 15
A.3 EVALUATION METRIC DETAILS
- The metric appendix defines logged human sessions as R and simulator-generated sessions as S, distinguishing Replay predictions from complete-session Rollouts.The five-level protocol and its metrics are introduced in Section 2.Source: §A.3 · p. 15
A.3.1 L0: ENVIRONMENT COMPLIANCE (REPLAY AND ROLLOUT)
- RVA and OVA measure the share of Replay and Rollout outputs, respectively, that parse into executable actions, with invalid outputs and fallbacks counted as failures.All outputs remain in the denominator.Source: §A.3.1 · pp. 15–16
A.3.2 L1: LOCAL BEHAVIORAL FIDELITY (REPLAY)
- L1 uses equally weighted action-type Macro-F1 and exact-match accuracy on retained action types, excluding four scroll actions from accuracy.Invalid predictions count as misses; scroll actions are excluded because their exact-match test duplicates the action-type signal.Source: §A.3.2 · p. 16
A.3.3 L2: SEQUENTIAL BEHAVIORAL FIDELITY (ROLLOUT)
- L2 compares item-page next-decision distributions with base-2 Jensen–Shannon divergence and session lengths with trajectory error.The decision set includes return, cart, buy, end, and open.Source: §A.3.3 · p. 16
A.3.4 L3: PERSONALIZED POPULATION FIDELITY (ROLLOUT)
- L3 population fidelity compares cohort behavioral signatures across real and simulated sessions, using equal weighting across eligible cohorts and schemes.Cohorts are based on prior activity, purchase propensity, and category breadth; signatures include session length, actions, information depth, item-page behavior, conversion, category, and price.Source: §A.3.4 · p. 17
A.3.5 L4: OUTCOME FIDELITY (ROLLOUT)
- L4 measures count-based click-through and add-to-cart rate errors, plus divergence between terminal session-outcome distributions.Terminal outcomes are purchase, cart only, browse only, or truncated.Source: §A.3.5 · p. 17
B.2 DATA PROCESSING AND IMPLEMENTATION
The experiments test production-effect calibration, inference cost, and qualitative trajectory examples alongside the data-processing and training setup. OPTD most closely matches the production policy effect, while its fidelity gains do not require systematically higher per-step latency.
B.2 DATA PROCESSING AND IMPLEMENTATION
- Users are split 8:1:1 with all sessions from each user kept together, and test trajectories are retained once without resampling.Replay reconstructs recorded states independently; Rollout advances only through environment-accepted actions.Source: §B.2 · p. 18
B.3 TRAINING HYPERPARAMETERS
- OPTD uses a frozen teacher initialized from an SFT-warmed checkpoint, with the student initialized from that same checkpoint.A larger model generates short first-person rationales for recorded actions, and the training setup uses distributed generation and optimization.Source: §B.2 · pp. 18–19, §B.3 · p. 19
B.4 PROMPT STRUCTURE
- Trajectory-level intent is appended only to the frozen teacher during training; the student receives current-decision information without that intent at inference.The student prompt includes user context, executed history, current interface state, and executable actions.Source: §B.4 · p. 19, Table 5 · p. 19
C.1 PRODUCTION A/B VALIDATION DETAILS
- Offline validation compares the same policy pair in simulation and controlled production buckets, using 10,000 users split evenly between policy arms.PV-CTR counts click actions per exposure page view, including repeated exposures to an item.Source: §C.1 · p. 20
- OPTD achieves a production policy-effect ratio of 0.713× and error of 0.287, with 50.2% lower error than the next-closest simulator.Other task-specific simulators have ratios of 0.020×–0.297×, while GPT-6-Sol and GPT-6-Luna overestimate the response; OPTD still underestimates it by 28.7%.Source: §C.1 · p. 20
C.2 EFFICIENCY ANALYSIS
- OPTD generally uses fewer total tokens than STA, while their latency curves remain closely aligned and OPTD requires no systematic increase in per-step latency.The efficiency analysis aggregates model calls, including internal calls and invalid-action retries, within trajectory steps.Source: §C.2 · pp. 20–21, Figure 6 · p. 21
C.3.1 REPRESENTATIVE USER TRAJECTORIES
- A three-visit example shows persistent beauty-product interest alongside shifts in product focus and engagement across chronologically ordered sessions.The displayed trajectory preserves all 26 actions and links four clicks to their preceding feed observations.Source: §C.3.1 · pp. 21–22, Figure 7 · p. 22
C.3.2 TRAJECTORY-LEVEL INTENT EXAMPLE
- A four-visit example represents trajectory-level intent as six typed anchors that capture recurring preferences and tendencies without specifying a product or next action.The same intent conditions the teacher at each state, while the student receives neither future observations nor trajectory intent.Source: §C.3.2 · p. 23, Figure 8 · p. 23
C.4 TURING-STYLE HUMAN EVALUATION DETAILS
The study uses blinded pairwise human judgments to compare simulated and real shopping trajectories, reporting both source-attribution rates and annotator agreement.
C.4 TURING-STYLE HUMAN EVALUATION DETAILS
- Three annotators independently assessed 200 held-out trajectory pairs per simulator, with each pair combining a real and generated trajectory for the same user context.Sources were anonymized as A and B, their display order randomized, and annotators could select Unsure.Source: §C.4 · p. 24
- OPTD had the lowest Human win rate (38.7%) and highest Model win rate (26.3%) among four simulators; Fleiss’ κ was 0.528.The authors note that κ measures annotator consistency, not attribution correctness or human-likeness.Source: §C.4 · p. 24
- The interface showed shared user context, action timelines, and step-level interface observations, and required annotators to provide a rationale.Annotators chose which trajectory was machine-generated or selected Unsure.Source: Figure 9 · p. 24, §C.4 · p. 24