PaperlayerView exact revision on arXiv

Source-linked AI summary

RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce

Xinwei Yang, Kelong Mao, Yudong Guo, Sulong Xu, Simiu Gu, Chen Huang, Wenqiang Lei

arXiv:2609.38974v1cs.AI

TL;DR

Real-world shopping requires multi-turn decisions as users reveal or revise constraints, but existing benchmarks leave that process under-evaluated. REALWORLDSHOP evaluates grounded shopping sessions, and REALSHOP_AGENT adds explicit session control; it consistently outperforms strong baselines.

  • Problem

    Existing benchmarks under-evaluate full conversational shopping, where users reveal and revise constraints and coordinate goals across grounded, multi-turn decisions.

    Source: §1 · p. 1

  • Method

    The paper introduces REALWORLDSHOP for session-level evaluation and REALSHOP_AGENT, an executable framework with state management, shopping-flow control, catalog-grounded retrieval, and runtime guards.

    Source: §1 · p. 2

  • Results

    REALSHOP_AGENT consistently outperforms strong baselines on REALWORLDSHOP, improving success from 0.71 to 0.82 over the strongest baseline.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that conversational shopping agents require explicit mechanisms for state tracking, planning, and session-level control.

    Source: §7 · p. 9

    Because the benchmark uses simulated shoppers, its results are a controlled approximation of real-world shopping behavior, not a complete substitute for live-user studies.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Large language models are reshaping ecommerce from static recommenders into interactive shopping assistants, yet real-world shopping requires session-level decision support: users reveal and revise constraints, coordinate multiple goals, and expect product-grounded recommendations over a full conversation. Existing benchmarks are mostly outcome-oriented or execution-oriented, leaving this evolving decision process under-evaluated. We introduce REALWORLDSHOP, a benchmark built on 3.28M grounded products, structured shopping episodes, a profile-grounded and actioncontrolled user simulator, and role-play evaluation. Our analysis shows that current systems produce locally plausible responses but struggle with state tracking, constraint updating, and grounded convergence, especially under ambiguous intent, bundle, and multi-intent scenarios. We further propose REALSHOP_AGENT, an executable session-control framework with explicit state management, shopping-flow control, catalog-grounded retrieval, and runtime guards. Experiments show that REALSHOP_AGENT consistently outperforms strong baselines on REALWORLDSHOP.

1 Introduction

Real-world shopping requires agents to elicit and update constraints, coordinate goals, and ground decisions across full conversations, but existing benchmarks under-evaluate that process. REALWORLDSHOP exposes session-level weaknesses, while REALSHOP_AGENT adds explicit control and outperforms strong baselines.

1 Introduction

  • The paper frames shopping as an evolving decision process because users reveal or revise needs, compare alternatives, and shift purchase targets.Recommendations must also be grounded in catalog evidence such as category, price, attributes, compatibility, and fulfillment constraints.Source: §1 · p. 1
  • Existing benchmarks often simplify product spaces and intents or emphasize tool execution, leaving conversational decision processes under-evaluated.The missing capabilities include hidden-constraint elicitation, state updating, grounded recommendations, and coordination of bundles and multiple intents.Source: §1 · p. 1
  • Current agents perform better at turn-level responses than session-level control, with failures in state preservation, constraint revision, subgoal coordination, and grounded convergence.These failures are especially pronounced in ambiguous-intent, bundle, multi-intent, and intent-shift scenarios.Source: §1 · p. 2
  • REALSHOP_AGENT improves success from 0.71 to 0.82 over the strongest baseline, with explicit session control and catalog-grounded decisions.Its modules manage evolving state, control shopping flow, retrieve catalog evidence, and guard against unsupported or stale recommendations.Source: §1 · p. 2
  • REALWORLDSHOP evaluates multi-turn shopping with a 3.28M-SKU inventory, structured episodes, an action-controlled user simulator, and role-play evaluation.Episodes include evolving intents and constraints, while role-play assesses local responses, process quality, and final convergence.Source: §1 · pp. 1–2

2 Related Work

Prior benchmarks assess final-item relevance or shopping actions, while REALWORLDSHOP evaluates full conversational trajectories under partial observability and catalog grounding. The paper presents executable session control as complementary to prior agent-building approaches.

Conversational Shopping Agent Benchmark

  • Outcome-oriented benchmarks assess preference elicitation and final-item relevance, whereas execution-oriented benchmarks test actions, tool calls, or task-stage completion.Neither line directly evaluates full-session conversational decisions as users reveal needs, revise constraints, reject candidates, or shift goals.Source: §2 · p. 2
  • REALWORLDSHOP evaluates complete multi-turn trajectories with catalog grounding and partial observability, covering response quality, process quality, outcomes, and early exits.Table 5 compares it with representative benchmarks.Source: §2 · p. 2

Improving Conversational Shopping Agents

  • Recent methods use profiling, strategic action selection, feedback modeling, tools, reinforcement learning, and recommendation representations; REALSHOP_AGENT adds executable session control.Its modules manage state, action timing, catalog grounding, and guards against stale or unsupported recommendations.Source: §2 · pp. 2–3

3 RealWorldShop Benchmark

REALWORLDSHOP assesses shopping agents as session-level decision makers through grounded products, evolving episodes and profiles, and role-play evaluation of complete interactions.

3 RealWorldShop Benchmark

  • The benchmark combines real-product inventory, structured shopping episodes, a profile-grounded simulator, and role-play measures of turn quality, process, and final convergence.The evaluation moves beyond single-turn recommendation to test session-level decision making.Source: §3 · p. 3

3.1 Product Inventory

  • The 3.28M-product inventory retains titles, prices, attributes, categories, descriptions, and fulfillment signals for catalog-faithful recommendation checks.These fields support checking preference match and fidelity to product evidence and constraints.Source: §3.1 · p. 3

3.2 Shopping Episode Construction

  • Shopping episodes combine a user profile, scenario, and session plan, with archetypes and reveal schedules testing clarification, state tracking, updates, and coordination.Scenarios include bundles, upgrades, repairs, restocking, mixed errands, and urgency-driven purchases; plans vary intent ambiguity, subgoal count, and when changes are revealed.Source: §3.2 · pp. 3–4

3.3 User Profile Construction

  • Profiles separate an agent-visible prior from latent user state and encode hard, soft, or negotiable preferences and constraints.They cover explicit and ambiguous intents and may be conditioned on catalog evidence; unsupported constraints are downgraded to soft or negotiable preferences.Source: §3.3 · p. 4

3.4 Evaluation Protocol

  • Role-play limits agents to dialogue history and visible profile information, and sessions end in success, early exit without convergence, or max-turn failure.Judging separately scores turn-level response quality, session process, and holistic purchase outcomes; validation reports QWK/κ of 0.76–0.89 and exact agreement of 0.88–0.97 across dimensions.Source: §3.4 · p. 4

4 Benchmark Analysis

REALWORLDSHOP evaluates shopping agents across held-out sessions, interaction phenomena, scenarios, and user styles. Results show that strong turn-level responses often fail to translate into coherent, grounded session decisions, especially when state must be revised or scenarios become demanding.

4.2 Benchmark Findings

  • Session-level scores trail turn-level scores across systems: average general LLMs drop from 1.08 to 0.80, and GPT-5 from 1.45 to 1.20.Turn Avg. measures local response quality; Sess. Avg. measures trajectory-level shopping competence.Source: Table 2 · p. 5, §4.2 · pp. 5–6

4.1 Benchmark Setup

  • The held-out diagnostic set samples six scenario types, six interaction phenomena, and six user profiles, with each system evaluated over five runs.Early-exit and non-exit sessions are grouped by realized outcomes rather than predefined sampling labels.Source: §4.1 · p. 5

4.2 Benchmark Findings

  • Early-exit sessions show a sharper state-tracking decline than clarification decline: 0.98 to 0.23 versus 1.43 to 1.25.The contrast suggests abandonment is more closely associated with failures to remember, update, or act on revealed information than with insufficient questioning.Source: Figure 2 · p. 5, §4.2 · p. 6
  • Bundle re-planning is especially difficult, with success falling to 48% and constraint-updating and decision-pacing scores dropping to 0.64 and 0.50.Revision-heavy interactions require propagating new information through session state, candidate sets, and downstream plans.Source: §4.2 · p. 6, Figure 3 · p. 6
  • Explicit single-item sessions achieve 72% success and a 0.11 local-to-session gap, while ambiguous single-item and long-horizon sessions show a gap of 0.33.Ambiguity, bundles, multi-intent coordination, and long-range dependencies expose weaknesses that turn-level quality alone does not capture.Source: §4.2 · p. 6, Figure 4 · p. 6
  • Low-patience, high-skepticism, and efficiency-first users reduce session-level scores to 0.63–0.70 and raise early-exit rates to 27%–32%, versus 12%–19% under more tolerant styles.Source: §4.2 · p. 6, Figure 5 · p. 6

5 REALSHOP_AGENT

REALSHOP_AGENT treats shopping conversations as state-driven decision processes rather than open-ended response generation. Its control loop maintains evolving state, selects shopping actions, grounds decisions in catalog evidence, and applies runtime checks.

5.1 M1: Session State Manager

  • The Session State Manager tracks revealed intents, active subgoals, constraints, rejected options, confirmed preferences, and stale dependencies.New constraints override long-term preferences, and dependent subgoals are marked stale when key budgets, candidates, compatibility requirements, or bundle anchors change.Source: §5.1 · p. 7

5.2 M2: Shopping-Flow Controller

  • The Shopping-Flow Controller selects among six actions—CLARIFY, RETRIEVE, VERIFY, COMPARE, RECOMMEND, and CONFIRM—based on session state and information needs.The policy times progression from elicitation through search and verification to recommendation and commitment, aiming to reduce premature recommendations, redundant questions, unsupported comparisons, and delayed convergence.Source: §5.2 · p. 7

5.3 M3: Catalog-Grounded Retriever

  • The Catalog-Grounded Retriever returns structured product evidence, including SKU identifiers, categories, attributes, prices, fulfillment signals, relevance scores, and retrieval metadata.Claims must trace to current records, while evidence tied to stale intents, rejected options, or superseded constraints is invalidated.Source: §5.3 · p. 7

5.4 M4: Runtime Execution Guards

  • Runtime guards limit redundant retrieval, rewrite unsupported SKU-level claims into uncertainty-aware statements, and block recommendations based on stale subgoals or constraints.Together, they form the final consistency layer before response generation.Source: §5.4 · pp. 7–8

5.5 Backbone Adaptation

  • The Qwen3.5-27B backbone is initialized from an SFT checkpoint and further optimized with reinforcement learning over tool-using shopping rollouts.The adaptation targets shopping-specific action selection and catalog-grounded reasoning under evolving constraints.Source: §5.5 · p. 8

6 Experiment

REALSHOP_AGENT is evaluated on REALWORLDSHOP using turn-level, session-level, and final-outcome metrics. The full system outperforms its prompt-only and adapted backbones, while diagnosis and ablation results indicate gains across session control and product grounding.

6.1 Experimental Setup

  • Experiments use REALWORLDSHOP’s role-play protocol and three-stage judging pipeline, reporting turn-level, session-level, and final-outcome metric groups.Table 3 reports the main results and ablations.Source: §6.1 · p. 8, §6.2 · p. 8

6.2 Overall Performance

  • The full REALSHOP_AGENT reaches a Sess. Avg. of 1.39, success rate of 0.82, Gap of 0.07, and Early Exit rate of 0.09.The prompt-only Qwen3.5-27B Base scores 0.70 Sess. Avg. and 0.55 success; SFT/RL adaptation raises these to 1.00 and 0.65.Source: §6.2 · p. 8

6.3 Diagnosis-Oriented Evaluation

  • The overall Early Exit rate decreases from 0.25 to 0.09, while State Tracking rises from 0.51 to 1.33 and Constraint Updating from 0.47 to 1.24.On the metadata-defined early-exit-risk subset, Sess. Avg. increases from 0.60 to 1.21.Source: §6.3 · p. 8
  • Under constraint updates and recommendation rejections, success rises from 0.55 to 0.83 and from 0.50 to 0.78, respectively.The full agent consistently outperforms both prompt-only and adapted backbones across revision-heavy interaction phenomena.Source: §6.3 · p. 8
  • In ambiguous-bundle sessions, success rises from 0.50 to 0.74 and Gap falls from 0.34 to 0.11; in multi-intent shopping, success rises from 0.49 to 0.76 and Gap falls from 0.35 to 0.07.The agent also reduces early exits for efficiency-first, high-skepticism, and low-patience users.Source: §6.3 · p. 8

6.4 Ablation Study

  • Removing the Session State Manager increases Gap from 0.07 to 0.24 and reduces State Tracking from 1.45 to 1.02 and Constraint Updating from 1.34 to 0.87; removing flow control drops Sess. Avg. from 1.39 to 1.17 and success from 0.82 to 0.66.These ablations implicate explicit state preservation and action timing in the full system’s performance.Source: §6.4 · p. 9
  • Removing the Catalog-Grounded Retriever reduces Rec. Acc. from 1.31 to 1.09 and Personalization from 1.22 to 0.85; removing Runtime Execution Guards lowers Sess. Avg. from 1.39 to 1.23 and raises Early Exit from 0.09 to 0.15.The reported ablations indicate that grounding and runtime checks contribute alongside state management and flow control.Source: §6.4 · p. 9

7 Conclusion

REALWORLDSHOP provides a large-scale benchmark for realistic conversational shopping, and REALSHOP_AGENT improves shopping outcomes while narrowing the gap between turn-level and session-level performance.

7 Conclusion

  • REALWORLDSHOP combines a 3.28M-SKU inventory, about 1,200 structured shopping scenarios, over 2,000 synthesized profiles, and session-level role-play evaluation.The benchmark evaluates shopping agents under realistic shopping conditions.Source: §7 · p. 9
  • REALSHOP_AGENT improves task success, reduces early exits, and narrows the gap between local response quality and session-level effectiveness.The conclusion frames explicit state tracking, planning, and session-level control as requirements for future conversational shopping agents.Source: §7 · p. 9

Limitations

The authors qualify their findings by noting limits in LLM-based judging, simulated-user realism, and benchmark scenario coverage.

Limitations

  • LLM-based judges may respond to surface fluency, response style, or prompt wording, affecting absolute scores and sometimes relative model-family comparisons.The study includes stratified human validation and repeated judging passes, but calls for caution and further human evaluation and cross-judge validation.Source: Limitations · pp. 9–10
  • Simulated users support controlled evaluation of hidden constraints, preference updates, and intent shifts but may not capture real shoppers’ behavioral diversity.The authors characterize benchmark results as a controlled approximation, not a complete substitute for live user studies.Source: Limitations · p. 10
  • REALWORLDSHOP does not fully cover after-sales service, returns and refunds, promotion-driven shopping, cross-platform price comparison, customer lifecycles, or live commerce.The authors suggest extending the benchmark to these scenarios for more comprehensive evaluation.Source: Limitations · p. 10

Ethical Considerations

The authors describe privacy protections in benchmark construction and frame the benchmark as a research-evaluation resource with safeguards against unsupported recommendations.

Ethical Considerations

  • Benchmark construction followed an IRB-approved protocol, applicable access permissions, and data-use requirements, retaining only product information needed for construction.Personally identifying or sensitive information associated with users or sellers is removed, anonymized, or desensitized; no personally identifiable user information is used.Source: Ethical Considerations · p. 10
  • Simulated profiles are constructed for research evaluation and do not represent real individuals; the benchmark evaluates agent behavior rather than inferring or reproducing identities.The simulator is validated for profile faithfulness and action controllability.Source: Ethical Considerations · p. 10
  • REALWORLDSHOP is intended for research evaluation, while REALSHOP_AGENT guards against unsupported SKU-level claims and stale-state recommendations.The authors say deployed systems should also incorporate platform-specific safety policies, consumer-protection requirements, and human oversight where appropriate.Source: Ethical Considerations · p. 10

A Additional Results

Additional analyses locate session-level weaknesses across early exits, revision-heavy dialogue, complex scenarios, and demanding user styles; the full agent improves across these diagnostic slices.

A Additional Results

  • Early exits are associated with broad process degradation, with the largest drops in state tracking, constraint updating, and decision pacing.The analysis suggests abandonment is not mainly caused by lack of clarification alone; failures to remember revealed information, update constraints, or make timely progress also matter.Source: A Additional Results · p. 13
  • Constraint updates, preference overrides, recommendation rejections, and bundle re-planning are more challenging than simpler refinement or comparison cases.These revision-heavy phenomena require feedback to propagate through session state, candidate sets, and downstream plans.Source: A Additional Results · p. 13
  • Ambiguous, bundle, multi-intent, and long-horizon scenarios add hidden constraints, subgoal dependencies, and longer-range memory demands, making local response quality less predictive of successful trajectories.Explicit single-item sessions are easier because the target is directly searchable and state revision is limited.Source: A Additional Results · p. 13
  • Efficiency-first, high-skepticism, and low-patience users expose whether agents identify blocking constraints, avoid redundant questions, verify candidates, and converge without repeated mistakes.These styles show higher early-exit rates and larger turn-to-session gaps, motivating evaluation under convergence pressure rather than uniformly cooperative users.Source: A Additional Results · p. 13
  • Compared with prompt-only and adapted backbones, REALSHOP_AGENT consistently improves results on early-exit-prone episodes, revision-heavy phenomena, complex scenarios, and demanding user profiles.The authors report that backbone adaptation helps but is insufficient without explicit state management, action timing, catalog-grounded verification, and runtime guards.Source: A Additional Results · p. 13

B Comparison with Existing Benchmarks and Methods

REALWORLDSHOP evaluates full, grounded shopping trajectories and provides a profile-grounded, action-controlled simulator. Its episode and profile construction makes interaction dynamics and evolving constraints available for controlled analysis.

B Comparison with Existing Benchmarks and Methods

  • 0.10 Full Agent Early Exit Rate on low-patience sessions, versus 0.34 for Base.Figure 5 reports these values for the low-patience interaction style.Source: B Comparison with Existing Benchmarks… · p. 14
  • REALWORLDSHOP evaluates clarification, state tracking, constraint updating, pacing, multi-intent coordination, and grounded convergence over full multi-turn trajectories.Prior benchmarks mainly assess final-item outcomes or shopping actions, rather than the conversational decision process behind them.Source: B Comparison with Existing Benchmarks… · pp. 13, 15

C Additional Details of the REALWORLDSHOP Benchmark

  • The appendix details inventory construction, episode generation, profile synthesis, simulator design, and role-play evaluation to make benchmark construction more explicit and reproducible.The stated aim is to clarify and support reproducibility of the construction procedure.Source: C Additional Details of the… · p. 16

C.1 Product Inventory

  • The fixed snapshot contains 3.28M deduplicated products with catalog and fulfillment fields, but does not model changing prices, stock, promotions, or availability.Evaluation uses the same product pool and metadata for all systems.Source: §C.1 · p. 16

C.2 Episode Construction

  • Episodes pair a profile, scenario, and session plan; category-anchored scenarios and plans specify task archetypes, constraints, subgoals, and expected interaction dynamics.Diagnostic labels are assigned during construction, not inferred from model outcomes.Source: §C.2 · p. 17

C.3 User Profile Construction

  • Structured profiles distinguish visible priors from latent intents and constraints, with catalog grounding and interaction-style factors supporting controlled, partially observable sessions.Ambiguous-intent profiles reveal goals gradually; interaction factors support stress tests of clarification, pacing, comparison, and constraint updating.Source: §C.3 · pp. 17–18

C.4 Simulator Design and Validation

  • The simulator selects a planned user action before verbalizing it, and validation checks profile faithfulness and whether utterances realize intended actions.This action-first design aims to make simulated interaction dynamics controllable and interpretable.Source: §C.4 · pp. 18–19

C.5 Evaluation Protocol

The protocol evaluates shopping conversations at turn, session, and outcome levels, with role-play exposing trajectory-level behavior. Additional checks assess judge reliability and simulator consistency.

C.5 Evaluation Protocol

  • Role-play evaluation separates local response quality, session process, and holistic outcome, scoring turn-level understanding and recommendations alongside session-level clarification, tracking, updating, pacing, and personalization.The holistic judge labels success, partial success, or failure and may diagnose convergence quality.Source: §C.5 · pp. 19–20

C.6 Automatic Judge Reliability

  • Judge reliability is assessed through human agreement on 150 sessions and repeated-judging stability measures, including ICC(2,1), score variation, and label flips.The supplied passages describe the validation measures but do not report their results.Source: §C.6 · p. 21, Table 7 · p. 21, Table 8 · p. 21

D Simulator Validation

  • 0.93 average profile-level faithfulness and 0.95 average action-level score were reported for Kimi-K2.6 on sampled simulator validation profiles.Action validation also reports 0.94 intended-action matching, 0.97 state-valid transitions, and 0.92 confirm/end control.Source: D Simulator Validation · p. 22

E Implementation Details of REALSHOP_AGENT

REALSHOP_AGENT combines explicit session state, controlled shopping actions, catalog-grounded evidence, and runtime guards. Its full execution harness outperforms the prompt-only and adapted backbones in the stated comparison.

E Implementation Details of REALSHOP_AGENT

  • The Session State Manager tracks intents, constraints, preferences, rejections, and stale dependencies, while the flow controller selects actions and the retriever supplies state-matched catalog evidence.Runtime guards limit redundant retrieval and prevent unsupported product claims or recommendations based on stale constraints.Source: E Implementation Details of… · pp. 22–23
  • The full REALSHOP_AGENT outperforms both the prompt-only backbone and the adapted backbone without the complete control loop.The paper describes adaptation and executable control as complementary: one improves learned behavior, while the harness structures the interaction process.Source: E Implementation Details of… · p. 24

F.1 Training Setup

  • The Qwen3.5-27B shopping policy is adapted with supervised fine-tuning followed by online reinforcement learning using trajectories from the executable runtime.The held-out REALWORLDSHOP evaluation episodes are disjoint from the training prompts.Source: §F.1 · pp. 24–25

F.2 Training Tasks and Rewards

  • Task-specific scalar rewards encourage completion, relevance, constraint satisfaction, grounding, and correct tool use while penalizing unsupported claims, invalid tool paths, redundant retrieval, and unsafe behavior.The active reward terms depend on the task type.Source: §F.2 · p. 25

F.3 GRPO Optimization

  • The policy uses clipped GRPO optimization, excluding rollout groups dominated by infrastructure failures or with zero reward variance.The method samples trajectory groups and computes group-relative advantages.Source: §F.3 · p. 25

G Prompt Design

  • The appendix includes prompts for profile synthesis, controlled user simulation, shopping-agent decisions, and trajectory judging.The figures identify the role of each prompt in REALWORLDSHOP.Source: Figure 6 · p. 26, Figure 7 · p. 27, Figure 8 · p. 28

H Case Study

  • Representative case studies cover compatibility-sensitive shopping, ambiguous intent, multi-intent coordination, low patience, and split-order decisions.The cases are used to illustrate session-level decision-control stresses.Source: H Case Study · p. 30

H.1 Representative Profiles

  • Profiles separate hard, soft, and negotiable requirements and include explicit or ambiguous intents, interaction styles, and catalog-grounded constraints.Profiles are normalized and validated for schema consistency, multi-intent structure, category anchoring, and catalog grounding.Source: §H.1 · p. 30

Related papers

Loading 2609.38974v1…