PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Open-Weight E-Commerce Agents with Environment-Grounded Verification

Nimit Shah, Haitz Sáez de Ocáriz Borde

arXiv:2609.16093v1cs.LG

TL;DR

Final-cart success cannot show how an e-commerce agent reached, missed, or exceeded a customer's intended cart. The paper precommits scenario evidence and retains interaction state to verify actions and claims. Across eight agents, capability profiles distinguish retrieval, cart-management, and grounded-description differences.

  • Problem

    A final cart does not reveal the route taken, the catalog facts available, intermediate cart states, or whether a customer had requested an item when it was searched.

    Source: §1 · p. 1

  • Method

    The environment precommits a target cart and reveal schedule against a fixed catalog, retains tool traces and state, and uses deterministic verification plus judges for language-dependent questions.

    Source: §1 · p. 1, §7 · p. 9

  • Results

    Across eight open-weight agents, capability profiles exposed differences in retrieval, cart management, and grounded descriptions; Qwen3.5-35B had recall 0.85 and goal-achievement rate 0.66 but cart precision 0.89.

    Source: §1 · p. 2, §5.2 · p. 7

  • Takeaways & Limitations

    The profiles identify distinct repair targets, and the paper argues that evaluations should report these distinctions when deployment and repair decisions depend on how an agent fails.

    Source: §7 · p. 9

    The fixed target cart and reveal schedule cannot represent unsettled or changing preferences or multiple acceptable products, and the simulator is not a human proxy.

    Source: §6 · p. 9

Abstract

from arXiv · show

A shopping conversation has many routes to the same cart, and a task-success rate reduces all of them to one score. We build a deterministic and reproducible e-commerce environment that precommits each trial's customer and trajectory parameters, including the persona, difficulty, target cart, and an item reveal schedule. A simulated consumer attempts to buy a target cart from the environment with assistance from the evaluated model. The environment guides the simulator's actions and records every assistant action alongside the environment state at that point. After the trial, these records allow the evaluator to assess individual parts of the conversation against the retained evidence. For example, the evaluator penalizes a search for failing to surface a target product only when the customer has already mentioned that product. We further use this evidence to apply different penalties to tool calls depending on how the assistant's actions compare with an expected tool-call set. Our environment also interacts with the simulator bidirectionally, reading its output to stop the trial when the simulator determines that the customer has become too frustrated and injecting directives in real time that specify when to explore, defer buying an item, or recall a previous exchange. This interaction creates an open-ended and verifiable simulation. Across eight open-weight agents from 20B to 35B parameters, with 160 trials per agent and 44 metrics, the resulting capability profiles distinguish under-action, over-purchase, unsupported product attributes, and poor search, all of which terminal success obscures.

1 Introduction

Final-cart success cannot reveal whether an agent acted appropriately or succeeded accidentally because it omits what the agent knew and how the cart changed. The paper proposes a precommitted, evidence-retaining environment and a verification framework that separates these behaviors.

1 Introduction

  • The environment precommits a hidden target cart and item reveal spans, then retains catalog state, tool traces, and reveal evidence throughout open-ended trials.The target cart is generated from a 10,000-product control catalog, whose category mix is fitted to aggregate retail data.Source: §1 · pp. 1–2
  • The verifier separates failures obscured by final-cart scores, including missed requested items, unwanted additions, unsupported product claims, and turns that do not advance the purchase.Deterministic reconstruction handles state-grounded questions, while model-based judging is reserved for language-dependent questions.Source: §1 · pp. 1–2
  • Across eight open-weight agents from 20B to 35B parameters, the resulting profiles expose differences in retrieval, cart management, and grounded descriptions.The study also examines shopping pace across difficulty tiers using termination reasons and tool-call density.Source: §1 · p. 2
  • The evaluation contributes 44 metrics for shopping outcomes, tool use, and customer-support quality, plus cohort-specific scoring that avoids averaging incompatible tasks.A standard registry supports new metrics, and each trial kind has its own rubrics and applicability map.Source: §1 · p. 2
  • A verification ablation identifies which questions can be settled from a final cart, transcript, environment state, or the full simulator record.The paper also reads agents as capability profiles and uses termination reasons and tool-call density to diagnose difficulty-tier outcomes.Source: §1 · p. 2

2 Related work

Prior shopping benchmarks score task outcomes or process information, but differ in when requirements are supplied and what action-level evidence is standardized. This paper positions precommitted request boundaries as a basis for grading intermediate tool calls.

2 Related work

  • WebShop [Yao et al., 2022] starts with a complete shopping request and scores product satisfaction, but does not publish standardized correctness labels for individual actions.Its authors separately analyze search activity, products visited, and example trajectories.Source: §2 · p. 2
  • ShoppingBench, WebMall, WebArena, and Mind2Web [Wang et al., 2026, Peeters et al., 2026, Zhou et al., 2024, Deng et al., 2023] use task specifications supplied up front, while conversational benchmarks reveal preferences incrementally.ShopperBench uses persona profiles and ChatShop reveals attributes of one target product; neither precommits per-item request boundaries for state-grounded checks.Source: §2 · p. 2
  • Process-oriented work evaluates progress or logs, while stateful benchmarks differ in their temporal evidence: ToolSandbox [Lu et al., 2025] snapshots selected events, and τ-bench [Yao et al., 2025] scores final database state.COMPASS [Qin et al., 2025] records when travel constraints become fully specified, but does not assign each constraint a reveal turn before dialogue; this paper adds precommitted request-relevant boundaries for intermediate tool-call grading.Source: §2 · p. 2
  • LLM text evaluation has documented position and length biases and imperfect success classification [Zheng et al., 2023, Liu et al., 2023, Wang et al., 2024, Dubois et al., 2024, Lù et al., 2025].Claim decomposition offers a complementary factuality approach: FActScore checks atomic claims against a knowledge source, while FacTool combines extraction with task-specific tools [Min et al., 2023, Chern et al., 2023].Source: §2 · p. 2

3 The Evaluation Environment

The environment fixes the evidence needed for verification before each trial while leaving the surrounding customer interaction open-ended. It creates scenarios, reveals requests on a precommitted schedule, and retains trial records for downstream checks.

3 The Evaluation Environment

  • The environment precommits verification evidence before a trial while keeping the customer interaction open-ended.The scenario drives the interaction, and retained evidence supports fixed, model-based, and hybrid checks that form a capability profile.Source: §3 · p. 3, Figure 1 · p. 5

3.1 The storefront

  • The storefront uses a held-out control catalog of 10,000 products across 19 categories, with category shares approximated from six retailers; agents act through eight tools.Retrieval ranks normalized sentence-transformer embeddings by cosine similarity, and each trial uses a fresh cart.Source: §3.1 · p. 3

3.2 Scenario: what the customer wants, and who they are

  • Scenarios draw hidden, in-stock target products by category and price, while difficulty, persona, hints, and turn budgets shape basket breadth and customer communication.Goal items always come from the catalog, so a reported no-match is scored as retrieval failure rather than an empty shelf; difficulty tiers and grocery trials have different breadth ranges.Source: §3.2 · p. 3, Table 1 · p. 4

3.3 The reveal schedule

  • A precommitted reveal schedule stamps when each item is first requested, using batch, linear, or nonlinear release styles; nonlinear beats can interleave exploration, deferral, and return.Large batch trials may release bounded waves, and every nonlinear item has an essential add beat so each target remains achievable.Source: §3.3 · p. 4

3.4 Running a trial

  • A user simulator sends structured customer turns and expected tool-call chains, while the controller checks the ground-truth cart and manages termination.The simulator cannot see tool messages as the customer, and search metrics count a target only after the customer has requested it; Figure 1 traces the scenario-to-verifier evidence flow.Source: §3.4 · pp. 4–5, Figure 1 · p. 5

4 Verification signals

The verification framework uses state replay and catalog records for questions they can settle, reserving judges for language-dependent meaning and quality. It also evaluates request-relative tool calls with differentiated penalties and aggregates metrics by applicability.

4.2 The need for model-based judging

  • The verifier combines deterministic checks with judges for customer-facing meaning: it decomposes claims, checks actions and numeric claims against records, and sends attribute support and conversational rubrics to language judges.The user simulator separately rates frustration; the finished transcript judge evaluates claims such as whether a reply advanced the purchase or understood the request.Source: §4 · p. 5, §4.2 · p. 6

4.1 Deterministic signals

  • State replay reconstructs cart quantities and compares them with the target to measure exact-cart success, precision, recall, goal achievement, and progress.Goal progress is the maximum fraction of exactly satisfied target products after a cart mutation, so over-adding can reduce it; search metrics count only requested, revealed targets.Source: §4.1 · p. 5

4.2 The need for model-based judging

  • Table 3 shows two agents differ fourfold on the attribute channel while agreeing on the numeric channel.The verifier reports the three hallucination-channel rates separately.Source: §4.2 · p. 6

4.3 Tool-call correctness

  • Tool-call correctness compares expected calls with actual calls, using exact-match checks first and batched judging for differing same-name pairs and unmatched actual calls.Cart writes receive no free allowance, read-only lookups are forgiven in bulk, and additional extras increase the penalty.Source: §4.3 · p. 6

4.4 Aggregation

  • The harness reports trajectory and conversational metrics separately and skips metrics inapplicable to a trial kind rather than recording zero.Aggregates include the mean, sample standard deviation, and valid count; Boolean means are interpreted as rates.Source: §4.4 · p. 6

5 Verifiable simulation for e-commerce agents

Across eight agents, retained environment evidence supports checks that become more specific from terminal cart outcomes to request-relative actions and simulator-recorded frustration. These checks expose distinct capability and pace failures that exact-cart success alone does not distinguish.

5 Verifiable simulation for e-commerce agents

  • The study evaluates eight open-weight agents on the same 160-trial sweep, using a primary judge and fixed tools, scoring contract, controller, and stopping policy.The four retained-evidence conditions range from final cart and transcript to environment state and the full simulator record.Source: §5 · pp. 6–7, §5.1 · p. 7

5.1 Verification ablation

  • With the full simulator record, the verifier checks request-relative tool appropriateness and search success, beyond checks available from final cart or environment state.Environment state supports progress, executed-action, catalog-claim, and termination checks; final cart supports terminal outcomes.Source: §5.1 · p. 7

5.2 E-commerce state turns outcomes into verifiable diagnoses

  • Exact-cart success ranges from 0.30 to 0.57, and tier outcomes do not form a stable agent ranking.Gemma4-26B falls from 0.59 on direct purchases to 0.27 on light exploration, while GPT-OSS reaches 0.67 on multi-item baskets.Source: §5.2 · p. 7
  • Qwen3.5-35B has the highest recall (0.85) and goal achievement (0.66), but its final-cart precision is 0.89 and it makes 44.1 calls per conversation.The target cart and state replay show that it finds wanted items but also adds items absent from the target cart.Source: §5.2 · p. 7
  • GPT-OSS and Qwen3-30B fabricate attributes at 0.236 and 0.252 versus a cluster near 0.055, while numeric-claim rates remain 0.033 and 0.029.Qwen3-30B also has the highest phantom-action rate, 0.158; the diagnostic channels distinguish judge-checked attributes from deterministic action and numeric checks.Source: §5.2 · p. 8, Table 3 · p. 8
  • Qwen3-30B makes the fewest calls (15.4) and is also the lowest exact-cart agent, with 63% of trials exhausting the turn budget.The traces also identify GPT-OSS additions made before customer confirmation on 15.3% of additions; every search returned candidates, so bad-search rates measure relevance.Source: §5.2 · p. 8
  • Gemma4-26B exhausts the budget on 22 of 33 light-exploration trials but only 1 of 17 complex trials, while Qwen3.5-35B makes a median 66 calls on complex baskets.Four of 17 of Qwen3.5-35B’s complex-basket trials end in abandonment.Source: §5.2 · p. 8
  • In a 300-trial, two-agent cross-judge check, tool-call correctness has the closest agreement (mean difference 0.002, MAE 0.030, r = 0.924), while clarification has the lowest correlation (r = 0.490).GPT-5.4 scores hallucination 0.052 lower on average; the check does not calibrate either judge against human labels.Source: §5.2 · p. 9

6 Limitations

The evaluation depends on precommitted targets and a cooperative simulator, and its controlled storefront omits important live-retail conditions. Some scores also depend on a single judge and predicted expected tool calls.

6 Limitations

  • The eight-agent sweep uses one primary judge, neither judge is calibrated against human labels, and errors in expected-call prediction can change tool-call correctness scores.The cross-judge check covers only two agents and five rubrics.Source: §6 · p. 9
  • The method requires a target cart and reveal schedule fixed before the conversation, so it cannot represent unsettled preferences, changing preferences, or several acceptable products.The simulator knows the hidden target and may disclose an exact product name after a failed search, potentially rescuing it more cooperatively than a live customer.Source: §6 · p. 9
  • Results characterize controlled, tool-mediated catalog shopping, not live retail traffic, where prices and inventory change and merchant policies, delivery, returns, competing listings, and interface friction are absent.The turn budget is a benchmark convention, not an estimate of customer patience; another budget or tool-mediation policy may change measured profiles.Source: §6 · p. 9

7 Conclusion

The conclusion argues that final-cart success cannot reveal the route an agent took, and that retained environment evidence supports more actionable evaluation and repair. The accompanying construction details describe a retailer-averaged control catalog and seeded procedures for baskets, personas, and item reveals.

7 Conclusion

  • Environment-grounded verification preserves intermediate evidence to evaluate agent behavior, turning terminal outcomes into actionable diagnoses and supporting reproducible pre-deployment tests.The authors recommend reporting distinct failure profiles when deployment and repair decisions depend on how an agent fails; they say the controlled test complements validation on live retail traffic.Source: §7 · p. 9

A Per-retailer catalog composition

B Basket construction parameters

  • Basket construction samples distinct in-stock products category-first, with price-weighted representatives and category depth truncated at 12 products.Repeated category slots draw from products not already selected; selected products receive quantities under separate grocery and non-grocery weighting regimes.Source: B Basket construction parameters · pp. 12–13

C Persona dimensions

  • The simulated customer is sampled along six independent categorical dimensions using realistic-frequency weights, and those traits populate its persona prompt.The sampled dimensions make moderate specificity and conversational style common while garbled input is rare.Source: C Persona dimensions · p. 13, Table 5 · p. 13

D Reveal-style sampling and journey construction

  • Reveal journeys vary by realized basket size and trial kind, with batch waves, nonlinear deferrals and returns, and turn-budget compression preserving essential add beats.Grocery never draws nonlinear; micro-batches release the next wave only after all items in the current wave reach the cart.Source: D Reveal-style sampling and journey… · pp. 13–14, Table 6 · p. 13

D.1 Recorded reveal journeys

Recorded journeys illustrate how the reveal schedule supports nonlinear returns, bounded waves, and sequential choices. The tool-call rubric separately scales penalties by call category and treats unjustified cart mutations more seriously than extra reads.

D.1 Recorded reveal journeys

  • A Gemma4-31B interaction defers and revisits requests for three items before adding all three to the cart.The recorded exchange shows the controller maintaining unresolved items across natural deferrals and returns.Source: §D.1 · p. 14
  • A Qwen3.6-35B journey releases seven target items in waves of [2, 2, 2, 1], rather than as one opening list.The customer’s requests introduce new categories in bounded groups.Source: §D.1 · pp. 14–15
  • A second Qwen3.6-35B example introduces a flannel, polo, and pruning shears in sequence, each after the preceding choice.This contrasts with wave-based and nonlinear disclosure.Source: §D.1 · p. 15

E Tool categories and extra-call penalties

  • Unjustified extra-call costs grow linearly per extra call and quadratically in cumulative penalty; unmatched cart mutations incur the full penalty immediately.Read-only and exploratory calls receive limited allowances, and the penalty schedule is organized by registry category.Source: E Tool categories and extra-call… · p. 15, Table 7 · p. 15

F Verification case studies

Verification case studies show that exact-cart outcomes can conceal unsupported claims, unperformed actions, and substantial over-purchase. The surrounding evaluation reports verification costs, latency patterns, a 44-metric battery, and directional refusal-probe results.

F Verification case studies

  • An exact cart can still contain an unsupported product description: one trial had exact-cart success 1.0 but attribute-hallucination rate 1.0.The catalog described the licorice twists as chewy, not “slightly crunchy”; the checker identified this separate repair target.Source: F Verification case studies · pp. 15–16
  • A completion claim without a preceding cart-addition call was labeled phantom, despite later cart calls; the trial’s phantom-action rate was 1.0.The trial also ended at the turn budget with recall 0.25 and cart precision 0.50.Source: F Verification case studies · p. 16
  • A trial achieved every target requirement yet had 28 cart products for a 15-SKU target, recall 1.0, precision 0.344, and false exact-cart success.One product had 9 units in the cart when the target requested one.Source: F Verification case studies · p. 16

G Verification cost

  • Across the eight-model sweep, assistants averaged 417K tokens per trial versus 347K for the simulator and judge combined; metered simulator and judge calls averaged $94 per 160-trial run.That metered cost is $0.59 per trial; the assistant tokens come from the self-hosted inference path.Source: G Verification cost · p. 16

H Model identity and latency

  • All five sparse-mixture agents were faster per round trip than all three dense agents, while per-trial latency did not follow that ordering exactly.Per-trial latency also depends on the number of exchanges an agent uses.Source: H Model identity and latency · p. 17, Table 10 · p. 17

I The metric battery

  • The run reports 44 metrics grouped into deterministic outputs, inline simulator ratings, and post-hoc judge scores, with values averaged over valid scores.The harness stamps the scoring-contract version on each transcript and score artifact.Source: I The metric battery · p. 17, Table 11 · p. 18

J Refusal probes

  • In ten-probe refusal cohorts, agents searched on 0.80–1.00 of probes and hard-refusal rates ranged from 0.00 to 0.20.These results support only directional comparisons because each model has ten probes; unavailable shopping targets are scored under a policy-aware rubric, not as shopping failures.Source: J Refusal probes · p. 19, Table 12 · p. 20

K Cross-judge agreement

A two-agent cross-judge check found closer agreement for tool-call correctness and response helpfulness than for clarification and hallucination. The authors caution that the targeted review cannot establish a general judge ordering and that neither judge was calibrated against human labels.

K Cross-judge agreement

  • Tool-call correctness had a mean difference of 0.002, MAE 0.030, and trial-level correlation 0.924; helpfulness differed by at most 0.1 on 91.3% of paired trials.GPT-5.4 scored helpfulness 0.027 lower on average.Source: K Cross-judge agreement · p. 20, Table 13 · p. 20
  • Clarification had the largest standard-trial MAE, 0.111, and lowest correlation, 0.490, while hallucination scores differed directionally by 0.052 on average.GPT-5.4 scored hallucination lower on average; the authors say clarification and claim assessment require stronger calibration within this two-agent subset.Source: K Cross-judge agreement · p. 20, Table 13 · p. 20
  • The targeted review cannot establish a universal ordering of judge strictness or accuracy, and neither judge was calibrated against independent human labels.The manual review covered several high-disagreement trials.Source: K Cross-judge agreement · p. 20

L Difficulty-tier outcomes and shopping pace

Exact-cart success is reported overall and across four difficulty tiers, with small tier samples limiting precision. The tiered turn budgets also reflect shopping pace, revealing different ways agents use the available time.

L Difficulty-tier outcomes and shopping pace

  • Table 14 reports exact-cart success overall and by generator tier; wide intervals from small tier samples support identifying behavioral variation, not ranking agents within tiers.The table reports 95% bootstrap confidence intervals.Source: L Difficulty-tier outcomes and… · p. 21, Table 14 · p. 21
  • The turn budget tests whether an agent matches the required shopping pace as well as assembles the target cart.Larger baskets and less specific requests receive proportionally more turns.Source: L Difficulty-tier outcomes and… · p. 21
  • Gemma4-26B uses a median 1.14 tool calls per turn on light exploration versus 2.04 on complex baskets, while the complex-tier budget allows 2.25 turns per item.On light exploration, 22 conversations reach the turn budget and 11 reach the goal; only one complex trial reaches the budget. Early rapport and required size or gender clarification can push confirmation past the light-exploration budget.Source: L Difficulty-tier outcomes and… · p. 21
  • Qwen3.5-35B issues a median 66 and up to 239 calls on complex baskets, while 4 of 17 trials end in customer abandonment.Termination reasons and tool-call density indicate whether an agent should move more directly from request to cart action or reduce unnecessary tool use.Source: L Difficulty-tier outcomes and… · p. 21

M Full metric tables

The complete outcome and tool-use results distinguish under-action from over-purchase, while typed rubric results expose attribute hallucinations that an aggregate score does not localize.

M Full metric tables

  • Qwen3-30B has the lowest exact-cart success, fewest calls, and highest turn-budget-exhaustion rate, while Qwen3.5-35B has the highest goal-achievement rate and cart recall, most calls, and lower cart precision.These profiles are consistent with under-action and over-purchase, respectively; Table 15 covers 100 shopping trials per model, and turns to goal average only over trials reaching the goal.Source: M Full metric tables · p. 21, Table 15 · p. 22
  • GPT-OSS and Qwen3-30B have much higher attribute-hallucination rates than other agents, while their numeric-hallucination rates remain within the observed range.Table 16 normalizes rubrics to [0, 1], with higher scores better; its per-trial macro-mean failure rates are lower-is-better.Source: M Full metric tables · p. 21, Table 16 · p. 22
  • The aggregate hallucination score alone does not localize the attribute-grounding failure reported for GPT-OSS and Qwen3-30B.Typed failure rates in Table 16 separate attribute hallucinations from numeric hallucinations.Source: M Full metric tables · p. 21, Table 16 · p. 22
Loading 2609.16093v1…