PaperlayerView exact revision on arXiv

Source-linked AI summary

EcomBench: Towards Holistic Evaluation of Foundation Agents in E-commerce

Rui Min, Zile Qiao, Ze Xu, Jiawen Zhai, Wenyu Gao, Xuanzhong Chen, Haozhen Sun, Zhen Zhang, Xinyu Wang, Hong Zhou, Wenbiao Yin, Bo Zhang, Xuan Zhou, Ming Yan, Yong Jiang, Haicheng Liu, Liang Ding, Ling Zou, Yi R. Fung, Yalong Li, Pengjun Xie

arXiv:2512.08868v2cs.AI

TL;DR

Many agent benchmarks focus on academic or artificial scenarios, leaving real-world e-commerce tasks underexplored despite their diverse interactions and changing conditions. EcomBench builds a human-curated benchmark from genuine user demands, with multiple task types and difficulty levels. Evaluations show performance declines as difficulty rises, including a fall to 46% in Level 3 for leading models.

  • Problem

    Most existing agent-evaluation studies focus on academic puzzle-style benchmarks, leaving real-world e-commerce tasks involving changing trends and domain knowledge largely unexplored.

    Source: §1 · pp. 1–2

  • Method

    EcomBench uses human-in-the-loop curation of genuine e-commerce demands and three difficulty levels, including tool-hierarchy-selected questions requiring complex reasoning and long-horizon planning.

    Source: §1 · p. 2

  • Results

    Performance declines as difficulty increases; ChatGPT-5.1 and Gemini DeepResearch, each above 90% accuracy in Level 1, fall to 46% in Level 3.

    Source: §4.2 · p. 5

  • Takeaways & Limitations

    EcomBench provides a rigorous testbed for measuring agent capabilities in realistic, dynamic e-commerce scenarios.

    Source: §7 · p. 9

    EcomBench currently evaluates question-answering rather than explicit interaction, omits predictive problems such as product selection and market forecasting, and requires substantial human effort for maintenance.

    Source: §7 · p. 9

Abstract

from arXiv · show

Foundation agents have rapidly advanced in their ability to reason and interact with real environments, making the evaluation of their core capabilities increasingly important. While many benchmarks have been developed to assess agent performance, most concentrate on academic settings or artificially designed scenarios while overlooking the challenges that arise in real applications. To address this issue, we focus on a highly practical real-world setting, the e-commerce domain, which involves a large volume of diverse user interactions, dynamic market conditions, and tasks directly tied to real decision-making processes. To this end, we introduce EcomBench, a holistic E-commerce Benchmark designed to evaluate agent performance in realistic e-commerce environments. EcomBench is built from genuine user demands embedded in leading global e-commerce ecosystems and is carefully curated and annotated through human experts to ensure clarity, accuracy, and domain relevance. It covers multiple task categories within e-commerce scenarios and defines three difficulty levels that evaluate agents on key capabilities such as deep information retrieval, multi-step reasoning, and cross-source knowledge integration. By grounding evaluation in real e-commerce contexts, EcomBench provides a rigorous and dynamic testbed for measuring the practical capabilities of agents in modern e-commerce.

1 Introduction

EcomBench addresses the limited coverage of real-world tasks in agent benchmarks by evaluating authentic e-commerce questions that combine domain expertise, information retrieval, and reasoning. It combines expert curation, broad task and difficulty coverage, and regular updates; evaluations show that existing models have room to improve on practical e-commerce tasks.

1 Introduction

  • EcomBench targets the gap left by puzzle-style evaluations with composite question-answering tasks reflecting authentic users’ daily e-commerce issues.It assesses domain expertise, precise information retrieval, user-intent understanding, and useful conclusions.Source: §1 · pp. 1–2
  • Human-in-the-loop curation starts from genuine demands collected in global e-commerce ecosystems and refines questions with experts for depth, clarity, and verifiable answers.The benchmark’s principles also include expert-written and peer-validated questions.Source: §1 · pp. 2–3
  • Three difficulty levels include high-difficulty questions selected with a Tool Hierarchy approach to test complex reasoning, long-horizon planning, tool use, and e-commerce expertise.The approach uses e-commerce-specific tools to identify questions that cannot be solved in just a few action steps.Source: §1 · p. 2
  • Existing models still have considerable room to improve on practical e-commerce tasks.The introduction points to Figure 1 for its comparison of model performance on EcomBench.Source: §1 · p. 2, Figure 1 · p. 2
  • EcomBench spans e-commerce tasks such as policy consulting, cost analysis, and marketing strategy, with multiple-choice and open-ended questions calibrated by difficulty.The benchmark is also regularly updated to track market trends and reduce potential data-contamination risks.Source: §1 · p. 3

2 Human-in-the-loop Data Curation

The curation process turns real e-commerce demands into answerable, verifiable benchmark questions, then selects difficult tasks using a hierarchy of tools. Human review shapes the questions, while tool-hierarchy filtering targets multi-step reasoning and planning.

2.1 Data Collection from Real-World User Demands

  • Collected user demands are filtered into seed questions with verifiable answers, then refined by e-commerce experts and independently labeled by at least three experts.Questions with inconsistent expert answers are discarded to improve reliability.Source: §2.1 · p. 3
  • Question reconstruction and labeling rely primarily on human effort rather than direct LLM synthesis, at higher cost, to retain genuine human demands and domain insight.The source demands come from real e-commerce interactions, including searches for policy information, cost estimates, product selection, and business decisions.Source: §2.1 · pp. 3–4

2.2 Question Selection with Tool Hierarchy

  • Level-3 questions are selected with specialized e-commerce tools and rejection sampling to retain tasks requiring complex reasoning chains beyond a few action steps.The tool hierarchy contrasts advanced tools with atomic search or browsing; tasks can demand extensive action sequences, long-horizon reasoning, and adaptive tool use.Source: §2.2 · p. 4

3 In-depth Analysis of EcomBench

EcomBench organizes real-world e-commerce questions into seven task categories and three difficulty levels. Examples illustrate that harder questions extend beyond simple retrieval to require information gathering, regulatory interpretation, and coherent decisions.

3.1 Fine-Grained Task Category

  • The benchmark covers seven categories: Policy Consulting, Cost and Pricing, Fulfillment Execution, Marketing Strategy, Intelligent Product Selection, Opportunity Discovery, and Inventory Control.Table 1 is described as presenting task categories and descriptions.Source: Table 1 · p. 4, §3.1 · p. 5

3.2 Case Study on Difficulty Levels

  • Level 1 comprises 20% of tasks, Level 2 30%, and Level 3 50%, with progressively more demanding reasoning and planning capabilities.Level 1 tests foundational expertise and basic tool operations; Level 2 requires decomposition across multiple action steps; Level 3 tests cross-source integration, deep retrieval, and long-horizon reasoning.Source: §3.2 · p. 5
  • Examples from Policy Consulting and Cost and Pricing show tasks requiring diverse-source information gathering, interpretation of domain-specific regulations, and coherent decisions.The examples are sampled across all three difficulty levels.Source: §3.2 · p. 5

4 On Evaluating Diverse Agents on EcomBench

EcomBench evaluation combines model comparisons with LLM-based binary correctness scoring and manual checks. Performance declines with task difficulty, and model strengths vary across e-commerce categories.

4.1 Evaluation Settings

  • The evaluation covers a diverse set of leading models and scores answers with an LLM-assigned binary correctness measure, with manual inspection of a subset.Average correctness is reported for each model.Source: §4.1 · p. 5

4.2 Evaluation across Difficulty Levels

  • Level 1 scores range from 80% to 95%, but even ChatGPT-5.1 and Gemini DeepResearch fall to 46% at Level 3.For the remaining assessed models, Level 3 scores generally stay below 35%.Source: §4.2 · pp. 5, 7, Figure 3 · p. 7

4.3 Evaluation across Task Categories

  • ChatGPT-5.1 leads overall, but SuperGrok excels in finance-related tasks and Gemini DeepResearch surpasses it in strategy-related tasks.The category comparisons show distinct model strengths across policy-related, finance-related, and strategy-related domains.Source: Figure 4 · p. 7, §4.3 · p. 7

5 Dynamic Maintenance and Updating of EcomBench

EcomBench is updated quarterly to keep its questions aligned with both agent progress and changes in e-commerce. Its planned task expansion moves toward more analytical and predictive work.

  • Quarterly updates replace questions that become too easy as agents improve and revise items made outdated by policy, market, or product changes.The updates aim to keep difficulty aligned with contemporary models while reflecting current e-commerce practice.Source: §5 · pp. 7–8
  • Planned task expansion includes market analysis and forecasting, moving beyond concise, verifiable answers toward analytical, decision-oriented, and predictive tasks.Source: §5 · p. 8

6 Related Work

Related work traces the development of deep research agents and a shift toward evaluating agents in realistic settings. EcomBench is positioned as covering broad e-commerce expertise and decision-driven tasks grounded in practical scenarios.

  • Deep research agents build on retrieval and agentic frameworks, extending them to long-horizon information seeking through specialized tools and structured reasoning.The passage describes ReAct as a conceptual backbone for modern deep research agents and notes automated synthesis approaches for complex tasks.Source: §6 · p. 8
  • Agent evaluation has expanded from static retrieval toward multi-step tool use, expert reasoning, procedural reliability, and economic impact.The passage gives GAIA as a pivotal shift toward multi-step, tool-dependent tasks with verifiable answers.Source: §6 · p. 8
  • Recent benchmarks address realistic prediction, finance, and trading; unlike persona-based ECom-Bench, EcomBench targets broader e-commerce expertise and decision-driven tasks.The passage describes ECom-Bench as evaluating multimodal agents through persona-based user simulations.Source: §6 · pp. 8–9

7 Conclusion and Limitations

EcomBench is a human-curated, quarterly updated benchmark of authentic e-commerce demands, spanning multiple task categories and three difficulty levels. Its current scope and upkeep have limitations.

  • EcomBench combines authentic user demands, domain-expert curation, multiple task categories, three difficulty levels, and quarterly updates.The authors present it as a testbed for evaluating foundation agents in realistic, dynamic e-commerce scenarios.Source: §7 · p. 9
  • The benchmark currently focuses on question-answering rather than explicit interaction environments, and predictive tasks and ongoing human maintenance remain limitations.The authors note that regular problem design and verification increase dataset-construction costs.Source: §7 · p. 9
Loading 2512.08868v2…