PaperlayerView exact revision on arXiv

Source-linked AI summary

MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations

Qiming Shi, Yulong Tao, Linbo Jin, Zhaolu Kang, Yibo Dou, Jiawen Zhu, Tianjun Pan, Shaokang Fu, Chengyu Wang, Siyue Li, Yaping Cheng, Di Weng, Chengfu Huo

arXiv:2607.28956v2cs.AI

TL;DR

Existing agent benchmarks provide limited evidence about sustained objective pursuit in persistent, long-horizon environments. MerchantBench evaluates this capacity through 365-day seller-side e-commerce operations, finding a substantial performance gap between LLM agents and human participants.

  • Problem

    Many agent benchmarks emphasize bounded tasks with immediate completion, limiting evidence about coherent objective pursuit across persistent, extended-horizon environments.

  • Method

    MerchantBench evaluates long-term coherence in a persistent 365-day seller-side e-commerce simulation with interdependent decisions and differently delayed operational feedback.

  • Results

    LLM agents exhibit a substantial system-level performance gap from the Human baseline across 48 runs involving eight models and two agent frameworks.

  • Takeaways & Limitations

    Trace analyses associate weaker outcomes with declining operational activity, premature goal abandonment, and strategy changes uncalibrated to accumulated evidence.

  • Takeaways & Limitations

    Activity volume alone is insufficient because models differ in how effectively they translate actions into business value.

Abstract

from arXiv · show

Large language model agents are increasingly evaluated as autonomous tool users, yet most benchmarks focus on bounded tasks with immediate success criteria. Real-world deployments often require Long-Term Coherence, the capacity to preserve purposeful behavior across extended horizons while adapting decisions to accumulated evidence. Evaluating this capacity requires a persistent environment in which actions constrain future choices, feedback arrives at heterogeneous delays, and incoherent behavior produces measurable cumulative effects. Seller-side e-commerce provides a suitable setting for this evaluation through recurrent and interdependent decisions over Product Sourcing, Listing and Pricing Control, Cash-Flow Management, and Mixed-Latency Feedback Adaptation. We introduce MerchantBench, a 365-day order-level simulation grounded in 98,843 real e-commerce product records and equipped with 26 tools for agent interaction. MerchantBench couples promptly observable Upstream Supplier Events with delayed Downstream Order Outcomes, requiring agents to follow individual order lifecycles and revisit earlier decisions. We evaluate eight LLMs under two agent frameworks in 48 runs, each spanning 365 simulated days. Our results reveal a substantial gap between even the latest LLMs and human participants, with the best LLM configuration attaining only 27.3\% of the mean final net assets achieved by human participants.

Introduction · Related Work

MerchantBench extends agent evaluation from bounded tasks to persistent seller-side e-commerce, where agents must sustain coherent decisions under partial observability, cash constraints, changing opportunities, and delayed order feedback. It addresses gaps in existing commerce and long-horizon benchmarks through a 365-day, order-level simulation grounded in real product data.

  • Introduction: Existing agent benchmarks commonly evaluate tool use, web interaction, application control, and state-changing workflows as bounded tasks with immediate completion criteria.The paper argues that many real-world deployments instead require sustained objective pursuit and decision consistency over extended horizons.
  • Introduction: Seller-side e-commerce tests Long-Term Coherence through repeated product selection, listing and pricing control, cash management, and revision of earlier decisions.These decisions evolve with market conditions, supplier states, and order outcomes throughout the operating horizon.
  • Introduction: Individual order lifecycles create temporal asymmetry: orders commit available cash immediately, while fulfillment failures and after-sales outcomes surface later.Agents must connect delayed outcomes to earlier listing or pricing decisions as evidence accumulates.
  • Introduction: Changing opportunities require a large Product Catalog with full-year demand trajectories, because promising products emerge while existing choices lose value.Agents must continually identify opportunities and revise their portfolios using market signals and realized order outcomes.
  • Introduction: MerchantBench formulates persistent seller-side e-commerce operation as a partially observable decision-making problem over 365 simulated days.Its Product Catalog is grounded in 98,843 real e-commerce product records, and demand becomes individual orders progressing through fulfillment and after-sales stages.
  • Introduction: The benchmark is structured around four interdependent decision components and includes partial observability, cash constraints, Upstream Supplier Events, and delayed Downstream Order Outcomes.The paper reports 48 runs of 365 simulated days across eight LLMs and two agent frameworks.
  • Related Work: Existing commerce benchmarks cover shopping, storefront interaction, customer support, merchant workflows, negotiation, market competition, and economic-agent transactions.Across these categories, evaluation centers on tasks, dialogues, transactions, or competitive episodes rather than continuous operation of the same online store.
  • Related Work: Long-horizon benchmarks assess sustained reasoning and action across workplace workflows, exploration, virtual worlds, web navigation, computer use, inventory, fulfillment, and interactive economies.The supplied passage also identifies Vending-Bench and Vending-Bench 2 as emphasizing long-horizon business evaluation.

MerchantBench

MerchantBench models seller-side e-commerce operations as a partially observable, 365-day simulation in which agents coordinate sourcing, store control, cash flow, and adaptation under mixed-latency feedback. It combines real-world product and demand data, supplier events, individual order lifecycles, and 26 tools to evaluate long-term coherence and net asset outcomes.

  • Environment: MerchantBench formulates store operation as a finite-horizon partially observable Markov decision process.The simulator advances hourly across a 365-day control horizon, with decision windows every 12 steps.
  • Environment: 365 days and 8,760 hourly steps create an extended environment where demand, suppliers, listings, finances, orders, and pending events evolve over time.Agents receive a decision window once every 12 steps, while the latent state includes the simulation clock, product demand profiles, supplier conditions, store listings, finances, active orders, and pending events.
  • Data and simulation: 98,843 products anchor MerchantBench in real e-commerce records containing product and supplier attributes, 365-day demand histories, and platform quality and fulfillment signals.The benchmark also incorporates 365 date-aligned daily market reports from 1688 for product sourcing.
  • Feedback dynamics: Three Upstream Supplier Events—Price Change, Product Delisting, and Shipment Delay—alter procurement prices, suspend procurement, or extend supplier dispatch times.Event probabilities are calibrated from real platform fulfillment signals, and supplier inventory replenishes over time.
  • Feedback dynamics: The downstream simulation converts daily demand traces into individual orders whose realized outcomes update store ratings and follow supplier and logistics delays.Order-level demand depends on linked daily demand, hourly category patterns, price elasticity, store rating, and listing exposure.
  • Agent interaction: 26 merchant tools expose a shared observation protocol for sourcing, listing and pricing, cash-flow management, and adaptation to supplier events and order outcomes.Agents receive summaries of simulated time, store status, and recent supplier and order changes at each decision window while interacting with identical environment dynamics.

Experiments

MerchantBench evaluates eight LLMs across two agent frameworks, alongside rule-based and human baselines, using business, reliability, and long-horizon activity metrics over 365 simulated days. Hermes generally outperforms ReAct, but results vary substantially in stability, sustained activity, and agents’ ability to adapt to delayed evidence and maintain coherent operation.

  • Experimental Setup: Eight LLMs are evaluated with ReAct and Hermes, using three runs per model-framework pairing, alongside a rule-based baseline and three human participants.Each run begins with RMB 2,000 cash, a RMB 1,000 security deposit, and capacity for 50 active listings.
  • Evaluation Metrics: The evaluation measures business performance, store reliability, and long-horizon activity, including Final Net Assets, GMV, Total Fines, SWR, and Total Tool Calls.SWR is the minimum share of scheduled decision windows containing an environment tool call across rolling 30-day periods.
  • Performance Variability: 55.1% is Qwen3.7-Max’s coefficient of variation under Hermes despite its highest mean final net assets, while GPT-5.6 Sol under ReAct has the lowest within-framework variation at 3.3%.Claude Opus 4.8 under Hermes has the lowest coefficient of variation within Hermes at 10.0%.
  • Framework Analysis: 53.3% higher final net assets, 71.5% higher GMV, and 71.2% more orders are achieved by Hermes than ReAct on average across models.Hermes improves mean final net assets for seven of eight models, while Kimi K2.6 is the exception at 4.1% lower under Hermes.
  • Evidence-Calibrated Adaptation: Delayed order outcomes produce divergent interventions: some agents replace risky listings or reprice products, while others retain or fail to revise affected products.GPT-5.6 Sol and Kimi K2.6 replaced responsible listings; Qwen3.7-Plus retained a risky product, and DeepSeek-V4-Pro did not revise affected listings after refunds.
  • Operational Coherence: 100% is the human SWR, compared with LLM ranges of 10.6%–99.4% under ReAct and 17.8%–66.1% under Hermes.Qwen3.7-Max’s quarterly Effective Window Rate falls from 62% to 37% under Hermes and from 68% to 23% under ReAct, alongside fewer environment tool calls.

Conclusion

MerchantBench evaluates Long-Term Coherence in persistent seller-side e-commerce operations over 365 days. Across 48 runs, LLM agents showed a substantial system-level performance gap from the Human baseline, with weaker outcomes accompanying declining activity and premature goal abandonment.

  • Conclusion: MerchantBench evaluates Long-Term Coherence through persistent seller-side e-commerce operation in a 365-day order-level environment.The environment is grounded in 98,843 real e-commerce product records.
  • Conclusion: 48 runs involving eight LLMs and two agent frameworks revealed a substantial system-level performance gap from the Human baseline.The comparison concerns system-level performance across the evaluated agents and human baseline.
  • Conclusion: Trace analyses linked weaker outcomes with declining operational activity and premature goal abandonment.The passage also mentions strategy changes, but the supplied text truncates before specifying their characterization.

Demand and Rating Dynamics

MerchantBench models listing exposure with a cold-start ramp followed by exponential decay, while store ratings update daily from eligible order outcomes and influence subsequent demand through star-level multipliers.

  • Listing Exposure: Listing exposure rises through a cold-start ramp, then decays exponentially toward a minimum level as listing age increases.The exposure factor is defined from elapsed days since activation and uses a linear ramp up to a threshold followed by exponential decay.
  • Store Rating Dynamics: Store reputation updates after each simulated day using outcome-dependent scores and evidence weights from eligible completed orders.Cancellations and insufficient-balance failures are excluded; older outcomes receive discounted weight, while a prior stabilizes sparse evidence.
  • Store Rating Dynamics: The continuous store rating maps to a discrete star level whose demand multiplier affects subsequent order generation.The simulator also computes listing-level diagnostic ratings, but those ratings do not affect demand.

Data Collection and Filtering · Full Operational Coherence Profiles

MerchantBench filters real-world 1688 product records into a 365-day, ten-category catalog with varied risks, prices, and seasonal demand histories. Its operational coherence profiles show differing activity trajectories across humans and eight models under Hermes and ReAct.

  • Data Collection and Filtering: 1688-sourced Product Records contain 365-day order histories spanning ten first-level product categories.Records are filtered for missing identifiers or names, out-of-scope categories, and missing or nonpositive prices.
  • Data Collection and Filtering: The filtered catalog retains substantial cross-category variation in record coverage and supplier prices.Its calibrated joint risk distribution combines diverse Downstream Order Outcome probabilities with varied Upstream Supplier Event intensities.
  • Data Collection and Filtering: 365-day demand histories preserve major shopping peaks, seasonal changes, and distinct product-specific demand cycles.Representative cycles are shown for red envelope, electric fan, and hot water bag products.
  • Full Operational Coherence Profiles: Figures 13 and 14 compare monthly trajectory diagnostics for Human and all eight models under Hermes and ReAct.The diagnostics extend operational activity comparisons across both agent frameworks.
  • Full Operational Coherence Profiles: 81% and 77% are GPT-5.6 Sol’s final-two-month activity levels under Hermes, while DeepSeek-V4-Flash remains between 70% and 87% throughout the year.Under Hermes, GPT-5.6 Sol remains almost fully active through February, whereas DeepSeek-V4-Flash maintains a narrower annual range.
  • Full Operational Coherence Profiles: 64% to 31% marks Qwen3.7-Max’s Hermes decline, while Claude Opus 4.8 contracts from 46 to eight month end active listings.Kimi K2.6 recovers to 78% and 74% effective windows only in the final two months under Hermes.

Net Asset Curves … Full Monthly Product Sourcing Analysis

Across extended runs, LLM agents show variable financial performance: sustained tool use and product exploration correlate with higher final net assets, but activity alone does not ensure business value. Monthly demand alignment also varies substantially, with Human strongest and Claude Opus 4.8 improving under both frameworks.

  • Net Asset Curves: Net asset curves compare daily trajectories across ReAct and Hermes, all eight models, Human and Rule-based baselines, plus variation across individual runs.Figures 15–18 provide the complete daily curves and repeated-run variation.
  • Scale and Unit Profitability: Scale alone does not guarantee highest return: one Kimi K2.6 ReAct run reaches 3,004 orders at only RMB 11.1 net profit per order.Qwen3.7-Max with Hermes achieves the largest total profit in one run through high profit per order and moderate scale.
  • Tool Use and Product Selection Analysis: Activity volume alone is insufficient because models differ in how effectively they translate actions into business value.Dispersion around both fitted trends shows that identical levels of activity do not imply identical financial outcomes.
  • Full Monthly Product Sourcing Analysis: The Monthly Demand Alignment Percentile weights products by active listing hours against each month’s complete-catalog demand ranking.A score of 80 means an average listed product-hour belongs to a product exceeding 80% of the catalog in demand.
  • Full Monthly Product Sourcing Analysis: Human maintains the strongest demand alignment, while Rule-based stays near the catalog median with little seasonal change.Claude Opus 4.8 improves under both frameworks, whereas several other combinations plateau or decline during the year.

Time-aware Sourcing Gain · Hermes Case Study

Time-aware Sourcing Gain measures seasonal portfolio reallocation against a fixed annual-mix counterfactual, and higher gain is positively associated with final net assets across 48 LLM runs. The Hermes case study shows sharply varied code use and frequent creation and later use of named skills.

  • Time-aware Sourcing Gain: Time-aware Sourcing Gain compares actual monthly demand alignment with a counterfactual that holds each run’s annual product mix fixed.An unchanged product mix yields a gain of zero.
  • Time-aware Sourcing Gain: Higher Time-aware Sourcing Gain is positively associated with final net assets across the 48 LLM runs.Final net assets also depend on pricing, cash flow, and order management decisions.
  • Hermes Case Study: The Hermes case study examines programmatic code use and the creation and revision of skills.These capabilities are listed in Tables 4 and 5.
  • Code Use: 15 execute_code calls across two runs and 13 in one run: GPT-5.6 Sol dominates native code use across the 24 Hermes runs.At the first decision window, it computed prices for 50 candidate products using p = max(1.70c, c+6), rounding upward to prices ending in 0.9.
  • Code Use: 261 memory calls across three runs: Claude Opus 4.8 channels its analysis through memory rather than extensive native code use.Claude Opus 4.8, Qwen3.7-Max, and DeepSeek-V4-Pro each invoke execute_code once; Qwen3.7-Max and DeepSeek-V4-Flash each invoke terminal once.
  • Skill Evolution: 17 of 24 runs and seven of the eight models contain a run-created RealShop skill in their final Hermes profiles.Seventeen of the 18 created skills record at least one subsequent use, and the only unused skill was created by Qwen3.7-Max.
  • Skill Evolution: All three runs create a skill for GPT-5.6 Sol, Kimi K2.6, and Qwen3.7-Plus, whereas Claude Opus 4.8 creates none.The passage reports model-level differences in skill creation across the three runs per model.

Detailed Experimental Configuration

The experimental configuration defines a 365-day evaluation protocol with rule-based and human runs, persistent context management, store-level constraints, and metrics for profitability, performance, anomalies, and sustained tool use. It also specifies demand, supplier, order-lifecycle, and rating parameters governing the simulation.

  • Evaluation Protocol: The protocol includes three rule-based runs and three inexperienced human-participant runs, each spanning 365 simulated days.The rule-based baseline checks abnormal states daily, delists products under specified conditions, and fills available slots using keywords from daily market reports.
  • Store Configuration: Each store starts with RMB 2,000 cash, RMB 1,000 security deposit, and up to 50 active listings, with fixed fines ranging from RMB 3 to RMB 8.Fines apply to Return and Refund, Bad Review, Stockout, insufficient balance, and Late Shipment; Cancellation and Returnless Refund incur no additional fine.
  • Context Management: Both frameworks manage the 365-day horizon through history compression, with ReAct summarizing at 160,000 tokens before truncation to 30,000 and Hermes using its default procedure.Hermes uses the evaluated model as its summary model.
  • Evaluation Metrics: Net Profit Margin is terminal net profit divided by GMV, while Average Store Rating and Average Active Listings are daily means.Order Anomaly Rate measures affected generated orders, and Sustained Window Rate measures scheduled decision windows containing at least one environment tool call.
  • Order Lifecycle Configuration: The simulation uses a 48-hour promised shipment time, with settlement transition and post-delivery outcome realization each sampled within 168 hours.Supplier initialization, replenishment, dispatch, logistics, recovery, price-change, and shipment-delay ranges are also explicitly configured.
  • Rating Configuration: Store ratings start at R0 = 4.0 with α = 20 and γ = 2−1/30, while outcome scores and evidence weights differ across lifecycle and adverse events.Demand multipliers are mapped to five rating intervals using thresholds 2.50, 3.30, 3.80, and 4.20.

Tool Sets and Skills

MerchantBench agents interact with the store through a fixed set of 26 tools. The get_daily_report tool provides date-specific evidence that is current only through the previous simulation date, while Hermes additionally has built-in tools.

  • MerchantBench tools: 26 tools comprise the complete MerchantBench tool set used by ReAct to access observable fields and control the store.Table 2 lists the full set of tools.
  • MerchantBench tools: get_daily_report returns the report published for the current simulation date.Its evidence is current only through the previous date.
  • Agent tool access: Hermes uses the same MerchantBench tools as ReAct together with built-in tools.The passage states that Hermes provides the same MerchantBench tools in addition to built-in tools.

Agent Inputs and Observable Fields

Each evaluated agent receives one system prompt at registration and a compact observation every 12-hour decision window. Both frameworks use the shared MerchantBench task prompt and access the same task information.

  • Agent Inputs: Agents receive one system prompt at registration and a compact observation at every 12-hour decision window.These inputs define the recurring information available during interaction.
  • Shared Task Information: Both frameworks receive the shared MerchantBench task prompt and access the same task information.The task prompt is identified in Table 6.

26 MerchantBench

MerchantBench evaluates merchant agents through a 26-tool action space implemented under ReAct and Hermes, with Hermes additionally providing built-in tools and skills. The environment is partially observable: agents receive current public and realized operational evidence, while future demand, event hazards, and presampled outcomes remain hidden.

  • Agent interfaces: ReAct uses the complete MerchantBench prompt and 26 tools as its entire action space, while Hermes embeds the prompt in its runtime template.Hermes also augments the 26 MerchantBench tools with built-in tools and skills.
  • Partial observability: MerchantBench exposes current public and realized operational evidence but withholds future demand, event hazards, and presampled outcomes.Some visible order fields remain empty until their corresponding lifecycle events are realized, whereas hidden fields are never returned through merchant tools.
  • Evaluation outputs: The benchmark reports monthly operational-coherence profiles and daily net-asset curves across human, rule-based, Hermes, and ReAct runs.The profiles cover effective window rate, environment tool calls, month-end active listings, and monthly net profit over the simulated horizon.
Loading 2607.28956v2…