PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoVend: Benchmarking LLM Agents on Long-Horizon Autonomous Vending Operations

Qiming Shi, Yikun Li, Beidi Luan, Rui Sun, Yan Ma, Jing Li, Daxin Jiang, Zuo Bai

arXiv:2610.36980v1cs.AIcs.CL

TL;DR

Evaluating seller-side agents requires measuring whether Web decisions yield sustained returns when physical execution is costly. AutoVend tests this in a 180-day vending operation, where four of five LLM agents underperform a rule-based inventory-control baseline and operational gains do not consistently translate into stronger economic growth.

  • Problem

    Seller-side decisions affect inventory and cash beyond a single interaction, while online-to-offline returns depend on costly physical execution and coordinated inventory commitments.

    Source: §1 · pp. 1–2

  • Method

    AutoVend evaluates one LLM agent managing 20 vending machines across five site types for 180 simulated days in a data-grounded simulator with shared inventory and paid replenishment.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    Four of five evaluated LLM agents underperform the rule-based inventory-control baseline, and gains in individual metrics do not consistently yield stronger economic growth.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Operational and trajectory analyses identify coordination, execution, feasible-action, inventory-commitment, and policy-revision difficulties in this benchmark setting.

    Source: §5.3 · p. 9, §6 · p. 10, §5.6 · p. 10

    Changing environmental conditions and inventory states prevent temporal comparisons from causally identifying whether feedback improves agent decisions.

    Source: §5.5 · p. 9

Abstract

from arXiv · show

Large language model (LLM) agents are extending agent-mediated commerce on the Web from shopping for consumers to operating businesses for sellers. In online-to-offline (O2O) commerce, decisions are issued through Web-based platforms but realized through costly physical execution. Evaluating such agents requires going beyond individual transactions to ask whether decisions made online turn into sustained economic returns. We introduce AutoVend, a benchmark in which one LLM agent manages 20 vending machines across five site types through a Web-based operating interface for 180 simulated days. Its data-grounded simulator combines a shared warehouse, procurement lead times, paid replenishment, inventory expiration, and recurring operating costs. We evaluate five LLM agents against a rule-based inventory-control baseline and diagnose their site coordination, execution, and performance over time. Four of five evaluated LLM agents fall short of this baseline, and returns per dollar of inference cost rank models differently from returns alone. Operational analyses show that gains in individual metrics do not consistently yield stronger economic growth, as inventory and execution costs erode these gains. Trajectory analysis further reveals difficulties in grounding feasible actions, managing inventory commitments, and converting feedback into effective policy revisions.

1 Introduction

Seller-side commerce requires decisions whose effects persist beyond a single transaction, while online-to-offline returns depend on costly physical execution. AutoVend provides a long-horizon benchmark for testing whether Web-based operating decisions produce sustained economic outcomes.

1 Introduction

  • In online-to-offline commerce, decisions occur on Web platforms but returns depend on physical execution costs and constraints.AutoVend also measures the inference cost of generating returns through the interface.Source: §1 · p. 1
  • Sustained operation requires coordinating heterogeneous site demand and shared resources, planning procurement and paid replenishment, and revising decisions as sales feedback accumulates.Advance ordering must account for delivery lead times, while stocking and site visits incur costs and can expose inventory to expiration.Source: §1 · pp. 1–2
  • Existing benchmarks do not establish how well agents coordinate shared inventory across heterogeneous physical retail sites when replenishment incurs costs and delays.Source: §1 · p. 2
  • AutoVend has one agent manage 20 vending machines across five site types through a Web-based interface.The agent selects products, sets prices, orders procurement, and arranges replenishment from a shared warehouse.Source: §1 · p. 2
  • The evaluation combines terminal net assets with analyses of site coordination, replenishment costs and availability, and operating performance over time.Source: §1 · p. 2
  • Its simulator uses real vending records to model heterogeneous demand, procurement lead times, inventory aging, and paid site visits, and validates demand behavior against observed sales.Source: §1 · p. 2
  • Four of five evaluated LLM agents fall short of the rule-based inventory-control baseline.Operational diagnostics and trajectory analyses identify challenges in inventory commitments, execution costs, and policy revision.Source: §1 · p. 2

2 Related Work

Related work evaluates commerce agents across completed transactions, sustained business operation, and long-horizon tasks. AutoVend focuses on accumulated economic outcomes in distributed physical retail with shared inventory and paid on-site actions.

Agent Evaluation in Commerce

  • Commerce benchmarks often end with a completed task or transaction, whereas seller-side benchmarks assess sustained business operation through accumulated economic outcomes.Source: §2 · p. 2
  • Prior seller-side settings include single online stores, one cross-border shop, up to four online stores sharing a warehouse, a single supermarket, and a single vending machine.AutoVend instead has one agent operate 20 heterogeneous sites from a shared warehouse, with every on-site action requiring a paid visit.Source: §2 · pp. 2–3

Long-Horizon Agent Evaluation

  • Sustained-operation benchmarks fix a simulated calendar span in which costs accrue and the environment evolves, so evaluation measures accumulated economic results rather than goal completion.Related evaluations add progress, capability, action-attribution, and continual-learning analyses; AutoVend ties operational dimensions to sites and on-site execution over time.Source: §2 · p. 3

3 The AutoVend Benchmark

AutoVend specifies a 180-day, 20-machine operation with hidden, data-grounded demand and a shared warehouse, then details its simulator, validation, and operating interface.

3.1 Task Formulation

  • Across 180 days, one agent manages 20 machines and 1,372 products, maximizing terminal net assets under hidden demand and unmet sales.The objective accounts for cash and unexpired inventory, whose value changes through sales, costs, fines, and write-offs.Source: §3.1 · p. 3

3.2 World Simulator

  • Shared stock, delayed procurement, paid visits, and expiring inventory make product availability trade off against recurring operating costs.Orders arrive three days after placement, and a visit costs 35 CNY per machine per day; batches expire 30 or 90 days after warehouse arrival.Source: §3.2 · p. 3

3.3 Real-World Data Grounding

  • Calibration draws on 220,128 sales units across 99 real machines, estimating product popularity, site preference, subcategory alignment, and machine scale.The selected simulated network uses 20 machines across five site types, and its catalog covers 1,372 products in 18 subcategories.Source: §3.3 · p. 4
  • WAPE is 0.473 versus 0.606 for the naive reference and 0.561 for persistence, while simulated total volume is within 0.7% of observed.TVD is 0.153 versus 0.206 for the naive reference, and median within-machine product-sales Spearman correlation is 0.577; price and environmental responses remain unvalidated.Source: §3.3 · p. 5, Figure 3 · p. 6

3.4 Agent Interface

  • Each day, arrivals and recurring fees precede agent observations and tool actions; sales and visit costs settle after the agent closes the action window.Daily reports include cash and net assets, recent profit and loss, stock, holidays, and machine inventory, sales, and expiry status; product details are retrieved through queries.Source: §3.4 · p. 6

4 Experiments

The experiments compare five LLM agents with a rule-based inventory-control baseline over repeated runs and report operational and economic measures. Four of five models underperform the baseline, while returns and API efficiency rank models differently.

4.1 Experimental Setup

  • Across three runs, agents use the same 20-machine network for 180 simulated days, with identical prompts, tools, and daily interaction budgets.The baseline combines rolling-demand inventory targets, weekly replenishment, lead-time-aware procurement, sales-based assortments, and local gross-profit price trials.Source: §4.1 · pp. 6–7
  • Net Assets is terminal cash plus unexpired inventory valued at cost; other measures include assortment, waste, pricing regret, shelf utilization, capital efficiency, tool calls, and API efficiency.API Efficiency averages per-run net asset growth per USD of estimated API expenditure.Source: §4.1 · p. 7

4.2 Main Results

  • Four of five models underperform the rule-based baseline, and all five have lower assortment scores than the baseline.All three GPT executions exceed the baseline, whereas none of the other 12 does.Source: §4.2 · p. 7
  • Kimi K3 has a 45.6% coefficient of variation across runs, while GPT-6.1 Sol, GLM-5.3-Flash, and Claude Sonnet 5.5 each show around 11%.The paper distinguishes consistency from economic effectiveness.Source: §4.2 · p. 7
  • GPT-6.1 Sol has the highest terminal assets, but DeepSeek has the highest net-asset growth per dollar of estimated API expenditure.GPT's mean estimated API expenditure is approximately 80 times DeepSeek's, while its net-asset growth is about 2.9 times as large.Source: §4.2 · p. 7

5 Analysis

Most LLM executions finish above initial capital, but reliable baseline outperformance is limited; operational and trajectory analyses identify coordination, execution, and policy-revision challenges.

5.1 Operational Performance Profiles

  • No model leads all five operational dimensions, and stronger profitability does not consistently coincide with better inventory control, pricing, or capital efficiency.The profiles normalize terminal net assets, assortment quality, reversed waste rate, reversed pricing regret, and daily gross profit per unit of inventory capital across model means and the baseline.Source: §5.1 · p. 7, Figure 4 · p. 8
  • Claude Sonnet 5.5 has empty shelves on 25.21% of site-days, while GPT-6.1 Sol has conditional pricing regret of 30.18 CNY per machine per day, the largest gap.DeepSeek V4.1 Flash has the highest waste rate at 1.85% and machines empty on 1.67% of site-days.Source: §5.1 · p. 8

5.2 RQ1: Can LLM agents effectively manage long-horizon business operations?

  • All 15 LLM executions finish above initial capital, but only GPT-6.1 Sol exceeds the baseline in all three executions.Trajectories also distinguish sustained growth from early gains followed by stagnation, particularly for Kimi K3.Source: §5.2 · p. 8, Figure 5 · p. 9

5.3 RQ2: Can agents coordinate resources across sites?

  • GPT’s reassignment gap is 7.05%, versus 2.61% for the baseline; a small gap can still accompany weak assortments.The diagnostic reassigns existing assortments at six 30-day checkpoints and introduces no new products.Source: §5.3 · p. 9

5.4 RQ3: Can agents plan physical execution?

  • GPT and DeepSeek avoid completely empty shelves, but GPT earns approximately twice DeepSeek’s visit-cycle contribution per site-day; Sonnet and Kimi retain empty-site exposure despite warehouse stock.Central inventory requires paid on-site transfer to reach customers, and visit-cycle returns exclude costs other than visit fees and final incomplete cycles.Source: §5.4 · p. 9

5.5 RQ4: Can agents improve through feedback?

  • Daily asset growth falls from days 16–45 to days 136–165 for all five model means and 13 of 15 executions.DeepSeek generates more gross profit, but additional warehouse and visit expenses offset most of the gain; changing conditions and inventory states prevent causal attribution to feedback.Source: §5.5 · p. 9, Figure 6 · p. 10

5.6 Failure Modes

  • Trajectory inspection identifies failures in action feasibility, inventory-cost planning, physical execution, and policy revision.These observed cases illustrate behavior but do not establish prevalence or causal effects.Source: §5.6 · p. 10
  • Cases include 49 rejected allocations, repeat purchases despite 2,158 unexpired units, a 35 CNY visit after 5.14 CNY gross profit, and infeasible allocations recurring five days later.The recurring allocation pattern was repaired within each interaction but reappeared on day 168 after day 163.Source: §5.6 · p. 10

6 Conclusion

AutoVend evaluates long-horizon autonomous business management across a distributed vending network. Four of five LLM agents underperform a rule-based baseline, with operational analyses revealing coordination and cost challenges.

6 Conclusion

  • AutoVend evaluates autonomous business management across 20 heterogeneous vending sites with shared inventory, procurement lead times, and paid physical execution.The evaluation spans five LLM agents over 180 simulated days.Source: §6 · p. 10
  • Four of five LLM agents underperform the rule-based inventory-control baseline, while individual-metric gains do not consistently yield stronger economic growth.Trajectory cases identify difficulties in verifying feasible allocations, adapting procurement, consolidating visits, and retaining corrections across decisions.Source: §6 · p. 10
  • The findings draw on three runs per model in one environment; the authors call for tests across varied conditions and controlled interventions.Suggested interventions include explicit state verification and persistent policy updates.Source: §6 · p. 10

A Prompts

The system prompt defines a 180-day vending operation whose agent must move products from procurement through warehouse and machine shelves while maximizing net assets. Daily prompts provide observations and require the agent to end each day without human guidance.

A.1 System Prompt

  • 20 machines run for 180 days, starting with 40000 in cash; all shelves and the warehouse begin empty, with 1372 catalog SKUs and no sales history.Nothing sells until the agent orders stock, waits for delivery, and restocks a machine on site.Source: §A.1 · p. 13
  • Net assets equal cash plus the cost value of unexpired units on shelves, in the warehouse, or in transit.Expired units are worth nothing toward net assets.Source: §A.1 · p. 13
  • Stock flows from order to transit, warehouse, machine shelf, and sale; ordered units are debited immediately, and the supplier promises three-day delivery that may vary.In-transit units cannot be shelved or sold.Source: §A.1 · p. 13
  • Warehouse rent is charged per unit per day, an 80-per-day fee applies whether or not the agent acts, and on-site actions cost 35 per device touched each day.Ordering, repricing, discarding, and queries are remote and free.Source: §A.1 · p. 13
  • Expired shelf stock can still sell but incurs a 1000 penalty per unit sold; expired warehouse stock keeps accruing rent and can only be removed by discarding.Delisting expired shelf units stops fine exposure but does not make them sellable again.Source: §A.1 · p. 13
  • Buying stock moves value from cash into inventory rather than creating a gain, while stock incurs rent and cannot be sold during its three-day delivery period.Net assets increase only through selling above unit cost and decrease through rent, labor, daily fees, expired-sale fines, and stock expiring or being discarded unsold.Source: §A.1 · p. 13
  • Demand varies by site and product: fuller devices sell faster, free volume depresses demand, and sales stop increasing after a few units on the shelf.Daily sales are noisy, longer observation windows estimate underlying rates, and product fit and price response differ by location type.Source: §A.1 · pp. 13–14

A.2 User Prompt

  • Each day supplies an observation and a prompt to keep running the business, with no human prompting or approval; the agent must call end_day when work is done.The environment is Shanghai, amounts are in CNY, weather and holidays affect demand, and saved notes are limited to 6,000 characters and are not shortened.Source: §A.1 · p. 14, §A.2 · p. 14

B Tools

The tools support batched inventory and pricing changes, with per-item validation for structurally valid requests and visit charges for physical operations.

B Tools

  • Structurally malformed batches are rejected before execution, while valid batches are checked item by item so accepted items can execute alongside rejected ones.Results include accepted and rejected items with rejection reasons; rejected items do not alter inventory or trigger a visit.Source: B Tools · p. 15
  • The six inventory and pricing mutation tools accept batches of 1–100 items, with quantities specified as positive integers.Query windows are limited to 1, 3, 7, or 30 days; paginated queries default to 20 rows and allow at most 50.Source: B Tools · p. 15
  • The three physical tools charge one visit per machine per day regardless of successful operation count, while an oversized notebook write is rejected without replacing existing notes.A question mark denotes an optional parameter.Source: B Tools · p. 15

C Ethical Use of Data

The simulator is grounded in authorized commercial vending records, and the released materials exclude both personal information and the raw records.

C Ethical Use of Data

Related papers

Loading 2610.36980v1…