RetailBench: Evaluating Long-Horizon Autonomous Decision-Making and Strategy Stability of LLM Agents in Realistic Retail Environments

Motivation

• Long-horizon evaluation asks whether agents sustain persistent objectives, adapt to evolving feedback, and recover from imperfect decisions over extended interactions.Prior studies report that local errors can compound through delayed consequences and reshape future states.Source: §1 · p. 1
• Prior retail benchmarks typically isolate subproblems, leaving a need for an open benchmark of tool-using agents facing coupled operational pressures over extended horizons.Store operators must make decisions whose effects may emerge only after several days.Source: §1 · p. 2

Contributions

• RetailBench evaluates LLM agents in a data-grounded supermarket simulator spanning pricing, replenishment, supplier selection, feedback, and financial dynamics.Agents also manage information acquisition, memory, and day termination amid delayed deliveries, inventory aging, stockouts, reviews, and returns.Source: §1 · p. 2
• The evaluation finds current agent systems remain far from stable, high-performing long-horizon operational policies.The paper identifies incomplete evidence acquisition, surface-level decision making, and lack of a consistent long-horizon policy as recurring failure modes.Source: §1 · p. 2

2.1 Interface

• The POMDP abstraction exposes tool-facing observations and actions while the simulator privately maintains delayed inventory, demand, supplier, feedback, and cash-flow consequences.The agent aims to sustain operation while improving sales, gross profit, net worth, and operating duration.Source: §2.1 · pp. 2–3
• Agents interact through read, action, memory, and code tools with a partially observable state; invoking end_today advances the simulation by one day.The day-level transition processes deliveries, demand, sales, feedback, returns, expiration, rent, cash flow, and supplier and news changes.Source: Figure 2 · p. 3, §2.1 · p. 3

2.2 Data-Grounded Simulator

• Real grocery records anchor products, prices, sales, traffic, and procurement-cost priors, while review and financial-news corpora supply dynamic feedback and events.The Dominick’s dataset supports the product universe and estimates of demand, traffic, price sensitivity, and category substitution patterns.Source: §2.2 · p. 3
• RetailBench is a reproducible, data-grounded stress test, not a full-fidelity simulator for forecasting real-store financial outcomes.Some reviews, news, supplier adaptation, returns, and demand components are synthetic or rule-based.Source: §2.2 · p. 4

2.3 Modules

• RetailBench couples seven modules through daily transitions that update deliveries, shelf visibility, sales, feedback, news effects, inventory aging, and cash-flow accounting.The modules are Inventory, Suppliers & Orders, Shelf, Customer Demand, Reviews & Returns, News Events, and Finance & Records.Source: §2.3 · p. 4

2.4 Scope

• Unlike benchmarks that expose narrower decision surfaces or isolate operational subproblems, RetailBench requires coordinated decisions across closed-loop store operations.The scope includes pricing, inventory, suppliers, shelf placement, customer feedback, external events, and cash-flow survival.Source: §2.4 · p. 4

3.1 Environment Configuration

• The fixed environment contains 96 products across 20 categories, five candidate suppliers per product, and specified budget, rent, inventory, and shelf capacities.The configuration uses an initial budget of 30,000, daily rent of 600, inventory capacity of 15,000 units, and shelf capacity of 40 products.Source: §3.1 · p. 4

3.2 Models and Agent Frameworks

• Seven contemporary LLMs are evaluated, with each model represented by its strongest completed run among available framework settings and a privileged oracle as a non-agent reference.Frameworks include ReAct, Reflection, and Plan-and-Act when available; GPT-5.5 is evaluated only under ReAct because of API cost constraints.Source: §3.2 · p. 4

3.3 Metrics

• Operational viability and sales metrics include final net worth, survival days, cumulative units sold, distinct products sold per day, and return ratio.Higher values are preferred for final net worth, survival days, sales, and daily products sold; lower return ratios are preferred.Source: §3.3 · pp. 4–5
• Inventory metrics track stockout days and expired-unit share, while tool-call and token measures characterize interaction behavior and resource use.Tool-use metrics are diagnostic rather than direct optimization objectives; fewer expired units and stockout days and lower token use are preferred.Source: §3.3 · pp. 4–5

4.1 Main Benchmark Results

• The reported LLM results select each model’s longest-surviving rollout, using final net worth and total sales as tie-breakers; the oracle is a privileged reference.This protocol emphasizes operational viability and does not make the oracle a directly comparable language-agent baseline.Source: §4.1 · p. 5

4.2 Performance

• 58–180 days is the survival range across selected LLM runs, averaging roughly 107 days; DeepSeek-V4-Pro and GPT-5.5 reach the full 180-day horizon.Kimi-K2.6 survives 130 days, while the remaining selected runs end after 58–73 days.Source: §4.2 · p. 5
• 131,510.42 final net worth and 267,998 total sales for the oracle exceed the best selected LLM results by 107,159.44 in net worth and 103,581 in sales.GPT-5.5 has the highest selected LLM net worth at 24,350.98; DeepSeek-V4-Pro has the highest selected LLM sales at 164,417.Source: §4.2 · p. 5

5 Analysis

• The four diagnostics cover product selection, evidence acquisition, action conversion, and temporal follow-up across the retail-management loop.Together, they examine choosing products, gathering decision-relevant evidence, translating it into actions, and revisiting delayed consequences.Source: §5 · p. 5

5.1 Incomplete Evidence Acquisition

• GPT-5.5 reaches query depth 0.9922 and evidence completeness 0.9689, while Grok-4.3’s evidence completeness is 0.0000.Agents often order or update prices without inspecting demand, inventory, supplier conditions, reviews, and external events; stronger evidence acquisition generally aligns with better outcomes, but does not guarantee effective decisions.Source: §5.1 · p. 6

5.2 Surface-Level Decision Making

• The oracle’s mean raw price distance is 1.98%, versus 17.92% for GPT-5.5; selected LLM mean supplier-quality ranks range from 2.55 to 3.89.Agents more often choose the cheapest supplier than the raw-quality-best one: QualityFirst is 21.5% across LLM runs versus 55.6% for PriceFirst.Source: §5.2 · p. 6

5.3 Lack of Consistent Long-Horizon Policy

• Selected LLM runs act on 0.95–7.89 products daily and miss 32.86%–88.18% of high-demand opportunities; DeepSeek-V4-Pro is strongest on these diagnostics.The narrow action set reflects weak portfolio-level management: many products sold each day never receive attention.Source: §5.3 · p. 7
• Oracle follow-up is 0.9936 with 0.0040 unresolved events; DeepSeek-V4-Pro records 0.7316 follow-up and 0.0028 unresolved events, while GPT-5.5’s follow-up is 0.4443.The diagnostics measure whether acted-on products receive another action within seven days and whether stockout, return, or expiration events remain unattended.Source: §5.3 · pp. 7–8
• Agents may make reasonable local decisions yet fail to sustain store operation because they do not maintain product attention or revisit delayed consequences.The analysis identifies persistent stateful policies as necessary to revise operational decisions after feedback arrives.Source: §5.3 · p. 8

5 Analysis

• RetailBench failures cluster around incomplete evidence acquisition, weak evidence-to-action conversion, and inconsistent long-horizon policy.The authors argue that improvement requires better evidence selection and reasoning plus more reliable stateful policies.Source: §5.3 · p. 8

Retail, e-commerce, and supply-chain decision benchmarks

• Retail and supply-chain benchmarks typically isolate promotion, demand modeling, shopping, or inventory decisions, whereas RetailBench couples these decisions with feedback, events, and cash flow over time.Source: §6 · p. 8

Long-horizon agent benchmarks

• Recent long-horizon benchmarks study persistence, memory, exploration, and workflow decisions, but generally lack data-grounded retail dynamics over many days.Source: §6 · p. 8
• RetailBench complements those benchmarks with closed-loop product-level operation and diagnostics of evidence acquisition, evidence-to-action conversion, and policy consistency.Source: §6 · p. 8

7 Conclusion

• RetailBench couples pricing, replenishment, supplier selection, shelf assortment, customer feedback, external events, and financial constraints over extended episodes.The benchmark is presented as a diagnostic testbed for whether agents maintain operational state, revisit delayed consequences, and align tool calls with long-term business outcomes.Source: §7 · p. 8
• Even LLMs that survive the full horizon remain far below the oracle in net worth and sales, with recurring bottlenecks in evidence acquisition, surface-level decisions, and long-horizon policy.The paper concludes that future agents need stronger evidence selection, evidence-to-action reasoning, stateful policy maintenance, and feedback adaptation.Source: §7 · p. 8

• The single-store setting omits multi-store coordination, competitive markets, strategic supplier behavior, labor constraints, promotions, and interactions among multiple autonomous agents.Source: §8 · pp. 8–9
• Reviews, news, supplier adaptation, returns, and parts of the demand model are synthetic or rule-based, limiting RetailBench's fidelity to retail economics.The authors characterize it as a controlled stress test rather than a full-fidelity simulator.Source: §8 · p. 9
• The evaluation covers prompting-based agents under one fixed environment configuration, and selected runs characterize strongest observed behavior rather than average performance across designs or settings.Source: §8 · p. 9
• The failure analysis is diagnostic rather than causal and does not fully isolate each factor's causal contribution.Source: §8 · p. 9

• Daily transitions couple delayed deliveries, shelf assortment, realized demand, reviews and returns, news effects, inventory aging, and cash-flow accounting.The seven modules are Inventory, Suppliers & Orders, Shelf, Customer Demand, Reviews & Returns, News Events, and Finance & Records.Source: §A.1 · p. 11
• Inventory tracks stock, item age, sellable quantities, capacity, stockouts, expirations, and waiting items, constraining procurement, shelving, and sales.Source: §A.1 · p. 11
• Each product has five procurement candidates with cost, quality, and delivery-time profiles; orders arrive later when inventory capacity allows.Supplier choice also affects expenditure, delivery timing, return risk, reviews, and future availability; shelf capacity limits the customer-facing assortment.Source: §A.1 · p. 11
• Customer choice follows an MNL model, with category demand based on shelf-visible attraction and then allocated across visible products.Source: §A.1 · pp. 11–12
• Sales can generate product reviews that update later demand, while supplier-quality-dependent returns are logged and deducted from revenue.Source: §A.1 · p. 12
• News carries hidden metadata for demand- or supply-side effects, while Finance & Records tracks revenue, costs, returns, rent, cash, inventory and pending-order value, and net worth.Daily transitions also apply deliveries, sales, feedback updates, expiration, rent, cash-flow updates, and supplier or news changes.Source: §A.1 · p. 12

• RetailBench selects 96 products from 20 Dominick’s categories with the most complete and informative sales records.The observations support modeling the relationship between pricing decisions and realized demand.Source: §A.2 · p. 12
• The simulator defines products, their categories, five suppliers per product, and the subset placed on the shelf each day.Source: §A.2 · p. 12
• The state space covers product attributes and prices, inventory and pending deliveries, shelf assortment, suppliers, demand, external news, and financial accounting.Source: §A.2 · p. 12
• Agents can query store observations, use constrained analysis tools, and take actions including ordering, pricing, setting shelf products, managing notes, and ending the day.Source: §A.2 · p. 12

A.2.1 Product State Sprod; A.2.2 Inventory State Sinv; A.2.7 Financial State Sfin

• Product state tracks identifiers, descriptions, shelf lives, prices, historical sales, and review aggregates; inventory tracks unit age and removes units that exceed shelf life, with age-based depreciation in net worth.Available funds and inventory value contribute to net worth, and inventory value uses linear shelf-life depreciation.Source: §A.2.1 · p. 12, §A.2.2 · pp. 12–13, §A.2.7 · p. 14

A.2.3 Supply Chain State Ssup

• Each product has suppliers with procurement prices, quality levels, and sampled delivery times; enforced diversity provides a highest-quality option, a lowest-price-and-quality option, and intermediate price–quality choices.Procurement price tiers use Dominick’s cost signals, with observed positive correlations between price tier and quality.Source: §A.2.3 · p. 13

A.2.4 Demand Signals and Demand Generation; A.2.6 Simulator Update Equations

• Demand depends on traffic, prices, shelf-visible assortment, reviews, and news; category demand is sampled against an outside option, allocated by relative product attraction, and capped by sellable inventory.Products absent from the active shelf assortment cannot receive demand; sold units can generate reviews that affect later demand.Source: §A.2.4 · p. 13, §A.2.6 · p. 14

A.2.5 External Information Sext; A.2.6 Simulator Update Equations

• Active news events have scope, side, sign, magnitude, and duration; they affect demand utilities and/or supplier prices, with positive supply news lowering procurement cost and negative supply news raising it.News texts are grounded in financial-news articles and expanded through LLM-assisted rewriting and metadata annotation.Source: §A.2.5 · p. 13, §A.2.6 · pp. 13–14

A.2.8 Action Space Details; A.3 Environment Setting

• Typed tools separate read-only observations and constrained analysis from state-changing orders, price updates, shelf assortment changes, notes, and end-of-day transitions; evaluations use one released environment configuration.That configuration specifies initial funds of 30,000, daily rent of 600, inventory capacity of 15,000, and shelf capacity of 40 products.Source: §A.2.8 · pp. 14–15, §A.3 · p. 15

A.4 Oracle Policy

• The deterministic quality-based oracle is a privileged, non-optimal reference with direct access to structured simulator fields; its reported configuration uses ncat = 2, Cshelf = 40, and B = 6.Its rule selects historically profitable products, chooses suppliers by quality and cost, and sets prices using historical profit records.Source: §A.4 · pp. 15–16

A.5 Four-Stage Diagnostic Metrics

• The four-stage diagnostics assess product selection, evidence gathering, action conversion, and follow-up using tool traces, simulator records, and delayed events; they are retrospective, not rewards or decision-time observations.They use a seven-day follow-up window and a three-day lookback for high-demand coverage unless otherwise noted.Source: §A.5 · pp. 16–17
• Evidence-acquisition diagnostics test whether actions follow relevant same-day queries: orders require inventory, sales, supplier prices, and quality proxies, while price changes require price, cost, inventory, and sales/profit history.QDepth measures partial evidence coverage, Completeness requires all action-critical categories, and the oracle is assigned 1 for both because its structured inputs include the required fields.Source: §A.5 · pp. 16–17
• LLM orders are more price-first than quality-first, while the oracle explicitly prioritizes supplier quality; PriceFirst and QualityFirst measure choices of the cheapest and raw-quality-best supplier.Raw supplier quality is hidden from agents, though return rates and customer ratings provide proxies; a high PriceFirst rate with low QualityFirst may increase returns and harm future reviews.Source: Figure 4 · p. 17, §A.5 · p. 17
• Stage 4 tracks whether acted-on products receive another state-changing action within seven days and whether delayed stockout, return, or expiration events prompt a later query or action.Additional measures include revisit rates, response latency, repeated events without intervention, and adjacent-day action continuity, distinguishing one-shot daily policies from policies maintaining product-level state across delayed feedback.Source: §A.5 · p. 17