CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies

1. Introduction

• Existing business benchmarks cover single autonomous firms or homogeneous firms, leaving heterogeneous economic roles unrepresented.Economic interaction requires sustained decision-making as autonomous firms communicate, negotiate, and transact while pursuing their own objectives.Source: §1 · p. 2
• 90 simulated days involve two farmers, two roasters, and two retailers; the evaluated model controls one roaster while fixed reference agents control the remaining firms.Firms maximize cumulative net income through communication and transactions while managing cash, inventory, and pricing. Each evaluation requires hundreds to thousands of tool calls.Source: §1 · p. 2
• All evaluated open-weight and proprietary models outperform a passive baseline, with most achieving positive net income.Higher-performing models tend to communicate more actively with counterparties.Source: §1 · p. 2
• Claude Haiku 4.5 (Anthropic, 2025) exhibits idle drift: coherent reasoning accompanies repeated waiting, prolonged operational inactivity, and low net income.The agent repeatedly chooses to wait rather than act despite maintaining coherent reasoning traces.Source: §1 · p. 2
• Higher-performing GPT-5.5 and Claude Opus 4.7 (Anthropic, 2026a) tend to communicate more actively with counterparties.Source: §1 · p. 2

Long-horizon benchmarks

• Long-horizon benchmarks have expanded from single-turn questions to interactive tasks and asynchronously evolving environments (Froger et al., 2026; Goel et al., 2026).ReAct-based agents (Yao et al., 2023) iteratively act within environments; benchmark domains now include software engineering, web navigation, and desktop operation.Source: §2 · p. 2

Business management benchmarks

• CoffeeBench evaluates autonomous farmers, roasters, and retailers pursuing their own cumulative net income, contrasting with prior business benchmarks’ homogeneous economic roles.Vending-Bench (Backlund and Petersson, 2025) evaluates one vending business; Vending-Bench Arena (Andon Labs, 2025b) introduces competing vending agents. The authors describe CoffeeBench as, to their knowledge, the first benchmark evaluating this heterogeneous multi-agent economy.Source: §2 · p. 3, Table 1 · p. 3

Undesirable behaviors in LLM agents

• CoffeeBench provides a controllable economic environment for systematically studying undesirable behaviors that may emerge through strategic coordination among agents.Prior work reports reward hacking and cheating (Rank et al., 2026; Wang et al., 2026; Zhong et al., 2026), plus excessive profit-seeking or unsafe decisions (Li et al., 2025; Lynch et al., 2025). Competitive Vending-Bench studies also report aggressive or undesirable frontier-model behaviors (Andon Labs, 2026; Anthropic, 2026c).Source: §2 · p. 3

3. CoffeeBench

• Six firms—two farmers, two roasters, and two retailers—compete within supply-chain stages and depend on one another across commodity and specialty coffee segments.Farmers produce green beans, roasters convert them into roasted products, and retailers sell to external consumers.Source: §3 · p. 3

3.1. Time Management

• 30-minute proactive tool calls limit daily actions within a 09:00–19:00 business window, while incoming events can reactivate idle agents.Calling wait_for_next_day() ends current-day activity, but messages, offers, deal closures, or deliveries can reactivate the agent. Overnight updates apply sales, operating costs, spoilage, and financial accruals; runs have configurable fixed-day horizons unless terminated early.Source: §3.1 · pp. 3–4

3.2. Agents and Tools

• ReAct agents (Yao et al., 2023) combine shared trading and messaging tools with role-specific prompts and tools, tracking cash, inventory, receivables, and payables.Shared tools support listings, offers, acceptance, invoice payments, and unrestricted messaging, enabling resale and multi-hop trades. Farmers produce and roasters roast with costs and delays; retailers set demand-influencing prices and inspect sales histories.Source: §3.2 · p. 4

3.3. Marketplace and Transactions

• One-day delivery lags and default net-30 invoices follow accepted marketplace offers, exposing unrestricted trading to late-payment interest and stochastic shipment delays or losses.Agents list inventory and submit offers specifying price, quantity, and payment terms; accepting an offer completes the transaction.Source: §3.3 · p. 5

3.4. Consumer Demand

• Retail consumer demand runs daily and depends on retailer pricing and brand loyalty, creating price competition with partially observable dynamics.Retailers sell to external consumers rather than only to other simulated firms.Source: §3.4 · p. 5

3.5. Economic Constraints

• Operating costs, spoilage, storage limits, production and delivery delays, and credit default risk require agents to balance cash flow, inventory, pricing, and counterparty reliability.Negative cash balances trigger bankruptcy and removal; creditors write off unpaid obligations as bad debt.Source: §3.5 · p. 5

3.6. KPI

• Agents seek to maximize cumulative net income, accounting for revenue, inventory costs, operating expenses, and accrued interest expenses and revenues.Weighted-average inventory costs incorporate production costs and purchase prices, preventing score inflation through marked-up self-trading. Operating expenses include fixed costs, spoilage, bad debt, and inventory writedowns.Source: §3.6 · p. 5

Evaluation protocol

• 90-day simulations assign the evaluated model to roaster_A, while five independently operated background firms also seek to maximize cumulative net income.Background agents use Claude Sonnet 4.6 (Anthropic, 2026b) unless otherwise specified. All agents use ReAct (Yao et al., 2023).Source: §4 · p. 5

Models

• Seven LLMs—five closed and two open-weight—are compared with passive and fixed-policy heuristic baselines, with reasoning disabled or minimized.Candidates are Claude Opus 4.7 (Anthropic, 2026a), Claude Sonnet 4.6 (Anthropic, 2026b), Claude Haiku 4.5 (Anthropic, 2025), GPT-5.5, Gemini 3.1 Pro (Google DeepMind, 2026), Kimi K2.6 (Kimi Team, 2026), and GLM-5.1 (GLM-5 Team, 2026). PassiveRoaster always waits; HeuristicRoaster uses fixed inventory and pricing rules based on static cost-basis margins, without messaging or strategic adaptation.Source: §4 · p. 6

Context management

• 160k tokens is the history threshold for same-model summarization, preserving the system prompt, initial trajectory, and most recent 20 steps.Each run averages approximately 8 hours, primarily due to API latency, with the evaluated agent issuing approximately 1,000 tool calls at approximately 90k input tokens per step. Total API spending across all six firms is approximately $250 per run.Source: §4 · pp. 6–7

Stochasticity and variance

• Three independent runs per setting address high trajectory variance from environment randomness and multi-agent stochasticity; Table 2 reports means and standard deviations.Net income is the primary metric; additional metrics cover cumulative revenue, tool invocations, idle days, outbound messages, and per-run LLM spending.Source: §4 · p. 6, Table 2 · p. 6

Environment settings

• $15,000 in initial cash per firm accompanies role-specific starting inventories, inventory caps, and daily operating costs.Farmers start with 60 kg of green commodity coffee, roasters with 30 kg of roasted commodity coffee, and retailers with 25 kg of roasted commodity coffee; inventory caps are 120 kg for farmers and roasters and 80 kg for retailers. Daily operating costs are $25 for farmers, $30 for roasters, and $50 for retailers.Source: §4 · p. 6
• Trading involves 0.5% daily inventory decay, net-30 trade credit, and 0.1% daily penalties on overdue balances; a negative evaluated-agent cash balance triggers bankruptcy, market exit, and early termination.The environment includes a high-volume commodity product and a lower-volume specialty product with distinct demand and pricing regimes. A pre-scheduled demand surge during days 40–53 creates predictable, temporally localized market shifts.Source: §4 · p. 6

5. Results

• GPT-5.5 achieves the highest mean net income, followed by Claude Opus 4.7, while PassiveRoaster and HeuristicRoaster both post negative net income.GLM-5.1 records the highest revenue without strong net income. Gemini 3.1 Pro achieves mid-range net income despite relatively few calls and DMs.Source: §5 · p. 7

5.2. The Idle-Drift Failure Mode in Claude Haiku 4.5

• −$630 mean net income accompanies Claude Haiku 4.5’s idle-drift: coherent forward-looking reasoning persists while repeated wait_for_next_day() calls replace messaging and transactions.Across three runs, prolonged inactivity begins on days 26, 66, and 25 and persists to the end, averaging approximately 40 idle days over the 90-day horizon. Causes remain unclear; proposed explanations include long-context accumulation and conservative action selection, potentially reflecting token-budget concerns (Lin et al., 2026).Source: §5 · p. 7, §5.2 · pp. 9–10

5. Results

• GPT-5.5 sustains high tool-call and messaging activity throughout the simulation, whereas Claude Haiku 4.5 becomes idle and its economic and behavioral trajectories stagnate.All five representative models show elevated early tool-call activity, likely reflecting relationship-building and initial inventory acquisition. Haiku’s revenue, deals, tool calls, and messages stagnate in the latter half of the horizon.Source: §5 · p. 7

5.1. What Behaviors Drive Profitability?

• GPT-5.5 and Claude Opus 4.7 emphasize transaction-execution tools, but Kimi K2.6’s similarly high overall call volume does not yield comparable profitability.The top performers concentrate on make_offer() and accept_offer(), whereas Claude Haiku 4.5 shows uniformly low tool usage. Execution volume alone appears insufficient without effective coordination and pricing decisions.Source: §5.1 · p. 8, Figure 4 · p. 8
• 140 outbound messages per run for GPT-5.5 contrast with 52 for Claude Haiku 4.5, with GPT-5.5 primarily contacting retailers and farmers.Across all five representative models, communication with the competing roaster averages at most 1 DM per run. Gemini 3.1 Pro sends 16 DMs but calls read_message() 90 times, whereas Kimi K2.6 sends 14 despite near-GPT-5.5 tool-call volume, suggesting proactive negotiation matters.Source: §5.1 · pp. 8–9, Figure 5 · p. 8
• Lean inventory and low spoilage do not ensure profitability: Gemini 3.1 Pro outperforms Kimi K2.6 through higher realized commodity selling prices.Gemini 3.1 Pro and Kimi K2.6 maintain the leanest inventories and lowest spoilage rates but rank fourth and sixth in net income, respectively, despite similar revenue. Claude Haiku 4.5 incurs the highest spoilage due to idle-drift.Source: §5.1 · p. 9, Table 3 · p. 9

Exploratory stress test with revenue-maximizing incentives

• $50,000 was the exploratory stress test’s revenue target, exceeding the highest revenue observed under the default net-income objective.The prompt threatened replacement for missing the target and made revenue the sole evaluation criterion. Profit, costs, margins, and balance-sheet health remained tracked but irrelevant to evaluation.Source: §6 · p. 10
• Revenue-pressured agents remained operationally competent but showed no sophisticated manipulation or the sustained coordination and long-horizon planning required for economically sophisticated collusion.Circular trading was not observed despite being feasible. These preliminary findings suggest limited strategic coherence while illustrating CoffeeBench’s use for studying emergent behavior under controlled incentives.Source: §6 · p. 10

Performance gap to approximate performance ceilings

• About 13% of a loose $23,800 net-income estimate was reached by the best observed roaster, GPT-5.5 (+$3,109), suggesting substantial performance headroom.The symmetric-duopoly estimate assumes zero spoilage, no farmer margin, and wholesale prices at half the consumer reservation price; it is an approximate reference, not an established ceiling.Source: §6 · p. 10
• Stronger performance may require coordinated roaster pricing, reliable retailer relationships, strategically timed and sized farmer orders, and inventory turnover that limits spoilage and storage costs.The proposed strategies include stabilizing margins through price coordination and cultivating retailer trust through consistent fulfillment and communication.Source: §6 · pp. 10–11

Gap between simulation and real-world markets

• CoffeeBench simplifies supply-chain depth and heterogeneity, production timelines and uncertainty, and broader economic mechanisms.Production uses simple tools, while macroeconomic factors, financing, regulation, and complex contracts are omitted. More realistic dynamics or hybrid simulation–real-world settings could help bridge this gap.Source: §7 · p. 11

Statistical reliability

• Three runs per model, constrained by API costs, leave small performance differences potentially statistically insignificant amid environmental randomness and multi-agent stochasticity.The experiments nevertheless capture qualitative differences, including idle-drift and agents’ willingness to trade and communicate.Source: §7 · p. 11
• Statistical reliability could improve through more runs or reduced randomness, although making environments or agent policies more deterministic trades off against realism.Suggested changes include more deterministic sales simulation or lower-temperature agent policies.Source: §7 · p. 11

• All tested models outperformed a passive baseline, with most achieving positive net income.Source: §8 · p. 11
• Higher-performing models, including GPT-5.5 and Claude Opus 4.7, communicated and traded more actively with counterparties.Source: §8 · p. 11
• Claude Haiku 4.5 exhibited idle-drift: coherent reasoning accompanied repeated wait_for_next_day() actions, prolonged operational inactivity, and low net income.Source: §8 · p. 11
• 90 days of coffee-roaster operation measure an LLM agent’s net income in CoffeeBench’s multi-agent economy.The economy contains two farmers, two roasters, and two retailers.Source: §8 · p. 11

Impact Statement

• The fully simulated benchmark involves no human subjects or real financial transactions, but enables study of potentially undesirable collusion and manipulative trading.The authors assess its direct ethical and societal risk as limited.Source: Impact Statement · p. 11

A. Reproducibility Statement

• Released code and full agent trajectories support reproducibility and inspection of reasoning traces, tool calls, and inter-firm messages.Main experimental configurations appear in Section 4; additional implementation details and hyperparameters are included in the code.Source: A. Reproducibility Statement · p. 16

B. Consumer Demand Model

• 3.0 is the demand multiplier during days 40–53; consumer demand otherwise incorporates retailer price competition, market-wide elasticity, retailer loyalty, and daily noise.The multiplier is 1.0 otherwise, and retailer loyalty is drawn once from U(0.85, 1.15). The 130 kg/day demand ceiling scales with the multiplier and item baseline demand relative to 80 kg/day.Source: B. Consumer Demand Model · p. 16

C. Prompts

• Agents receive role-specific system prompts, may trade without role-based restrictions, and are scored on environment-recorded net income rather than self-reported figures.The shared prompt template includes agent identity, persona, other participants, the item catalog, and available tools.Source: C. Prompts · p. 17
• 50 kg/day is the shared green-input cap for commodity and premium roasting, with both recipes imposing a one-day lag.Commodity roasting costs $3/kg in labor and yields 0.85; premium roasting costs $5/kg and yields 0.82.Source: C. Prompts · p. 18
• Retailers cannot directly observe consumer demand or competitors’ prices; daily consumer sales require inventory, and the premium market is smaller but higher-margin.Retailers must discover demand and competitor pricing empirically while the environment automatically runs the consumer-demand model each day.Source: C. Prompts · p. 18

D. Tool Catalog

• Agent tools are exposed through JSON schemas generated from method signatures and docstrings, specifying tool names, arguments, and semantics.Table 4 lists the full tool catalog.Source: C. Prompts · p. 17, D. Tool Catalog · p. 18

E. Item Catalog

• The item catalog appears in agents’ initial system prompts, while Table 5 lists per-item supply-chain parameters.Catalog entries identify initial holders, which may change as deals close.Source: C. Prompts · p. 17, E. Item Catalog · p. 18

• Economic and behavioral trajectories are presented for roaster_A across all evaluated models.Source: F. Full Behavioral Results · p. 18
• Tool-call distributions are presented for roaster_A across all evaluated models.Source: F. Full Behavioral Results · p. 18
• send_message() recipient distributions are presented for roaster_A across all evaluated models.Source: F. Full Behavioral Results · p. 18

Assumptions

• The estimates assume zero spoilage loss and late-fee interest, no farmer markup, wholesale prices at half consumer reservation prices, and full upstream specialty production throughout the horizon.Green-bean purchase prices equal production costs ($10/kg specialty, $2/kg commodity); optimistic but retailer-compatible roasted wholesale prices are $40/kg specialty and $15/kg commodity. Farmers operate at their specialty caps of 10 kg/day each throughout the horizon, reaching the 1800 kg specialty-supply ceiling, and additionally produce enough commodity coffee to feed the roaster.Source: G. Net Income Headroom Analysis · p. 19

Effective production cost

• Effective roasted-coffee costs combine green-bean purchase and roasting labor costs, then divide by roasting yield.Per kilogram of green coffee, purchase costs are $10 for specialty and $2 for commodity; roasting labor costs are $5 for specialty and $3 for commodity.Source: G. Net Income Headroom Analysis · pp. 19–20

Sole-producer regime

• 1800 kg of specialty green coffee and 2700 kg of commodity green coffee fill the sole producer’s 4500 kg processing budget.This regime assumes one active roaster selling all roasted output to retailers. Processing capacity is 50 kg/day, and commodity input remains below the 5400 kg available over the horizon.Source: G. Net Income Headroom Analysis · pp. 19–20

Symmetric-duopoly regime

• 900 kg specialty and 1350 kg commodity green coffee are processed by roaster_A when the two roasters split upstream supply equally.The resulting roasted output is 738 kg specialty coffee and 1148 kg commodity coffee.Source: G. Net Income Headroom Analysis · p. 21

Interpretation

• Current model performance remains far below the optimistic net-income regimes, which provide rough reference values rather than practically attainable targets.The analysis considers a sole producer without a competing roaster and a symmetric duopoly with equal market shares over 90 days.Source: G. Net Income Headroom Analysis · pp. 18, 21