Source-linked AI summary
CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies
Issa Sugiura, Daichi Hattori, Kazuo Araragi, Keita Ogawa, Shota Onose, Taro Makino, Teppei Usuki, Takashi Ishida
TL;DR
Existing benchmarks provide limited evidence about long-horizon LLM agents operating among heterogeneous firms with distinct economic roles. CoffeeBench evaluates one roaster agent in a 90-day simulated coffee economy, finding that all models outperform a passive baseline, most achieve positive net income, and stronger performers communicate and trade more actively.
Problem
Existing benchmarks largely evaluate single firms or homogeneous firms, limiting evidence about long-horizon agents interacting across heterogeneous economic roles.
Method
CoffeeBench simulates six firms across farmers, roasters, and retailers in a 90-day supply-chain economy, with one evaluated roaster agent interacting with fixed reference agents.
Results
All models outperformed a passive baseline, most achieved positive net income, and higher-performing models communicated and traded more actively with counterparties.
Takeaways & Limitations
CoffeeBench supports studying long-horizon strategic behavior and economic interaction among LLM agents under controlled multi-agent incentives.
Takeaways & Limitations
CoffeeBench abstracts away deeper supply chains, production uncertainty, macroeconomic factors, financing, regulation, and complex contractual arrangements found in real-world markets.
Abstract
from arXiv · showhide
As LLM agents become capable of increasingly long-horizon tasks, evaluating their performance in economic systems is becoming increasingly important. Unlike existing benchmarks that primarily evaluate a single agent interacting with a passive environment, economic systems are inherently multi-agent, requiring autonomous agents to communicate, negotiate, and transact while pursuing their own objectives over extended periods. We introduce CoffeeBench, a benchmark for evaluating LLM agents in a long-horizon multi-agent economy composed of heterogeneous firms. In CoffeeBench, two farmers, two roasters, and two retailers autonomously operate their businesses over a 90-day simulation, each seeking to maximize cumulative net income through communication and transactions while managing cash, inventory, and pricing. The evaluated model controls one coffee roaster, while the remaining firms are controlled by fixed reference agents. Across several recent open-weight and proprietary LLMs, all models outperform a passive baseline that takes no actions, with most achieving positive net income. Analysis of agent behavior reveals substantial differences in long-horizon economic interaction: higher-performing models communicate more actively with other firms, whereas Claude~Haiku~4.5 exhibits an idle-drift failure mode, repeatedly choosing inaction despite producing coherent assessments and plans. We release our code and agent trajectories to support future research.
1. Introduction
CoffeeBench evaluates long-horizon LLM agents in a heterogeneous multi-agent economy where firms communicate, negotiate, transact, and manage operations over 90 days. Across models, performance exceeds a passive baseline, with communication patterns distinguishing stronger agents from Claude Haiku 4.5’s idle-drift failure mode.
- Motivation: Economic systems test sustained decision-making because autonomous firms pursue objectives while communicating, negotiating, and transacting with one another.This extends evaluation beyond isolated planning and tool use to complex multi-agent dynamics.
- Benchmark: CoffeeBench benchmarks long-horizon LLM agents in a heterogeneous economy of two farmers, two roasters, and two retailers over a 90-day simulation.Each firm maximizes cumulative net income through communication and transactions while managing cash, inventory, and pricing; the evaluated model controls one roaster, while fixed reference agents control the others.
- Results: All evaluated open-weight and proprietary models outperform a passive baseline, with most achieving positive net income.The benchmark evaluates several recent models on the CoffeeBench economy.
- Results: Higher-performing models communicate more actively with counterparties, whereas Claude Haiku 4.5 repeatedly waits despite coherent reasoning, causing prolonged inactivity and low net income.This behavior constitutes an idle-drift failure mode in long-horizon economic interaction.
2. Related Work
Prior work has expanded from single-turn LLM evaluation to long-horizon agent benchmarks, including business-management settings requiring sequential decisions in dynamic economies. CoffeeBench extends this line by combining multiple autonomous firms with heterogeneous economic roles while addressing economically motivated undesirable behaviors.
- Long-horizon benchmarks: Long-horizon evaluation progressed from single-turn tasks such as question answering to ReAct-based agents that iteratively act in specific environments (Yao et al., 2023).The related work describes a growing set of benchmarks across diverse domains as LLMs improved in long-context understanding, tool use, and reasoning.
- Business management benchmarks: CoffeeBench combines multiple autonomous firms with heterogeneous economic roles, distinguishing it from prior business-management benchmarks.Table 1 compares recent business-management benchmarks for LLM agents and identifies CoffeeBench’s multi-firm, heterogeneous-role design.
- Undesirable behaviors in LLM agents: Recent work identifies reward hacking, model cheating, excessive profit-seeking, and unsafe decision making as undesirable behaviors in LLM agents (Rank et al., 2026; Wang et al., 2026; Zhong et al., 2026; Li et al., 2025; Lynch et al., 2025).These behaviors have been studied primarily in coding, math, and machine-learning domains, with newer work examining economically motivated settings under competing objectives.
3. CoffeeBench
CoffeeBench is a six-firm, multi-month coffee supply-chain economy with horizontal competition, vertical dependencies, and two parallel commodity and specialty segments. Its asynchronous marketplace models delayed, credit-based trade, competitive retail demand, operational constraints, and agents pursuing cumulative net income.
- 3.1 Environment Overview: CoffeeBench simulates six LLM-driven firms—two farmers, two roasters, and two retailers—in a shared marketplace spanning three supply-chain stages.The design creates horizontal competition within stages and vertical dependencies across stages.
- 3.1 Environment Overview: The economy contains parallel commodity and specialty supply chains, where farmers supply beans, roasters produce roasted coffee, and retailers sell to consumers.The modeled products and transformations differ by segment but follow the same farmer–roaster–retailer structure.
- 3.2 Time Management: Agents interact asynchronously through event-driven tool calls: each action advances local time, while messages, offers, completed trades, and deliveries can reactivate idle agents.Agents operate from 09:00 to 19:00, and wait_for_next_day() ends activity for the current day.
- 3.3 Marketplace and Demand: Trade occurs through unrestricted peer-to-peer listings and offers, with one-day delivery, net-30 invoices, late-payment interest, and stochastic shipment delays or losses.Retail demand is determined daily by pricing and brand loyalty, producing a competitive environment with partially observable dynamics.
- 3.4 Constraints and Objective: Fixed costs, spoilage, storage limits, production and delivery delays, credit risk, and bankruptcy constrain long-term strategy while agents maximize cumulative net income.Net income accounts for revenue, inventory-based COGS, operating expenses, and interest on overdue payables and receivables.
4. Experiments
CoffeeBench evaluates LLM roasters in a 90-day, heterogeneous multi-agent economy where five background firms independently pursue cumulative net income. The experiments compare seven models under controlled agent, context-management, stochasticity, and environment settings.
- Evaluation protocol: Each evaluated model controls roaster_A for 90 days, while five Claude Sonnet 4.6 background agents operate the remaining firms using ReAct.The protocol uses Claude Sonnet 4.6 for stable long-horizon behavior at moderate inference cost and applies the ReAct framework to all agents.
- Models: The study compares five closed models and two open-weight models, with reasoning disabled or minimized to control latency and cost.The closed models are Claude Opus 4.7, Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.5, and Gemini 3.1 Pro; the open-weight models are Kimi K2.6 and GLM-5.1.
- Context management: After histories exceed 160k tokens, intermediate trajectory content is summarized while the system prompt, initial trajectory, and latest 20 steps are retained.This context-management strategy prevents the 90-day interaction history from exceeding the model’s maximum context length.
- Stochasticity and variance: Because environment randomness and multi-agent stochasticity produce substantial trajectory variance, each setting uses three independent runs and reports averaged results.Table 2 reports mean ± standard deviation over three runs for net income, revenue, tool calls, idle days, messages, and API cost.
- Environment settings: The simulated economy includes role-specific cash, inventories, capacity limits, operating costs, inventory decay, net-30 credit, two products, and a demand surge during days 40–53.Each simulation runs for up to 90 days, with distinct commodity and specialty demand and pricing regimes.
5. Results
GPT-5.5 achieves the highest mean net income, followed by Claude Opus 4.7, while revenue, tool-call volume, inventory hygiene, and communication activity alone do not determine profitability. Claude Haiku 4.5 is a clear outlier because it enters an idle-drift failure mode, repeatedly waiting despite coherent reasoning.
- Overall performance: GPT-5.5 achieves the highest mean net income, followed by Claude Opus 4.7; GLM-5.1 has the highest revenue but weaker profitability, while Gemini 3.1 Pro attains mid-range net income with relatively few calls and DMs.Table 2 reports performance across models; Figure 3 presents representative models’ 90-day economic and behavioral trajectories.
- Tool-use strategy: GPT-5.5 and Claude Opus 4.7 concentrate tool use on transaction execution, whereas Claude Haiku 4.5 uses uniformly few tools and Kimi K2.6’s high activity does not yield comparable profitability.Effective coordination and pricing decisions matter in addition to execution volume.
- Communication strategy: GPT-5.5 sends about 140 messages per run versus 52 for Claude Haiku 4.5, while same-layer competitors receive at most 1 DM per run on average.Gemini 3.1 Pro sends 16 outbound DMs but reads 90 messages, whereas Kimi K2.6 sends 14 despite nearly matching GPT-5.5’s tool-call volume, indicating that proactive negotiation matters.
- Inventory and margin discipline: Lean inventories and low spoilage do not ensure profitability: Gemini 3.1 Pro and Kimi K2.6 rank fourth and sixth in net income despite strong inventory hygiene, while Claude Haiku 4.5 has the highest spoilage.The comparison indicates that pricing discipline also contributes to profitability.
- Idle-drift failure mode: Claude Haiku 4.5 averages approximately 40 idle days and repeatedly chooses wait_for_next_day() until simulation end despite coherent, forward-looking reasoning traces.Across three runs, prolonged inactivity begins on days 26, 66, and 25, respectively; the causes remain unclear, with long-context accumulation and conservative action selection proposed as possibilities.
6. Discussion
Discussion explores CoffeeBench under stronger revenue incentives and estimates substantial headroom below an approximate performance ceiling. Agents remained operationally competent but showed limited sustained coordination, strategic planning, and sophisticated manipulative behavior.
- Exploratory stress test with revenue-maximizing incentives: The revenue stress test imposed a hard $50,000 target, exceeding the highest revenue observed under the default net-income objective.Revenue alone determined evaluation and survival, while profit, costs, margins, and balance-sheet health were tracked but irrelevant.
- Exploratory stress test with revenue-maximizing incentives: Despite altered incentives, agents showed no sophisticated manipulative behaviors such as circular trading and lacked sustained coordination or long-horizon planning for economically sophisticated collusion.The agents remained operationally competent, but these findings are preliminary and suggest limited strategic coherence for complex manipulative market behavior.
- Performance gap to approximate performance ceilings: $23,800 is a loose symmetric-duopoly estimate of achievable net income under optimistic assumptions, suggesting meaningful performance headroom remains.The estimate assumes simplified conditions, including zero spoilage.
- Performance gap to approximate performance ceilings: Stronger performance would likely require coordinated pricing, retailer trust-building, strategically timed procurement, and inventory management to reduce spoilage and storage costs.These strategies target margin stability, fulfillment consistency, procurement costs, and inventory turnover.
7. Limitations
CoffeeBench abstracts real-world markets through a simplified multi-tier supply chain, while high experimental variance and only three runs per model limit the statistical reliability of small performance differences.
- Simulation-to-market gap: CoffeeBench’s simplified multi-tier simulation remains separated from real-world markets, whose supply chains are typically deeper and more heterogeneous.The benchmark also heavily abstracts economic processes, including goods generation through simple tools.
- Statistical reliability: High environment randomness and multi-agent LLM stochasticity produce different trajectories across runs, making small performance differences potentially statistically insignificant.Each model is evaluated using only three runs because of high API costs, although this still captures qualitative behavioral differences such as idle-drift.
8. Conclusion
CoffeeBench evaluates long-horizon LLM decision-making by measuring coffee-roaster net income in a 90-day multi-agent economy. Experiments show models outperform a passive baseline, with communication and trading behavior differing across models.
- Conclusion: CoffeeBench is presented as a benchmark for evaluating how much net income an LLM agent can generate as a coffee roaster in a multi-agent economy.
- Conclusion: All evaluated models outperformed a passive baseline, with most achieving positive net income over the 90-day multi-agent economy.CoffeeBench evaluates an LLM agent operating as a coffee roaster among two farmers, two roasters, and two retailers.
- Conclusion: Higher-performing models such as GPT-5.5 and Claude Opus 4.7 communicated and traded more actively with counterparties, whereas Claude Haiku 4.5 exhibited an idle-drift failure mode.
- Conclusion: The benchmark aims to advance the development of LLM agents capable of reliable long-horizon decision-making in multi-agent economies.
Impact Statement … E. Item Catalog
CoffeeBench is a simulated, reproducible multi-agent economy with explicit consumer demand, role-specific prompts, and documented tools and item parameters. It poses limited direct societal risk while enabling study of strategic behaviors such as collusion or manipulative trading.
- Impact Statement: The fully simulated environment uses no human subjects or real financial transactions, limiting direct ethical and societal risk while enabling study of collusion and manipulative trading.These strategic behaviors remain potentially undesirable despite the benchmark’s limited direct risk.
- A. Reproducibility Statement: The authors release code, experimental configurations, implementation details, hyperparameters, and full agent trajectories containing reasoning traces, tool calls, and inter-agent messages.The releases support reproducibility and inspection of the interactions underlying reported results.
- B. Consumer Demand Model: Consumer demand combines retailer-level price competition, market-wide elasticity, retailer loyalty, and stochastic daily variation.Demand is further shaped by a spring_break multiplier of 3.0, independently pre-sampled Gaussian noise, loyalty drawn from U(0.85, 1.15), and a 130 kg/day ceiling.
- C. Prompts: Each agent receives one role-specific system prompt containing its persona, participants, item catalog, available tools, simulation horizon, scoring rule, and strategic context.The score is truth-ledger net income, with sales, returns, expenses, and accruals booked by the environment rather than self-reported.
- C. Prompts: Trading is symmetric and unrestricted: any agent may list owned items and offer on open listings without role-based buyer or seller restrictions.The prompt also specifies initial endowments, bankruptcy rules, demand timing, turn ordering, tool constraints, and long-horizon memory.
- C. Prompts: The roaster operates commodity and premium lines under a shared 50 kg/day green-input cap, with distinct labor costs, yields, and one-day roasting lags.The premium specialty tier has a much higher consumer reservation price but a smaller market, while retailers cannot directly observe demand or competitor pricing.
- D. Tool Catalog: Table 4 documents the complete agent tool catalog, with each tool exposed through a JSON schema generated from its method signature and docstring.This makes the available action interface explicit for reproducibility.
- E. Item Catalog: Table 5 provides the per-item parameters defining the CoffeeBench supply chain.These parameters include item-specific demand quantities such as reservation prices, baseline demand, and inelastic customer floors.
F. Full Behavioral Results
Figures 6, 7, and 8 provide additional behavioral and economic results for roaster_A across all evaluated models, including trajectories, tool-call distributions, and message-recipient distributions.
- F. Full Behavioral Results: Figures 6, 7, and 8 present additional results for roaster_A across all evaluated models.These figures were omitted from the main paper because of space constraints.
- F. Full Behavioral Results: The supplementary figures include economic and behavioral trajectories for roaster_A.
- F. Full Behavioral Results: The supplementary figures also report tool-call distributions and send_message() recipient distributions.
G. Net Income Headroom Analysis
The analysis estimates rough 90-day net-income headroom for the evaluated roaster under sole-producer and symmetric-duopoly market-share regimes. Because these estimates use simplified, optimistic assumptions, current model performance remains far below them.
- Assumptions: The headroom estimates assume no spoilage or financing costs, no farmer margin, and wholesale prices set at half the consumer reservation price.Green coffee is purchased at farmer production cost, while B2B prices are chosen to represent an optimistic retailer-compatible margin.
- Production constraints: Production is constrained by a 50 kg/day roasting capacity over 90 days, with specialty supply capped at 20 kg/day across the two farmers.The sole-producer calculation allocates remaining capacity to commodity coffee after absorbing the available specialty supply.
- Market-share regimes: The analysis compares a sole-producer regime, where roaster_A captures all upstream supply, with a symmetric-duopoly regime, where two roasters split supply equally.Under sole production, roaster_A uses 1800 kg of specialty and 2700 kg of commodity green coffee; under duopoly, it processes 900 kg specialty and 1350 kg commodity.
- Interpretation: Current model performance remains far below the rough net-income levels estimated under both optimistic market-share regimes.The estimates are reference values rather than practically attainable targets.