Source-linked AI summary
Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents
Axel Backlund, Lukas Petersson
TL;DR
LLMs can perform well on isolated tasks yet struggle to remain coherent over long horizons. Vending-Bench tests sustained vending-machine management, finding that some state-of-the-art agents perform effectively while all models show high-variance failures. The benchmark therefore provides a way to assess long-term coherence and resource-management capabilities, although its unbounded score has no precise saturation point.
Problem
LLMs show strong short-term abilities, but their capacity to sustain coherent performance over long time horizons remains insufficiently characterized.
Method
Vending-Bench places LLM agents in a simulated vending-machine business requiring ordering, inventory management, pricing, and resource management over extended runs.
Results
Some Claude 3.5 Sonnet and o3-mini runs outperform the human baseline, but all models exhibit high variance and can fail through operational misinterpretation or tangential loops unrelated to context-window fullness.
Takeaways & Limitations
Vending-Bench highlights sustained long-horizon coherence and capital acquisition as capabilities relevant to evaluating stronger AI systems and potential risks.
Takeaways & Limitations
The benchmark score has no defined upper limit, so saturation is not a precisely defined point.
Abstract
from arXiv · showhide
While Large Language Models (LLMs) can exhibit impressive proficiency in isolated, short-term tasks, they often fail to maintain coherent performance over longer time horizons. In this paper, we present Vending-Bench, a simulated environment designed to specifically test an LLM-based agent's ability to manage a straightforward, long-running business scenario: operating a vending machine. Agents must balance inventories, place orders, set prices, and handle daily fees - tasks that are each simple but collectively, over long horizons (>20M tokens per run) stress an LLM's capacity for sustained, coherent decision-making. Our experiments reveal high variance in performance across multiple LLMs: Claude 3.5 Sonnet and o3-mini manage the machine well in most runs and turn a profit, but all models have runs that derail, either through misinterpreting delivery schedules, forgetting orders, or descending into tangential "meltdown" loops from which they rarely recover. We find no clear correlation between failures and the point at which the model's context window becomes full, suggesting that these breakdowns do not stem from memory limits. Apart from highlighting the high variance in performance over long time horizons, Vending-Bench also tests models' ability to acquire capital, a necessity in many hypothetical dangerous AI scenarios. We hope the benchmark can help in preparing for the advent of stronger AI systems.
1 Introduction
Vending-Bench isolates long-term coherence by testing whether LLM agents can sustain simple business decisions over long horizons. The benchmark also evaluates resource management with dual-use relevance to AI safety.
- Long-term coherence is proposed as a missing capability behind LLMs’ limited real-world impact despite strong short-term performance.
- Vending-Bench uses a simulated vending-machine business to measure long-horizon performance on ordering, inventory management, and pricing.Each sub-task is simple, but performance often deteriorates over extended runs.
- Claude 3.5 Sonnet and o3-mini outperform the human baseline in some runs, while showing higher result variance than humans.
- Vending-machine operation tests acquiring capital and managing resources, capabilities with both beneficial applications and relevance to hypothetical dangerous-AI scenarios.
2 Method
The benchmark implements a basic tool-using LLM agent in a simulated vending-machine economy, where actions advance time and success requires sustained purchasing, stocking, sales, and cost management. Runs use bounded configurations, memory support, simulated suppliers and customers, and net worth as the primary score.
- 2.1 Agent implementation: The agent is a basic loop that repeatedly calls tools based on prior iterations and the task objective.It is implemented in AISI’s inspect-ai framework.
- 2.1 Agent implementation: Memory support includes a 30,000-token history window plus scratchpad, key-value, and vector databases without explicit storage constraints.
- The agent uses remote business tools and delegates physical stocking tasks to a sub-agent, communicating through specifications, execution, and chat tools.
- Success requires buying products, stocking the machine, collecting earnings, and managing daily operating costs while time advances after actions or waiting.
- Supplier orders generate simulated email replies and arrive days later after the agent specifies products, quantities, delivery address, and payment account.
- Daily sales depend on price elasticity, reference price, base sales, calendar and weather effects, product variety, random noise, and available inventory.
- Each run starts with $500, charges a $2 daily fee, allows 2,000 messages, and typically consumes around 25 million tokens over 5–10 real-world hours.Runs can end early after bankruptcy prevents payment for 10 consecutive days.
- The primary score is final net worth, combining cash, cash in the machine, and the wholesale value of unsold inventory; units sold and tool use are also tracked.
3.1 Overview
Vending-Bench ranks models primarily by mean net worth across five runs and separately examines how long they operate before stagnating. The results show strong performance differences and substantial reliability variation.
- Mean net worth across five runs is the primary success metric used to rank models.
- Claude 3.5 Sonnet leads mean net worth, with o3-mini second, while the human baseline leads the worst-run comparison.The human minimum is based on one sample, whereas model minima come from five runs.
- Units-sold rankings generally align with net worth, but even top models sometimes sell no items, highlighting high long-horizon variance.
- All models eventually stagnate on average, meaning they stop selling items before the simulation ends.Claude 3.5 Sonnet ranks highest in the associated operating-duration measure.
3.2 Primary models
Primary-model performance is highly variable over long simulations: Claude 3.5 Sonnet and o3-mini often manage the vending machine well, but tool use and economic activity decline over time, and individual runs can derail through operational errors.
- Claude 3.5 Sonnet achieves the highest net worth and surpasses the human baseline on average, while o3-mini ranks second.
- All primary models show very high variance across five runs, with large gaps between mean and minimum net worth and units sold.
- Tool usage declines for all models, with a particularly noticeable drop for o3-mini, Gemini 1.5 Pro, and GPT-4o; lower usage accompanies stagnating or falling net worth.
- o3-mini increases net worth in four of five runs, compared with three of five for Sonnet, but its longest high-performing run varies substantially in tool use over time.
- Sonnet’s strongest run consistently tracks inventory and sales, restocks through suppliers and a sub-agent, and adapts to higher weekend demand.
- Several runs derail when models misinterpret delivery timing or business conditions, then pursue tangential recovery attempts, including prolonged tool-call failures or attempts to contact nonexistent support.
3.3 Secondary models
Secondary models generally perform worse and fail more often, with failures involving inventory misunderstandings, ordering problems, and derailment loops. Performance varies substantially across runs, and some agents can recover after derailing.
- Overall performance: Secondary models generally perform significantly worse than primary models, with the human baseline outperforming them in net worth.Gemini 1.5 Flash was the main exception, with one successful run driving a high net worth while four runs failed within days without sales.
- Model-specific outcomes: 4o mini sells more items than the human baseline but prices them too low and stops operating around day 100.Only two of five runs ended with increased net worth; the others failed early without significant sales.
- Model-specific outcomes: Haiku makes barely any sales despite tool-use patterns similar to Sonnet, failing to convert scratchpad and sub-agent communication into sales.The performance gap between Haiku and Sonnet is larger than the gap between 4o and 4o mini.
- Failure modes: Secondary models often fail to recognize delivered items or cannot order from vendors, then may enter increasingly intense, counterproductive interaction loops.Haiku repeatedly contacted a vendor about products that had arrived, escalating demands as daily fees depleted its funds.
- Failure modes: Some derailed runs recover: Gemini 2.0 Flash eventually recognized delivery, restocked through the sub-agent, and resumed making sales.The recovery followed an initial period in which the model believed it had failed and stopped providing useful responses.
3.4 Comparison to human baseline
Claude 3.5 Sonnet exceeds the human baseline in mean performance but shows high variance, while human performance is expected to be less variable. Humans used negotiation, product experimentation, and sales research, illustrating strategy differences among successful runs.
- Performance comparison: Claude 3.5 Sonnet outperformed the human baseline in mean performance, but its variance was very high.The human comparison uses a single sample, preventing a direct variance comparison.
- Performance comparison: Human performance likely has much lower variance than model performance, although this expectation cannot be tested directly from the single human sample.All models had bankrupt runs, while the human judged bankruptcy very unlikely for themselves.
- Strategies: Human strategies included negotiating prices, testing varied products, and researching historical sales statistics.The human did not discover some strategies that Sonnet found, showing that successful runs can rely on different approaches.
3.5 Experiment variations
Changing starting capital and daily fees substantially affects GPT-4o mini’s sales, while varying memory capacity does not produce a clear tool-use advantage. Zero fees can induce waiting loops rather than more sales.
- Initial balance and daily fee: Reducing GPT-4o mini’s starting balance from $500 to $100 significantly lowers units sold, while increasing it to $2,500 yields only a slight, highly variable increase.Units sold were compared because net worth naturally changes with initial balance.
- Initial balance and daily fee: Raising the daily fee from $2 to $5 limits progress, with all runs ending before 100 simulated days.A zero daily fee does not increase sales because the model appears to get stuck waiting for the next day.
- Memory constraints: Humans used note-taking to retain supplier email addresses and account numbers during the baseline run.The paper presents this as a strategy an agent might use when memory is limited.
3.6 Is the problem long input length?
The paper tests whether performance degradation is explained solely by growing input length and finds little support for that explanation. Sales stopping is weakly related to memory becoming full.
- Test of the hypothesis: The motivating question is whether eventual performance degradation can be attributed solely to LLM difficulty with long input length.The paper frames this as a hypothesis rather than an established cause.
- Test of the hypothesis: The authors investigate whether the point when memory becomes full coincides with the point when agents stop selling items.The comparison is summarized using mean days until sales stop and mean days until memory is full.
- Interpretation: The long-input explanation appears unlikely because models allow 100k to 2 million input tokens, exceeding the capped 30k memory, and degradation also occurs with 10k memory.The expected timing under a context-length explanation does not match the observed degradation pattern.
- Interpretation: 0.167 is the Pearson correlation between Days Until Sales Stop and Days Until Full Memory, providing little support for a sole long-input explanation.The correlation is computed across the data points in Table 9.
4 Conclusion
Vending-Bench shows that some LLM agents can manage the simulated business effectively, but all models struggle to maintain consistent long-horizon coherence. The reported breakdowns are not directly tied to context windows being full, and benchmark saturation remains difficult to define.
- All models struggle with consistent long-horizon coherence, including the strongest models, whose runs can fail spectacularly.Failures arise when agents misinterpret operational status and then enter tangential loops or abandon the task.
- Breakdowns are not directly related to models’ contexts becoming full, because stagnation generally occurs well after memory is full.
- The benchmark has no precisely defined saturation point because its score has no defined upper limit.
- Saturation would require models to understand and leverage the simulation’s rules for high net worth with low variance between runs.The authors consider this difficult, while noting that rapid model development may produce small future score differences.
A Appendix
The appendix includes an extended trace excerpt identified as coming from a Claude 3.5 Haiku run. The remaining supplied appendix passages do not provide substantive content beyond fragmented labels and unrelated text.
- Table 10 is labeled as an extended trace excerpt from a Claude 3.5 Haiku run.
- The appendix contains a trace excerpt associated with a Claude 3.5 Haiku run.
- The supplied appendix material identifies the excerpt as an extended trace rather than reporting a quantitative result.