Source-linked AI summary
EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economies
Xavier Hu, Jinxiang Xia, Shengze Xu, Kangqi Song, Yishuo Yuan, Guibin Zhang, JinCheng Ren, Boyu Feng, Li Lu, Tieyong Zeng, Jiaheng Liu, Minghao Liu, He Zhu, Yuchen Eleanor Jiang, Wei Wang, Wangchunshu Zhou
TL;DR
Existing long-horizon evaluations are often episodic, narrow, or weakly grounded in persistent economic dynamics. EcoGym introduces an open benchmark with three unified interactive economies and effectively unbounded plan-and-execute horizons. Across scenarios, no single LLM consistently dominates, and models show suboptimality in either strategy or action execution.
Problem
Existing economic benchmarks are narrow, often closed, and insufficiently representative of heterogeneous, interdependent business processes.
Method
EcoGym evaluates continuous plan-and-execute decisions across Vending, Freelance, and Operation using unified interfaces, effectively unbounded horizons, latent mechanics, and economic outcomes.
Results
No single model consistently dominates across all scenarios, and models exhibit significant suboptimality in high-level strategies or efficient action execution.
Takeaways & Limitations
EcoGym provides an open, generalizable testbed for transparent evaluation of long-horizon agents in interactive economies.
Takeaways & Limitations
The framework relies on rule-based abstractions that may not fully capture irrational human behavior.
Abstract
from arXiv · showhide
Long-horizon planning is widely recognized as a core capability of autonomous LLM-based agents; however, current evaluation frameworks suffer from being largely episodic, domain-specific, or insufficiently grounded in persistent economic dynamics. We introduce EcoGym, a generalizable benchmark for continuous plan-and-execute decision making in interactive economies. EcoGym comprises three diverse environments: Vending (adapted from the closed-source Vending-Bench, with full open-source release), Freelance (new), and Operation (new), implemented in a unified decision-making process with standardized interfaces, and budgeted actions over an effectively unbounded horizon (1000+ steps if 365 day-loops for evaluation). The evaluation of EcoGym is based on business-relevant outcomes (e.g., net worth, income, and DAU), targeting long-term strategic coherence and robustness under partial observability and stochasticity. Experiments across eleven leading LLMs expose a systematic tension: no single model dominates across all three scenarios. Critically, we find that models exhibit significant suboptimality in either high-level strategies or efficient actions executions. EcoGym is released as an open, extensible testbed for transparent long-horizon agent evaluation and for studying controllability utility trade-offs in economic settings.
1 Introduction
EcoGym addresses limitations in existing long-horizon economic benchmarks by providing an open, unified, effectively unbounded testbed across three interactive economies. Experiments show that current LLMs remain suboptimal in strategic planning or action execution, with no model consistently leading across scenarios.
- Motivation: Existing economic benchmarks are narrow, often closed, and insufficiently representative of heterogeneous, interdependent business processes.Prior testbeds include vending and stylized game settings, while some use proprietary evaluation pipelines.
- Benchmark: EcoGym evaluates continuous plan-and-execute decision making in Vending, Freelance, and Operation through a unified interface.Vending is adapted from the closed-source Vending-Bench with a full open-source release, while Freelance and Operation are new.
- Benchmark: 1000+ steps can be evaluated across 365 day-loops, requiring agents to sustain strategic coherence in an effectively unbounded horizon.The environment also includes latent economic mechanics that agents must discover while managing resources sustainably.
- Results: No single model consistently dominates all economic scenarios, while models remain significantly suboptimal in high-level strategies or efficient action execution.The evaluation includes eight diagnostic experiments covering context windows, behavior patterns, memory modules, and human baselines.
- Evaluation: EcoGym grounds evaluation in tangible economic outcomes rather than short-horizon task completion.Its assessment emphasizes economic returns and cumulative optimization in market settings.
Freelance
Figure 2 presents EcoGym’s design principles, its three environments, and a detailed Vending-environment description, linking the designs with golden leader lines.
- Freelance: Figure 2 organizes EcoGym’s design principles, three environments, and detailed Vending mechanics in one overview.Golden leader lines indicate how the designs reflect the stated principles.
2 Related Work
Related work evaluates long-horizon planning across diverse domains and increasingly places LLM agents in economic simulations. These studies span general planning benchmarks and transaction-level economic decision making.
- Long-horizon Planning Evaluation: Long-horizon planning is evaluated in embodied environments, GUI navigation, autonomous driving, and deep research tasks.Examples include ALFWorld, SciWorld, AndroidWorld, WebArena, PCA-Bench, MetaAD, xBench, and BrowseComp.
- Agents in Economic Simulations: Economic-agent research includes micro-economic execution agents that make transaction, budget-allocation, and survival-oriented decisions.Generative Agents additionally introduced persistent memory and planning for sustained social and economic interactions.
3 EcoGym
EcoGym combines compact action spaces, unbounded horizons, economic environments, latent mechanics, and partial observability in three unified scenarios. Its environments require sustained resource management, exploration, and feedback-driven decisions.
- Design Principles: EcoGym uses a compact action space with an effectively infinite interaction horizon to emphasize cumulative strategic planning.Agents typically choose among 4-5 discrete primitives while maintaining coherence over thousands of decision steps.
- Design Principles: The benchmark grounds evaluation in economic activities involving resource allocation, labor management, and operational efficiency.The three scenarios assess LLMs as economic actors rather than only code generators or abstract reasoners.
- Design Principles: Latent mechanics require agents to explore, test hypotheses, and infer undisclosed environmental relationships from interaction and feedback.In Operation, the relationship between content quality, user engagement, and DAU is hidden from the agent.
- General Formulation: Vending, Freelance, and Operation share a unified decision-making process under partial observability.At each step, agents receive structured observations, pursue a semantic goal, select discrete actions, and trigger stochastic state transitions.
- Vending: Vending maximizes net worth through procurement and dynamic pricing while managing hidden market parameters and delayed logistics.Its actions include market exploration, inventory procurement, and price adjustment, with demand driven by hidden seasonality and price sensitivity.
- Freelance: Freelance maximizes income while avoiding burnout through task discovery, labor execution, settlement, and wellness actions.The environment tracks money, energy, stress, skills, tasks, and a hidden burnout threshold; failures can trigger a stress-driven death spiral.
- Data Construction: Freelance tasks are assembled from cross-domain datasets, difficulty-filtered, strategically mutated, and checked for solvability before deployment.Coding tasks receive scenario injection, quantitative tasks undergo logic mutation, and both LLM and human checks are required.
- Operation: Operation optimizes DAU in a digital content platform where user activity naturally decays to zero without intervention.The agent acts as a platform operator and targets average daily active users.
4 Experiments
Experiments evaluate diverse LLMs across EcoGym’s economic scenarios and diagnostics, revealing environment-dependent performance, stochastic instability, and distinct strategic and execution behaviors.
- Main results: No single model dominates EcoGym: Gemini-3 leads Vending and Freelance, while Claude-Sonnet-4.5 ranks first in Operation.Vending averages five runs because of high variance; Freelance and Operation use three runs.
- Stochastic stability: Vending has high performance variance, whereas Freelance and Operation produce relatively stable agent trajectories.The different variance profiles motivate averaging five Vending runs versus three runs for the other environments.
- Context window length: Extending context beyond k = 128 produces inconsistent effects: Gemini-3-Flash rebounds at k = 1024, while Gemini-3-Pro degrades after peaking at k = 128.The divergent trajectories indicate instability when models process super-long contexts.
- Failure modes: The analysis distinguishes strategic choices from execution efficiency, including Claude-Sonnet-4.5 prioritizing scale and Gemini-3-Pro using actions more effectively than competitors.In Operation, Claude-Sonnet-4.5 generated 643 items at 0.566 quality, while the runner-up generated 326 items at 0.762 quality.
- Behavioral evolution: Gemini-3-Pro’s action frequencies reveal phase-dependent behavior across Vending, Freelance, and Operation rather than static action selection.In Operation, the agent shifts from acquisition_boost to moderation_tighten and then creator_incentive.
5 Conclusion
EcoGym evaluates long-term economic viability through sustained interaction with resource scarcity and stochastic dynamics. The authors report strong short-term reasoning but difficulty maintaining strategic coherence, while noting important scope boundaries.
- 5 Conclusion: EcoGym moves beyond atomic task success to evaluate long-term economic viability under sustained interaction.The framework exposes agents to resource scarcity and stochastic dynamics over extended periods.
- 5 Conclusion: Frontier models demonstrate impressive short-term reasoning but struggle to maintain strategic coherence over long-time decisions.
- 5 Conclusion: The framework relies on rule-based abstractions that may not fully capture irrational human behavior.
- 5 Conclusion: The current study focuses on individual agent-environment interactions, leaving multi-agent game-theoretic dynamics as a next step.
B.2 Freelance
The Freelance environment models a gig economy in which agents maximize income and human capital while maintaining survival constraints. Its dynamics combine economic costs, physiological feedback, task evolution, and LLM-based auditing.
- Metric Calculation: The primary goal is to maximize Income (Mfinal), calculated from total revenue minus daily, refresh, and relaxation costs.
- Metric Calculation: The Composite Score (Stotal) combines final income with skills, energy, and stress to represent long-term sustainability.The human-capital term rewards higher skills and energy and lower stress.
- Task Auditing: Tasks are submitted to an LLM-based Auditor that evaluates trajectory quality and determines payment.
- Prompt Design: The environment uses a dual-prompt structure separating the freelancer agent from the auditor that evaluates work quality and payment.
- Agent Objective: Freelance agents must maintain Money > 0, Energy > 0, and Stress < 100 while maximizing Career Score.
D Action Input/Output Schema
The action schema documentation specifies environment tools and their returned observations, with Vending represented through a dedicated action-schema table.
- Action Schemas: Tables 7, 8, and 9 list exact tool function names, required arguments, and returned-observation structures for the environments.
- Vending: Table 7 provides the Vending action schema.
E Implementation Details on Memory
EcoGym’s hybrid explicit memory architecture combines working, symbolic, and episodic memory for long-horizon interaction. It uses time-decayed retrieval and a trust hierarchy to synthesize context while preserving structured state reliability.
- Memory Architecture: The architecture combines working, symbolic, and episodic memory to address fixed context windows, attention decay, and numerical instability.
- Working Memory: Working Memory stores recent interaction history in a token-limited FIFO buffer and supports asynchronous consolidation.
- Symbolic Memory: Symbolic Memory extracts structured state variables such as asset balances, plan stack, and task progress from interactions.
- Episodic Memory: Episodic Memory retrieves historical interaction fragments using semantic similarity combined with time decay.The retrieval mechanism favors recent relevant experiences while retaining distant critical lessons.
- Conflict Resolution: The trust hierarchy prioritizes symbolic memory over working memory and episodic recall when information conflicts.
- Tool Use: Temporal tool-use ridge plots visualize day-level call frequencies over the full 365-day simulation horizon.
G GUI in Human Performance Testing
The paper uses a GUI for human performance testing in Operation tasks, with executable actions presented as buttons. Additional figures track tool-usage frequencies over time across all three environments for Gemini-3-Flash and GPT-5.2.
- Figure 10 presents the GUI used for human performance testing in Operation tasks.Human experts execute actions through buttons located in the bottom-left corner.
- Figure 8 shows temporal tool-usage frequencies for Gemini-3-Flash in Vending, Freelance, and Operation.The three environments are arranged left, middle, and right, respectively.
- Figure 9 shows temporal tool-usage frequencies for GPT-5.2 in Vending, Freelance, and Operation.The three environments are arranged left, middle, and right, respectively.