PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentOdyssey: Open-Ended Long-Horizon Text Game Generation for Test-Time Continual Learning Agents

Zheyuan Zhang, Zehao Wen, Alvin Zhang, Andrew Wang, Jianwen Xie, Daniel Khashabi, Tianmin Shu

arXiv:2606.24893v1cs.CL

TL;DR

Existing benchmarks inadequately capture how agents explore, learn, remember, and plan over long horizons at test time. AgentOdyssey evaluates these abilities in procedurally generated open-ended text games, finding substantial agent limitations while showing benefits from short-term memory.

  • Problem

    Existing benchmarks and training paradigms inadequately evaluate test-time continual learning agents in realistic settings requiring exploration, memory, and long-horizon planning.

  • Method

    AgentOdyssey procedurally generates open-ended text games with rich entities, world dynamics, long-horizon tasks, and diagnostic measures of knowledge, memory, exploration, action diversity, and cost.

  • Results

    34.8% increase after playing the game in World Knowledge QA is achieved by the GPT-5 Long Context agent, while short-term memory improves SFT performance.

  • Takeaways & Limitations

    Short-term memory is an important component of test-time training, while limited exploration and repetitive behavior constrain effective long-horizon agent performance.

  • Takeaways & Limitations

    Agents exhibit insufficient object and action coverage, often overlooking intermediate resources and failing to systematically explore available actions.

Abstract

from arXiv · show

For agents to learn continuously from interaction with the world at test time, they must be able to explore effectively, acquire new world knowledge and skills, retain relevant episodic experiences, and plan over long horizons. To evaluate these key abilities of test-time continual learning agents, we introduce AgentOdyssey, a novel evaluation framework that procedurally generates open-ended text games with rich entities, world dynamics, and long-horizon tasks. Critically, AgentOdyssey goes beyond the conventional machine learning assumption that learning does not occur at test time by placing agents in a continuous, long-horizon setting that interleaves learning and inference throughout deployment. We further propose a multifaceted evaluation methodology that measures not only game progress but also offers diagnostic tests on world knowledge acquisition, episodic memory, object and action exploration, action diversity, and model cost. We evaluate diverse agent paradigms in the generated games. Our experimental results reveal critical limits in agents' key abilities, as well as factors that influence their meaningful horizon. Although performance scales with stronger base models, even the top agent remains far below human performance, leaving substantial headroom for improvement. Among agent mechanisms, we find that short-term memory benefits multiple agent paradigms and is an important component of agent test-time training.

1. Introduction

The introduction frames test-time continual learning as a gap in existing agent benchmarks and presents AGENTODYSSEY as an open-ended text-game framework for studying it. The framework combines procedurally generated dynamic worlds with multifaceted diagnostics of agents’ learning, memory, exploration, planning, and evolution over long horizons.

  • Motivation: Existing benchmarks inadequately study realistic test-time learning because they evaluate static performance or train agents across episodes before evaluation rather than learning continuously during deployment.These paradigms obscure the challenge of acquiring knowledge and skills after initial training while interacting with the world.
  • Motivation: Successful test-time continual learning agents require exploration, episodic memory, world knowledge acquisition, skill learning, and long-horizon planning.The abilities reinforce one another: exploration and planning produce experiences, learning unlocks future experiences, and memory preserves useful knowledge and skills.
  • AGENTODYSSEY: AGENTODYSSEY procedurally generates open-ended text games with diverse entities, world dynamics, rich mechanics, and long-horizon objectives for evaluating these five abilities.Its games require agents to explore and acquire world knowledge, learn skills such as note-taking, use episodic memory, and plan across multi-step objectives.
  • AGENTODYSSEY: The game-generation engine uses LLM-based entity and rule synthesis grounded in an ontology, and verifies and automatically fixes generated games through program synthesis.It can generate diverse locations, objects, NPCs, action rules, and environmental step rules.
  • Evaluation: The evaluation methodology measures game progress and diagnoses world knowledge, episodic memory, object and action exploration, action diversity, and model cost.These diagnostics quantify agent evolution over a long horizon beyond what game progress alone reveals.
  • Findings: The empirical study finds critical limits in current agents, factors shaping their meaningful horizon, and an important role for short-term memory in agent test-time training.The study evaluates diverse agent paradigms in the generated games.

2. Related Works

Prior agent-evaluation environments span video games, embodied simulators, and web or OS settings, but they do not fully assess the five abilities targeted for test-time continual learning. Related work also emphasizes memory as central, with agents using several memory paradigms.

  • Text Games for Evaluating Agents: Agent evaluation has used video games, embodied simulators, and web or OS environments.
  • Text Games for Evaluating Agents: These environments do not fully support evaluating the five key abilities proposed for test-time continual learning agents.
  • Test-Time Continual Learning Agents: Memory is central to test-time continual learning agents and is roughly categorized into 5 classes.The passage identifies RAG-based, Long Context, and Fixed Size Memory agents among these classes.

3. AGENTODYSSEY

AgentOdyssey is an evaluation framework that procedurally generates open-world, long-horizon text games for studying test-time continual learning agents. Its ontology, dynamics, quests, reward signals, and diagnostic tests are designed to assess learning, memory, exploration, planning, and game progress.

  • Framework motivation: AgentOdyssey procedurally generates open-world text games to evaluate five key abilities of test-time continual learning agents.The framework targets open-ended, long-horizon settings where later-task success depends on experience and learning from earlier interaction.
  • Environment ontology: The game ontology specifies entities, world graphs, observations, actions, dynamics, rewards, and time-dependent state evolution.Worlds contain locations, objects, and NPCs; each step combines agent actions with environment-driven rules, with 10 minutes of simulated time per step.
  • Game entities and world graph: Declarative entity specifications and level-conditioned world-graph sampling create structured progression in difficulty and resource availability.Entities include attributes such as NPC health and attack strength, and higher-level entities are more likely to appear in higher-level areas.
  • World dynamics: Modular action and step rules create long-range dependencies, persistent processes, stochastic transitions, and time-based strategic challenges requiring episodic memory.Examples include later crafting with objects obtained after combat and day-night changes in enemy strength and aggressiveness.
  • Evaluation and generation pipeline: Quests, supplementary rewards, LLM-based game generators, automated runtime validation, and diagnostic tests jointly support multifaceted agent evaluation.Rewards measure main and side quests, explored areas, unique objects crafted, and unique NPCs defeated; diagnostics evaluate abilities beyond direct game progress.

4. Agent Paradigms

AgentOdyssey evaluates LLM-based agents through a universal interface spanning six paradigms and two baselines, with primary experiments focused on four paradigms and optional memory-related augmentations. The paradigms differ in how they retain or update experience, while all agents use ReAct prompting and a shared final action-generation prompt.

  • Evaluation scope: The evaluation groups LLM-based agents into 6 paradigms and includes two additional baselines: no memory and random action.Agents are evaluated through a universal interface across different base models.
  • Memory mechanisms: Long Context Agents append observations, reasoning, and actions at each step, whereas Fixed Size Memory Agents retain information through sliding-window or bounded self-updating buffers.The sliding-window variant is described as short-term memory.
  • Learning mechanisms: RAG Agents retrieve relevant observation-reasoning-action tuples from an external embedding-text database, while SFT Agents encode experience into LoRA adaptor weights through supervised fine-tuning.The RAG implementations include Vanilla RAG, Mem0, Raptor, and Voyager.
  • Learning mechanisms: RL Agents update LoRA adapter weights with PPO using environment rewards, while Latent Agents store experience as learnable latent memory tokens in model hidden states.Latent memory enables persistent storage and retrieval.
  • Evaluation scope: The primary evaluation focuses on Long Context, Fixed Size Memory, RAG, and SFT Agents, optionally augmented with reflection, summarization, or short-term memory.Short-term memory is implemented as a fixed-size context window storing a specified number of recent observations.
  • Implementation: All agents use ReAct prompting and the same final-stage prompt to output an action from the action space; evaluated backbones include GPT-5, Qwen-3, and LLaMA-3.LLaMA-3 is included because MemoryLLM and MPlus are compatible only with LLaMA 3/3.1-8B.

5. Experiment 1 - Diagnosing Five Key Abilities of Agents

Experiment 1 evaluates five key abilities of test-time continual-learning agents in a procedurally generated, long-horizon game. Results show that Long Context performs best but agents still face exploration, memory, knowledge, skill-learning, planning, and efficiency failures.

  • Experimental Setup: AGENTODYSSEY generates a long-horizon game environment to investigate five key abilities of test-time continual-learning agents.Agents are run for 500 steps, with progress visualized using normalized cumulative reward.
  • Overall Performance: The Long Context agent with GPT-5 achieves the highest game performance and strongest diagnostic results, including a 34.8% increase in World Knowledge QA accuracy.It also achieves the highest Episodic Memory QA accuracy, indicating stronger semantic knowledge acquisition and episodic retention.
  • Diagnostic Abilities: World knowledge increases through step 300 before flattening from insufficient exploration, while Long Context shows the strongest object exploration and action diversity correlates with game performance.STM and SFT exhibit sharp decreases in action diversity that coincide with cumulative-reward plateaus.
  • Cost and Reasoning Efficiency: Long Context model cost increases quadratically with steps, reducing its meaningful horizon under a budget, whereas other agents use fewer tokens with linear growth.Many paradigms and models also rely on excessive context and reasoning tokens, raising inference costs and slowing decisions.
  • Failure Patterns: Agents insufficiently explore objects and actions, often ignoring prerequisite crafting resources and restricting long-term planning.This reflects myopic exploration over objects and the available action space.
  • Failure Patterns: Agents exhibit repetitive action loops, semantic-memory hallucinations, weak procedural skill acquisition, and poor maintenance or switching among concurrent goals.Only Long Context partially adapts to deterministic adversary behavior, while agents commonly fail to re-anchor on primary objectives.

6. Experiment 2 - Effect of Agent Mechanisms on Test-Time Training

Experiment 2 evaluates Qwen3-4B agents over 500 steps in a simpler game to isolate test-time training mechanisms. Short-term memory substantially improves SFT, while diagnostic results reveal collapse in Long Context and memory weaknesses in SFT.

  • Agent mechanisms: The Long Context agent performs well with strong models but poorly with Qwen3-4B, whose limited capability motivates evaluating a simpler game.The simpler game features reduced crafting hierarchies, weaker enemies, and no side quests to avoid goal switching.
  • Agent mechanisms: Short-term memory substantially improves the SFT agent, producing the strongest overall performance and outperforming the vanilla Short-Term Memory agent.The result highlights test-time parametric weight updates as a form of long-term memory.
  • Experimental setup: Experiment 2 evaluates four SFT agents and three comparison agents, all powered by Qwen3-4B, using game progress, diagnostic testing, and model cost.The SFT agents include a vanilla baseline and three mechanism-augmented variants; comparisons include Long Context, RAG, and Short-Term Memory agents.
  • Diagnostic evaluation: After gameplay, Long Context achieves zero accuracy on World Knowledge QA and Episodic Memory QA, while action diversity progressively declines to a single action.These diagnostic patterns suggest collapse and degeneration; SFT also shows decreased World Knowledge QA accuracy and very low Episodic Memory QA accuracy.

7. Conclusion

AgentOdyssey introduces an open-ended text-game framework and multifaceted diagnostics for evaluating test-time continual learning agents in long-horizon, non-resettable environments. Experiments reveal scaling with memory and reasoning ability, persistent capability limitations, and short-term memory as an effective agent component.

  • Contributions: AGENTODYSSEY evaluates test-time continual learning agents in long-horizon, non-resettable environments using open-ended text-game generation and multifaceted diagnostics beyond task rewards.The diagnostics probe key abilities beyond task rewards.
  • Findings: Performance scales with memory capacity and backbone reasoning ability, but agents retain limitations in exploration, episodic memory, world knowledge acquisition, skill learning, and long-horizon planning.These limitations persist across diverse agent paradigms.
  • Findings: Short-term memory is identified as an effective component of agent test-time training.The framework’s experiments identify short-term memory as beneficial across agent paradigms.
  • Limitations and Future Work: The current environment supports only textual observations and a single agent, while its turn-based design assigns each action a fixed duration.These choices simplify the environment but constrain studies of visual perception and grounding, multi-agent interaction, and world dynamics modeling.
  • Limitations and Future Work: Future work could extend AGENTODYSSEY to multi-agent games with visual rendering and richer temporal dynamics.These extensions target limitations in visual perception, grounding, multi-agent interaction, and world dynamics modeling.

Appendix · A1. More Results for Experiment 1 and Experiment 2

The appendix extends Experiments 1 and 2 with additional backbone, frontier-model, memory-size, and seeding analyses. It also reports world-knowledge progress over time and compares cumulative reward with human performance.

  • A1. More Results for Experiment 1 and Experiment 2: Table A1 reports Experiment 1 results using proprietary LLM backbones across quest, supplementary-reward, world-knowledge, and episodic-memory metrics.The table defines Q, SQ, E, C, D, WK.b, WK.a, and Epi.a for interpreting these results.
  • A1. More Results for Experiment 1 and Experiment 2: Figure A2 tracks World Knowledge QA accuracy every 100 steps.The figure focuses on the evolution of world-knowledge performance during gameplay.
  • A1. More Results for Experiment 1 and Experiment 2: Figure A1 compares cumulative reward from additional frontier LLMs using the Long Context agent with human performance in the Experiment 1 game.The comparison concerns cumulative reward in the Experiment 1 setting.
  • A1. More Results for Experiment 1 and Experiment 2: Table A2 reports Experiment 1 results with open-weight LLM backbones across progress, supplementary rewards, knowledge, and episodic-memory measures.Its metrics include Q, SQ, E, C, D, WK.b, WK.a, and Epi.a.
  • A1. More Results for Experiment 1 and Experiment 2: Table A3 reports Experiment 2 results with Qwen3-4B using the framework’s progress, supplementary-reward, knowledge, memory, and exploration metrics.The table defines Q, SQ, E, C, D, WK.b, WK.a, Epi.a, OE, and AE.
  • A1. More Results for Experiment 1 and Experiment 2: Table A4 examines how short-term memory size affects agent performance across rewards, knowledge, memory, exploration, diversity, efficiency, and inference-related metrics.The table includes Q, SQ, E, C, D, WK.a, Epi.a, OE, AE, AD, and AT.
  • A1. More Results for Experiment 1 and Experiment 2: Table A5 reports variance across multiple runs with different random seeds for rewards, knowledge, memory, exploration, diversity, efficiency, and inference-related metrics.The reported measures include Q, SQ, E, C, D, WK.a, Epi.a, OE, AE, AD, AT, and I.

A2. Environment Design and Implementation Details

This section indicates that further details on the environment design and implementation are provided below.

  • The paper states that additional environment design and implementation details follow.

A2.1. Tasks

AgentOdyssey organizes evaluation around sequential main quests that test five key abilities and shorter, flexible side quests that leverage accumulated world knowledge and experiences.

  • Main Quest: The main quest consists of interdependent chapters and stages evaluating the agent’s five key abilities, with strictly sequential progression.A new stage unlocks only after the preceding stage is completed.
  • Main Quest: Main-quest stages can require intermediate exploration, object gathering, and weapon crafting before defeating a boss.The boss example illustrates how tasks are composed of several dependent sub-tasks.
  • Side Quest: Side quests have shorter horizons than the main quest, can be completed in any order, and include collect, talk, craft, and trade objectives.Each area unlocks four side-quest types.
  • Side Quest: Side quests draw on accumulated world knowledge and experiences, such as remembering where a previously encountered NPC was found.Their rewards facilitate progress in the main quest.

A2.2. Game Generation

AgentOdyssey generates open-ended games by expanding entities, rules, and quests together, while enforcing semantic coherence, balanced state distributions, and long-horizon task structure.

  • Entity Generation: Entity generation expands the world definition through in-context learning while enforcing hard semantic constraints and balancing heuristics.These controls preserve coherence and a well-shaped state distribution as new entities are added.
  • Rule Generation: Rule generation synthesizes action and step rules that jointly evolve the state space, world dynamics, object affordances, and NPC behaviors.Both sub-generators can inspect and modify the world definition and world-graph instantiation process.
  • Quest Generation: Quest generation produces multi-stage quest chapters emphasizing objective diversity, hierarchical goal decomposition, long-horizon dependencies, and calibrated difficulty progression.The framework is designed to evaluate the five key abilities of test-time continual learning agents.

A2.3. Additional Features … A7. Generated Game Details

AgentOdyssey combines structured game features, explicit agent interfaces, synthesized entities and dynamics, and procedurally generated games to support long-horizon continual-learning evaluation. Its appendix documents environment constraints, agent implementations, prompt and observation formats, game-generation examples, and statistics across experiments and additional games.

  • A2.3. Additional Features: A tutorial room introduces the action space while withholding some actions to test out-of-distribution learning and novel-affordance discovery.The tutorial and subsequent quests are implemented as environment step rules.
  • A2.3. Additional Features: The environment enforces physical constraints, supports online expansion after reachable areas are explored, and provides an interactive trajectory visualizer.Examples include limited container capacity, two-handed object holding, constrained writing, asynchronous generation of one to two new places, and replay over the world graph.
  • A3. Additional Agent Implementation Details: The RL agent adapts PPO-based training to test-time continual learning by partitioning interactions into episodes while persisting world state.The actor is Qwen3-4B and the critic is Qwen3-1.7B; results are reported under three configurations.
  • A4.1. Agent Prompt Template: The prompt requires one action per turn and a single JSON object containing reasoning and action, with the action exactly matching the declared action space.The observation format exposes time, location, inventory and nearby entities, agent statistics, and neighboring areas.
  • A4.2. Observation: Example observations combine current time and location with event feedback, held and nearby objects, NPCs, agent statistics, and neighboring areas.One example reports a crafted wood_plank, a torch and small_bag_0, level 1, attack 10, defense 0, health 93, and experience 30.
  • A5. Examples of Synthesized Game Entities; A6. Examples of Synthesized Game Dynamics; A6.1. Synthesized Action Rule: Synthesized games represent places, areas, objects, NPCs, initializations, custom events, and features, while synthesized rules validate actions and update world state.Examples include Sapphire Caves with levelled areas, craftable tools and materials, enemy combat patterns, GPT-5 generation, and online expansion.
Loading 2606.24893v1…