PaperlayerView exact revision on arXiv

Source-linked AI summary

Cogito, Ergo Ludo: An Agent that Learns to Play by Reasoning and Planning

Sai Wang, Yu Wu, Zhongwen Xu

arXiv:2509.25052v1cs.AIcs.LG

TL;DR

Deep reinforcement learning can require immense experience and encode knowledge opaquely, while many LLM agents lack structured continuous learning. CEL uses an LLM to reason, plan, and refine explicit rules and strategy from interaction; it learns across three grid-world tasks, with reflection reported as critical to success.

  • Problem

    Deep RL often requires immense experience and stores strategy opaquely, while LLM agents may lack continuous adaptation and interpretable models of environment mechanics.

    Source: §1 · p. 1

  • Method

    CEL starts without game-rule knowledge and uses an LLM to plan during episodes, then reflect on trajectories to refine explicit dynamics and a strategic playbook.

    Source: §1 · p. 2, §3.2 · pp. 4–5

  • Results

    CEL learns effective strategies across Minesweeper, Frozen Lake, and Sokoban; Frozen Lake reached a 97% success rate within the first 10 episodes.

    Source: §1 · p. 2, §4.3 · p. 8

  • Takeaways & Limitations

    The reported results support language-based reasoning as a way to build agents that perform tasks while producing a transparent, auditable decision process.

    Source: §5 · p. 12

    CEL assumes a tabula rasa start: its initial rule set is empty, and subsequent knowledge is derived purely from interaction.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

The pursuit of artificial agents that can learn to master complex environments has led to remarkable successes, yet prevailing deep reinforcement learning methods often rely on immense experience, encoding their knowledge opaquely within neural network weights. We propose a different paradigm, one in which an agent learns to play by reasoning and planning. We introduce Cogito, ergo ludo (CEL), a novel agent architecture that leverages a Large Language Model (LLM) to build an explicit, language-based understanding of its environment's mechanics and its own strategy. Starting from a tabula rasa state with no prior knowledge (except action set), CEL operates on a cycle of interaction and reflection. After each episode, the agent analyzes its complete trajectory to perform two concurrent learning processes: Rule Induction, where it refines its explicit model of the environment's dynamics, and Strategy and Playbook Summarization, where it distills experiences into an actionable strategic playbook. We evaluate CEL on diverse grid-world tasks (i.e., Minesweeper, Frozen Lake, and Sokoban), and show that the CEL agent successfully learns to master these games by autonomously discovering their rules and developing effective policies from sparse rewards. Ablation studies confirm that the iterative process is critical for sustained learning. Our work demonstrates a path toward more general and interpretable agents that not only act effectively but also build a transparent and improving model of their world through explicit reasoning on raw experience.

1. Introduction

The introduction identifies a need for agents that learn an interpretable model of their environment through experience, then presents CEL’s reasoning-and-planning approach and reported learning results.

1. Introduction

  • Deep RL can require immense experience and encode strategy opaquely, while LLM agents often lack continuous adaptation and learned world models can remain uninterpretable.The introduction frames the need for agents that both act effectively and comprehend environments transparently.Source: §1 · p. 1
  • Figure 1 contrasts reward-trained implicit policies and static zero-shot LLM decisions with CEL’s persistent Rule & Playbook passed across episodes.The figure describes CEL’s policy as RL-trained alongside that persistent knowledge base.Source: Figure 1 · p. 2
  • CEL uses an LLM to reason and plan, refining an explicit language-based environment model and strategic playbook through interaction and post-episode reflection.Reflection analyzes the episode trajectory to induce rules and summarize successful and unsuccessful behavior into guidance for later episodes.Source: §1 · p. 2
  • CEL learns to master the evaluated tasks through autonomous rule discovery and strategy development; ablations identify iterative reflection as critical to learning success.The authors also report human-readable rulebooks and strategic heuristics generated from raw interaction.Source: §1 · p. 2

2. Related Work

The related work positions CEL against deep RL, learned world models, language-based RL, and LLM agent architectures, emphasizing its explicit, experience-refined cognitive components.

The Apex of Deep Reinforcement Learning

  • Large-scale deep RL can achieve superhuman performance but uses vast experience and opaque neural policies; unlike AlphaZero, CEL learns game rules through interaction.The cited systems are described as requiring thousands of GPU-years and billions of game frames, while AlphaZero receives a perfect environment model.Source: §2 · p. 3

Planning with Learned World Models

  • MuZero and Dreamer plan using latent world models, whereas CEL infers human-readable rules and transition dynamics that it can refine in natural language.Both approaches use learned models to predict environmental dynamics for planning.Source: §2 · p. 3

Language as the Algorithm vs. Language in the Architecture

  • Natural Language Reinforcement Learning reformulates RL components in language, while CEL uses the LLM to orchestrate distinct language-grounded modules.In NLRL, state values are descriptive text and policy improvement involves LLM reasoning over linguistic value judgments.Source: §2 · p. 3

LLMs as Agent Architectures

  • GEM and LMGame-Bench provide evaluation harnesses, Gemini 2.5 Pro demonstrates zero-shot Pokémon play, and PORTAL uses an LLM to generate behavior trees.CEL instead uses the LLM for planning, acting, and accumulating knowledge through direct interaction with environments.Source: §2 · p. 3

Our Approach

  • CEL maintains interconnected components—an explicit world model, game rules, strategic playbook, and language-based value function—refined through post-episode reflection.Reflection analyzes trajectories to update rules and strategy simultaneously.Source: §2 · p. 3

3. Method

CEL uses one LLM to reason and plan with explicit, language-based representations of environment dynamics and strategy. It alternates in-episode action selection with post-episode reflection that updates both its rules and playbook.

3. Method

  • One LLM manages CEL’s cognitive functions, representing states, actions, rewards, inferred dynamics, and strategic guidelines as natural-language strings.The framework makes its reasoning explicit through a chain-of-thought process.Source: §3 · p. 4

3.1. Language-based World Model

  • The Language-based World Model predicts a next state and immediate reward for each candidate action, using the current state, action, and learned rules.Its predictions are descriptive natural-language strings used to simulate possible outcomes for planning.Source: §3.1 · p. 4

3.2. Induction of Environmental Dynamics

  • After an episode, Rule Induction analyzes its state-action-reward trajectory against prior rules to refine the environment model.The initial rule set is empty; subsequent knowledge is derived from interaction.Source: §3.2 · pp. 4–5

3.3. Strategy and Playbook Summarization

  • After each episode, the LLM uses the trajectory and final outcome to distill successful and unsuccessful patterns into actionable playbook advice.The explicit text playbook can be incorporated into the next episode’s prompt context.Source: §3.3 · pp. 5–6

3.4. Language-based Value Function

  • The Language-based Value Function qualitatively assesses a state’s long-term potential for success, conditioned on current rules and the strategic playbook.This assessment serves as a planning heuristic.Source: §3.4 · p. 6

3.5. The Agent’s Operational Cycle

  • Within an episode, CEL evaluates states and simulates one-step outcomes for available actions; afterward, it updates rules and strategy from the full trajectory and trains the LLM using the final outcome.The two phases separate step-by-step action selection from offline knowledge consolidation.Source: Figure 2 · p. 5, §3.5 · p. 6

4. Experiments

CEL is evaluated on three sparse-reward grid-world tasks without supplied game rules. It improves across tasks, while the Minesweeper ablation shows that sustained learning is strongest with iterative rule refinement.

4.1. Game Environments

  • The tasks are Minesweeper, Frozen Lake, and Sokoban, each configured to reward only successful completion with +1; no game rules are provided.The agent must learn dynamics through interaction, with knowledge limited to the available actions.Source: §4.1 · p. 7

4.2. Implementation Details

  • Experiments use Qwen3-4B-Instruct with rLLM/verl and GRPO outcome-reward post-training, averaging 256 playthroughs per game and updating rules every five episodes.The 256 playthroughs comprise 32 seeds with eight trials per seed.Source: §4.2 · p. 7

4.3. Results

  • CEL improves from no explicit rules across three tasks, reaching 54% Minesweeper success versus 26% for the ground-truth-rule baseline, 84% in Sokoban, and 97% in Frozen Lake within 10 episodes.Sokoban’s increase follows an initial exploration period; Frozen Lake reaches its near-perfect rate within the first 10 episodes.Source: §4.3 · pp. 7–8

4.4. Ablation Study

  • In Minesweeper, the full model with iterative rule refinement outperforms both the no-rules variant and the variant that induces rules only once.The no-rules curve stagnates, while the once-induced rules initially improve before stagnating and degrading.Source: §4.4 · p. 8, Figure 4 · p. 8

4.5. Case Study

CEL uses language-based planning during episodes and reflection afterward to build a rulebook and strategic playbook. Its case studies show decisions grounded in learned rules, and generalization to unseen layouts and other games.

4.5.1. In-Episode Decision-Making

  • At each step, the Language-based Value Function assesses the current state, and the Language-based World Model predicts each action’s consequences for one-step lookahead.In the Minesweeper example, it predicts that action “(0, 3)” solves the puzzle, then selects the most favorable predicted outcome.Source: §4.5.1 · pp. 8–9, Figure 5 · p. 9

4.5.2. Autonomous Rule Discovery

  • The agent’s Minesweeper rulebook is induced from interaction and covers symbol meanings through the game objective, grounding its world model and value function.The rulebook is described as comprehensive and accurate.Source: §4.5.2 · p. 9, Figure 6 · p. 10

4.5.3. Emergent Strategy and Playbook Generation

  • The playbook distills gameplay into tactical methods and high-level principles, then conditions the value function in the next episode.Examples include Constraint Propagation and Safe Exploration.Source: §4.5.3 · p. 9, Figure 7 · p. 10

4.6. Generalization

  • CEL generalizes to unseen layouts and new games: it maintains performance on new seeds and learns across games while its core model weights remain frozen.The cross-game agents refine their explicit rulebooks and playbooks every five episodes.Source: §4.6 · p. 11, Table 1 · p. 11, Figure 8 · p. 11

5. Conclusions

The conclusion presents CEL as an agent that learns through explicit reasoning, in-episode planning, and post-episode reflection. Across several environments, it reports task mastery alongside transparent, auditable decision-making.

5. Conclusions

  • CEL combines in-episode planning with post-episode reflection to construct a human-readable world model and strategic playbook from interaction, starting tabula rasa.The conclusion characterizes this as learning by thinking.Source: §5 · p. 12
  • Across several environments, the authors report that CEL masters complex tasks while producing a transparent and auditable decision-making process.Source: §5 · p. 12
  • The authors present CEL as a path toward hybrid systems combining its explicit understanding with traditional architectural efficiency.Source: §5 · p. 12

A. Details of Environments

The experiments use three grid-world tasks with distinct rules and configurations: Minesweeper, Frozen Lake, and Sokoban.

Minesweeper

  • Minesweeper requires clearing non-mine cells without detonating mines; the experiments use a 5×5 grid with three randomly placed mines.Source: A. Details of Environments · p. 15

Frozen Lake

  • Frozen Lake is configured as 6×6 navigation from a start tile to a goal while avoiding six holes, with each action moving exactly one cell.The deterministic setting removes the environment’s usual slippery stochasticity.Source: A. Details of Environments · p. 15

Sokoban

  • Sokoban requires pushing boxes to designated goals without pulling; the study’s version uses a 6×6 grid and a single box.The push constraint creates a complex search space and requires careful long-horizon planning to avoid irreversible states.Source: A. Details of Environments · p. 15

B. Analysis on Rollout Outcomes for Action-only Model

The Action-only model fails to train with GRPO because its rollout batches lack the mixed outcomes needed to produce a learning signal.

B. Analysis on Rollout Outcomes for Action-only Model

C. Performance with an Expanded Training Set

CEL’s peak Minesweeper success rate rose to 62% with 128 unique training seeds, compared with 54% in the main experiments using 32 seeds. The authors interpret this as evidence that more diverse layouts support generalizable reasoning and planning.

C. Performance with an Expanded Training Set

  • The additional experiment increased the number of unique training-layout seeds from 32 to 128.The 32-seed set was used in the main experiments.Source: C. Performance with an Expanded… · p. 16
  • 62% was CEL’s new peak Minesweeper success rate, up from 54% reported in Figure 3 after expanding the training set.Figure 10 shows the new peak; the main experiments used 32 unique seeds, while the expanded set used 128.Source: C. Performance with an Expanded… · p. 16, Figure 10 · p. 16
  • The authors suggest that exposure to more diverse scenarios enhances core reasoning and planning and reduces overfitting to specific layouts.They characterize the result as evidence of a robust, generalizable problem-solving model for Minesweeper.Source: C. Performance with an Expanded… · p. 16

D. Prompt Templates

The Phase 1 prompt template guides the LLM’s in-episode decisions by asking it to assess the current state, strategic value, and possible action outcomes.

D. Prompt Templates

  • In Phase 1, the LLM evaluates the current state to support in-episode decision-making.The prompt template is shown in Figure 11.Source: Figure 11 · p. 17
  • The prompt asks the LLM to assess strategic value using LVF.LVF is named in the Phase 1 prompt-template description.Source: Figure 11 · p. 17
  • The prompt asks the LLM to predict action outcomes using LWM.LWM is named in the Phase 1 prompt-template description.Source: Figure 11 · p. 17

E. Additional Results

The additional results illustrate CEL’s decision-making, learned environmental rules, and strategic playbooks across Minesweeper, FrozenLake, and Sokoban. Examples include Sokoban guidelines on repositioning boxes and avoiding deadlocks.

E. Additional Results

  • The Phase 2 prompt template guides trajectory analysis and refinement of the explicit model of environmental dynamics.This is the Rule Induction and Playbook Summarization process shown in Figure 12.Source: Figure 12 · p. 18
  • The Sokoban decision-making example illustrates an in-episode decision process.The example is shown in Figure 13.Source: E. Additional Results · p. 17, Figure 13 · p. 19
  • The examples include learned environmental rules for Minesweeper, FrozenLake, and Sokoban.These are presented as examples of the agent’s explicit knowledge base.Source: E. Additional Results · p. 17, Figure 14 · p. 19, Figure 15 · p. 20
  • The playbook examples include strategic guidelines for FrozenLake and Sokoban.Figures 17 and 18 show examples from the agent’s playbook.Source: E. Additional Results · p. 17, Figure 17 · p. 22, Figure 18 · p. 22
  • Sokoban playbook guidance recommends repositioning to push a stuck box from another direction and avoiding moves that create deadlocks.The examples label repositioning as key to unlocking progress and advise against trapping blocks between walls or other blocks.Source: E. Additional Results · p. 22

Related papers

Loading 2509.25052v1…