PaperlayerView exact revision on arXiv

Source-linked AI summary

Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks

Haoyu Zhao, Zhengxu Yu, Zhiyuan He, Meng Fang, Rasul Tutunov, Haitham Bou-Ammar, Weilin Luo, Jun Wang

arXiv:2610.11794v1cs.AIcs.CLcs.CVcs.LG

TL;DR

Limited observations can leave multiple world models consistent with past interactions but divergent on unseen states. Memento 3 uses an external rulebook and executable model, revised and verified while the LLM stays fixed. It clears all levels of 25 ARC-AGI-3 public games and demonstrates a Pong controller that wins three evaluated episodes without further LLM calls.

  • Problem

    Limited interaction histories can support multiple world models that fit observed transitions yet predict differently in unobserved states.

    Source: §1 · p. 2

  • Method

    Memento 3 maintains a natural-language rulebook and executable world model, revising and verifying them against interaction evidence while keeping the underlying LLM fixed.

    Source: §1 · p. 2, §2.2 · p. 10, §7 · p. 27

  • Results

    On ARC-AGI-3, the single-model agent clears every level of all 25 public games, reaches a mean RHAE of 100.0, and uses 44% of the human action count; its Pong controller wins 21:0 in three episodes.

    Source: §1 · p. 3, §7 · p. 27

  • Takeaways & Limitations

    Verified world-model updates guide subsequent planning and exploration, and the population extension lets multiple models use shared interaction evidence to guide exploration.

    Source: §1 · pp. 2–3, §7 · p. 27

    Model selection assumes a proper prior over rulebook-faithful pairs that favors shorter combined descriptions.

    Source: §2.3 · p. 11

Abstract

from arXiv · show

Learning to act in unfamiliar environments requires agents to infer how the world works and revise that understanding as new evidence arrives. Yet limited observations can support multiple world models that explain past interactions but predict different outcomes in unseen states. We introduce Memento 3, building on the Memento series to enable frozen LLM agents to continually learn explicit world models through external memory. The agent maintains a natural-language rulebook as persistent semantic memory, recording revisable hypotheses about environment dynamics while leaving unknown aspects underspecified. It compiles this rulebook into executable code for prediction and planning. Through a continual loop of observation, reflection, rule revision, compilation, and verification, the agent uses prediction errors to refine both the rulebook and its code. Updated code is accepted only when the LLM judges it faithful to the rulebook and cell-exact replay reproduces the observed transitions. We investigate this process as a model-based route to recursive self-improvement (RSI): the agent autonomously explores the environment, revises its world model, and uses verified updates to guide subsequent interaction and learning, while the underlying LLM remains fixed. A population extension maintains multiple world models in parallel, sharing interaction evidence and using their predictions to guide exploration. On ARC-AGI-3, the single-model agent clears every level of all 25 public games, achieves a mean Relative Human Action Efficiency (RHAE) of 100.0, and uses 44% of the human action count. In an Atari Pong case study, a learned feedback controller wins 21:0 in each of three evaluated episodes with different openings, without further LLM calls.

1. Introduction

Memento 3 extends reflective learning from policy memory to explicit world models, using a revisable rulebook and executable code to guide learning in unfamiliar environments. It evaluates this model-based approach on ARC-AGI-3 and Atari Pong.

1. Introduction

  • Replay can reject programs inconsistent with observed transitions, but finite evidence may leave multiple consistent programs that predict differently in unseen states.The agent therefore needs to maintain and revise its generalisation hypotheses as new evidence arrives.Source: §1 · p. 2
  • Memento 3 extends prior Memento work on policy and procedural memory to the complementary problem of learning environment rules.Earlier Memento systems focused on improving decisions or reusable procedures rather than explicit world models.Source: §1 · p. 2
  • Memento 3 represents its evolving world model as a natural-language rulebook of environment hypotheses and executable code for prediction and planning.The rulebook is revised as new observations arrive; the underlying LLM remains fixed.Source: §1 · pp. 2–3
  • The reflective loop revises and verifies the rulebook and code, then uses verified updates to guide planning and exploration and diagnose prediction errors.The agent also extends this setup to parallel world models that share interaction evidence and guide exploration.Source: §1 · pp. 2–3
  • On ARC-AGI-3, the single-model agent clears every level of all 25 public games, reaches a mean RHAE of 100.0, and uses 44% of the human action count.In Atari Pong, its learned feedback controller wins 21:0 in three evaluated episodes with different openings, without further LLM calls.Source: §1 · p. 3

2. Model-Based Recursive Self-Improvement

This section presents a model-based approach in which a frozen LLM revises rulebooks and executable world models from interaction, then uses accepted models to guide planning, exploration, and further learning. It covers both a single-model controller that tests unresolved hypotheses and a population that retains alternative models, shares interaction evidence, and can guide different action sequences.

2.1. Problem formulation

  • The agent must infer hidden transition rules and goal conditions from interaction, reaching a goal within a limited action budget without access to the underlying state or environment rules.Every real action, including exploration and failed attempts, counts toward the budget.Source: §2.1 · pp. 4, 5, 8
  • A persistent rulebook records hypotheses about entities, dynamics, goals, and termination, while executable code implements them for prediction and planning; the frozen LLM revises both as evidence accumulates.The rulebook can leave unobserved mechanics unspecified, and implementation repairs may change code without changing the rulebook.Source: §2.1 · pp. 5–6

2.2. Reflective rule revision, compilation, and verification

  • At each interaction, the frozen LLM can retain the current pair, revise rules, or repair code; a proposal is accepted only if the LLM judges it faithful to the rulebook and exact replay matches every recorded next observation.Verification checks the entire interaction record, so a repair must explain earlier evidence as well as the latest discrepancy.Source: §2.2 · pp. 9–10

2.3. Model selection and planning

  • Among admissible rulebook–executable pairs, the idealized selection objective favors the shortest combined description, which the frozen LLM approximates by proposing simpler rules and source.Each proposed simplification is checked again for rulebook fidelity and replay consistency.Source: §2.3 · p. 11
  • The single-model agent follows a shortest goal-reaching plan from its reconstructed state when one fits the remaining action budget; otherwise, it explores or requests model revision.Under the selected deterministic model, each planned action reduces shortest-path cost by one; no plan is returned when the goal is unreachable.Source: §2.3 · p. 12
  • When no goal plan is available, the LLM selects an intermediate target and an auxiliary planner finds a route; execution stops at a prediction mismatch and returns evidence for reflection.Both planned and exploratory observations extend the shared record and update the agent’s world-model components, guiding subsequent action selection and reflection.Source: §2.3 · p. 12

2.4. Population extension: from one to many

  • A single-model plan can remain self-confirming when its actions never expose errors, even if the model would predict incorrectly after other histories or actions.Repeating the same hypothesis in language alone does not resolve this behaviour.Source: §2.4 · pp. 12–13
  • The population maintains parallel rulebook–executable hypotheses over one shared interaction record, checks them against recorded trajectories, and repairs falsified members.For action selection, it samples among distinct nonempty plan classes and follows the selected plan until success, falsification, or replanning, broadening exploration beyond one early model choice.Source: §2.4 · pp. 13–14

3. Realisation on ARC-AGI-3

The ARC-AGI-3 agent infers game dynamics and goals through interaction, representing its hypotheses in a natural-language rulebook and executable code. It revises these artefacts throughout play and preserves earlier versions in Git for inspection and recovery.

3.1. Environment and scoring

  • ARC-AGI-3 levels are modeled as deterministic, goal-directed POMDPs; the agent observes a 64 × 64 grid and interaction history but not the game source or a language description.Completion is prioritized, and environment actions—including resets—count toward the budget; model construction and replay do not.Source: §3.1 · pp. 14–15

3.2. Rulebook and executable artefacts

  • The agent stores hypotheses about objects, actions, dynamics, goals, and alternatives in Markdown, then compiles code that predicts the next grid and level completion.The planner uses an executable only after the LLM checks rulebook fidelity and replay matches every cell of recorded frames.Source: §3.2 · p. 15, Figure 4 · p. 16

3.3. Continual operation

  • Reflection runs throughout each game: new evidence can revise the rulebook before the executable, or change implementation alone when the rulebook remains intact.Rules and code persist across attempts and levels, so earlier inferences can guide later prediction and planning.Source: §3.3 · p. 15

3.4. Git memory

  • Git commits each iteration’s rulebook and modules, preserving the learning trajectory so the agent can inspect, compare, recover, or restore earlier versions.Figure 5 shows observations resolving competing hypotheses and later evidence reversing an earlier cursor rule.Source: §3.4 · pp. 15, 17, Figure 5 · p. 17

4. Experiments

On ARC-AGI-3, the single-model agent clears every level of all 25 public games at mean RHAE 100.0, using fewer actions than the human baseline. Ablations and a two-model run examine the rulebook’s contribution and the population extension’s action efficiency.

The 25 public ARC-AGI-3 games

  • The evaluation covers all 25 public ARC-AGI-3 games, counts every environment action including failed attempts and post-reset actions, and uses Claude Opus 5 with extra-high reasoning effort.Baseline values come from official ARC Prize scorecards, with columns counted on the same basis.Source: §4 · p. 18
  • Mean RHAE is 100.0 across all 25 public games, and CODE-AS-MODEL clears every level; baseline1 is the closest compared system at 99.0.Table 1 reports per-game RHAE; NOOA scores 85.1, OPINE-World 78.4, DreamTeam 38.1, and Continual Harness 20.5.Source: §4 · pp. 18–19, Table 1 · p. 19
  • 7,518 actions—0.44 of the human baseline of 17,135—clear all 25 games, while baseline1 also finishes every game using 8,347 actions (0.49 of the human budget).Other systems leave games unfinished, so their totals reflect incomplete runs rather than solution costs.Source: §4 · p. 19, Table 2 · p. 20

Ablations

  • 617 rather than 677 actions, a 9% reduction, and 678 rather than 830 agent turns, an 18% reduction, are reported with the rulebook across three difficulty bands.Each run clears every level and reaches RHAE 100; the comparison removes the natural-language rulebook while keeping the harness, verifier, executor, backbone, and reasoning effort identical.Source: §4 · pp. 19–20, Table 3 · p. 21

Population

  • On wa30, the two-model population clears all nine levels at RHAE 100 and reduces scored actions from 899 to 597 versus the verified single-model trajectory.It matches or improves per-level action efficiency on eight of nine levels; the models begin with different hypotheses and converge as shared evidence resolves disagreements.Source: §4 · pp. 20–21, Table 4 · p. 21

5. Case Study on Atari Pong

The Pong case study tests whether a learned executable world model can support feedback control in an adversarial game. The controller wins 21:0 in each of three evaluated episodes, while revisions to its bounce rules refine its predictions.

Setup

  • The agent learns both physical dynamics and opponent behaviour, then uses a feedback policy to predict candidate-action consequences and choose actions as observations arrive.Before each action, it reconstructs its state from recent frames and actions; trajectory, paddle-motion, and bounce discrepancies guide revisions.Source: §5 · p. 22

Result

  • 21:0 in all three episodes with different openings; the learned controller reaches the maximum score without LLM calls during evaluation.The controller is fixed during evaluation; the three episodes require 3,358, 4,014, and 5,039 actions, respectively.Source: Figure 8 · p. 23, §5 · pp. 23, 25

Baselines

  • Baselines include a direct LLM agent without persistent memory or an executable model, a random policy, and model-free and model-based reinforcement-learning methods.The model-based RL baselines use a training budget of 100,000 actions, equivalent to 400,000 emulator frames.Source: §5 · pp. 23–25

Learning and model revision

  • Observed contacts revise a shared bounce rule from a four-band hypothesis to a centred-deflection formula and then an asymmetric band map.A centre hit permits vy = 0; later contacts show that the formula overestimates vertical return speeds, prompting the asymmetric map.Source: Figure 9 · p. 24, §5 · p. 25

Result

  • 9,504 emulator frames is the learning cost for the maximum episode score; reported model-free and model-based baseline costs are approximately 2.1 × 10^4 and 42 times higher, respectively.The controller compares candidate return trajectories using the learned ball and opponent dynamics without testing each one in the environment.Source: §5 · p. 25

6. Related Work

Related work spans executable world models, language-guided program construction, reflective textual memory, and uncertainty-aware exploration. Memento 3 combines these strands around a persistent rulebook and executable model, while prior ARC-AGI-3 results motivate evaluation beyond public-set completion.

World models and executable representations

  • Programs expose transitions as functions that planners can execute and test, complementing neural world models used for imagined rollouts, model-predictive control, and tree search.Dyna combines real interaction with model-generated experience; Dynalang incorporates language as a predictive modality in a multimodal latent model.Source: §6 · p. 25

Language-guided program induction

  • Language-guided program induction builds explicit models through decomposition, hypothesis translation, counterexample repair, candidate-model search, and execution-based verification.Examples include Parsel, Hypothesis Search, WorldCoder, GIF-MCTS, Code World Models for General Game Playing [21], and Code-as-World [41].Source: §6 · p. 25

ARC-AGI-3 systems

  • ARC-AGI-3 systems use persistent or natural-language hypotheses and replay checks, but stronger replay need not improve decisions when objectives are misidentified or models are used inappropriately.Prior work also reports that requiring an executable model does not uniformly improve action performance within tested configurations.Source: §6 · p. 26

Language rules and reflective memory

  • Textual-memory approaches range from reflections and reusable manuals to editable environment rules, while Code as Model jointly revises a persistent rulebook and executable realization.Its LLM assesses code–rulebook fidelity, and replay checks the executable against recorded transitions.Source: §6 · p. 26

Model uncertainty and exploration

  • The population extension keeps multiple rulebook–executable hypotheses against shared evidence and samples a member to guide exploration, approximating a version space rather than a calibrated Bayesian posterior.Its stated purpose is to reduce dependence on a single early model construction.Source: §6 · p. 26

Continual adaptation and evaluation

  • External-workspace adaptation includes editable prompts, skills, memory, and executable modules; public ARC-AGI-3 completion alone cannot isolate a world-model architecture’s contribution.The related work argues comparisons should also consider model choice, reasoning budget, action efficiency, repeated-run stability, and held-out evaluation.Source: §6 · pp. 26–27

7. Conclusion

Memento 3 uses a natural-language rulebook and compiled code as a revisable world model while keeping the underlying LLM fixed. Its single-model agent completes all 25 public ARC-AGI-3 games and its learned Pong controller wins all three evaluated episodes.

Conclusion

  • On ARC-AGI-3, the single-model agent clears every level of all 25 public games, reaches the RHAE ceiling, and uses 44% of the human action count.The conclusion reports 7,518 actions and mean RHAE of 100.0.Source: §7 · p. 27
  • Code as Model records environment hypotheses in a natural-language rulebook, compiles them into code for prediction and planning, and revises both through interaction with the LLM fixed.The population extension maintains multiple world models, updated and verified against shared interaction history.Source: §7 · p. 27
  • In Atari Pong, the learned model supports a feedback controller that wins 21:0 in three episodes with different openings and no LLM calls during execution.Verified model updates guide subsequent planning and exploration, shaping evidence for further revisions.Source: §7 · p. 27
Loading 2610.11794v1…