PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Evolving World Models for LLM Agent Planning

Xuan Zhang, Wenxuan Zhang, See-Kiong Ng, Yang Deng

arXiv:2606.30639v2cs.AIcs.CL

TL;DR

Deployment shifts can make world-model predictions unreliable, and unreliable foresight may mislead agent decisions. WORLDEVOLVER revises deployment-time context with episodic and semantic memory and selectively exposes predictions while keeping the agent and parameters frozen. It achieves the strongest Word2World prediction accuracy across three backbones and improves downstream agent success rates in the reported evaluations.

  • Problem

    Deployment shifts can cause frozen world models to mispredict transitions, while unreliable foresight can mislead the downstream agent.

    Source: §1 · pp. 1–2

  • Method

    WORLDEVOLVER revises deployment-time context through retrieved realized transitions, mismatch-derived heuristic rules, and confidence filtering while the agent and model parameters remain frozen.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    WORLDEVOLVER achieves the strongest prediction accuracy on Word2World across three backbones and improves downstream agent success rates.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The reported results suggest that reliable foresight depends on how environmental signals are processed and presented to the agent.

    Source: §5 · p. 9

    Selective Foresight depends on token-level probabilities that some closed-model APIs lack and assumes confidence tracks accuracy, a relationship that can vary across environments and backbones.

    Source: Limitations · p. 10

Abstract

from arXiv · show

World models offer a principled way to equip long-horizon LLM agents with foresight: predictions of action consequences before execution. However, unreliable foresight can be ignored, misused, or even degrade downstream decision-making. In this paper, we introduce WorldEvolver, a self-evolving world model framework that revises its deployment-time context while keeping the downstream agent and all model parameters frozen. WorldEvolver integrates three modules: (i) Episodic Memory, which exploits real action transitions through retrieval-based simulation; (ii) Semantic Memory, which extracts persistent heuristic rules from prediction-observation mismatches; and (iii) Selective Foresight, which filters low-confidence predictions before integrating them into agent reasoning context. We evaluate WorldEvolver on ALFWorld and ScienceWorld, measuring world model prediction accuracy on Word2World and downstream agent success rate on AgentBoard. Extensive experiments show that WorldEvolver achieves the highest prediction accuracy across three backbones and leads other world model baselines on downstream agent success rate, demonstrating that test-time memory revision enhances both predictive fidelity and planning performance.

1 Introduction

World-model foresight can complement memory-based adaptation, but deployment shifts can make frozen models unreliable and unreliable predictions can mislead agents. WORLDEVOLVER revises deployment-time context through three mechanisms while leaving the agent and model parameters frozen.

1 Introduction

  • Distribution shifts can make frozen world models mispredict transitions, while online parameter updates are costly at LLM scale and risk over-editing or catastrophic forgetting.Realized transitions and prediction-observation mismatches provide deployment-time evidence that can be retained as context instead.Source: §1 · pp. 1–2
  • The oracle diagnostic compares no, noisy, and perfect foresight by scoring generated actions against teacher actions using exact action accuracy.The supplied passages describe the conditions and metric but do not report the figure’s values.Source: Figure 2 · p. 2
  • Episodic Memory retrieves realized action transitions, Semantic Memory derives persistent heuristic rules from mismatches, and Selective Foresight filters low-confidence predictions before they reach the agent.The loop stores executed transitions and updates semantic memory from the corresponding prediction-observation mismatch.Source: §1 · p. 2
  • WORLDEVOLVER revises deployment-time world-model context while keeping the downstream agent and all model parameters frozen.Its design addresses grounding predictions in realized transitions, turning mismatches into reusable knowledge, and filtering unreliable foresight.Source: §1 · p. 2
  • The paper benchmarks WORLDEVOLVER against RAWM-ϕ and ITP-I on Word2World, ALFWorld, and ScienceWorld, evaluating prediction alignment and downstream planning improvements.World models are presented as a complement to memory-based adaptation because they anticipate future outcomes.Source: §1 · pp. 1–2

2 Related Work

Prior work explores world models, memory, and self-evolving agents through varied modeling and adaptation strategies. The paper distinguishes WORLDEVOLVER by evolving the predictive model’s deployment-time context rather than the agent policy or model parameters.

2 Related Work

  • LLM-agent world models span planner-simulators, next-state predictors, explicit pre-action prediction, task-level knowledge models, offline training, and joint optimization.The cited approaches include Hao et al. (2023), Chae et al. (2025), Fu et al. (2025), Qiao et al. (2024), Fang et al. (2025), Yang et al. (2025), and Ding et al. (2026).Source: §2 · p. 3
  • Agent self-evolution includes verbal feedback, skill libraries, distilled experience, persistent context, autonomous task generation, and learning from failures or interaction data.Examples include Wang et al. (2024a), Packer et al. (2023), Zhao et al. (2024), Huang et al. (2025), and Jung et al. (2025).Source: §2 · p. 3
  • WORLDEVOLVER starts with empty memory, accumulates executed transitions and mismatch-derived rules online, confidence-gates one-step predictions, and keeps the downstream policy frozen.RAWM-ϕ instead retrieves from a fixed transition library with a PPO-trained head.Source: §2 · p. 3
  • Most existing methods evolve the agent policy or external context rather than the world model that supports future prediction under changing environments or unreliable foresight.Closer approaches retrain world models on rollouts, update abstract state models during exploration, or alternate neural and symbolic updates.Source: §2 · p. 3

3 Methodology

The method formulates planning as prediction over textual interaction histories while the agent and world model remain frozen. WORLDEVOLVER revises contextual evidence through episodic and semantic memory, filters predictions by confidence, and aligns updates with executed actions.

3.1 Problem Formulation

  • The task is a partially observed interaction process: the agent sees a textual history of observations and actions rather than the hidden environment state.The world model predicts future observations from the current interaction state and a candidate action; the method focuses mainly on one-step foresight.Source: §3.1 · p. 3

3.2 WORLDEVOLVER

  • WORLDEVOLVER augments a frozen world model with contextual evidence rather than changing its parameters, using Episodic Memory, Semantic Memory, and Selective Foresight.Episodic Memory reuses realized transitions; Semantic Memory converts prediction-observation mismatches into persistent textual heuristics.Source: §3.2 · p. 4, Figure 3 · p. 4
  • Episodic Memory retrieves prior realized transitions using action-token Jaccard similarity and renders selected observation-action-outcome records as context.Retrieval is based on previously accumulated experience, since new records are appended only after execution.Source: §3.2 · p. 4
  • Semantic Memory factorizes predicted and observed outcomes into tuples, turns mismatches into candidate rules, and updates rule evidence as later observations support or contradict them.Only rules with positive evidence scores are added to context; updates can aggregate a mini-batch of mismatch cases.Source: §3.2 · pp. 4–5
  • Selective Foresight exposes a prediction only when its confidence meets a threshold, otherwise withholding it from the agent to reduce the risk of misleading decisions.When token probabilities are available, confidence is derived from the geometric mean token probability.Source: §3.2 · p. 5

3.3 Agent Planning with World Models

  • The planning loop drafts an action, retrieves transitions, predicts its consequence, and conditions the policy on the prediction only if it passes the confidence threshold.If the executed action differs from the draft, the model predicts the executed action again; after execution, the realized transition is stored and its mismatch informs semantic-memory updates.Source: §3.3 · pp. 5–6

4 Experiment

The experiments assess next-observation prediction and closed-loop planning across two environments, multiple backbones, and agent styles. WORLDEVOLVER leads the world-model baselines in planning and achieves the highest prediction accuracy across the evaluated prediction settings; the discussion examines memory choices, online learning, and limits on translating prediction gains into task success.

4.1 Setups

  • Evaluations use Word2World transition data and AgentBoard tasks in ALFWorld and ScienceWorld, with five trials per task and up to 30 steps per trial.Prediction tests contain 195 trajectories per environment; planning uses 134 ALFWorld tasks and 90 ScienceWorld tasks.Source: §4.1 · p. 6

4.3 Experiments on Agent Planning

  • WORLDEVOLVER is the strongest world-model method across all eight planning settings, and selective foresight improves or ties its no-foresight variant in every setting.Relative to RAWM-ϕ, the no-selective-foresight variant improves average Success Rate by 3.67 points.Source: Table 2 · p. 7, §4.3 · pp. 7–8

4.2 Experiments on World Model Prediction

  • The full WORLDEVOLVER achieves the highest prediction accuracy across both environments and all three backbones, with gains from combining episodic retrieval and semantic rules.Semantic Memory alone gives modest gains over Zero-Shot, while the Episodic Memory variant provides substantially larger improvements and outperforms RAWM-ϕ.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.4 Discussion

  • Increasing episodic retrieval size from 1 to 5 yields the largest prediction gains, while most semantic batch-size choices differ by within two Exact Match points.The selected settings are kME = 5 and kMS = 1.Source: §4.4 · p. 8, Figure 5 · p. 8
  • On ScienceWorld with Gemma-4-26B-A4B, WORLDEVOLVER variants increasingly outperform RAWM-ϕ and ITP-I as trial index grows.Episodic and semantic memories accumulate across trials and tasks within the same environment, supporting later replanning.Source: §4.4 · p. 8, Figure 4 · p. 8
  • Resetting memory before each trajectory substantially reduces prediction accuracy, but resetting it before each task changes best-of-5 planning success by only 0.5 to 1.1 points.The planning controls retain memory across the five trials within each task.Source: §4.4 · p. 9, Table 3 · p. 9, Table 4 · p. 9
  • One-step prediction accuracy does not directly translate into higher terminal task success because prediction errors affect planning only through their effects on actions and subsequent states.Planning scores task completion, whereas prediction evaluation directly scores one-step forecasts.Source: §4.4 · p. 9
  • GPT-5.4-mini’s selective-foresight results are mixed, with confidence intervals overlapping the no-world-model results in all four settings.The discussion attributes its smaller gains to stronger baseline reasoning and less headroom for additional foresight.Source: §4.4 · p. 9, Figure 6 · p. 9

5 Conclusion

WorldEvolver revises world-model context at test time through episodic memory, semantic memory, and selective foresight rather than updating parameters. It reports stronger prediction accuracy and downstream planning performance.

5 Conclusion

  • The framework revises world-model context at test time through episodic memory, semantic memory, and selective foresight without updating model parameters.The downstream agent and model parameters remain frozen.Source: §5 · p. 9
  • Experiments on ALFWorld and ScienceWorld show improvements in both world-model fidelity and downstream planning performance.Source: §5 · p. 9
  • WorldEvolver achieves the strongest Word2World prediction accuracy across three backbones and improves downstream agent success rates.The conclusion says these results suggest that reliable foresight depends on how environmental signals are processed and presented to the agent.Source: §5 · p. 9

Limitations

The evaluation and confidence-filtering design impose scope and deployment constraints. The study uses two controlled text environments, while token-probability availability and confidence-to-accuracy alignment can limit filtering.

Limitations

  • Evaluation covers controlled long-horizon text environments, ALFWorld and ScienceWorld, not web navigation, code generation, robotics, or multimodal interaction.The authors identify extending evaluation to those settings as future work.Source: Limitations · p. 10
  • Token-level prediction probabilities may be unavailable through some closed-model APIs, requiring alternative confidence estimators.The authors mention self-consistency and learned calibration models as alternatives.Source: Limitations · p. 10
  • Dynamic filtering assumes prediction confidence correlates with accuracy, but that relationship can vary across environments and backbone models.The authors leave more robust confidence estimation and adaptive filtering for future work.Source: Limitations · p. 10

Ethical Considerations

The authors provide code and identify the benchmarks as publicly available for research use. They also disclose auxiliary AI assistance for coding and paper writing.

Ethical Considerations

A Experimental Setups

The experimental setup compares inference-only world-model baselines and tests foresight with two downstream agent policies. The baselines differ in prompting, retrieval, and adaptive imagination.

A Experimental Setups

  • The compared world-model baselines are inference-only and do not use gradient updates.Source: A Experimental Setups · p. 13
  • Zero-Shot predicts the next observation from the task description, current state, and proposed action in a direct prompt.Source: A Experimental Setups · p. 13
  • RAWM-ϕ retrieves the top-1 similar stored transition from a fixed library and formats it as an in-context prediction example.The retrieval source is the Word2World test split, and retrieval uses Qwen3-Embedding-8B across both environments.Source: A Experimental Setups · p. 13
  • ITP-I moves adaptive horizon selection and imagination into the world model, choosing k from 0 to kmax=5.Prediction evaluation restricts ITP-I to one-step imagination; multi-step imagination is used only for agent-planning evaluation.Source: A Experimental Setups · p. 13
  • ReAct and ReflAct are the two evaluated agent policies for testing transfer across agent types.ReflAct adds goal-state reflection before action selection.Source: A Experimental Setups · p. 13

B Implementation Details

Implementation uses shared generation settings and persistent memories, with selective foresight gated by confidence thresholds calibrated on prediction-quality curves.

Generation settings

  • All generations use temperature 0, top-p 0.5, random seed 42, and a 32,768-token context window.The mismatch critic and factorized-tuple mapping function share the world-model backbone.Source: B Implementation Details · p. 13

Selective Foresight

  • Selective Foresight applies when geometric-mean token probability q_t exceeds threshold τ, with calibration curves relating confidence to Exact Match and Token F1.The procedure is also used for GPT-5.4-mini; thresholds are selected per cell, with values near 1−10−5 performing well.Source: B Implementation Details · p. 14, Figure 7 · p. 14, Table 5 · p. 14

Persistent memory

  • Episodic and semantic memories persist across tasks within each environment to support deployment-time continual learning.The memories are denoted M_E and M_S, respectively.Source: B Implementation Details · p. 14

C Evaluation and Analysis

Across evaluation settings, WORLDEVOLVER offers a favorable runtime–prediction trade-off and improves planning most visibly on unsaturated, transition-sensitive tasks. Its prediction and planning behavior is examined across deployment order, confidence, and evaluation seeds.

Runtime and prediction

  • WORLDEVOLVER provides the best trade-off between runtime and prediction performance across evaluated backbones.For Gemma-4-26B-A4B, runtime rises from 1.05s to 1.48s per transition, versus 1.42s for ITP-I.Source: C Evaluation and Analysis · p. 14, Table 6 · p. 14

Memory evolution and order robustness

  • WORLDEVOLVER stays above Zero-Shot, RAWM-ϕ, and ITP-I in trajectory-macro Exact Match across environments and backbones, with a mid-deployment lift on ScienceWorld and stable high accuracy on ALFWorld.Qwen3.5-9B has lower, noisier local Exact Match; performance is consistent across five seeded trajectory orders.Source: C Evaluation and Analysis · pp. 15–16, Figure 9 · p. 16

Difficulty breakdown

  • 46.00 to 54.00 success on GPT-5.4-mini ScienceWorld ReflAct hard tasks is the clearest reported gain; WORLDEVOLVER improves Gemma-4-26B-A4B across both ScienceWorld agent types.The GPT-5.4-mini gain is without Ft; with Ft, success is 50.00.Source: C Evaluation and Analysis · p. 15, Figure 8 · p. 15

Action-level effect

  • For Gemma-4-26B-A4B, the study measures how often visible selective foresight changes the draft action across all foresight-exposed decision steps in AgentBoard planning rollouts; all Table 9 rows report the w/ Ft condition.Source: C Evaluation and Analysis · p. 15, Table 9 · p. 16

Task-type breakdown

  • WORLDEVOLVER gains are clearest on transition-sensitive ALFWorld PICK2 and ScienceWorld Lifespan, Thermom., and Chemistry, while PICK is nearly saturated and State Change remains near zero.CLEAN, COOL, and LOOK remain difficult, especially for Gemma-4-26B-A4B.Source: C Evaluation and Analysis · pp. 15–16

Foresight confidence

  • Higher-confidence predictions generally have higher Exact Match, and WORLDEVOLVER stays above Zero-Shot, RAWM-ϕ, and ITP-I across confidence-retention levels.The confidence gate therefore filters a stronger predictive signal across the reported settings.Source: C Evaluation and Analysis · p. 16

Statistical uncertainty and Success@1

  • WORLDEVOLVER w/ Ft has the highest best-of-5 point estimate in all four settings, though confidence intervals overlap, and the highest Success@1 among world-model methods in all four.Across all methods, its Success@1 is highest in both ScienceWorld settings, ties the no-world-model baseline on ALFWorld ReflAct, and trails it on ALFWorld ReAct.Source: C Evaluation and Analysis · p. 16, Table 12 · p. 17

Retrieval failures

  • Representative retrieval failures show that task-relevant memories can still miss fine-grained executability or command-validity constraints.The examples include a predicted pickup where the gold observation is “Nothing happens” and a predicted circuit connection rejected by the simulator.Source: C Evaluation and Analysis · p. 17

D Case Study

The case study shows that episodic retrieval can find task-relevant transitions yet still overgeneralize when action success depends on fine-grained executability or command validity.

Cross-environment pattern

  • In both environments, Episodic Memory retrieves task-relevant transitions but predicts success for actions the simulator cannot execute.The cited cases concern ALFWorld pickup and ScienceWorld circuit construction.Source: D Case Study · p. 18, Table 15 · p. 18, Table 16 · p. 18

ALFWorld

  • In ALFWorld, nearby successful pickup and location memories lead to a predicted pickup even though the specific target command produces no state change.The failure turns on a fine-grained executability condition.Source: D Case Study · p. 18, Table 15 · p. 18

ScienceWorld

  • In ScienceWorld, circuit-transition memories identify the relevant task family but do not ensure that a particular connection-command phrasing is valid.The model overgeneralizes from nearby valid circuit updates and predicts a successful connection.Source: D Case Study · p. 18, Table 16 · p. 18

E Prompts

The appendix presents the agent and world-model prompts, along with the prompts used to update Semantic Memory.

Agent prompts

  • ALFWorld ReAct and ReflAct prompts are shown alongside ScienceWorld prompts, which enumerate six command groups and use a per-turn Thought/Action format.The ALFWorld ReflAct prompt replaces ReAct Thought with Reflection.Source: E Prompts · p. 18, Figure 14 · p. 21, Figure 15 · p. 21

World-model prompts

  • World-model prompt figures cover Zero-Shot, RAWM-ϕ, ITP-I, Episodic Memory, Semantic Memory, and WORLDEVOLVER configurations.WORLDEVOLVER combines episodic and semantic grounding with post-generation confidence filtering.Source: E Prompts · p. 18, Figure 18 · p. 22, Figure 19 · p. 22

Memory-update prompts

  • The observation factorizer and preservation-rule extractor prompts support Semantic Memory updates from prediction-observation mismatches.The factorizer forms triples whose set difference identifies mismatches; the extractor processes batches and returns rules appended to memory.Source: E Prompts · p. 18, Figure 24 · p. 24, Figure 25 · p. 25
Loading 2606.30639v2…