PaperlayerView exact revision on arXiv

Source-linked AI summary

EvoBelief: Falsifiable World Models for Self-Improving LLM Agents

Janet Wang, Jiang Guo, Yi Xiang, Jiaqing Yuan, Mingwen Dong, Hideo Kobayashi, Shuaichen Chang, Chandan K. Reddy, Jiarong Jiang

arXiv:2610.12789v1cs.AI

TL;DR

LLM agents can fail despite procedural capability when their beliefs about environment dynamics are incomplete, incorrect, or outdated. EvoBelief evolves those beliefs as falsifiable hypotheses, validating and revising rules before they guide decisions. Across evaluated regimes, it improves task performance, reuses learned rules, adapts to hidden dynamics changes, and discovers quantitative laws.

  • Problem

    Agents that improve their procedures can still fail when their beliefs about environment dynamics are incomplete, incorrect, or outdated.

    Source: §1 · pp. 1–2

  • Method

    EvoBelief forms falsifiable hypotheses, validates them with independent evidence, and revises evidence-linked rules in an external world model while keeping the language model frozen.

    Source: §1 · p. 2

  • Results

    Across evaluated regimes, EvoBelief improves task performance, reuses learned rules, adapts to hidden dynamics changes, and discovers quantitative laws.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Testable, revisable environment knowledge offers a route to self-improvement without model-parameter updates.

    Source: §1 · p. 2, §5 · p. 10

    The experiments use text-based environments with transitions that can be logged and predictions checked when outcomes are observable.

    Source: A LIMITATIONS AND FUTURE WORK · p. 14

Abstract

from arXiv · show

Self-evolving LLM agents typically improve how they act through accumulated experience, reflection, or skills. Yet even procedurally capable agents can fail when their beliefs about environment dynamics are incomplete, incorrect, or outdated. We study a complementary axis of self-evolution: evolving those beliefs during deployment. Such beliefs can support multiple tasks, but incorrect ones can also propagate across them. We introduce EvoBelief, a self-evolving world model that represents environment knowledge as falsifiable hypotheses. Unexpected outcomes and decision-relevant unknowns trigger hypothesis formation. Later observations and controlled experiments provide independent validation before a rule can guide predictions or plans within its supported scope. Counterexamples narrow or retire rules as the environment changes. The language model remains frozen, and each rule remains linked to its evidence and the decisions it changes. Across four benchmarks spanning static dynamics, hidden drift, and scientific discovery, EvoBelief improves task performance and reuses learned rules across tasks. On ALFWorld without actor-side transition memory, it improves success over the matched Frozen World Model by 23.9% and 31.3% with Claude Haiku 4.5 and GPT-5.4, respectively. On MiniGrid, it revises stale rules and recovers after unannounced dynamics changes. In DiscoveryWorld's Reactor Lab, it discovers and validates quantitative laws, completing 83.3% of Normal-tier and 58.3% of Challenge-tier instances, versus 0% for each evaluated baseline. These results demonstrate self-improvement through testable, revisable environment knowledge rather than model-parameter updates.

1 INTRODUCTION

EvoBelief addresses failures caused by incorrect beliefs about environment dynamics, complementing agents that improve how they act. It forms and validates revisable beliefs that can guide decisions across tasks while the language model remains frozen.

1 INTRODUCTION

  • Existing self-evolving agents improve procedures, but a capable agent can still fail when its environmental prior is wrong.A task-specific correction may fix one failure without capturing a property that matters to shipping, storage, and analysis.Source: §1 · p. 1
  • Belief evolution updates a persistent model of environment dynamics rather than only adapting task-specific behavior, so corrected beliefs can support multiple tasks.The paper distinguishes a policy, which specifies how an agent acts toward a goal, from a world model, which captures how the environment responds.Source: §1 · pp. 1–2
  • EvoBelief forms falsifiable hypotheses after prediction mismatches or decision-critical unknowns, validates them with later evidence, and revises them when contradicted.Only validated rules can guide future predictions and plans.Source: §1 · p. 2
  • The language model stays frozen while an external, persistent world model stores evidence-linked beliefs and the conditions under which each rule applies.This design makes it possible to trace how a validated rule changed a prediction or plan.Source: §1 · p. 2
  • Across four benchmarks, EvoBelief improves task success, transfers learned beliefs across tasks, adapts to dynamics shifts, and discovers quantitative laws.The evaluation spans static and changing dynamics as well as scientific discovery.Source: §1 · p. 3

2 RELATED WORK

Related work improves agent behavior through experience and studies world models that support prediction or planning. EvoBelief distinguishes itself by maintaining evidence-scoped beliefs about environment dynamics that gain and lose authority through validation.

Belief states and world models

  • EvoBelief separates episode-local state estimation from persistent, interaction-learned dynamics beliefs, unlike a hand-specified belief over current state (Kumar et al. (2026)).The related work also covers world models used for simulation, planning, rule learning, and transition memories.Source: §2 · p. 3

Experience, memory, and self-evolving agents

  • LLM agents commonly improve through reflection, episodic memories, reusable lessons, procedures, or skills, whereas EvoBelief changes beliefs about deployed environment dynamics (Shinn et al., 2023; Zhao et al., 2024; Park et al., 2023; Wang et al., 2023).The paper also distinguishes approaches that modify prompts, scaffolds, search procedures, or agent code while keeping the model fixed (He et al., 2026; Li et al., 2026; Zhang et al., 2026a; Robeyns et al., 2025; Lee et al., 2026; Xu et al., 2026).Source: §2 · p. 3
  • EvoBelief's distinguishing lifecycle lets learned claims gain behavioral authority only after independent validation and lose it when later counterexamples challenge them.The paper frames this as a requirement for continual adaptation when transition dynamics change and earlier knowledge becomes stale (Wołczyk et al., 2021; Kessler et al., 2023).Source: §2 · p. 4

3 METHOD

EvoBelief separates local episode state from persistent environment-dynamics beliefs, represents those beliefs as testable hypotheses, and gathers passive or active evidence to validate them. Only applicable rules with sufficient independent support guide predictions and plans; counterevidence can narrow, demote, or retire them.

3.1 PROBLEM FORMULATION

  • EvoBelief maintains episode-local situational state z_t and persistent environment-dynamics beliefs W_t, which survives episode resets and changes as evidence arrives.The language model parameters stay fixed, leaving W_t to store environment-specific knowledge learned during deployment.Source: §3.1 · p. 4

3.2 WORLD KNOWLEDGE AS TESTABLE HYPOTHESES

  • Each hypothesis states that condition C and action or procedure A should produce observable consequence Y, making it testable by establishing C, acting, and checking Y.Broader scope is earned through evidence across varied bindings rather than assumed in advance.Source: §3.2 · p. 4
  • Grounded prediction mismatches and decision-critical unknowns trigger hypotheses, while rival explanations may remain pending until evidence distinguishes them.Forming a hypothesis is not evidence for it; an unvalidated claim cannot justify a task-directed action as established knowledge.Source: §3.2 · p. 4

3.3 TESTING HYPOTHESES WITH PASSIVE AND ACTIVE EVIDENCE

  • Hypotheses gain evidence through ordinary transitions that test them or active experiments admitted when their outcomes can change a decision and their setup is executable.Support and falsification criteria are fixed before an experiment; inconclusive tests count as neither support nor counterevidence.Source: §3.3 · pp. 4–5

3.4 FROM EVIDENCE TO AUTHORITY: THE RULE LIFECYCLE

  • Reusable authority requires independent support across episodes, bindings, and tests; counterevidence narrows scope and can demote or retire a rule.Demoted rules retain their evidence, so later support can restore authority; repeated counterexamples retire a rule only when no support remains.Source: §3.4 · p. 5

3.5 USING THE WORLD MODEL FOR PREDICTION AND PLANNING

  • At each decision, applicable validated rules can revise candidate-action predictions, veto actions, or chain into short plans; without an applicable rule, the agent falls back to its W-blind choice.Decision-relevant unresolved claims may instead prompt an admissible experiment, and records of rule-changed actions do not count as new independent support.Source: §3.5 · pp. 5–6

4 EXPERIMENTS

The experiments assess EvoBelief across static environments, hidden dynamics drift, and scientific discovery. Results show strong task performance, adaptation after a silent rule change, and validated law discovery, alongside analyses of rule substrates and behavior.

4.1 SETUP

  • The evaluation covers ScienceWorld, ALFWorld, MiniGrid, and DiscoveryWorld, comparing EvoBelief with ReAct, WorldEvolver, and a matched Frozen WM control.ALFWorld and ScienceWorld are evaluated with and without actor-side transition memory; MiniGrid and DiscoveryWorld omit it.Source: §4.1 · p. 6

4.2 REGIME I: STATIC, LARGELY PRIOR-ALIGNED ENVIRONMENTS

  • EvoBelief has the highest task success rate in every reported ScienceWorld and ALFWorld setting, with validated rules correcting action mechanics and prerequisites on ALFWorld.Across four ALFWorld configurations, over 80% of decisions under a validated rule occur in a family other than the one where it was validated.Source: §4.2 · pp. 6–7
  • ScienceWorld primarily uses live experiments to resolve decision-relevant unknowns, while validated rules guide a smaller number of later decisions.Together, the static benchmarks exercise complementary uses of persistent beliefs: cross-task rule reuse in ALFWorld and targeted evidence gathering in ScienceWorld.Source: §4.2 · p. 7

4.3 REGIME II: HIDDEN DYNAMICS DRIFT

  • On three held-out MiniGrid seeds, EvoBelief solves 69.6% vs 1.3% for Frozen WM, and 80.8% vs 7.1% when colors and room names are permuted.After the silent key switch, counterevidence retires the stale rule, experimentation validates a successor, and success returns to its pre-switch level.Source: §4.3 · p. 8

4.4 REGIME III: SCIENTIFIC DISCOVERY

  • No evaluated baseline completes a Reactor Lab instance or earns knowledge credit; EvoBelief completes most Normal-tier and a majority of Challenge-tier instances.Completed instances follow a lifecycle that tests rival quantitative laws, validates a survivor with a prediction frozen before a held-out trial, and uses the law to configure the reactor.Source: §4.4 · p. 9, Table 3 · p. 9
  • In Plant Nutrients, EvoBelief completes the most instances in both tiers and is the only method earning knowledge credit in both; completion differences from the strongest baselines are uncertain.The confidence interval for knowledge-score differences excludes zero, while the interval for completion differences includes zero.Source: §4.4 · pp. 9–10

4.5 ABLATION AND ANALYSIS

  • Explicit falsifiable rules outperform tested alternative persistent substrates, while EvoBelief leads on held-out state-change F1, exact match, and token F1 on both evaluated benchmarks.WorldEvolver has higher feasibility accuracy on ScienceWorld but lower content scores; behavioral routes differ across environments, with experiments providing most direct intervention in ScienceWorld.Source: Figure 7 · p. 9, §4.5 · p. 10

5 CONCLUSION & DISCUSSION

EvoBelief treats environment beliefs as testable, revisable knowledge rather than changing the agent’s parameters. Across the evaluated regimes, it improves performance, reuses learned rules, adapts to hidden changes, and discovers quantitative laws.

5 CONCLUSION & DISCUSSION

  • Validated rules guide predictions and plans, while counterexamples can revoke their authority when the environment changes.Unresolved hypotheses can also motivate live experiments.Source: §5 · p. 10
  • A verified correction to an environment belief can improve multiple tasks, including tasks the agent has not seen.Where episodic memory already supplies the missing prior, memory and rules partly substitute for one another.Source: §5 · p. 10
  • Across the evaluated regimes, EvoBelief improves task performance, reuses learned rules, adapts to hidden dynamics changes, and discovers quantitative laws.The belief state remains open to inspection.Source: §5 · p. 10

A LIMITATIONS AND FUTURE WORK

The experiments are limited to text-based environments with observable outcomes that can be logged and checked. The authors identify extensions to richer settings and the need to address their different risks and evidence conditions.

A LIMITATIONS AND FUTURE WORK

  • Experiments use text-based environments where transitions can be logged and required outcomes are observable.Extending the lifecycle to multimodal, embodied, and real-world software agents is a next step.Source: A LIMITATIONS AND FUTURE WORK · p. 14
  • Stochastic effects in future settings call for statistical tests rather than count thresholds when committing rules.Live experiments with costly or irreversible interventions would require explicit risk limits or human approval.Source: A LIMITATIONS AND FUTURE WORK · p. 14
  • When episodic memory already supplies what the prior lacks, it overlaps with explicit beliefs, leaving the choice between them as future work.The authors also note that evidence-linked beliefs could be shared across agents or distilled into model weights.Source: A LIMITATIONS AND FUTURE WORK · p. 14

B METHOD DETAILS

The appendix details how EvoBelief represents and tests hypotheses, assigns responsibilities across the frozen model and benchmark adapters, and applies rules during decisions. Its implementations constrain evidence, experiments, and rule commitment to preserve independent validation.

B METHOD DETAILS

  • Hypotheses are merged by a goal-free signature, with an append-only ledger retaining each evidence entry and the transitions it supports.The ledger tracks episode, binding, and test; lifecycle changes cite the entries that caused them.Source: B METHOD DETAILS · p. 14
  • Only environment-grounded observations count toward hypothesis formation or settlement; harness errors and the agent’s own summaries are excluded.Rival hypotheses are tested against the same realized evidence, and tests require independent later occurrences within bounded episode budgets.Source: B METHOD DETAILS · p. 14
  • Reuse requires evidence across distinct bindings and episodes, including one independent of decisions influenced by the hypothesis.For a single long DiscoveryWorld episode, commitment instead uses one preregistered held-out test with a prediction frozen beforehand.Source: B METHOD DETAILS · p. 14

C LANGUAGE-MODEL CALLS, PROMPTS, AND RETRIEVAL

  • The frozen model proposes actions and predictions, while benchmark adapters encode observations and enforce evidence accounting, rule applicability, retirement, and planning.The actor itself does not receive validated rule text; DiscoveryWorld alone may also make a rule-conditioned predictor call.Source: C LANGUAGE-MODEL CALLS, PROMPTS, AND… · p. 15

C.1 ROLES AND CALLS

  • Call patterns vary by benchmark: ALFWorld batches anchor predictions, while DiscoveryWorld uses rule-conditioned prediction only when a retrieved rule applies.The actor receives no rule, prediction, or plan; Table 5 summarizes language-model calls by role.Source: §C.1 · pp. 15–16, Table 5 · p. 16

C.2 CANONICAL PROMPTS BY ROLE

  • The actor chooses executable actions from observable information, while transition analysis updates typed state claims and reports whether predictions matched.Whether a verdict counts as evidence is decided by code; DiscoveryWorld’s knowledge judge scores the stored knowledge after the episode.Source: §C.2 · pp. 16–17

C.3 ACTOR-SIDE TRANSITION MEMORY

  • Transition memory retrieves up to four realized public transitions from completed earlier episodes, ranked by lexical overlap with the current goal and state.It is shared across methods when enabled, contains no outcome labels, and is not used by EvoBelief on MiniGrid.Source: §C.3 · p. 18

D DECISION-STEP ALGORITHM

  • Each decision forms candidates without access to the belief store, predicts outcomes, checks applicable rules, admits eligible experiments, plans, selects, executes, and absorbs evidence.Rules change predictions only when they support the change, and every route spends the same task budget.Source: D DECISION-STEP ALGORITHM · p. 18

E EXPERIMENTAL SETUP DETAILS

The experimental setup spans stationary, changing, and scientific-discovery environments, with task-specific protocols and matched comparisons. The authors describe benchmark settings, baselines, fairness controls, and diagnostic measures.

E EXPERIMENTAL SETUP DETAILS

  • The benchmarks cover stationary but partially hidden ScienceWorld and ALFWorld tasks, changing dynamics in MiniGrid, and scientific discovery in DiscoveryWorld.DiscoveryWorld tests hidden quantitative laws in Reactor Lab and nutrient rules in Plant Nutrients across Normal and Challenge tiers.Source: E EXPERIMENTAL SETUP DETAILS · p. 19
  • ALFWorld and ScienceWorld use actor settings with and without transition memory; MiniGrid EvoBelief runs omit it, and DiscoveryWorld methods use none.The DiscoveryWorld study uses twelve fresh instances per tier, four methods on the same tasks, and one attempt per instance.Source: E EXPERIMENTAL SETUP DETAILS · p. 19
  • Frozen WM is a matched prior-driven control, while ReAct, ITP-I, and WorldEvolver provide additional baselines under the shared harness.The paired Frozen WM comparison evaluates EvoBelief’s complete system, including its experiments; WALL-E 2.0’s published ALFWorld result is not directly comparable.Source: E EXPERIMENTAL SETUP DETAILS · p. 19
  • Main comparisons attempt each task once, with methods sharing task sets, order, seeds, interfaces, context limits, retry policy, and scoring.MiniGrid diagnostics split results at the hidden changepoint and record rule lifecycle events and decisions changed by validated rules.Source: E EXPERIMENTAL SETUP DETAILS · pp. 19–20

F ADDITIONAL RESULTS

The appendix consolidates what the belief machinery records across benchmarks and explains how paired comparisons and uncertainty intervals are calculated.

F.1 MECHANISM SUMMARY

  • The four benchmark implementations count different units, and their rows are not stages of one funnel.ALFWorld records rule branches, MiniGrid guarded rules and versions, ScienceWorld experiments and support records, and DiscoveryWorld candidate laws.Source: Table 6 · p. 20
  • Validated rules are far fewer than support records because repeated observations in one context add records but little independent evidence.For example, ScienceWorld records 573 experimental support records and 4,031 passive supports, alongside 54 experiments opened and 39 completed.Source: Table 6 · p. 20

F.2 PAIRED TESTS AND CONFIDENCE INTERVALS

  • The paired comparisons use exact two-sided McNemar tests and 10,000-resample percentile bootstrap intervals for success-rate differences.DiscoveryWorld uses the sealed campaign’s instance-paired analysis.Source: §F.2 · p. 20
  • MiniGrid episode-level tests and intervals do not account for dependence because episodes within a stream share an evolving belief state.Table 7 pairs episodes by stream and position, while post-versus-pre comparisons pair episode i + 20 with episode i.Source: Table 7 · p. 21

F.3 REGIME I: STATIC-ENVIRONMENT DETAILS

On ALFWorld, validated rules increasingly help across deployment, especially on task families where the prior misstates procedures. The appendix also describes rule confirmation and cross-task use.

F.3 REGIME I: STATIC-ENVIRONMENT DETAILS

  • EvoBelief’s cumulative advantage widens over matched tasks, fastest on process types where the prior misstates procedures, while solving more tasks with fewer decisions and model calls.Generic household rules are reused in later task types, and each process rule is committed within its first few tasks.Source: §F.3 · p. 21
  • Of 1,115 decisions changed by validated rules, 907 had their predicted effect fully confirmed in the next observation, and none was classified as an effect mismatch.199 outcomes were inconclusive because a closed receptacle made one predicted atom unobservable; nine refusals were recorded as evidence against a condition or as inconclusive.Source: §F.3 · p. 22
  • Rules persist across 134 unseen tasks, and validation requires independent evidence from at least two tasks, making same-task uses rare by construction.Table 8 classifies use by later task, family, unseen object type, changed decision, and confirmation; Table 9 lists validation tasks, later tasks, bindings, and rule effects.Source: §F.3 · p. 22, Table 8 · p. 22, Table 9 · p. 22

F.4 REGIME II: DRIFT DETAILS

MiniGrid tests whether EvoBelief can learn hidden rules and recover when dynamics change without announcement. The results describe rule revision, active experimentation, and committed planning across three task families.

F.4 REGIME II: DRIFT DETAILS

  • The prior-governed agent fails on MG-C and MG-M because each solution requires an unexpected step within a 24-step budget, while its MG-R control retains the decoy key and silent switch.Its MG-C attempts leave the door open before lifting; on MG-M it unlocks the door but then tries to walk through it while shut.Source: §F.4 · pp. 22–23
  • After the silent switch, EvoBelief reaches a similar level across all three families and recovers to or above pre-switch performance in the final dozen episodes of every stream.On MG-R, it loses nothing before the switch and pulls furthest ahead in late post-switch episodes after committing a successor rule; Frozen WM uses 24 decisions and 178 model calls per episode against EvoBelief’s 98 calls.Source: §F.4 · p. 23
  • On MG-C, executing committed plans is especially useful because the solution is a chain no single experiment completes; the shadow variant instead relies on prior actions and experiments.The no-commitment-gate variant has success and failed authorized-step rates close to the full method on this deterministic benchmark, a reported null result for the gate.Source: §F.4 · p. 23, Table 11 · p. 23
  • A stale MG-C rule is retired after a counterexample, a successor carries plans through episode 20, and two post-switch refutations open a new evidence epoch before a green-key rule is committed.On MG-M, the two-stage door is represented by separate unlock and empty-handed-open rules, with the first relearned after the switch.Source: §F.4 · pp. 23–24
  • The families require different hidden transition rules: MG-C requires a shut door and empty hands to lift the box, while MG-M requires separate unlock and open toggles.The working key changes from blue during the first 20 episodes to green afterward; the switch is not announced.Source: Figure 8 · p. 24

F.5 REGIME III: DISCOVERY DETAILS

DiscoveryWorld’s discovery results distinguish candidate-law testing from validation and task planning. The appendix also reports Plant Nutrients revisions, fresh-instance outcomes, and a baseline rerun under a higher output ceiling.

F.5 REGIME III: DISCOVERY DETAILS

  • On the 24 fresh Reactor Lab instances, a calibration law learned on disjoint instances did not solve tasks without local experimentation.A frozen belief preloaded with that law earned half the knowledge credit on four Normal and six Challenge instances but completed none.Source: §F.5 · p. 24
  • Reactor Lab hypotheses are candidate laws with predictions frozen before experiments, and a law is validated only by a confirmed held-out prediction.One executed experiment can settle multiple rival laws; the table records crystal measurements, frozen predictions, activation frequencies, plan outcomes, and knowledge scores.Source: Table 12 · p. 25, Table 13 · p. 26
  • The revised Plant Nutrients implementation gates absence-based counterexamples on a publicly created opportunity and applies validated rules across the full planting plan.The reported code was frozen before a blind run on 12 fresh Normal and 12 fresh Challenge instances; development used prior pilot instances.Source: §F.5 · p. 25
  • On Plant Nutrients, 19 task instances were completed; 17 through validated task plans, while one used a discriminating plan and one had no plan.Knowledge margins over ReAct, WorldEvolver, and the Hypothesizer were 58.3, 66.7, and 75.0 percentage points on Normal, and 33.3 percentage points over each on Challenge.Source: §F.5 · p. 25
  • The original Hypothesizer output ceiling was 3,072 tokens, and Challenge runs ended on truncation; reruns raised the ceiling to 8,192 tokens, with Plant Nutrients Normal completions moving from eight to six on the same 12 instances.The reruns changed only the ceiling and used one attempt per cohort; the reported original runs are described separately.Source: §F.5 · pp. 25–26, Table 14 · p. 26

F.6 HELD-OUT PREDICTION EVALUATION

Held-out structured prediction is evaluated on transitions from episodes disjoint from belief formation. EvoBelief leads on three metrics on both benchmarks, while Feasibility results are less consistent.

F.6 HELD-OUT PREDICTION EVALUATION

  • Each benchmark contributes 1,500 transitions from episodes disjoint from belief formation, evaluated with Haiku 4.5 and one frozen prediction prompt.Predictions use greedy decoding, and each transition is scored once.Source: §F.6 · p. 26, Table 15 · p. 27
  • On both benchmarks, EvoBelief leads every other row on State-∆F1, EM, and Token F1 with 95% intervals excluding zero; Feasibility evidence is mixed.WorldEvolver-full’s ScienceWorld Feasibility lead excludes zero, while every ALFWorld Feasibility difference has an interval including zero.Source: Table 15 · p. 27, §F.6 · p. 27
  • State-∆F1 scores canonical state-change atoms, Feasibility scores action executability, and EM and Token F1 compare structured state records.A correct prediction of no state change receives a State-∆F1 score of one.Source: §F.6 · p. 26, Table 15 · p. 27

Related papers

Loading 2610.12789v1…