PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Learning to Predict the World Help Agents Act? Auditing World-Model Post-Training

Xinyu Che, Hang Yan, Yanchen Liu, Haochen Liu, Ruifeng Li, Anran Shi, Heng Wang, Jun Liu

arXiv:2609.33335v1cs.CLcs.AI

TL;DR

Agents must anticipate how actions change their environments, yet existing world-model post-training comparisons do not establish whether gains depend on correct prediction targets. The paper tests this with mismatched observations and independent random rewards, finding task gains despite reduced prediction accuracy and coverage gains without environment-informed rewards.

  • Problem

    Before-and-after world-model post-training comparisons combine learning from correct prediction targets with additional training, leaving open whether task gains depend on correct environment information.

    Source: §1 · p. 2

  • Method

    The paper compares true and in-distribution mismatched next-observation targets, plus rewards independent of the environment and response, while measuring prediction accuracy and task performance.

    Source: §1 · p. 2

  • Results

    Mismatched targets lower prediction accuracy yet retain task gains, while random-reward training raises VisualWebArena pass@64 by 14.3% relative to BASE.

    Source: §1 · p. 3, §6 · p. 9

  • Takeaways & Limitations

    Coverage gains can occur without environment information in the reward, including on VisualWebArena without matched-observation rewards or an external multimodal teacher.

    Source: §1 · p. 2, §6 · p. 10

Abstract

from arXiv · show

Predicting how an environment will change before acting is a natural route to better decision making for agents. Recent post-training methods therefore require agents to predict the next observation and turn that prediction into a reward or a direct supervision signal, which is called world model. Existing next-observation training methods help the agent to learn the environmental content. However, they additionally involve an optimization process, which may introduce several effects other than learning to predict the world. Consequently, where the performance gain comes from during the training process remains an open question. We answer this research question through replacing true next-observation targets with in-distribution mismatched observations during the training process. Across two interactive text environments, mismatched targets lower prediction accuracy by 15.3-61.6% relative to ground-truth targets, yet retain substantial task gains over the base model. Compared with the base model, trained models consider more candidate actions and exhibit less looping. We also introduce a setting that replaces prediction-based rewards with independent random signals. This training expands task coverage (pass@64) even when the reward carries no environment information. We also generalize this finding to VisualWebArena, where random-reward training raises pass@64 by 14.3% relative to the base model, without observation-matching rewards or an external multimodal teacher for reward construction.

1 INTRODUCTION

Agents need to anticipate action-conditioned changes in their environments, but standard next-token pretraining does not directly supervise those transitions. This paper tests whether world-model post-training gains depend on correct environment information or can also arise from other effects of training.

1 INTRODUCTION

  • Standard next-token pretraining predicts text continuation but does not directly supervise how an action changes the environment, despite agents needing to anticipate those changes.Recent post-training methods use prediction-based rewards or direct prediction supervision, often assuming more accurate predictions should improve action.Source: §1 · p. 1
  • The paper asks whether task gains depend on correct environment information or can also arise from other effects introduced by training.Before-and-after comparisons combine learning from correct prediction targets with additional training, so improved task performance alone does not establish that more accurate prediction caused the gain.Source: §1 · p. 2
  • On VisualWebArena, random-reward training raises pass@64 by 14.3% relative to BASE, without observation-matching rewards or an external multimodal teacher for reward construction.This extends the text-environment finding and suggests a simpler training design for multimodal agents.Source: §1 · pp. 2–3
  • Controlled experiments use mismatched in-distribution observations and independent random rewards to test the contribution of correct environment information while measuring prediction accuracy and task performance.The conditions compare BASE, true next-observation targets, mismatched targets from the same training distribution, and rewards independent of the environment and response.Source: §1 · pp. 2–3
  • Mismatched targets substantially reduce prediction accuracy while preserving task gains, and random-reward training also expands task coverage, showing gains cannot be attributed solely to improved environment prediction.The study also reports more candidate-action consideration and less looping among trained agents than BASE.Source: §1 · pp. 2–3

2 ATTRIBUTION DESIGN

The attribution design compares world-model post-training methods and controls observation content or reward information while holding training procedures fixed. It measures prediction accuracy, task selection and coverage, and action-generation behavior across text-environment settings.

2.1 METHODS UNDER STUDY

  • RL-based post-training predicts the next observation and optimizes the policy with GRPO using predicted-versus-observed state agreement; the distillation method supervises the same prediction through a teacher’s token distribution.The distillation comparison follows the on-policy self-distillation procedure in COMAP, with the teacher receiving the true or mismatched next observation.Source: §2.1 · p. 3

2.2 EXPERIMENTAL DESIGN

  • The RL conditions hold the training setup fixed while varying reward construction, and the distillation comparison changes only next-observation supervision between GT and MIS.BASE is untrained; GT uses true next observations; MIS substitutes another in-distribution observation; COIN uses a reward independent of environment and response. Prediction accuracy uses an independent LLM judge, while pass@1 measures selection and pass@64 measures coverage across attempts.Source: §2.2 · pp. 3–4

2.3 EXPERIMENTAL SETUP

  • Experiments use ALFWorld and ScienceWorld with Qwen2.5-7B-Instruct, and the distillation comparison is conducted on ALFWorld.ALFWorld is evaluated with and without supplied action candidates; ScienceWorld is evaluated only without them. Prediction accuracy uses 1,200 held-out transitions, and task evaluation samples 64 trajectories per task.Source: §2.3 · p. 4

3 ANALYSIS RESULTS

Across text environments, correct observation content strongly affects prediction accuracy but has smaller or absent effects on task coverage; random-reward training also expands coverage. Training is associated with broader action search and less looping, including when predictions are mismatched or rewards are random.

3.1 PREDICTION ACCURACY AND TASK PERFORMANCE DO NOT MOVE TOGETHER

  • 15.3–61.6% lower prediction accuracy under MIS accompanies only 0.51–2.75% absolute pass@1 differences and at most 3.0% pass@64 differences across three settings.On ALFWorld, GT exceeds MIS by 2.75% in list-regime pass@1 while pass@64 barely changes; the pattern also holds in ScienceWorld and privileged OPSD.Source: §3.1 · p. 4, Table 2 · p. 5

3.2 RANDOM REWARD ALSO EXPANDS COVERAGE

  • COIN raises ALFWorld list-regime pass@64 from 56.20% for BASE to 87.23%, and GT’s pass@1 advantage over COIN narrows as k increases.In ALFWorld’s list regime, GT reaches 37.30% pass@1 versus 30.41% for COIN; correct targets primarily improve single-attempt selection, while random-reward training expands coverage.Source: §3.2 · pp. 4–5, Figure 2 · p. 5, Table 2 · p. 5

3.3 CORRECT CONTENT DOES NOT IMPROVE FREE-FORM GENERATION

  • In no-list ALFWorld, GT reaches 52.92% pass@64 while COIN reaches 74.09%, and GT has a higher action failure rate than MIS, 65.32% versus 49.61%.ScienceWorld shows the same pass@64 ordering, and Figure 2 shows GT solving the fewest tasks among the three core GRPO conditions in both environments.Source: Figure 2 · p. 5, Table 2 · p. 5, §3.3 · p. 6

3.4 TRAINING RESHAPES ACTION GENERATION WITHOUT REQUIRING CORRECT OBSERVATION CONTENT

  • After complete CoT, GT, MIS, and COIN final-action distributions become more similar to one another and diverge jointly from BASE, despite GT exceeding MIS by 9.19 nats in prediction preference.The prediction-preference difference has a paired bootstrap 95% interval of [7.65, 10.71]; Figure 3 reports prediction preferences and action-distribution overlap.Source: §3.4 · p. 6, Figure 3 · p. 6
  • Across trained conditions, action revision and multiple-candidate generation increase in both regimes, while the list regime further increases both rates and COIN reduces looping from 24.7% to 1.9%.In the no-list regime, entity-reference rates rise from 7.5% for BASE to 27.9–43.3% across trained conditions; changes in search behavior also occur under mismatched supervision and random reward.Source: §3.4 · pp. 6–7, Figure 4 · p. 7, Table 3 · p. 7

4 FROM ATTRIBUTION TO APPLICATION

The VisualWebArena evaluation retains the GRPO pipeline but uses COIN’s environment-independent reward, without matched-observation targets or an additional multimodal teacher. COIN improves task coverage over BASE and is also associated with shorter, less-looping trajectories.

4.1 TRAINING DESIGN AND EVALUATION SETUP

  • The VisualWebArena evaluation uses Qwen2.5-VL-7B, splits tasks by template, and samples 64 trajectories for each of 201 evaluation tasks per condition.Agents still condition decisions on screenshots, while training uses neither an additional multimodal teacher nor a matched-observation target; Figure 5 depicts reward construction and pass@k.Source: §4.1 · pp. 7–8, Figure 5 · p. 8, Table 4 · p. 8

4.2 TESTING THE ATTRIBUTION FINDING ON VISUALWEBARENA

  • COIN improves VisualWebArena pass@64 by 14.3% relative to BASE, with higher pass@k across evaluated sampling budgets and a widening gap as k increases.Coverage gains come from Shopping and Classifieds; across all three websites, the loop ratio falls from 56.86% to 46.14%, with fewer interaction turns per episode.Source: §4.2 · pp. 8–9, Figure 5 · p. 8, Table 4 · p. 8

5 RELATED WORK

The related work spans agent benchmarks, world-model post-training, and methods for distinguishing informative from uninformative training signals. The paper positions its audit as separating effects of environment content from other effects of post-training.

Agent tasks

  • Agent benchmarks cover text interaction, rendered interfaces and cross-application workflows, and code or terminal environments, where success combines several capabilities.The cited benchmarks include ALFWorld, ScienceWorld, VisualWebArena, OSWorld, SWE-bench, and Terminal-Bench.Source: §5 · p. 9

World-model post-training for agents

  • World-model post-training includes reward agreement, auxiliary dynamics supervision, and supervised or self-distilled prediction, but end-to-end gains do not isolate the contribution of correct environment content.Prior work also studies environment prediction in GUI, Android, software-engineering, and terminal agents; a Rubik’s-cube study links probe accuracy with later GRPO gains.Source: §5 · p. 9

Uninformative rewards and content attribution

  • Uninformative-reward controls and large-k pass@k motivate separating content effects on single-attempt selection from changes in task coverage.The paper applies mismatched targets and environment-independent rewards to agent post-training, combining prediction accuracy with pass@1 and pass@64.Source: §5 · p. 9

6 CONCLUSION

The audit finds that observation content changes prediction accuracy more than task performance across text-environment settings, while random-reward training can expand coverage. On VisualWebArena, coverage still expands without matched-observation targets or an additional multimodal teacher.

6 CONCLUSION

  • Across two text environments and RL and distillation, observation content changes prediction accuracy far more than task performance, while correct content primarily improves selection and random-reward training expands coverage.The action analysis associates coverage expansion with broader candidate-action search and less trajectory looping.Source: §6 · p. 9
  • On VisualWebArena, COIN expands coverage without an additional multimodal teacher or matched-observation target.The paper frames the findings as decomposing world-model post-training gains into content and optimization effects, identifying dependencies that can be omitted in new environments.Source: §6 · p. 10

A.1 DATA AND TRAINING SETTINGS

The experiments use text-environment trajectories and VisualWebArena transition samples to train models with prediction-based rewards or distillation. Training settings and reward construction vary by method and environment.

A.1 DATA AND TRAINING SETTINGS

  • For text environments, 20,000 transitions per environment are sampled with seed 42; VisualWebArena samples are collected with Qwen3-VL-32B and filtered.ALFWorld samples use the official training split, while ScienceWorld excludes trajectories from the 200 AgentGym evaluation tasks.Source: §A.1 · p. 14
  • ALFWorld and ScienceWorld use Qwen2.5-7B-Instruct, while VisualWebArena uses Qwen2.5-VL-7B-Instruct.The final checkpoint from each run is evaluated, and inputs exceeding length limits are removed before training.Source: §A.1 · p. 14
  • Table 6 reports optimization and rollout settings, with batch sizes counting prompts and micro-batch size counting responses per GPU.Training-sample counts are reported before length filtering.Source: Table 6 · p. 14
  • GT and MIS use embedding-based binary rewards, while COIN uses an independent Bernoulli(0.5) reward.GT/MIS reward is 1 when predicted and target observations have cosine similarity above 0.8; MIS permutes targets across samples.Source: §A.1 · p. 14
  • OPSD compares fixed-teacher and student distributions over the vocabulary, applying the loss only to the student’s generated next-observation span.The teacher receives the true next observation in GT or a mismatched observation in MIS.Source: §A.1 · p. 14

A.2 EVALUATION SETTINGS

Evaluation measures task success and next-observation prediction under specified interaction and sampling settings. Confidence intervals are computed by resampling tasks or shared decision points.

A.2 EVALUATION SETTINGS

  • Task evaluation uses temperature 1.0, a 30-step episode limit, and benchmark-specific success evaluators.Per-step output limits are 1,024 tokens for ALFWorld and VisualWebArena and 512 for ScienceWorld; VisualWebArena also uses top-p 0.9.Source: §A.2 · p. 15
  • Prediction accuracy is evaluated on 600 ALFWorld transitions from each test pool and 1,200 ScienceWorld transitions from the held-out AgentGym task pool.Samples use seed 42 and are shared across conditions within each environment; predictions use greedy decoding with a 640-token limit.Source: §A.2 · p. 15
  • Pass@k intervals use 10,000 task-bootstrap estimates, while prediction-preference and action-overlap intervals resample shared decision points 2,000 times.Each sampled task retains all 64 rollouts, with the same task draws across conditions.Source: §A.2 · p. 15

B ADDITIONAL TASK RESULTS AND REWARD-PAIRING DIAGNOSTICS

Reward permutation tests whether task gains depend on assigning each prediction the score computed for it. Shuffling scores reduces task performance, while prediction-accuracy changes do not track those task decreases.

B ADDITIONAL TASK RESULTS AND REWARD-PAIRING DIAGNOSTICS

  • All four pass@64 decreases after reward shuffling are significant, with Holm-adjusted p < 0.012.In ALFWorld’s list regime, pass@1 decreases are 8.87% for GT and 2.82% for MIS; the decreases are closer in the no-list regime.Source: B ADDITIONAL TASK RESULTS AND… · p. 15
  • Shuffling scores reduces pass@1 and pass@64 in both ALFWorld regimes, indicating a task benefit from assigning each score to the prediction it evaluates.The score values remain unchanged, but a prediction can receive a score computed for another prediction.Source: B ADDITIONAL TASK RESULTS AND… · p. 15
  • GT prediction accuracy falls 5.00% after shuffling, whereas MIS accuracy rises 22.33%, despite task decreases in both conditions.The MIS result is another case where better next-observation prediction accompanies worse task performance.Source: B ADDITIONAL TASK RESULTS AND… · p. 15

C ACTION AND REASONING ANALYSIS

The analysis compares prediction preferences and action distributions at shared ALFWorld decision points, then characterizes chain-of-thought outputs and action-choice behaviors. It also reports an independent review of action-revision labels.

C.1 PREDICTION PREFERENCES AND ACTION DISTRIBUTIONS

  • At 710 shared decision points, prediction preference is the true observation’s log probability minus the mismatched observation’s log probability.The probe holds the task, history, current observation, and executed action fixed, and fixes the mismatched next observation across conditions.Source: §C.1 · p. 15
  • Action overlap sums shared probability mass across two distributions and ranges from zero for disjoint distributions to one for identical distributions.Figure 3 reports mean overlap across the shared decision points.Source: §C.1 · p. 16

C.2 CHAIN-OF-THOUGHT MEASUREMENTS

  • Table 3 characterizes chain-of-thought through output length, entity references, action revision, and comparison of candidate actions.Entity references count steps whose reasoning mentions a previously observed entity, identified by matching object names and identifiers.Source: §C.2 · p. 16
  • Action revision means proposing an action and then abandoning it or choosing another, while candidate comparison means weighing at least two alternatives.Both measures are percentages of sampled steps meeting their criterion.Source: §C.2 · p. 16
  • An independent review confirmed 40 of 60 action-revision labels, with false positives mainly involving where to begin a sequential search.The review identifies a limitation in the action-revision annotation.Source: §C.2 · p. 16

C.3 ADDITIONAL CASE STUDIES

These additional case studies examine agent action-candidate reasoning and VisualWebArena coverage, alongside evaluation procedures for coverage, judging predicted action effects, and labeling reasoning in agent transcripts.

C.3 ADDITIONAL CASE STUDIES

  • In the case study, MIS rules out examining a known safe, while COIN compares examining or closing it and then places the watch inside; both complete the task.Figure 6 illustrates candidate comparison and rule-outs under mismatched supervision and random rewards.Source: §C.3 · pp. 16–17, Figure 6 · p. 17

D VISUALWEARENA DETAILS

  • VisualWebArena evaluation trains on 4,478 transitions from 437 tasks and evaluates 201 held-out read-only tasks, split by intent template within each website.Tasks that could change shared website state, such as placing orders or publishing posts, are excluded from parallel evaluation.Source: D VISUALWEBARENA DETAILS · p. 17
  • The paired VisualWebArena coverage difference had p = 0.0414, with a 95% bootstrap interval of [1, 19] tasks for the net gain.The interval is based on 2,000 resamples of task pairs; the test was two-sided and exact McNemar.Source: D VISUALWEBARENA DETAILS · p. 17

E.1 PREDICTION TRAINING

  • Prediction-training templates condition on history, the current observation or page, and the action, and request step-by-step reasoning followed by a next-state prediction.VisualWebArena uses a screenshot with numbered element marks and includes the objective, action history, and current page.Source: §E.1 · pp. 18–19

E.2 TASK EXECUTION

  • Task-execution prompts specify environment-appropriate action formats; VisualWebArena provides marked screenshots and requires one valid action per response.Its response format includes reasoning, a one-sentence operation description, and an action command.Source: §E.2 · pp. 20–21

E.3 PREDICTION-ACCURACY JUDGING

  • The prediction-accuracy judge assesses whether a prediction captures the action’s effect, ignoring wording and unpredictable specifics but rejecting hallucinated events.The judge compares the predicted next observation with ground truth and outputs a correct or incorrect verdict.Source: §E.3 · p. 22
  • Prediction accuracy is judged by whether a prediction captures the action’s effect, ignoring wording and unknowable specifics but counting hallucinated events or wrong outcomes as incorrect.The verdict is CORRECT when the dynamics match without over-prediction, and INCORRECT when the predicted effect contradicts the ground truth.Source: §E.3 · p. 23

E.4 CHAIN-OF-THOUGHT ANNOTATION

  • For each reasoning step, annotators identify immediate action candidates and whether one was rejected, distinguishing predicted-consequence, history-memory, and other rejection bases.Sequential plans and sweeps count as one candidate, while a considered option that is later displaced still counts.Source: §E.4 · pp. 23–24
  • Annotators compare the final action with the last surviving candidate, labeling it last, earlier, or none, and return the annotations in strict JSON.The prompt also records candidate count, rejection status and basis, and evidence supporting a rejection.Source: §E.4 · p. 24

Related papers

Loading 2609.33335v1…