Current Agents Fail to Leverage World Model as Tool for Foresight

• Long-horizon tasks require agents to anticipate how environments evolve across interdependent decisions and potentially irreversible consequences.Source: §1 · p. 1
• Foresight is defined here as anticipating environmental evolution rather than generating goal-directed plans from the agent’s own perspective.Source: §1 · p. 1
• World models can serve as optional tools that agents query to simulate candidate actions before committing to real-world behavior.Source: §1 · p. 2, Figure 1 · p. 2
• Agents often hesitate to invoke world-model tools even when simulation could improve environmental foresight.Source: §1 · p. 2
• When agents do invoke simulations, they may produce one deterministic future, override predicted outcomes with confident internal reasoning, or fail to evaluate counterfactual branches.Source: §1 · p. 2
• The paper frames progress as calibrating when agents consult simulations, how they interpret them, and how they integrate simulated evidence with reasoning.Source: §1 · p. 2

• Cognitive science describes planning as mentally simulating actions and outcomes with internal world models.Source: §2 · pp. 2–3
• Recent agent research uses LLMs or VLMs as dynamics models, searches imagined trajectories, or embeds learned simulators into decision loops.Source: §2 · p. 3

• The protocol evaluates whether a decision-making agent can invoke a learned world model to simulate candidate actions before committing to them.Source: §3 · p. 3
• The agent’s trajectory records prior actions and observations while the real environment contains the current state and the world model approximates its transition dynamics independently.Source: §3 · p. 3
• A room-escape example uses simulation to compare breaking a reinforced door with crawling through a vent before selecting the viable exit.Source: §3 · p. 3
• The policy maps the current trajectory to an action and an execution context, choosing either the real environment or the world model.Source: §3.1 · p. 3
• Simulation predicts a next state without changing reality, and the simulated outcome is appended as a hypothetical observation for future reasoning.Source: §3.1 · p. 3

• The interaction process continues until the goal is reached or the maximum horizon is exceeded, retaining real and simulated outcomes in one trajectory.Source: §16 · p. 4

Interaction Modes

• Normal Mode makes the world model optional, allowing agents to query it or act directly in the real environment.Source: §3.2 · p. 4
• WM Invisible Mode removes access to the world model, so all actions execute directly in the real environment as a baseline.Source: §3.2 · p. 4
• WM Force Mode requires agents to simulate actions before executing them, testing whether compulsory planning helps or hinders performance.Source: §3.2 · p. 4

• The evaluation examines when simulation is worthwhile, how predictions are interpreted, and how simulated and real experience guide subsequent actions.Source: §3.2 · p. 4

Experimental Tasks

• The evaluation covers four interactive agent tasks and four visual question-answering benchmarks requiring spatial reasoning, visual grounding, or hypothetical transformations.The study compares embodied control with non-embodied perceptual reasoning to assess whether simulation helps across task types.Source: §4.1 · pp. 4–5

Test Models

• Nine vision-language models from GPT, Llama, and Qwen families are evaluated across multiple model sizes.All tested models have basic visual reasoning abilities, allowing differences to be attributed to world-model usage rather than insufficient perception.Source: §4.2 · p. 5

Evaluation Setup

• Agent tasks use cloned environments to simulate hypothetical actions, while evaluations compare optional, unavailable, and—in agentic tasks—mandatory world-model access.Performance is measured by task success for agents and answer correctness for VQA.Source: §4.2 · p. 5

Performance Results

• Across agent and VQA settings, world-model access does not reliably improve performance; agent results often worsen, while VQA gains remain marginal.Table 2 reports that almost all models fail to achieve higher agent-task performance with access, whereas Table 3 shows nearly unchanged VQA accuracy.Source: §4.3 · p. 5, Table 2 · p. 6, Table 3 · p. 6

Calling Behavior

• World-model usage is consistently low, especially in VQA, where rates remain below 0.1 for all but the Llama family.The findings indicate that models lack a clear strategy for deciding when and why simulation would improve predictions.Source: §4.3 · p. 5
• Calling behavior differs across families: Llama models query most proactively with little measurable benefit, while larger GPT models often bypass external help.Qwen models show a similar size-related pattern on agent tasks, with Qwen2.5-VL-7B as an unusually unwilling caller.Source: §4.3 · pp. 5–6

Fine-Grained Outcomes

• VQA cases show nearly equal rates of helpful and harmful world-model use, while agent tasks experience harmful rollouts more frequently than helpful ones.The case-level analysis concludes that current world-model access offers limited practical advantage and can be counterproductive.Source: §4.3 · p. 6

Attribution Analysis

• Models can extract value from simulations, but poorly calibrated use produces fragile gains and frequent regressions.The analysis motivates taxonomies of governance successes and failures across the cognitive pipeline.Source: §5.1 · p. 6

Agent Tasks

• Agent-task rollouts mainly improve planning and state understanding, but action loops, over-planning, and misinterpretation destabilize execution.Agents may repeatedly re-plan without meaningful progress, making foresight beneficial in isolated cases but detrimental over long horizons.Source: §5.1 · pp. 6–7

VQA Tasks

• VQA rollouts mainly aid state understanding and visual confirmation, while unclear requests and ambiguous scenarios can amplify uncertainty.These errors reflect failures in governance of meaning, where ambiguity propagates rather than resolves uncertainty.Source: §5.1 · p. 7

Governance

• Foresight governance requires deciding what to simulate, how to interpret simulations, and when to act on them; stable use depends more on governance than simulation.The proposed success pattern combines calibrated queries, unambiguous verification, and stable integration into actionable plans.Source: §5.1 · pp. 7–8, Figure 4 · p. 8, Figure 5 · p. 8

Task Perspective

• Agent tasks show higher world-model usage and benefit rates than VQA tasks, whose targeted disambiguation makes imprecise simulations more distracting.Current invocation patterns are better suited to dynamic, stateful environments than static perception queries.Source: §5.2 · p. 8

Call Perspective

• Repeated world-model calls correlate negatively with task success because models often re-query instead of integrating prior rollouts into stable plans.Excessive calls frequently coincide with action loops and degraded execution, especially in agent tasks.Source: §5.2 · p. 8, Figure 7 · p. 8

Usage Perspective

• Forced world-model invocation causes a steeper performance drop across agent tasks, increasing both the frequency and severity of harmful cases.Average performance declines for every task under the forced-use condition.Source: §5.2 · pp. 8–9, Figure 8 · p. 9

Hypothesis Testing

• Confirmation-style queries can reinforce an incorrect hypothesis, whereas simulating several plausible hypotheses supports structured discrimination.This shifts simulation from affirming one guess toward testing alternatives against predicted cues.Source: §6 · p. 9

Integration Modules

• Dedicated Decider, Reflector, and Memory modules could organize when to simulate, how to evaluate outcomes, and how to maintain task state.The proposed loop explicitly separates invocation decisions, feedback evaluation, and long- and short-term memory.Source: §6 · p. 9

Agent Training

• Training with online multi-turn rollouts could reward appropriate invocation, diverse queries, and information gain while penalizing excessive repetition.Supervised cold-start data and indirect objectives such as reductions in hypothesis entropy are proposed to improve credit assignment.Source: §6 · p. 9

Conclusion

• World-model access introduces hesitation, over-analysis, and misaligned interpretation because agents must manage hypothetical branches alongside real experience.The observed difficulties motivate more fine-grained governance of foresight.Source: §7 · p. 9

Future Work

• Promising directions include stable internal state across simulated and real trajectories, exploration-driven simulation policies, and hypothesis-refining interfaces.These directions aim to integrate simulation more directly into decision-making and support anticipatory cognition.Source: §7 · p. 10

Scope and Evaluation

• The evaluation excludes Gemini and Claude, uses different simulators for agentic and VQA tasks, and observes occasional WAN2.1 frame artifacts.These choices may introduce mild cross-task incomparability and affect downstream performance.Source: Limitations · p. 10
• Many failures trace to underspecified or ambiguous agent instructions that degrade simulation quality and downstream performance.Future work will expand model coverage and use stronger, more consistent simulators to better isolate agent effectiveness.Source: Limitations · p. 10

• Controlled experiments, rather than real-world deployment, evaluate world models as tools for VLM agents.Source: Ethical Statement · p. 10
• Simulated futures can be overtrusted or misread, potentially leading to unsafe decisions when transferred to high-stakes domains.Source: Ethical Statement · p. 10
• The paper recommends robust oversight, evaluation, and privacy protections before safety-critical deployment involving user data.Source: Ethical Statement · p. 10

• Prior work covers planning and long-horizon reasoning for agents operating in complex environments.Source: Moon, Hiroki Furuta, Gopala… · p. 11
• Other cited efforts study spatial reasoning, multimodal intelligence, robotic manipulation, and dynamic agent benchmarks.Source: Moon, Hiroki Furuta, Gopala… · p. 11
• The references also connect foresight to cognitive theories of mental models, prospection, and imagining the future.Source: Moon, Hiroki Furuta, Gopala… · p. 11
• Related research includes learned, pretrained, and generative world models for simulating environments and dynamics.Source: Moon, Hiroki Furuta, Gopala… · pp. 11–12

• Agentic tasks use frames produced by the environment while executing the model’s predicted simulation actions, with at most three frames incorporated per step.Source: A Evaluation Settings Details · p. 13
• VQA tasks use WAN2.1 as the world model and Qwen2.5-3B-VL for prompt augmentation.Source: A Evaluation Settings Details · p. 13
• Frames are filtered for consecutive similarity below 0.95, uniformly interpolated to three representatives, and always include the final generated frame.Source: A Evaluation Settings Details · p. 13
• Accuracy is averaged across data points, using success rate for agent tasks and multiple-choice accuracy for VQA tasks.Source: A Evaluation Settings Details · p. 13

Forced world-model usage

• Forcing world-model usage generally performs worse than allowing optional access, providing no reliable effectiveness improvement.Source: B Additional Experiment Results · p. 13

Usage compliance

• GPT-5’s world-model usage rate remains below 1.0 even when explicit instructions require simulation before each action.Source: B Additional Experiment Results · p. 14
• SAT tasks show particular reluctance to use the world model, including under forced-use conditions.Source: B Additional Experiment Results · p. 14

Agent behavior

• Many models frequently reach the 15-step episode limit, while step count and action validity show no meaningful differences across settings.Source: B Additional Experiment Results · p. 14

Comparative outcomes

• Across evaluated tasks and models, performance improvements from world models do not consistently exceed performance degradations.Source: B Additional Experiment Results · p. 14

Attribution Analysis

• 60 cases were manually analyzed to identify core categories explaining when world-model assistance succeeds or fails.The analysis used these findings to curate eight attribution categories.Source: C Attribution Analysis Details · p. 15

Automated Attribution

• GPT-4o compared trajectories with and without world-model access, assigning one primary attribution reason and any additional contributing reasons.The compared trajectories covered both optional and forced world-model settings.Source: C Attribution Analysis Details · p. 15

Reliability Check

• 81.67% primary-attribution agreement with human judgments supports the reliability of GPT-4o’s case-level annotations.The agreement was measured on the 60 cases previously reviewed and annotated by humans.Source: C Attribution Analysis Details · p. 15

Usage and Performance

• Higher world-model invocation does not consistently yield positive returns across model families, scales, agent tasks, and VQA settings.Smaller agent models sometimes obtain modest gains, whereas larger models often reduce usage and experience neutral or negative effects; VQA changes remain marginal.Source: D Additional Analysis Results · p. 16, Figure 9 · p. 16

Call Frequency

• Agent tasks induce substantially higher world-model call frequencies than VQA tasks, often by an order of magnitude.Most VQA models make fewer than one call per question on average.Source: D Additional Analysis Results · p. 16, Figure 10 · p. 17
• Higher world-model call counts are associated with lower success rates in both agent and VQA tasks, with sharper degradation in agent tasks.The analyses report negative trends and binned degradation, linking repeated querying to uncertainty and poor foresight integration.Source: D Additional Analysis Results · pp. 16–17, Figure 11 · p. 18, Figure 12 · p. 18

Forced Usage

• Mandatory world-model invocation in VQA consistently fails to improve performance and often causes modest, systematic degradation.Cases where forced use hurts outnumber cases where it helps, while nearly half of questions remain unaffected.Source: D Additional Analysis Results · p. 17, Figure 13 · p. 18

Comparative Analysis

• Tables 14–17 compare both-correct, world-model-helps, world-model-hurts, and both-wrong outcomes across normal and forced settings.These percentages are reported separately for agentic and VQA tasks, with each model-task combination summing to 100%.Source: Table 14 · p. 20, Table 15 · p. 20, Table 16 · p. 21

Evaluation Setup

• The appendix presents evaluation prompts using system, initial-task, and feedback templates for each mode and task.These templates include placeholders for action limits, separators, inserted states, and simulated images.Source: F Evaluation Instruction · p. 22