Source-linked AI summary
General Agentic Planning Through Simulative Reasoning with World Models
Mingkai Deng, Jinyu Hou, Zhiting Hu, Eric Xing
TL;DR
Reactive agent systems often choose actions without explicitly simulating future outcomes, a gap the paper addresses with SiRA, a world-model-based planning architecture. Across three task categories, simulative reasoning outperforms a matched reactive baseline by up to 124% in task-completion rate.
Problem
Reactive agentic systems select actions through fixed procedures, while chain-of-thought lacks an explicit mechanism to simulate and compare candidate futures.
Source: §1 · pp. 1–2Method
SiRA uses an LLM-based world model and natural-language belief states to simulate candidate actions in a goal-oriented, model-agnostic architecture.
Source: §1 · p. 2Results
Takeaways & Limitations
Persistent advantages across distinct task types provide initial evidence that the benefit stems from generalizable counterfactual evaluation rather than task-specific tuning.
Source: §1 · p. 3SiRA uses only webpage text observations, which can miss information in images and layout, such as occlusions.
Source: §5 · p. 13
Abstract
from arXiv · showhide
What does it mean to plan? Current agentic systems, whether scaffolded workflows or end-to-end policies, rely on reactive decision-making: selecting the next action via a fixed procedure with at most undifferentiated adaptive computation (e.g., chain-of-thought) lacking explicit modeling of future outcomes. This limits generalizability, as each new task demands re-engineering rather than transfer of shared reasoning capacity. Humans, by contrast, plan by mentally simulating consequences of candidate actions within an internal world model, a capacity known as simulative reasoning (System II) that supports flexible, goal-directed behavior across diverse contexts. We argue that simulative reasoning through a world model provides a general-purpose planning mechanism for agentic systems, improving upon reactive policies (System I) by grounding decisions in predicted future states rather than pattern-matched responses. To verify this, we introduce SiRA (Simulative Reasoning Architecture), a goal-oriented architecture instantiating simulative reasoning using an LLM-based world model with natural-language belief states, while remaining model-agnostic. We evaluate across three qualitatively distinct task categories: constrained navigation, multi-hop information aggregation, and general instruction following, in a web-browser environment. Across all categories, simulative reasoning achieves up to 124% higher task completion rates than a matched reactive baseline, and increases constrained navigation success from 0% to 32.2% compared to a representative open-web agent. The persistent advantage across distinct task types suggests the benefit stems from generalizable counterfactual evaluation rather than task-specific tuning.
1 Introduction
The introduction frames planning as explicit simulation of candidate futures, contrasting it with reactive decisions that lack a grounded way to compare outcomes. SiRA instantiates this approach and is evaluated across three task categories.
1 Introduction
- Reactive systems select actions through fixed procedures, while chain-of-thought does not provide a mechanism to propose, simulate, and compare candidate futures.The paper distinguishes internal computation from planning grounded in predicted consequences.Source: §1 · pp. 1–2
- Simulative reasoning models future states from candidate actions, evaluates their consequences, and selects behavior grounded in those predictions.The paper frames this as a general-purpose planning mechanism that can in principle transfer across tasks without re-engineering.Source: §1 · p. 2
- SiRA instantiates the approach with an LLM-based world model and compact natural-language belief states, in a model-agnostic architecture.Its components isolate perception, simulative planning, and action selection.Source: §1 · p. 2
- Up to 124% higher task-completion rate than the matched reactive baseline, with constrained-navigation success rising from 0% to 32.2% versus a representative open-web agent.The reported advantage persists across qualitatively distinct task types, which the authors suggest reflects generalizable counterfactual evaluation rather than task-specific tuning.Source: §1 · p. 3
2 Related Work
Related work contrasts reactive agent designs with model-based planning and reviews limitations in existing web-agent benchmarks. It presents FlightQA as an evaluation dataset for live-web reasoning under varying constraint complexity.
Reactive Agentic Systems
World Models and Model-Based Planning
- Model-based planning simulates candidate-action consequences, but prior work often uses holistic continuous state embeddings and targets narrow domains rather than a general-purpose planning pattern.The passage notes applications ranging from classic games [48] and continuous control to mathematical reasoning, open-ended games, and web browsing.Source: §2 · p. 3
Benchmarks and Evaluation
- Existing benchmarks such as WebArena [63], WebVoyager, MiniWoB++ [35], Mind2Web, and WebShop are described as having limits in realism, question currency, or task-completion measurement.FlightQA evaluates reasoning in live web environments with systematically varying constraint complexity.Source: §2 · p. 3
3 Simulative Reasoning for Agentic Planning
The section formalizes goal-directed planning as evaluating future outcomes, then develops simulative reasoning in which a world model predicts future belief states. SiRA applies this principle through concept-based representations and separate simulated planning and concrete execution.
3.1 Formulation of Agent-Environment Model
- The agent-environment formulation models an agent choosing actions from states and an environment producing next states; agent success is measured by expected future discounted reward.The value function expresses this long-term success in terms of possible future states and their probabilities.Source: §3.1 · pp. 3–4
3.2 Definition of Optimal Agent
- Simulative reasoning proposes multiple candidate actions, predicts their consequences, and selects behavior based on those predictions, unlike reactive policies that choose the next action directly.The deliberation process also creates opportunities to spot and correct errors.Source: §3.2 · p. 4
3.3 World Model for Simulative Reasoning
- A world model lets an agent simulate future belief states from observations and candidate actions, enabling planning without direct access to the true world state.The model predicts belief-state transitions over a chosen horizon, supporting reasoning about possible outcomes without interacting with the environment.Source: §3.3 · p. 5
- A general-purpose world model can simulate outcomes across domains without direct interaction, supporting transfer across tasks and environments.The paper argues that this state-transition-based reasoning offers an advantage that reactive policies cannot reproduce simply by using more internal computation.Source: §3.3 · p. 5
3.4 Instantiating Simulative Reasoning
- Concept-based natural-language state summaries let modules predict and evaluate individual state components, reducing hallucination and supporting more robust planning.Factoring the state into discrete concepts is intended to make targeted change prediction tractable where monolithic embeddings are less suited to it.Source: §3.4 · pp. 5–6
- Separating abstract simulated actions from concrete commands disentangles planning from action-space details and supports a richer range of intentions.Higher-level simulated actions can represent multiple execution steps, reducing rollout length and opportunities for error accumulation; natural-language proposals are reported to be more diverse and grounded.Source: §3.4 · p. 6
- SiRA encodes observations as belief states, plans by proposing actions, predicting states, and evaluating goal progress, then passes the selected plan to an actor for execution.The process can repeat until a simulated action sequence is selected.Source: §3.4 · p. 6
4 Experiments
The experiments compare simulative reasoning with reactive and other agent baselines across three task categories in a web-browser environment. Simulative reasoning outperforms the matched reactive policy across categories, with the largest reported task-completion gain reaching 124%.
4 Experiments
- The study compared full SiRA against a matched reactive policy that selected an action without simulation, isolating simulation from other architectural choices.The browser testbed was selected for its practical value, realistic complexity, partial observability, long-horizon interaction, and multimodality.Source: §4 · p. 7
- Up to 124% higher task-completion rate was achieved by simulative reasoning than by the matched reactive policy across three task categories.Figure 4 also reports near-zero constrained-navigation success for unconstrained chain-of-thought, including with o1 and o3-mini.Source: Figure 4 · p. 8, §4 · p. 8
4.1 Constrained Navigation
- FlightQA generates questions by adding constraints iteratively, yielding 90 questions across 15 sequences with three to eight constraints; responses are judged for groundedness and relevance.Because live flight availability changes, the authors cannot establish fixed ground-truth answers and instead use LLM judgments of interaction history and constraint satisfaction.Source: §4.1 · p. 9, Figure 6 · p. 10
- 32.2% constrained-navigation accuracy was achieved by simulative reasoning, versus 0% for OpenHands BrowsingAgent, alongside a 124% improvement over the matched reactive policy.The matched-policy improvement was significant at the 0.01 level.Source: Table 1 · p. 10, §4.1 · p. 10
- The simulative-reasoning advantage persisted as FlightQA constraint counts increased, though both methods’ performance rose sharply at seven constraints before falling again.The authors suggest the persistent gap indicates generalizable counterfactual evaluation rather than task-specific tuning; the non-monotonic pattern may reflect backend memorization or implicit constraints.Source: §4.1 · p. 11, Figure 7 · p. 11
4.2 Multi-Hop Information Aggregation / 4.3 General Instruction Following
- Multi-hop evaluation used FanOutQA’s first 100 dev examples to assess multi-source question answering, while WebArena tested diverse natural-language instructions using a random subset of 100 examples.Multi-hop experiments used gpt-4o-2024-05-13. WebArena allowed 15 steps and adapted the agent description to steer its answer format for evaluation.Source: §4.2 · p. 11, §4.3 · p. 12
4.2 Multi-Hop Information Aggregation
- 48.6% higher response rate and 47.5% higher fact-level accuracy were achieved by simulative reasoning than by the reactive policy on multi-hop information aggregation.On FanOutQA, the evaluation used the first 100 development examples; the authors report statistically significant differences between methods.Source: §4.2 · p. 11, Table 2 · p. 12
4.3 General Instruction Following
- 21.1% improvement over the reactive policy and up to 91.7% over BrowsingAgent were reported for general instruction following.The authors caution that success rates are not directly comparable to prior work because their OpenHands-based evaluation differs from the standard benchmark setup.Source: §4.3 · p. 12
5 Limitations
The paper notes higher inference-time computational cost and browser-tooling and observation constraints; the supplied passages also report a general instruction-following result.
5 Limitations
- Thorough exploration of multiple candidate plans and a modular pipeline make simulative reasoning more computationally costly than reactive policies at inference time.The paper identifies caching, abstention, and parallelization as future ways to speed up world-model-based reasoning.Source: §5 · p. 12
- Open-source browser agents may be blocked by Captcha or anti-scraping tools, while text-only webpage observations can miss images and layout information such as occlusions.The paper identifies deeper browser integration and multimodal perception with planning as areas to address.Source: §5 · pp. 12–13
- 21.1% over the reactive policy and 91.7% over BrowsingAgent are reported for simulative reasoning on general instruction following.The results use a random 100-sample WebArena subset.Source: Table 3 · p. 13
6 Conclusion
The conclusion reports consistent gains over a matched reactive baseline across three task categories and argues that grounded counterfactual evaluation may provide a general-purpose planning mechanism. It also identifies directions for extending and improving world-model-based reasoning.
6 Conclusion
- Across constrained navigation, multi-hop information aggregation, and general instruction following, simulative reasoning consistently outperforms the matched reactive baseline, including as task complexity increases.The authors suggest the benefit reflects generalizable grounded counterfactual evaluation rather than domain-specific tuning.Source: §6 · p. 13
- The authors argue that simulative reasoning through a world model offers a general-purpose planning mechanism by grounding decisions in predicted future states.SiRA instantiates this principle with an LLM-based world model and natural-language belief states.Source: §6 · p. 13
- Future directions include extending beyond web browsing, improving world-model accuracy, regulating simulation cost, using world models for continual reinforcement learning, and discussing transparency and alignment.The paper frames these as opportunities across embodied, social, physical, learning, and safety settings.Source: §6 · p. 13
A SiRA Inference Pseudo-Code
SiRA’s inference routine takes an observation and goal alongside encoder, proposer, world model, critic, actor, and planning-horizon components. It searches simulated plans, scores terminal belief states, and uses the best sequence to select an executable action.
A SiRA Inference Pseudo-Code
- The routine takes observation and goal inputs, encoder, proposer policy, world model, critic, and actor components, plus a planning horizon.The pseudo-code notes that replanning decisions and some procedures are omitted for simplicity.Source: A SiRA Inference Pseudo-Code · p. 18
- During planning, SiRA initializes a best score and action sequence, then searches a tree rooted at the current estimated state while its DFS budget remains.It proposes simulated actions over the planning horizon and evaluates the terminal belief state with a critic.Source: A SiRA Inference Pseudo-Code · p. 18
- After retaining the best simulated action sequence, SiRA infers an executable next action conditioned on the estimated state and that sequence.Source: A SiRA Inference Pseudo-Code · p. 18
B Details on Web Browsing Environment
The supplied browser-environment passages show a Google Flights page snapshot with navigation controls, flight-search fields, suggested destinations, and example fares.
B Details on Web Browsing Environment
- The snapshot identifies the page as Google Flights and includes navigation links for Explore, Flights, Hotels, and Vacation rentals.Source: B Details on Web Browsing Environment · p. 19
- The page snapshot includes flight-search controls such as passenger settings, departure and return fields, and an Explore destinations button.Source: B Details on Web Browsing Environment · p. 19
- The snapshot lists Cleveland, Columbus, and Akron as destination options and shows example fares of $128 to San Francisco and $68 to New York.The San Francisco listing also shows Jan 9–Jan 16, one stop, and 10 hr 30 min; the New York listing shows Dec 7–Dec 14.Source: B Details on Web Browsing Environment · p. 19
C Prompts for Web Browsing Implementation
Appendix C specifies prompts for implementing a web-browsing agent, covering webpage observation, state and action descriptions, outcome prediction, and constrained browser-action execution. It also describes how intents are clustered into browser actions and how the WebArena assistant responds and stops after completing user goals.
C Prompts for Web Browsing Implementation
- The agent is framed as an internet-browsing assistant expected to respond accurately, thoroughly, efficiently, and politely, and to stop after messaging the user.Source: C Prompts for Web Browsing… · p. 20
- Its observation combines webpage text and a viewport screenshot with information about page structure, interactive components, visibility, focus, and accessibility.Source: C Prompts for Web Browsing… · pp. 20–21
- The browsing prompt defines 13 action types and requires one API call at a time, using only elements in the current observation and valid function-call arguments.Source: C Prompts for Web Browsing… · pp. 21, 25, 26
- The state prompt requests a comprehensive account of page elements, values, task-relevant information, and errors, while the intent prompt requests one concise next action without changeable element IDs.Source: C Prompts for Web Browsing… · pp. 22–23
- The prompt asks the agent to infer the next webpage state after attempting its intent, use present tense, and update memory with observed changes and progress without listing planned actions.Source: C Prompts for Web Browsing… · pp. 23–24
- A separate evaluation prompt classifies execution as success or failure and indicates whether an unsuccessful attempt is still on track.Source: C Prompts for Web Browsing… · p. 24
- The action-clustering prompt presents a 16-action browser space and asks the agent to group natural-language intents into semantically equivalent clusters.Source: C Prompts for Web Browsing… · pp. 21, 26
- Intent clustering groups differently worded instructions that express the same action and map to the same Playwright browser API call, while minimizing the number of clusters.Source: C Prompts for Web Browsing… · p. 26
- The clustering results are represented as a JSON object with unique cluster identifiers and lists of candidate action IDs.Source: C Prompts for Web Browsing… · p. 27
- The prompt illustrates grouping five variants of a Google-homepage navigation intent into one cluster.Source: C Prompts for Web Browsing… · p. 27
C.1 Adaptation for WebArena Evaluation
- The WebArena assistant sends an answer once it has found a complete answer to a question.Source: §C.1 · p. 28
- For information-location tasks, it stops when the current page satisfies the request and tells the user it found the information.Source: §C.1 · p. 28
- For operations, it stops and reports completion once the requested operation has been done.Source: §C.1 · p. 28
- The assistant should try to achieve goals on the current site, be forthright when a task is impossible, and end after messaging the user.Source: §C.1 · p. 28
D Prompts for Generating and Evaluating on the FlightQA Dataset
The prompts define a flight-search benchmark by varying query diversity and constraint complexity, then evaluate whether answers are grounded in the interaction history and satisfy user constraints. They also specify procedures for generating queries with increasing numbers of constraints and assessing responses.
Prompts for Generating and Evaluating on the FlightQA Dataset
- Queries vary destinations, dates and trip durations, passenger counts, cabin classes, and preferences such as nonstop flights.The guidance calls for common and obscure destinations, varied travel periods, and solo, family, and group travel.Source: D Prompts for Generating and… · p. 29
- Complexity tests include multi-leg trips, time-bounded searches, conditional preferences, and budgets, alongside varied flight attributes.Listed attributes include route, dates and times, duration, airline, cabin, aircraft, and emissions.Source: D Prompts for Generating and… · p. 29
- Generated questions should use the specified number of constraints and repeat this process at least seven times.The instructions exclude passenger-count constraints and allow relative dates rather than fixed calendar dates.Source: D Prompts for Generating and… · p. 29
- The prompts expand starting queries by adding one constraint at a time until reaching a maximum number of constraints.Each successive question retains the previous constraints and adds another.Source: D Prompts for Generating and… · p. 31
- The benchmark evaluates whether flight-search responses are supported by the interaction history and satisfy user constraints to the extent allowed by search results.When all constraints cannot be met, the response should be assessed for how closely it follows them.Source: D Prompts for Generating and… · p. 32