PaperlayerView exact revision on arXiv

Source-linked AI summary

MCP-Cosmos: World Model-Augmented Agents for Complex Task Execution in MCP Environments

Giridhar Ganapavarapu, Dhaval Patel

arXiv:2605.09131v1cs.AIcs.MA

TL;DR

MCP agents face a gap between static planning that overlooks environment dynamics and reactive execution that lacks long-horizon foresight. MCP-Cosmos integrates world models into MCP planning so agents simulate tool interactions before execution, improving tool selection, parameter accuracy, and tool-calling success.

  • Problem

    Existing MCP agents lack a unified approach that accounts for dynamic environment behavior during planning while providing long-horizon foresight during execution.

  • Method

    MCP-Cosmos uses a BYOWM framework in which agents simulate complete tool orchestration in latent space before executing the selected plan.

  • Results

    World-model-augmented agents achieve better tool selection and parameter accuracy, higher tool-calling success rates, and more efficient tool usage than reactive baselines.

  • Takeaways & Limitations

    Execution Quality provides a benchmark for measuring whether world models guide targeted tool usage with minimal exploratory overhead.

  • Takeaways & Limitations

    The evaluation uses a static MCP-Bench snapshot, and the world models do not yet adapt online to evolving tool schemas or API behaviors.

Abstract

from arXiv · show

The Model Context Protocol (MCP) has unified the interface between Large Language Models (LLMs) and external tools, yet a fundamental gap remains in how agents conceptualize the environments within which they operate. Current paradigms are bifurcated: Task-level planning often ignores execution-time dynamics, while reactive execution lacks long-horizon foresight. We present MCP-Cosmos, a framework that infuses generative World Models (WM) into the MCP ecosystem to enable predictive task automation. By unifying three disparate technologies, namely MCP, World Model, and Agent, we demonstrate that a "Bring Your Own World Model" (BYOWM) strategy allows agents to simulate state transitions and refine plans in a latent space before execution. We conducted experiments using two strategies, namely ReAct and SPIRAL with 2 planning models and 3 representative world models over 20+ MCP-Bench tasks. We observed improvements in Agent's environment interaction KPI such as tool success rate and tool parameter accuracy. The framework also offers new metrics such as Execution Quality to generate new insights about the effectiveness of world models compared to baseline.

1 Introduction

MCP-Cosmos addresses the tension between static planning and reactive execution by integrating World Models into the MCP interaction loop. Its BYOWM architecture supports predictive tool planning, while benchmarking and Execution Quality evaluate efficient tool use and tool-call success.

  • Motivation: Existing MCP-agent evaluations expose a tension between planning-centric frameworks that use static tool definitions and execution-centric approaches that inadequately anticipate real-world dynamics.The supplied passage identifies static planning’s failure to account for environmental stochasticity before its text is truncated.
  • Framework: MCP-Cosmos integrates World Models into the MCP interaction loop so agents can simulate tool calls, assess their impact, and mitigate failures before real-time action.The framework’s core hypothesis shifts task automation from reactive execution toward predictive planning in a simulated latent space.
  • Evaluation: 300+ trajectories support a comparative analysis of world-model-infused agents against robust baselines using MCP-Bench tasks, tools, and evaluation metrics.The benchmarking uses an existing task dataset and MCP tools.
  • Evaluation: Execution Quality measures whether World Models guide efficient tool usage by penalizing excessive tool calls beyond those necessary for an input task.The paper claims empirical evidence for this metric’s effectiveness in Sections 4 and 5.

2 World Model Infused Planning and Execution in MCP

MCP-Cosmos separates tool orchestration into simulation-based planning and execution in the actual MCP environment. Agents iteratively revise simulated action sequences, select a final tool-call plan, execute it, and return the answer with execution artifacts.

  • Two-Phase Workflow: The framework uses two phases: proactive world-model planning without real tool interaction, followed by execution of the selected plan in the actual environment.This two-phase approach formalizes world-model integration into multi-turn planning and execution.
  • World-Model API: The abstract world-model API initializes model-specific configurations and provides a simulate method that maps a tool call and user request to a simulated observation.World models may implement different approaches for constructing simulation environments.
  • Simulation-Based Planning: During planning, the agent generates tool calls, simulates observations in latent space, and revises the plan until reaching a viable plan or termination criterion.Action–observation pairs accumulate in a world-model trajectory, enabling efficient exploration of multiple potential paths.
  • Plan Selection: A nondeterministic policy model such as an LLM or deterministic reward-based MCTS selects the final sequence of tool calls and parameters from the world-model trajectory.The selected plan is the artifact executed in the real environment.
  • Execution and Outputs: Execution runs real MCP tools and receives real observations; failed actions can trigger optional remaining-plan adjustment, though benchmarking excluded this expensive revision step.The algorithm records the final answer, execution plan P, and complete execution trajectory τ.

3 Experimental Setup and Results

MCP-Cosmos evaluates world-model-augmented agents on complex, multi-server MCP-Bench tasks using ReAct, ReAct-Plan-Exec, and SPIRAL-Exec architectures with three world-model configurations. World-model infusion improves overall performance and key tool-use metrics over ReAct, but introduces substantial token-consumption trade-offs.

  • Dataset and Scenarios: MCP-Bench is the primary dataset, covering 28 live MCP servers and 257 cross-domain tools through fuzzy-instruction tasks with rigorous rule- and judge-based evaluation.The selected 2- and 3-server scenarios emphasize cross-domain dependencies, state maintenance, and prediction of multiple tool outputs.
  • Agent Architectures: The benchmark compares ReAct as the no-world-model baseline with ReAct-Plan-Exec and SPIRAL-Exec, which use planning, simulation, or MCTS-assisted predictive execution.SPIRAL-Exec uses an LLM-based Planner and Simulator to generate observations before execution.
  • World Models: The evaluation uses gpt-oss-120b and claude-sonnet-4.6 as LLM world models and Arctic-AWM-4B as an MCP-environment-trained agent world model.Arctic-AWM-4B is described as the smaller version selected because of deployment size, availability, and deployment complications.
  • Results: World-model agents generally outperform ReAct overall, with SPIRAL-Exec + gpt-oss-120b-WM achieving 44.8% and ReAct-Plan-Exec + claude-sonnet-4.6-WM achieving 42.4%.AWM-4B-WM is less effective than general-purpose LLMs across most configurations, while ReAct remains strongest in Task Fulfillment.
  • Results: All world-model agents improve over ReAct in Parameter Accuracy, Parallel Efficiency, and Tool Selection, although ReAct benefits from full-environment access and error-recovery loops.The reported metrics aggregate task completion, tool selection, planning effectiveness, and overall task success.
  • Computation Cost: 49K tokens is ReAct’s average per-task consumption, versus 302K for SPIRAL-Exec + Arctic-AWM-4B-WM-WM, a 5x increase; total task consumption ranges from 745K to 7M tokens.Although Arctic-AWM-4B is small, it consumed 3x tokens with both SPIRAL-Exec and ReAct-Plan-Exec configurations.

4 Evaluation Gap Analysis and New Metrics

The evaluation exposes a gap between task fulfillment and execution efficiency: ReAct can complete tasks despite low tool-call success and repeated retries. Execution Quality addresses this gap by combining tool-call success with normalized, inverted call volume.

  • Evaluation Gap: ReAct records the highest Task Fulfillment (46.8) but the lowest tool call success rate (77.7), revealing a mismatch between completion and execution quality.Task fulfillment and dependency awareness are dynamic, making them harder to improve through simulator-driven proactive planning alone.
  • New Metric: Execution Quality averages tool call success rate and normalized average number of tool calls to penalize excessive retries and reward efficient execution.The normalized call score is computed as (max_avg_calls − agent_avg_calls) / (max_avg_calls − min_avg_calls) and is expected to range from 0-100.
  • Execution Quality Results: ReAct has the highest average number of tool calls (7.04) and a normalized score of 0, indicating repeated retries after unsuccessful tool calls.Despite this overhead, ReAct can still eventually complete task requirements.
  • New Metric: Execution Quality better distinguishes agents that solve tasks efficiently from agents that succeed only after substantial execution overhead.Fewer tool calls receive higher normalized scores because average call volume is min-max normalized and inverted.

5 Ablation study

The ablation study tests whether a stronger planner can compensate for removing an explicit world model. CLAUDE-SONNET-4.6 improves accuracy but substantially increases exploration, while world-model simulation constrains this behavior and preserves efficiency.

  • Planner capacity: CLAUDE-SONNET-4.6 improves accuracy over the GPT-OSS-120B ReAct baseline, but the stronger planner trades efficiency for accuracy.The ablation replaces GPT-OSS-120B with CLAUDE-SONNET-4.6 to isolate planner capacity from explicit world-model contributions.
  • Efficiency: 29.78 tool calls per task is the CLAUDE-SONNET-4.6 baseline, versus 7.04 for GPT-OSS-120B and 1.12–7.91 for world-model-augmented agents.Execution time also grows from 63.7s to 214.9s, a 3.4× slowdown, under identical tasks.
  • Planner capacity: 14.13 rounds and 29.78 calls under CLAUDE-SONNET-4.6 yield 2.1 tool calls per round, compared with 7.42 rounds and 7.04 calls under GPT-OSS-120B.This indicates speculative batching of multiple parallel calls per reasoning step, beyond simply retrying more.
  • World-model effects: World-model pairing suppresses aggressive exploration: SPIRAL variants use 1.83–1.92 tool calls, while ReAct-Plan-Exec variants use 6.91–7.91.Simulation forces the planner to commit to a vetted plan rather than probe the live environment.
  • Implications: A stronger planner does not substitute for an explicit world model on efficiency, and low-quality simulations paired with powerful planners are undesirable.The latency and cost of unconstrained exploration scale with planner capability and simulation quality; no conclusive ordering was observed between homogeneous configurations.

6 Limitations

MCP-Cosmos is limited by evaluation on static MCP-Bench environments and by World Models that lack real-time online learning for evolving tools and APIs. It also faces substantial computational costs and metrics that incompletely capture reasoning quality and representation interpretability.

  • Evaluation and adaptation: MCP-Bench relies on a static snapshot of server environments, while current World Models lack real-time online learning for evolving tool schemas or API behaviors.The paper notes that real-world ecosystems are inherently dynamic.
  • Computational cost: High-parameter models such as CLAUDE-SONNET-4.6 impose significant computational overhead, making their cost-performance trade-off potentially prohibitive for latency-sensitive or resource-constrained applications.The paper contrasts superior grounding with the associated computational cost.
  • Evaluation metrics: Current evaluation metrics focus primarily on task success and execution quality, without capturing the quality of intermediate reasoning steps or the interpretability of World Model representations.This leaves important aspects of the agent’s reasoning process and model representations unmeasured.

7 Related Work

Prior world-model research spans reinforcement learning, robotics, and multimodal generation, with emphasis on scalable prediction, test-time rollout, and evaluation. In MCP environments, recent work has introduced large-scale tool-use environment generation and reinforcement-learned models, motivating MCP-specific evaluation methods.

  • World Models: World models support planning, reasoning, and decision-making by learning predictive representations of environment dynamics across reinforcement learning, robotics, and multimodal generation.Recent research emphasizes scalable architectures, test-time rollout, and systematic evaluation.
  • MCP Environments: Recent MCP-environment work introduced an automated tool-use environment-generation pipeline with 1000 ready-to-use environments and three reinforcement-learning-trained models.The passage frames these contributions within growing MCP-based agentic systems and production-oriented world models.
  • MCP Environments: The growth of MCP-based world models creates a need for evaluation methods specific to world models operating in MCP environments.This need is linked to the increasing popularity of MCP, expanding agentic systems, and emerging production-targeted MCP world models.

8 Conclusion · A Technical appendices and supplementary material · A.1 System Requirements

MCP-Cosmos evaluates how infusing world models into agentic planning affects MCP task execution. Its augmented proactive-planning frameworks improve tool selection and parameter accuracy while supporting more parallel execution, with experiments tested primarily on an M1 Mac.

  • 8 Conclusion: MCP-Cosmos is an evaluation suite curated from the ecosystem-scale MCP-Bench.It investigates the impact of world model infusion on agentic planning.
  • 8 Conclusion: The suite investigates the impact of world model infusion on agentic planning.
  • 8 Conclusion: ReAct has traditionally served as the performance ceiling because of its trial-and-error recovery.
  • 8 Conclusion: ReAct-Plan-Exec and SPIRAL-Exec achieve better tool selection when augmented with powerful world models.
  • 8 Conclusion: ReAct-Plan-Exec and SPIRAL-Exec achieve better parameter accuracy when augmented with powerful world models.
  • 8 Conclusion: World-model augmentation increases the possibility of parallel execution through proactive planning.
  • A.1 System Requirements: Experiments ran on a 2021 M1 Max MacBook Pro with 32GB memory and were tested primarily on an M1 Mac.Arctic AWM 4B was deployed natively on the M1 system, while most other models were deployed in the cloud.

A.2 Token Usage Analysis - gpt-oss-120b Planner

Under pass@k=4, ReAct is the most token-efficient configuration, while world-model augmentation substantially increases token consumption through simulation overhead. The results show a trade-off between reduced reasoning iterations and higher inference cost, with model choice balancing capability and efficiency.

  • Token efficiency: 49,995 tokens per task is the baseline ReAct average, making it the most token-efficient configuration.The statistics report average per-task metrics across evaluated gpt-oss-120b planner configurations.
  • Token efficiency: 66%–504% higher per-task token consumption occurs for world-model-augmented agents than for baseline ReAct.The increase reflects the computational trade-off introduced by world models.
  • Token efficiency: 301,908 tokens per task is reported for SPIRAL-Exec + Arctic-AWM-4B-WM, approximately 6× the ReAct baseline.This configuration is at the upper end of per-task token consumption.
  • Token efficiency: 745K total tokens is reported for ReAct-Plan-Exec + claude-sonnet-4.6-WM, versus 7.2M for SPIRAL-Exec + Arctic-AWM-4B-WM across the benchmark.Total values are aggregated over the full benchmark.
  • Efficiency trade-offs: World models reduce reasoning iterations but introduce simulation overhead, while LLM-based and AWM-based models trade off capability against inference cost.The token analysis corroborates the efficiency findings reported in the main body.

A.3 Token Usage Analysis - claude-sonnet-4.6 Planner · B Algorithm

Under the claude-sonnet-4.6 planner, token costs vary substantially across agent configurations, while the WM-infused algorithm separates simulated planning from real execution and can adjust plans after failures. The algorithm selects an execution plan from world-model explorations and returns the final answer, plan, and trajectory.

  • A.3 Token Usage Analysis - claude-sonnet-4.6 Planner: The appendix reports token usage statistics across all evaluated agent configurations with the claude-sonnet-4.6 planner.Average values are measured per task, while total values aggregate the full benchmark.
  • A.3 Token Usage Analysis - claude-sonnet-4.6 Planner: 95,993 average total tokens per task makes ReAct-Plan-Exec + Arctic-AWM-4B-WM the most token-efficient configuration, below baseline ReAct at 99,126.The result indicates that adding AWM-based planning can improve token efficiency without exceeding the baseline’s cost.
  • A.3 Token Usage Analysis - claude-sonnet-4.6 Planner: 99K–103K token usage groups SPIRAL-Exec + gpt-oss-120b-WM, SPIRAL-Exec + claude-sonnet-4.6-WM, and ReAct in a tight cluster.ReAct-Plan-Exec variants with LLM world models instead use 106K–124K, reflecting additional planning and world-model prompting overhead.
  • A.3 Token Usage Analysis - claude-sonnet-4.6 Planner: 686,772 average tokens per task and 16.48M cumulatively make SPIRAL-Exec + Arctic-AWM-4B-WM a high-cost outlier.Prompt tokens dominate output tokens; for ReAct, the example is 93,888 prompt tokens versus 5,238 output tokens.
  • B Algorithm: The planning phase initializes state from task instruction u, generates actions with πplan, and uses the world model to simulate pseudo-observations without real-world execution cost.The state is updated from simulated observations, allowing reasoning about future states during planning.
  • B Algorithm: The algorithm accumulates the simulated trajectory τwm and selects an optimal execution plan P, stopping when Tmax is reached or continuet = false.The selected plan is then executed through actual MCP tool calls in the environment.
  • B Algorithm: When an action fails during execution, πplan_adjust can optionally modify the remaining plan, although expensive revisions were excluded from the plan.After execution, the algorithm generates a final answer and returns the answer, execution plan P, and complete trajectory τ.

C MCP Task and Tool Distribution · C.1 High-Coordination Scenarios (3 Servers) · C.2 Dual-Server Grounding Scenarios (2 Servers)

The MCP-Cosmos evaluation suite comprises 24 curated MCP-Bench scenarios emphasizing ecosystem-scale complexity and multi-server coordination. These include 6 three-server coordination tasks and 18 dual-server scenarios testing bilateral interactions and fuzzy-instruction handling.

  • C MCP Task and Tool Distribution: The evaluation suite selects 24 curated MCP-Bench scenarios to emphasize ecosystem-scale complexity and multi-server coordination.These scenarios form the MCP-Cosmos evaluation suite.
  • C.1 High-Coordination Scenarios (3 Servers): Six tasks require simultaneous grounding and dependency management across three distinct MCP servers.This defines the high-coordination scenario category.
  • C.1 High-Coordination Scenarios (3 Servers): The three-server tasks include paper_search_call_for_papers_wiki_000, paper_search_call_for_papers_wiki_002, and metropolitan_mus_huge_icons_wiki_000.These are three of the six listed high-coordination scenarios.
  • C.1 High-Coordination Scenarios (3 Servers): The remaining three-server tasks are metropolitan_museum_huge_icons_wiki_001, medical_calculator_wikipedia_fruit_000, and medical_calculator_wikipedia_fruit_001.Together with the preceding three tasks, these complete the six-task list.
  • C.2 Dual-Server Grounding Scenarios (2 Servers): Eighteen tasks evaluate bilateral server interactions and stability in handling fuzzy instructions.This defines the dual-server grounding scenario category.
Loading 2605.09131v1…