Source-linked AI summary
World Reasoning Arena
PAN Team, Qiyue Gao, Kun Zhou, Jiannan Xiang, Zihan Liu, Dequan Yang, Junrong Chen, Arif Ahmad, Cong Zeng, Ganesh Bannur, Xinqi Huang, Zheqi Liu, Yi Gu, Yichi Yang, Guangyi Liu, Zhiting Hu, Zhengzhong Liu, Eric Xing
TL;DR
Existing world-model benchmarks largely emphasize next-state prediction and visual fidelity, leaving richer simulation capabilities under-tested. WR-Arena evaluates instruction-following, long-horizon forecasting, and simulative reasoning and planning; its experiments find substantial capability gaps, while semantically actionable rollouts are associated with planning gains.
Problem
Existing world-model benchmarks focus largely on next-state prediction and visual fidelity rather than the simulation capabilities needed to assess reasoning, long-horizon coherence, and purposeful action.
Source: §1 · pp. 1–2Method
WR-Arena builds a task taxonomy and curated datasets to evaluate action simulation fidelity, long-horizon forecasting, and simulative reasoning and planning.
Source: §1 · p. 2, §6 · p. 15Results
Current world models show significant gaps in following high-level instructions and maintaining long-horizon consistency, and only semantically actionable rollouts significantly improve planning performance.
Source: §1 · p. 3, §6 · p. 15Takeaways & Limitations
WR-Arena serves as a diagnostic tool and guideline for developing world models that understand, forecast, and plan in complex real-world environments.
Source: §1 · p. 3
Abstract
from arXiv · showhide
World models (WMs) are intended to serve as internal simulators of the real world that enable agents to understand, anticipate, and act upon complex environments. Existing WM benchmarks remain narrowly focused on next-state prediction and visual fidelity, overlooking the richer simulation capabilities required for intelligent behavior. To address this gap, we introduce WR-Arena, a comprehensive benchmark for evaluating WMs along three fundamental dimensions of next world simulation: (i) Action Simulation Fidelity, the ability to interpret and follow semantically meaningful, multi-step instructions and generate diverse counterfactual rollouts; (ii) Long-horizon Forecast, the ability to sustain accurate, coherent, and physically plausible simulations across extended interactions; and (iii) Simulative Reasoning and Planning, the ability to support goal-directed reasoning by simulating, comparing, and selecting among alternative futures in both structured and open-ended environments. We build a task taxonomy and curate diverse datasets designed to probe these capabilities, moving beyond single-turn and perceptual evaluations. Through extensive experiments with state-of-the-art WMs, our results expose a substantial gap between current models and human-level hypothetical reasoning, and establish WR-Arena as both a diagnostic tool and a guideline for advancing next-generation world models capable of robust understanding, forecasting, and purposeful action. The code is available at https://github.com/MBZUAI-IFM/WR-Arena.
1 Introduction
WR-Arena evaluates world models beyond short-term prediction and perceptual fidelity, probing instruction-following, long-horizon simulation, and planning. Its findings show persistent weaknesses in these capabilities, including a gap from human-level hypothetical reasoning.
1 Introduction
- Current models struggle to control action simulations, particularly for environment-centric commands, exposing difficulty faithfully following high-level instructions.Source: §1 · p. 2
- Long-horizon simulation remains difficult for all models, as accumulated error degrades consistency.Source: §1 · p. 2
- Only models producing semantically actionable rollouts significantly improve planning; perceptual quality alone is insufficient for decision-making.Source: §1 · p. 3
- Models jointly optimizing understanding, prediction, and control, such as PAN, deliver the most balanced performance across evaluation dimensions.Source: §1 · p. 3
- These findings reveal a substantial gap between current world models and human-level hypothetical reasoning.WR-Arena is presented as a diagnostic tool and guideline for developing world models that understand, forecast, and plan in complex real-world environments.Source: §1 · p. 3
2 Preliminaries
The preliminaries define world models as transition-based simulators that can explore alternative futures for reasoning and planning. They contrast existing evaluation focused on short-term prediction or visual fidelity with WR-Arena’s three complementary simulation dimensions.
2.1 World Model
- A world model transforms a previous state s and action a into a next state s′, serving as a generative simulator of possible futures.Source: §2.1 · p. 3
- Iterated transitions generate trajectories for thought experiments over alternative scenarios, including counterfactuals, paralleling human reasoning with mental models.People may project different futures and act according to their expected reward.Source: §2.1 · p. 3
- For planning, an agent can roll out candidate action sequences from an initial state and choose the trajectory that best achieves its goal.The preliminaries also describe world models as supporting transfer to unfamiliar tasks through shared mechanistic regularities.Source: §2.1 · p. 3
2.2 World Model Evaluation
- Existing benchmarks emphasize low-level control, next-state prediction, and video fidelity, often in single-turn settings that miss extended coherence and instruction-following demands.Source: §2.2 · pp. 3–4
- Action Simulation Fidelity evaluates following semantically meaningful multi-step instructions targeting agents or environments and producing diverse, counterfactual futures.Source: §2.2 · p. 4
- Long-horizon Forecast evaluates coherent extended rollouts, including prediction accuracy, temporal smoothness, and error accumulation.Source: §2.2 · p. 4
- Simulative Reasoning and Planning evaluates goal-directed simulation of candidate actions in structured and open-ended environments.Source: §2.2 · p. 4
- Together, the three dimensions move evaluation beyond single-turn or perceptual tests toward how models map states to futures and act toward goals.Source: §2.2 · p. 4
3 Evaluation Framework
WR-Arena evaluates world models as simulators across instruction-following, long-horizon coherence, and goal-directed planning. Its framework combines multi-step action rollouts, extended-sequence metrics, and iterative planner–world-model tasks in open-ended and structured settings.
3 Evaluation Framework
- The framework focuses on advanced world-model capabilities needed for complex goal-directed applications such as autonomous driving and robot navigation.Source: §3 · p. 4
3.1 Action Simulation Fidelity
- Action Simulation Fidelity tests whether a model turns high-level, multi-step instructions into faithful state sequences and diverse counterfactual futures.The framework compares agent-targeted control with environment-level scene interventions, using action faithfulness and precision as criteria.Source: §3.1 · pp. 4–5
3.2 Long-horizon Forecast
- Long-horizon Forecast assesses coherent rollouts through transition smoothness and generation consistency, with additive penalties linking initial fidelity to later stability.Transition Smoothness scores optical-flow motion and acceleration around round boundaries, rewarding visible gradual motion while penalizing jerks; scores are normalized across videos.Source: §3.2 · pp. 6–7
3.3 Simulative Reasoning and Planning
- Simulative Reasoning and Planning pairs a VLM planner with a world model that simulates candidate actions so the planner can choose actions toward a goal.Evaluation includes immediate robot-manipulation predictions, open-ended household tasks, and structured tabletop tasks.Source: §3.3 · pp. 7–8
4 Experiment
WR-Arena evaluates interactive world models and video generators across action simulation, long-horizon forecasting, and planning. Results show persistent difficulty with environment interventions and extended rollouts, while PAN has the most balanced profile across the evaluated dimensions.
4.1 Baselines
- The evaluation covers interactive world models and open- and closed-source video generators, comparing performance across all three benchmark dimensions.The evaluated video generators include WAN-2.1 and WAN-2.2, KLING, MiniMax, and Gen-3.Source: §4.1 · p. 8, Table 1 · p. 9
4.2 Main Results
- 11.5% on average, Agent Simulation exceeds Environment Simulation; MiniMax scores 72.33% on Agent Simulation and 51.67% on Environment Simulation.No model exceeds 60% accuracy on Environment Simulation; PAN improves over WAN2.1 by 16.66% on Agent Simulation and 10.0% on Environment Simulation.Source: Table 1 · p. 9, Action Simulation Fidelity · p. 9, §4.2 · p. 9
Action Simulation Fidelity.
- Action–state aligned fine-tuning improves PAN over WAN2.1 on both agent- and environment-targeted simulation, while environment-level manipulations remain harder across models.The average Agent Simulation advantage over Environment Simulation is 11.5%.Source: Action Simulation Fidelity · p. 9
Long-horizon Forecast.
- No evaluated system exceeds 65% on Transition Smoothness or Generation Consistency; PAN performs best on both long-horizon metrics.Consistency degrades as action sequences lengthen; PAN’s curve remains comparatively flat and is highest after round 3, while WAN 2.1 falls from approximately 90% to 30% over nine rounds.Source: Long-horizon Forecast · p. 10, §4.3 · p. 11, Figure 3 · p. 11
Simulative Reasoning and Planning.
- With the same VLM planner, PAN raises trajectory-level success by 26.33% in open-ended settings and 23.40% in structured environments.The authors report inconsistent effects from Cosmos 1 & 2 and V-JEPA 2, which sometimes provide modest improvements and sometimes fail to guide planning.Source: Simulative Reasoning and Planning · p. 10
4.3 Analysis
- PAN has the most balanced profile across dimensions, whereas commercial video generators and open-source world models have different limitations in task adaptation, control, temporal stability, or planning gains.The analysis attributes PAN’s profile to VLM priors and action–state aligned fine-tuning; it motivates jointly optimizing understanding, prediction, and control.Source: Simulative Reasoning and Planning · p. 10
5 Qualitative Results
The qualitative examples show how models handle goal-directed block arrangement and controlled weather interventions, highlighting the demands of coherent planning and faithful simulation.
5.1 Simulative Reasoning and Planing
- PAN makes targeted moves and corrects earlier placements, while Cosmos 1 repeats centering actions and Cosmos 2’s more varied steps remain partly uncoupled from the line-arrangement goal.PAN’s rollout includes state-aware refinement; Cosmos 1’s longer sequence shows weaker goal decomposition, while Cosmos 2 simulates plausible transitions without consistently prioritizing goal progress.Source: §5.1 · p. 13
5.2 Simulation Fidelity
- In the driving intervention, PAN preserves scene identity and renders escalating rain cues coherently, while Cosmos outputs vary and may under-express physical cues.The comparison follows a dry forest road through light rain to intensifying rainfall.Source: §5.2 · pp. 13, 15, Figure 5 · p. 14
- Simulation fidelity requires both preserving scene identity and realizing the intervention with the correct type and magnitude of physical change.The section identifies these constraints as relevant to realistic long-horizon rollouts and evaluation of environment-dependent decision making.Source: §5.2 · p. 15
6 Conclusion
WR-Arena evaluates world models as next-world simulators across action fidelity, long-horizon forecasting, and simulative reasoning and planning. Experiments find significant gaps in high-level instruction following and long-horizon consistency, positioning the benchmark as a diagnostic tool and roadmap for future research.
6 Conclusion
- WR-Arena systematically evaluates action simulation fidelity, long-horizon forecast, and simulative reasoning and planning using diverse tasks and datasets.The benchmark tests high-level control, coherent multi-step rollouts, and comparison of alternative futures.Source: §6 · p. 15
- Current state-of-the-art world models show significant gaps in following high-level instructions and maintaining long-horizon consistency, with error accumulation degrading multi-turn rollouts.Environment-centric control is particularly difficult; only models producing semantically actionable rollouts and jointly optimizing understanding, prediction, and control meaningfully improve planning performance.Source: §6 · p. 15
- WR-Arena serves as a diagnostic tool and roadmap for developing world models that can understand, forecast, and plan across diverse real-world environments.The conclusion frames this as a direction for future research.Source: §6 · p. 15