Source-linked AI summary
PlayWorld: Benchmarking World Models with Agent Players over Long-Horizon Objectives
Kaixin Ding, Xi Chen, Minghong Cai, Zhiyuan Xu, Yiyang Wang, Yuxiang Lu, Junyi Li, Shuyang Chen, Yuan Gao, Xin Tao, Pengfei Wan, Hengshuang Zhao
TL;DR
Interactive world models are difficult to compare fairly because fixed action trajectories may reach different states across models. PlayWorld addresses this with multi-modal Agent Players pursuing shared long-horizon objectives, and evaluates nine models across four evolution and interaction dimensions. The benchmark exposes unreliable long-horizon behavior, especially in spatial consistency and persistent state evolution.
Problem
Existing benchmarks often use predefined trajectories, but differing action granularity can make identical controls reach incomparable states across interactive world models.
Method
PlayWorld uses multi-modal Agent Players to adapt action execution toward shared long-horizon objectives and scores four core capabilities with sample-specific VQA rubrics.
Results
Across nine representative world models, PlayWorld reveals capability boundaries in long-horizon interactive evaluation, with current systems remaining unreliable at sustained spatial and state evolution.
Takeaways & Limitations
Long-horizon objectives and adaptive interaction provide a practical basis for more consistent evaluation of video world models.
Takeaways & Limitations
Basic ability metrics can remain high without objective completion, so they do not by themselves establish strong long-horizon world-model capability.
Abstract
from arXiv · showhide
Video world models simulate future states conditioned on current observations and user actions. Recent systems have demonstrated impressive video consistency and action controllability over long sequences. However, fairly comparing these interactive models remains challenging. In practice, a human player typically evaluates a world model by pursuing long-horizon objectives through interaction. For example, a user may turn around 360 degrees to see whether the environment remains consistent, or walk into the water and inspect whether realistic water ripples are generated. The action sequence required to achieve the same objective may vary substantially between models, making fixed action-conditioned evaluation unsuitable for cross-model comparison. To address this, we employ multi-modal Agent Players to interact with world models toward specified long-horizon objectives. Building on this paradigm, we introduce PlayWorld, a benchmark providing 171 scenarios, each with a specified objective. To evaluate performance thoroughly, we assess models along four core dimensions: geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution. In addition, we incorporate basic ability metrics for video quality and controllability. Experiments across nine state-of-the-art world models reveal that current models remain unreliable on long-horizon interactive objectives, particularly in maintaining spatial consistency and persistent state evolution. Code and data are available at https://github.com/kxding/PlayWorld.
1 INTRODUCTION
Interactive world models require evaluation beyond video quality because fixed low-level trajectories can produce incomparable outcomes across models. PlayWorld instead uses adaptive Agent Players to pursue shared long-horizon objectives across 171 cases.
- Existing benchmarks measure action responses and world-model capabilities, but typically use customized trajectories predefined for individual cases.
- The same commands can complete a 360° rotation in one model but only a partial turn in another, making geometry scores incomparable.
- PlayWorld gives each case a shared basic action reference, then adaptively adjusts execution through Keep, Stop, Extend, Correct, or End decisions.
- The benchmark evaluates geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution with a VQA rubric verifier.
- 171 human-annotated cases were evaluated across nine representative world models to reveal current capability boundaries.
2 RELATED WORK
Prior benchmarks assess perceptual quality, control, and world-model capabilities, but predominantly rely on predefined low-level controls. PlayWorld introduces shared objectives with online action adaptation to preserve evaluation intent across models.
- Video benchmarks measure perceptual quality and alignment but generally do not assess responses to user actions.
- World-model benchmarks evaluate capabilities such as camera control, multi-view geometry, and embodied control, often without adapting controls to generated observations.
- Because models differ in action granularity and response dynamics, identical controls may fail to reach comparable evaluation states.
- PlayWorld specifies a shared long-horizon objective and adapts action execution online through a multi-modal Agent Player.
3 PLAYWORLD
PlayWorld combines an adaptive Agent Player with scene-grounded objectives and sample-specific VQA rubrics for long-horizon interactive evaluation. Its benchmark spans diverse cases and tests geometry, interaction, hidden-state, and visible-process evolution alongside basic video metrics.
- Agent Player: The Agent Player observes generated frames and execution history, then updates actions in a closed interaction loop toward the objective.
- Agent Player: A basic human-annotated action sequence provides an initial reference, reducing latency and improving comparability while the agent adapts execution online.
- Benchmark: The benchmark contains over 170 cases covering 50 action patterns, 10–60-second rollouts, nine world models, over 1,400 videos, and more than 820 VQA questions.
- VQA rubric verifier: The VQA verifier checks trajectory validity before dimension-specific criteria, including identity, causality, physical response, and state progression.
- Evaluation dimensions: Cases target geometry revisitation, physically plausible interactions, hidden-state changes during occlusion, and continuously observed state progression.
- Basic ability evaluation: Basic metrics complement rubric evaluation with video-quality and controllability measures, while trajectory validation excludes stationary insight-evolution cases.
4 EXPERIMENTS
Experiments evaluate nine representative world models and compare adaptive Agent Player control with alternative strategies. Results show persistent evolution, spatial consistency, and complex interaction remain difficult, while basic video-quality metrics can overstate long-horizon capability.
- Models and protocol: Nine representative world models are evaluated using shared objectives, action sequences, Agent Player adjustments, and rubric-based and basic-ability metrics.The benchmark evaluates models through web interfaces or local chunk-wise generation under a common protocol.
- Agent Player analysis: Preset + Agent achieves the highest Trajectory Score and Human Preference on both Genie 3 and HappyOyster.Its Agent-modified Action Ratio remains within 10–20% on both models, indicating targeted modifications rather than wholesale replanning.
- Agent Player analysis: Agent model choice produces only minor trajectory-quality differences, with Claude Haiku selected for the overall quality–efficiency tradeoff.Claude Sonnet has the highest Trajectory Score, while Claude Haiku has the highest Human Preference and lowest measured Decision Latency.
- World-model observations: Out-of-sight evolution and insight evolution consistently score lower than other dimensions, making persistent state evolution a primary bottleneck.Models more reliably preserve visible structure or produce immediate responses than sustain semantic changes across time and occlusion.
- World-model observations: Long-horizon revisitation exposes global spatial inconsistency, including repeated generation of landmarks at different viewpoints during 360-degree orbits.The failure is consistent with reliance on local appearance continuity and short-range motion cues rather than a persistent global 3D representation.
- World-model observations: Complex interactions remain unreliable: models may pass through obstacles, omit water responses, or bind controls inconsistently to characters.Simple collision responses are handled more successfully than physically or visually richer interactions.
- World-model observations: Strong trajectory control does not necessarily imply strong world-model ability, as SANA-WM can reach objective regions while retaining modest rubric scores.The gap reflects difficulties with memory, spatial consistency, and physically plausible interactions.
- Metric validation: Basic ability metrics are insufficient for long-horizon evaluation: high video quality or controllability can coexist with failure to reach objective-specific target states.Depth Stability and Subject Consistency may remain high when camera motion is absent, and HappyOyster’s highest Basic Ability Score does not establish strongest long-horizon capability.
5 CONCLUSION
PlayWorld uses multi-modal Agent Players to simulate human evaluation of interactive video world models. Its 171 human-annotated cases and automated evaluation system provide a basis for consistent assessment under shared long-horizon objectives.
- Conclusion: PlayWorld uses multi-modal Agent Players to simulate how human users evaluate interactive video world models.The benchmark evaluates models through interaction rather than only fixed action execution.
- Conclusion: 171 human-annotated cases and an end-to-end automated evaluation system support consistent evaluation under shared long-horizon objectives.The benchmark is presented as a practical basis for more precise evaluation of video world models.
A IMPLEMENTATION DETAILS
The implementation uses a closed-loop Agent Player interface, standardized metrics, rubric verification, and contact-sheet inputs. Radar charts display rubric dimensions and Basic Ability on different scales.
- Closed-loop execution: Each case gives models the same initial frame, long-horizon objective, and annotated action sequence, while the Agent Player adapts execution from generated frames and action history.The loop returns Keep, Stop, Extend, Correct, or End until termination or the interaction limit.
- Model execution: Models are evaluated through either user-facing web interfaces or local chunk-wise generation, with common action definitions and recorded observations.The protocol includes five web-based models and four locally executed models.
- Metrics: Basic Ability combines seven video-quality metrics with translation and rotation pass rates for action controllability.Translation passes below target-normalized error 0.3, while rotation passes below mean geodesic error 45°; pass rates use valid estimates only.
- VQA verification: The VQA verifier validates trajectories before scoring geometry consistency, interaction fidelity, and out-of-sight evolution; insight evolution uses stationary observation instead.Invalid rollouts receive the minimum dimension score of 1, while insight evolution does not require trajectory validation.
- VQA verification: The verifier receives a 10-FPS primary contact-sheet stream and a 0.5-FPS detail stream together with the objective and sample-specific rubric.The streams use 5 × 5 and 2 × 2 frame grids at different resolutions to balance temporal coverage and spatial detail.
- Scoring and visualization: The Overall score is the unweighted mean of geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution.These rubric dimensions are displayed from 1–3, whereas Basic Ability uses a 0–1 range, so their radar-chart scales are not directly comparable.
- Interface execution: The agent interface enters the initial condition, waits for interactivity, dispatches controls while capturing observations, and preserves final video and execution records at termination.This figure summarizes the operational sequence of an evaluation episode.
B VQA SCORING ROBUSTNESS
PlayWorld uses one fixed VQA scoring pass per rollout and separately checks robustness because Gemini outputs can vary across repeated calls.
- A single fixed VQA scoring pass per rollout is used for the reported benchmark results.An additional independent pass assesses robustness.
- 0.0112 mean per-model sample variance separates the two scoring passes across nine models.The authors interpret this low variance as evidence of stable aggregate results.
- The two-pass analysis is not a precise estimate of scoring uncertainty.The authors recommend averaging multiple independent VQA passes when the evaluation budget permits.
C AGENT INTERFACE FOR WEB-BASED MODELS
PlayWorld interfaces with web-based world models through authenticated browser sessions, capturing generated observations and executing the Agent Player’s adaptive controls.
- The interface connects to authenticated browser sessions because web-based world models lack a common local inference API.It initializes cases, dispatches model-specific controls, monitors generation, and captures observations.
- Keep, Stop, Extend, or Correct decisions trigger the next control, while End terminates the episode.This execution loop connects captured frames to subsequent Agent Player actions.
- For each case, the browser supplies the initial condition, selects an available perspective, waits for interactivity, and captures the rendered world after each control.The Agent Player determines only how the world should be controlled; interface operations handle navigation and rendering checks.
- Figure 7 presents a HappyOyster execution from caption entry to termination using direct crops from the recorded browser session.Browser chrome, system controls, and assistant overlays are excluded without altering generated world content.
D HUMAN ANNOTATION AND VALIDATION
Human annotators construct benchmark cases and rubrics from observable evidence, while a blinded pairwise study validates rubric-based rankings against human preferences.
- Benchmark annotation: Annotators screen initial worlds, verify captions, assign viewpoints, write objectives and action sequences, and construct sample-specific VQA rubrics.They remove questions whose answers cannot be determined from the generated rollout.
- Human-alignment study: Five participants independently judge 120 concealed-model video pairs, producing 600 valid pairwise judgments across four equally sampled dimensions.Each comparison permits selecting the left video, right video, or Tie.
- Preference aggregation: Human preference gives full credit to wins and half credit to ties, then normalizes each dimension’s nine-model preference scores to sum to 100%.Overall preference pools comparisons across the four dimensions.
- Validation results: 95.8% of pairwise items had agreement from at least three raters, while rubric-based VQA scores significantly correlated with human preferences across all dimensions.Overall Spearman correlation was ρ = 0.933; dimension correlations ranged from ρ = 0.745 to ρ = 0.983.