Source-linked AI summary
WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPG
Zhen Li, Zian Meng, Shuwei Shi, Wenshuo Peng, Yuwei Wu, Bo Zheng, Chuanhao Li, Kaipeng Zhang
TL;DR
Existing datasets provide limited semantic actions and often entangle actions with visual changes, complicating structured state-dynamics learning. WildWorld supplies explicit state annotations at scale and WildBench evaluates action following and state alignment; experiments still expose challenges in semantic action modeling and long-horizon consistency.
Problem
Existing datasets often lack diverse semantically meaningful actions and explicit states, while action effects remain tied directly to visual observations.
Method
WildWorld automatically collects over 108M gameplay frames with rich actions and synchronized annotations of actions, skeletons, world states, camera poses, and depth, then derives WildBench.
Results
WildBench experiments reveal persistent limitations in modeling semantically rich actions and maintaining long-horizon state consistency.
Takeaways & Limitations
WildWorld and WildBench provide a foundation for building and systematically evaluating state-aware interactive world models.
Takeaways & Limitations
State evolution may be stochastic because of random events, so ground-truth alignment is statistically meaningful over multiple samples.
Abstract
from arXiv · showhide
Dynamical systems theory and reinforcement learning view world evolution as latent-state dynamics driven by actions, with visual observations providing partial information about the state. Recent video world models attempt to learn this action-conditioned dynamics from data. However, existing datasets rarely match the requirement: they typically lack diverse and semantically meaningful action spaces, and actions are directly tied to visual observations rather than mediated by underlying states. As a result, actions are often entangled with pixel-level changes, making it difficult for models to learn structured world dynamics and maintain consistent evolution over long horizons. In this paper, we propose WildWorld, a large-scale action-conditioned world modeling dataset with explicit state annotations, automatically collected from a photorealistic AAA action role-playing game (Monster Hunter: Wilds). WildWorld contains over 108 million frames and features more than 450 actions, including movement, attacks, and skill casting, together with synchronized per-frame annotations of character skeletons, world states, camera poses, and depth maps. We further derive WildBench to evaluate models through Action Following and State Alignment. Extensive experiments reveal persistent challenges in modeling semantically rich actions and maintaining long-horizon state consistency, highlighting the need for state-aware video generation. The project page is https://shandaai.github.io/wildworld-project/.
1 Introduction
Existing world-modeling datasets often lack semantically rich actions and explicit states, making structured action-conditioned dynamics difficult to learn. WildWorld addresses this gap with a large-scale dataset and benchmark centered on explicit state annotations and long-horizon evaluation.
- Existing datasets commonly provide simple, weakly semantic actions whose effects are directly tied to visual changes.
- Implicit state transitions, such as ammunition depletion after shooting, can produce different visual outcomes for the same action.
- WildWorld contains over 108M frames and more than 450 semantically meaningful actions with synchronized frame-level state annotations.
- WildBench evaluates interactive world models using Action Following and State Alignment.
- The paper combines WildWorld, WildBench, and extensive experiments to provide insights into state transition modeling and long-horizon consistency.
2 Related Work
Prior work has advanced video generation, interactive world modeling, and latent-state representations, but WildWorld emphasizes explicit semantic states and annotations for analyzing state dynamics.
- Interactive video generation has incorporated prompt switching, keyboard actions, and camera poses on top of image-to-video generation.
- Recent approaches introduce latent state representations, but typically learn them implicitly from visual observations.
- World-modeling datasets increasingly include gameplay videos or action signals, while WildWorld focuses on explicit, semantically meaningful state annotations.
3 WildWorld Dataset
WildWorld is curated through automated gameplay and synchronized acquisition of actions, states, and observations, followed by filtering and annotation. It combines large scale with diverse entities, scenes, long sequences, and rich action representations.
- Curation Pipeline: The curation pipeline comprises data acquisition, automated gameplay, data processing, and caption annotation.
- Data Acquisition: The acquisition platform records actions, underlying world states, and visual observations at different stages of game execution.
- Data Acquisition: The platform records player and monster actions, locations, rotations, velocities, animation IDs, gameplay attributes, and skeletal poses.
- Automated Collection: Automated gameplay and timestamped multi-source recording support scalable collection while enabling synchronization across data streams.
- Data Processing: Filtering removes short, temporally discontinuous, unusually bright or dark, and camera-occluded samples.
- Dataset Scale: 108 million frames and 119 annotation columns per frame remain after processing and filtering.
- Dataset Diversity: The dataset covers 29 monster species, 4 player characters, 4 weapon types, 5 stages, and combat and mount-traversal contexts.
- Temporal and Spatial Dynamics: Most clips span 4,000–28,000 frames, some exceed 40,000 frames, and median camera-to-character and character-to-monster distances are 15.69 and 12.63 units.
4 WildBench Benchmark
WildBench evaluates interactive world models beyond visual quality by measuring camera control, action following, and state alignment from synchronized ground truth. Its metrics compare generated behavior and skeletal trajectories against reference data.
- Evaluation Dimensions: WildBench evaluates video quality, camera control, action following, and state alignment using WildWorld’s action and state ground truth.
- Video Quality: Video Quality uses Motion Smoothness, Dynamic Degree, Aesthetic Quality, and Image Quality to assess motion and appearance.
- Camera Control: Camera Control measures discrepancy between ground-truth camera trajectories and trajectories estimated from generated videos.
- Action Following: Action Following segments samples by frame-wise action IDs and scores whether generated and ground-truth clips express the same action.
- State Alignment: State Alignment compares generated and ground-truth skeletal keypoint trajectories using mean coordinate accuracy across pixel thresholds.
- Evaluation Assumption: State Alignment remains statistically meaningful across samples despite stochastic state evolution from random events.
5 Experiments and Analysis
Experiments evaluate camera-, skeleton-, and state-conditioned generation approaches on WildBench, using explicit annotations and interaction-focused metrics. Results show broad gains over baseline, metric saturation in generic video quality, and trade-offs between interaction control, image quality, and autoregressive stability.
- Compared Approaches: CamCtrl uses camera trajectories, SkelCtrl uses skeleton videos, and StateCtrl injects structured discrete and continuous states into video generation.StateCtrl encodes entity-level and global states, then injects a unified state representation into intermediate DiT layers.
- Compared Approaches: WildBench evaluates interactive models using Action Following, State Alignment, and video-quality measures, with lower ATE/RPE preferred and higher values preferred for other metrics.The benchmark evaluation compares approaches trained on WildWorld against a baseline using the dataset’s ground-truth annotations.
- Overall Evaluation: All approaches improve interaction-related metrics over baseline; CamCtrl reduces Camera Control ATE by △2.61 and RPE by △0.05, while SkelCtrl nearly doubles Action Following and State Alignment.StateCtrl and StateCtrl-AR improve across all three reported interaction-related metrics.
- Overall Evaluation: All methods exceed 95% on VBench Motion Smoothness and Dynamic Degree, yet differ substantially on Action Following and State Alignment.The results indicate that generic video-quality metrics are saturated relative to the finer-grained WildBench interaction metrics.
- Overall Evaluation: SkelCtrl gains more on interaction metrics than StateCtrl but has lower Aesthetic Quality and Image Quality, revealing a trade-off from directly using visual control signals.Qualitative comparisons show StateCtrl produces clearer foreground subjects, whereas SkelCtrl better reproduces occlusion effects such as splashing sand and gravel.
- Overall Evaluation: StateCtrl-AR performs comparably to StateCtrl but shows a noticeable Action Following drop, attributed to error accumulation during iterative next-state prediction.The autoregressive setting uses only the first-frame state and predicts subsequent states as control inputs.
6 Conclusion
WildWorld is presented as a large-scale video dataset with explicit state annotations for action-conditioned world modeling. It is collected from Monster Hunter: Wilds and provides a rich action space with frame-level annotations.
- Conclusion: WildWorld is a large-scale video dataset with explicit state annotations, automatically collected from Monster Hunter: Wilds through a scalable pipeline.The dataset targets the study of action-conditioned world modeling.
- Conclusion: WildWorld provides over 450 semantically meaningful actions and frame-level annotations including character skeletons, world states, camera poses, and depth.