PaperlayerView exact revision on arXiv

Source-linked AI summary

Wizard: Manipulate Any World in Real Time via Agentic Control

Yuzhe Wu, Zijun Lin, Yeying Jin

arXiv:2610.40339v1cs.CV

TL;DR

Existing game world models have limited control flexibility, generalization to unseen settings, and real-time interaction. Wizard combines agentic state control with geometry and skeleton proxies for customizable gameplay, and demonstrates zero-shot generalization across unseen scenes, characters, and motions.

  • Problem

    Existing approaches offer limited control flexibility, generalization to unseen scenes and characters, and real-time interaction.

    Source: §1 · p. 1

  • Method

    Wizard combines an agentic framework that updates scene and combat states with geometric scene proxies, articulated character skeletons, aligned training data, and adaptive reference-image attention masks.

    Source: §1 · pp. 1–3

  • Results

    Wizard demonstrates zero-shot generalization to unseen scenes, characters, and motions, with rollouts reflecting specified scene structure and character motions.

    Source: §5.2 · p. 9, §6 · p. 12

  • Takeaways & Limitations

    Players can compose character transformations and scene manipulations within a gameplay scenario to create customized worlds.

    Source: §5.3 · p. 9

    The motion library can make compatible motions difficult to find for uncommon body structures, and scene-proxy consistency may decline over longer rollouts.

    Source: §6 · p. 12

Abstract

from arXiv · show

Recent advances in video world models have enabled the generation of increasingly realistic and interactive virtual environments, opening new possibilities for player-customizable gameplay. Agentic frameworks further advance this direction by decoupling explicit game logic and controllable world representations from visual generation. However, existing approaches support only simple game settings and predefined motion patterns, generalize poorly to unseen environments and characters, and offer limited real-time interactivity. In this paper, we introduce Wizard, an agentic game world model that enables real-time, precise control over character motions while generalizing zero-shot across novel scenes, characters, and actions. To achieve this, we construct a diverse dataset spanning 750 scenes and 178 characters, incorporating skeleton and geometry information as structured representations. We leverage these representations as intermediate proxies to condition a video generation model, together with an adaptive masking mechanism that improves alignment between the conditioning signals and generated video rollouts. Furthermore, we introduce an agentic framework that explicitly maintains and updates game states, allowing players to freely configure their virtual worlds and exercise fine-grained control over character motions while preserving visual quality and gameplay consistency. By combining agentic control with structured proxy-based generation, our framework unlocks new possibilities for player-driven game creation with real-time, generalizable, and mechanically consistent world simulation.

1 Introduction

Existing game world models offer limited control flexibility, generalization, and real-time interaction. Wizard combines structured spatial control with an agentic framework to support customizable worlds and precise character motion.

1 Introduction

  • Existing models constrain players to predefined actions or training-seen motions, while agentic and engine-based approaches remain limited in articulated actions, scene flexibility, or interaction latency.These limitations motivate broader control over characters and scenes in unseen settings.Source: §1 · p. 1
  • Wizard combines fine-grained character control, zero-shot generalization, real-time interaction, and mechanically consistent gameplay across novel scenes, characters, and motions.Players can specify changes through reference images or free-form language.Source: §1 · p. 1
  • Geometric scene proxies encode depth and height, while articulated character skeletons provide motion controls within a unified visual proxy.User-provided scene images are converted into approximate traversable 2.5D environments, and character references into rigged, controllable skeletons.Source: §1 · p. 2
  • The agent updates scene geometry from camera inputs, animates skeletons from action inputs, retrieves and retargets motions, and tracks combat states to enforce gameplay rules.Tracked states include health, relative distance, and orientation.Source: §1 · p. 2
  • Wizard trains on 50.3 hours of aligned proxy–video pairs and uses character-aware adaptive attention masks to improve appearance alignment and preserve character identity.Causal adaptation and few-step distillation enable real-time video generation.Source: §1 · pp. 2–3
  • The dataset spans 750 scenes and 178 characters, with aligned skeleton and scene-geometry information to support generalization.The framework uses structured proxies for fine-grained control and mechanics-consistent gameplay.Source: §1 · p. 3

2 Related Work

Video world models and agentic world models address different challenges: interactive rendering and explicit state control, respectively. Wizard extends agentic control to articulated character animation and scene geometry while targeting zero-shot generalization and real-time gameplay.

2.1 Video World Model

  • Action-conditioned models support interactive gameplay, but pixel-level supervision can make persistent states, long-term history, and spatial consistency difficult to maintain.Existing methods also remain constrained to settings seen during training, whereas Wizard targets unseen scenes, characters, and motions.Source: §2.1 · p. 3

2.2 Agentic World Model

  • Agentic models separate world-state management from visual generation, but prior work emphasizes state consistency and entity control more than fine-grained character animation.Wizard adds scene geometry and articulated skeleton control; its diverse data and causal adaptation with few-step distillation support generalization and real-time generation.Source: §2.2 · p. 3

3 Data

Wizard’s dataset pairs video with scene geometry and articulated skeleton proxies to support fine-grained control and generalization. It combines multiple data sources and converts them into a unified conditioning representation.

3 Data

  • The dataset addresses the lack of diverse video-world-model data paired with both character skeletons and scene geometry, comprising 50.3 hours across 750 scenes and 178 characters.It covers varied environments, character morphologies, and motion patterns.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Diverse Data Sources

  • Four sources contribute gameplay recordings, synthetic renders, Unreal Engine simulations, and public gameplay footage to broaden coverage and support scalable data production.The sources contribute 8.8, 11.6, 3.3, and 26.6 hours, respectively.Source: §3 · p. 4, §3.1 · pp. 4–5

3.2 Dataset Processing Pipeline

  • Data acquisition aligns camera parameters, skeletons, and scene depth with video frames, using game-process extraction, simulation assets, or public annotations as available.Real-game scene depth is captured once per second and used to reconstruct geometry for rendering at each frame’s camera pose.Source: §3.2 · p. 5
  • The unified proxy jointly renders scene geometry and character skeletons under a shared camera, encoding height and camera-forward depth in color channels.At each pixel, the renderer retains the surface closest to the camera.Source: §3.2 · p. 5
  • Character slots and facing direction are encoded separately from appearance, with each slot linked to a reference image to support control of unseen characters.Synthetic samples are filtered for implausible configurations or rendering failures, then captioned alongside aligned proxy videos, RGB clips, and reference images.Source: §3.2 · p. 5

4 Method

Wizard separates explicit control of scene geometry, character animation, and combat state from video generation, which renders those controls through structured visual proxies. Its agentic framework handles gameplay decisions, while character-aware conditioning and a real-time training pipeline guide video generation.

4 Method

  • Wizard updates character motion, scene geometry, and combat state in response to player inputs and boss decisions, then renders the resulting state as video.Session initialization takes less than five minutes to reconstruct the scene and characters, rig skeletons, and build a target-specific motion set.Source: §4 · p. 5, Figure 3 · p. 6

4.1 Agentic Framework

  • Character animation uses semantically labeled skeletons to retrieve compatible motions, retarget them to each character, and filter clips for physical plausibility.Motion retargeting matches limb chains, adapts to limb lengths, and uses inverse kinematics to maintain ground contact.Source: §4.1 · pp. 6–7
  • The world agent reconstructs scene geometry from an image and applies natural-language-directed geometric edits, re-rendering the scene from the current camera viewpoint.Its geometry proxies encode world height and inverse depth, and the agent can extend terrain beyond the input view.Source: §4.1 · p. 7
  • A shared world state guides boss movement and attack selection, while a rule-based controller checks hitboxes, cooldowns, and health to resolve combat outcomes.Combat ends when either character’s health reaches zero, and the updated state informs the planner’s next decision.Source: §4.1 · p. 7

4.2 Model Training

  • The video generator conditions on visual proxies, the initial scene image, character references, and context from previously generated frames.Proxy latents are concatenated with noisy video latents, while first-frame and character-reference tokens are added for DiT processing.Source: §4.2 · p. 7, Figure 4 · p. 8
  • Skeleton-derived spatial masks restrict each character reference to its expanded region and update with character state to reduce appearance leakage during gameplay.The region is expanded by 25%; during transformation, attention to cached tokens for the previous identity is blocked.Source: §4.2 · p. 8
  • A three-stage causal adaptation and distillation pipeline preserves proxy and character-reference conditioning while producing rollouts at 24 FPS with two denoising steps.The stages are teacher-forced autoregressive adaptation, consistency distillation, and distribution matching distillation.Source: §4.2 · p. 8

5 Experiments

The experiments present qualitative evaluations using references unseen during training and the same trained model without additional fine-tuning. They show zero-shot generalization and gameplay examples that combine character and scene changes.

5.1 Experimental Setup

  • Training data are partitioned into non-overlapping 121-frame windows at 30 fps, each about four seconds; qualitative zero-shot and creative-gameplay examples use unseen references and the same trained model without fine-tuning.Examples pair user-provided scene or character references with agent-controlled visual proxies and generated gameplay frames.Source: §5.1 · p. 9

5.2 Zero-shot Generalization

  • Wizard generates real-time combat rollouts for novel combinations of scenes, characters, and motions, with proxy poses and spatial arrangements reflected in video frames.The examples use user-provided inputs unseen during training and illustrate visual consistency across rollouts.Source: §5.2 · p. 9, Figure 5 · p. 10
  • The agentic framework maintains game state and enforces combat rules, including ending combat and displaying the outcome when either character’s health reaches zero.The paper notes that additional gameplay rules can also be incorporated.Source: §5.2 · p. 9

5.3 Open-ended Creative Gameplay

  • Players can change a character’s appearance during gameplay by supplying a reference image, making appearance an interactive choice rather than a fixed initial selection.Figure 6 presents reference images, visual proxies, and corresponding video rollouts.Source: §5.3 · p. 9, Figure 6 · p. 11
  • Players can manipulate scene geometry and combine environment changes with character transformation to create scenarios within the same framework.Examples include inverting the environment and introducing scene elements through agent-controlled visual proxies.Source: §5.3 · p. 9, Figure 6 · p. 11

6 Conclusions and Discussions

Wizard provides fine-grained character and scene control while maintaining mechanics-consistent, real-time interaction, and demonstrates zero-shot generalization to unseen characters, scenes, and motions. Its limitations include difficulty finding compatible motions for uncommon body structures and declining scene-proxy consistency over longer rollouts.

  • Wizard enables fine-grained character and scene control through articulated skeletons and geometry maps while maintaining mechanics-consistent, real-time interaction.Source: §6 · p. 12
  • Wizard demonstrates strong zero-shot generalization to unseen characters, scenes, and motions, enabling players to create and manipulate customized game worlds.Source: §6 · p. 12
  • Wizard’s reliance on a motion library makes it difficult to find compatible motions for characters with uncommon body structures.Source: §6 · p. 12
  • Consistency between scene proxies and generated video may decline over longer rollouts.Source: §6 · p. 12
Loading 2610.40339v1…