Source-linked AI summary
Marionette: Predicting World States, Rendering Geometry, Painting Appearance
Zian Meng, Zhen Li, Chuanhao Li, Qiang Li, Kaipeng Zhang
TL;DR
Pixel- and latent-space game world models implicitly maintain structured properties, making long-horizon consistency and controllability fragile. Marionette predicts an explicit articulated world state, renders exact geometry with a fixed bridge, and synthesizes appearance with diffusion; its state responds directly to actions and supports state-level repair without detectable appearance-fidelity loss.
Problem
Pixel- and latent-space autoregressive world models implicitly maintain pose, geometry, occlusion, identity, and control effects, while errors compound over long rollouts.
Method
Marionette combines two-stage autoregressive state dynamics, a zero-parameter deterministic graphics bridge, and a control-conditioned video-diffusion observation model.
Results
31% root-aligned joint error change across 48 held-out segments demonstrates direct action responsiveness, while predicted-state appearance has FVD 831 versus 799 for recorded pose.
Takeaways & Limitations
Explicit state makes long-horizon behavior inspectable and repairable while routing appearance through predicted state without detectable fidelity loss.
Takeaways & Limitations
Appearance consistency remains limited by chunk-relay memory, and some recorded entities lack tracked joints, leaving content without state behind it.
Abstract
from arXiv · showhide
Interactive game world models typically autoregress visual observations directly in pixel or latent space, forcing structured properties such as pose, geometry, and occlusion to be implicitly maintained by the same generative sequence. Over long horizons, errors in these latent world properties accumulate, making consistency and controllability fragile. We explicitly model the evolving world state, delegate exact geometric computation to a fixed, zero-parameter renderer, and leave the neural model to synthesize appearance. We instantiate this idea as Marionette, a world model for interactive games with articulated characters. First, a two-stage autoregressive dynamics model predicts an explicit and interpretable 276-dimensional 3D world state comprising multi-entity articulated skeletons, metric root trajectories, and rotations. Second, a zero-parameter graphics bridge converts the predicted state into pose-control videos, computing world-space geometry and occlusion in closed form. Third, a control-conditioned video-diffusion observation model synthesizes photorealistic RGB observations from the resulting structured controls. Our experiments establish two properties of Marionette. First, the predicted world state is directly controllable. Forcing a mismatched action stream changes root-aligned joint error by 31% across 48 held-out segments. Second, long-horizon behaviour is determined in the state, and can be repaired there. Left free, the two generated characters drift to 21.2 m apart (recorded sessions stay near 5 m) and a third of frames show ground penetration. Two rules imposed on the explicit state, a terrain collider and a separation cap, cut penetration by 66% and keep the pair engaged, with no change to the observation model. Routing appearance through the predicted state costs no fidelity we can detect, at an FVD of 831 against 799 for recorded pose.
1 Introduction
Marionette separates explicit world-state dynamics from appearance generation, assigning exact geometric bookkeeping to a deterministic bridge and appearance synthesis to a neural observation model. This design targets long-horizon consistency and controllability, and experiments show direct action responsiveness without detectable appearance-fidelity loss.
- Motivation: Direct pixel- or latent-space autoregression implicitly maintains pose, geometry, occlusion, identity, and control effects, allowing errors to compound over long rollouts.The model repeatedly consumes its own outputs, so consistency and controllability degrade as the horizon grows.
- Motivation: An explicit world state separates dynamics from appearance into two conditionally independent sub-problems.The formulation treats video as a high-dimensional observation of a lower-dimensional state.
- Approach: Marionette predicts a 276-dimensional articulated state, renders its geometry with a zero-parameter bridge, and synthesizes photorealistic RGB with a control-conditioned video-diffusion model.The state includes metric root trajectories, joint structure, and rotations; the bridge computes geometry and occlusion in closed form.
- Results: 31% root-aligned joint error change over 48 held-out segments shows that mismatched action streams directly affect the predicted articulated dynamics.The experiment evaluates the effect of forcing an incorrect action stream.
- Results: FVD is 831 for appearance routed through predicted state versus 799 for recorded pose, with no detectable fidelity loss reported.The comparison uses the same observation-model framing described in the introduction.
2 Related Work
Related work spans pixel- and latent-space interactive generators, explicit structure or memory for video generation, controllable articulated motion, and several meanings of “world model.” Marionette differs by combining interpretable, render-ready state dynamics with direct action control and deterministic rendering for persistent multi-character games.
- Generative game world models: Interactive game world models commonly autoregress appearance in pixel or latent space while conditioning on user actions.This lineage ranges from Atari and GAN-based simulators to foundation-scale systems for open worlds and streaming.
- Separating structure from appearance: Other approaches ground video generation in explicit 3D, memory, cached content, retrieved context, or reconstructed geometry, while keeping forward dynamics in video latents.Related work also specifies camera trajectories and evaluates off-screen state preservation and action response.
- Controllable articulated motion: Controllable articulated-motion research includes text- and kinematic-conditioned generators, joint-level controls, and physics-simulator-based character driving.These lines address motion generation and scene or terrain conditioning but differ in their interfaces and settings.
- Marionette’s distinction: Marionette extends articulated-motion modeling with two interacting entities, self-predicted actions, single-token control overrides, and a deterministic bridge to rendered video.Its interface between decision and animation is a state sequence, followed by deterministic rendering beyond the skeleton.
- Meaning of “world model”: Unlike control-oriented or occupancy world models, Marionette targets persistent interactive games with a state that is both interpretable and render-ready.The same explicit state supports forward prediction and photorealistic synthesis.
3 Method
Marionette models articulated game dynamics in an explicit state, converts that state into exact pose-control geometry, and uses diffusion only for appearance. Its two-stage dynamics interface exposes discrete action control while the bridge preserves metric geometry and occlusion by construction.
- 3.1 Problem formulation: The 276-dimensional state stores root displacement, root-relative joint positions, continuous 6D root orientation, and weapon state for a monster and hunter.The representation describes the physical scene configuration at each frame.
- 3.1 Problem formulation: Root-relative joints and per-frame root deltas make the representation invariant to global position and heading while remaining render-ready.A fixed operator reconstructs metric world-space joints from the state.
- 3.1 Problem formulation: The method factors prediction, rendering, and appearance into a dynamics model p(s_t+1: | s_≤t, c), deterministic render operator R(s), and observation model p(obs | R(s)).Metric geometry, root integration, joint kinematics, and camera projection are delegated to R.
- 3.2 Two-stage dynamics: decision then animation: ActionGPT predicts one discrete action token per frame and entity, and control overrides that token without retraining or an auxiliary conditioning network.The model also regresses root motion and orientation from recent state summaries.
- 3.3 Deterministic graphics bridge: Figure 2’s top row shows pose-control frames from predicted state and its bottom row shows RGB rendered from those controls; shared geometry comes from the bridge.The rollout spans 12 seconds and includes terrain depth encoding with both entities’ skeletons.
- 3.2 Two-stage dynamics: decision then animation: PoseGPT maps chosen action tokens and recent body state to the next 258-dimensional body pose.This separates low-dimensional discrete decisions from high-dimensional continuous animation.
- 3.2 Two-stage dynamics: decision then animation: The autoregressive loop assembles overridden actions, root motion, and generated body pose into the next state, optionally using terrain conditioning.The resulting state is appended to the context and the process repeats within a bounded attention window.
- 3.3 Deterministic graphics bridge: The zero-parameter bridge converts rotations and root deltas into metric world-space skeletons through closed-form geometric operations.It reconstructs absolute roots by integration and places relative joints using root-frame rotations.
4 Experiments
Marionette evaluates an explicit-state world model through synchronized state and observation layers, testing controllability and long-horizon repair. Its experiments show that actions affect pose, state rules repair distinct failures, and explicit-state rendering remains close to recorded-state appearance quality.
- Setup and evaluation protocol: The benchmark combines state-layer and observation-layer metrics with fixed seed contexts, rollout horizons, and control-injection scripts.The observation layer applies to RGB-emitting models, while the state layer applies only to models exposing explicit state.
- Setup and evaluation protocol: The baseline is an end-to-end pixel-autoregressive model sharing the backbone family, footage, capacity, and RGB training data, differing in implicit versus explicit state representation.Marionette additionally consumes the state view through a deterministic renderer, which is the design difference under test.
- Controllability: 31%: mismatched action streams increase RA-MPJPE from 0.272 to 0.357 m, with Force-GT below Force-Shuf on 33 of 48 segments.The probe uses 48 held-out segments, deterministic decoding, fixed seeds and horizons, and root-aligned joint error.
- Long-horizon behaviour: 21.2 m: free rollouts separate the hunter and monster from 4.9 m initially, while recorded-state separation remains near 5 m.The camera’s per-body pose-control share falls from 1.81% to 0.57% as the generated characters drift apart.
- Long-horizon behaviour: 66%: the terrain collider reduces the collision-frame ratio from 0.337 to 0.114 without changing 21.2 m separation.The separation cap instead holds the pair at 5.1 m, leaves penetration at 0.114, and adds 7% foot-skate.
- Observation quality: 831 versus 799 FVD: generated-state rendering is close to recorded-state rendering, while all four rule-conditioned states remain in a narrow 747–831 FVD band.Per-chunk FVD rises by at most a fifth, and none collapses over the horizon.
5 Limitations
The explicit, render-ready state introduces limitations in appearance persistence, state coverage, and train–inference pose distributions. The paper pairs each limitation with a corresponding remedy or scope boundary.
- Appearance conditioning: Appearance shifts with horizon because geometry is fixed by the pose-control bridge while appearance relies on autoregressive observation-model memory.Persistent entity reference images, re-supplied at every chunk, are proposed to stabilize appearance conditioning.
- Coverage of the recorded state: Unrecorded entities such as AI companions and small monsters can appear without state support because the recordings lack their joints.Recording their already-tracked skeletons would bring them into the state and render.
- Ground-truth and generated pose: The observation model trains on ground-truth-state pose videos but runs on generated-state pose videos, creating a train–inference distribution shift.The paper relies on generalization across this shift and uses a bridge ablation to bound its observable effect.
- Remedies: The explicit render-ready representation assumes that infeasible intermediate states can be projected back into a feasible training distribution.The terrain collider and separation cap address individual failures; fuller rules would cover rig limits, contact, and joint ranges.
6 Conclusion
Marionette argues for recurring explicit world state rather than appearance latents, with exact geometry delegated to a fixed renderer and appearance left to a neural model.
- Marionette fits an explicit world state instead of directly modeling pixel distributions.
- A fixed renderer handles exact geometry, while a neural model synthesizes appearance.
- 31% mismatched-token control change moves generated pose, measured through root-aligned joint error.
- FVD is 831 for predicted pose versus 799 for recorded pose, with no detected appearance-fidelity cost.
- A differentiable body constraint failed by allowing skeleton deformation, whereas input conditioning and post-hoc projection held.
Reproducibility statement
The gameplay corpus and action/state annotations were previously released, while terrain scans and implementation code are new releases associated with this work.
- The recordings, per-frame action labels, and explicit per-entity state come from the previously released WildWorld dataset.
- Scanned terrain height fields consumed by the graphics bridge are new to this work and are released with the code.
- The planned code release covers the two-stage dynamics model, deterministic bridge, and associated components.
A Dataset and Data Integrity
The study pairs synchronized game-engine observations, actions, state, and terrain data with explicit scope choices and metric state representations for evaluating dynamics and rendering.
- Data Integrity: WildWorld records actions, state, and observations from identical game frames, enabling ground-truth correspondence for layered evaluation.
- Dataset: The dynamics corpus contains 1,395 motion segments, approximately 1.6×10^7 frames, and entity-specific action vocabularies of 173 and 689 entries.
- Scope: The dynamics model is trained against a single monster type, while the RGB observation model covers 27 monster identities.
- Scope: Experiments fix one-on-one combat and a weapon type despite a broader corpus spanning 29 species, four hunters, five stages, and larger parties.
- Terrain: Terrain is separately scanned as layered height fields because single-height grids cannot represent overhangs, ledges, bridges, or floors beneath upper surfaces.
- Dataset: The RGB view contains 673.8 hours across 4,008 unique clips, while the dynamics view contains 16.4M frames across 1,395 segments.
- Representation: Root-relative joints and per-frame local root motion provide invariance to global position and heading; absolute positions are recovered through the deterministic bridge.
- Representation: The 276-dimensional state stores metric positions in a Y-up, right-handed world frame for monster and hunter entities, plus weapon state.
C The Deterministic Graphics Bridge
The deterministic graphics bridge converts predicted state into metric, occlusion-aware pose controls, while terrain features and inference-time constraints help keep rollouts physically coherent.
- Bridge: The bridge maps state sequences to metric world-space skeletons and rendered pose-control videos without learnable parameters.
- Bridge: 6D root rotations are converted to matrices before local root displacements are rotated into world space and cumulatively integrated.
- Bridge: Root-relative joints are transformed into metric world-space joints for both articulated entities and weapon points.
- Bridge: The bridge tessellates scanned terrain, applies a deterministic follow-camera or evaluation camera, and rasterizes skeletons with fixed depth ordering.
- Bridge: Every bridge step is closed-form, preserving world-space consistency, metric scale, and occlusion ordering by construction.
- Pose control: The pose-control frame is a three-channel geometry buffer designed to make required quantities recoverable from color.
- Pose control: Terrain and bones occupy disjoint height ranges, while categorical identity and inverse depth preserve branch identity and visibility information.
- Terrain: The terrain representation uses layered 100 m × 100 m chunks with 1 m grids and reference-height queries to select the standing surface.
D Dynamics Model Details
Marionette’s dynamics model uses causal transformer stages with discrete action and continuous root controls, while a deterministic bridge integrates metric roots and renders pose controls.
- Dynamics architecture: ActionGPT predicts root displacement, 6D root rotation, and action-related state from a low-dimensional summary.Both dynamics stages use causal transformers with sinusoidal position encodings and bounded attention windows.
- Long-horizon dynamics: Figure 5 evaluates long-horizon state divergence using root distance to a recorded reference across 20 held-out free rollouts.The state-layer curve is a diagnostic in absolute units rather than a score.
- Control interfaces: Root and token controls address different quantities: tokens steer actions, while root commands pin travel and heading in metric world units.The two channels can be overwritten at individual frames and used together.
- Control interfaces: A root override fixes displacement and rotation, after which the animation stage articulates the body under the imposed root.Root control does not specify limb motion, so it may need to be combined with action control.
E Evaluation Protocol Details
The evaluation separates articulated-pose error, root drift, bone-length stability, and perceptual video quality, using distributional FVD for stochastic long rollouts.
- State-layer metrics: RA-MPJPE subtracts each entity’s root before comparing joints, isolating articulated-pose error from global translation.Root drift instead measures Euclidean distance between predicted and reference root positions over the rollout horizon.
- State-layer metrics: Long-horizon drift curves use 20 held-out free rollouts of 1,000 frames at 20 fps from a 64-frame seed context.These state-only rollouts are longer than the twelve rendered windows.
- State-layer metrics: Bone-length stability is measured as the coefficient of variation of each bone length over a rollout.This metric probes rig consistency in the predicted state.
- Observation-layer metrics: FVD is reported per chunk against chunk index because stochastic worlds may diverge legitimately from a single recorded reference.Beyond the seed chunk, distributional scoring avoids charging plausible divergence as frame-level error.
F State-Layer Controllability: Full Protocol and Results
The state-layer probe tests whether injected action tokens change generated pose under matched contexts, using deterministic decoding across 48 held-out segments.
- Protocol and aggregate results: The probe compares Free, Force-GT, and Force-Shuf streams after seeding the two-stage model with a common context.All other conditions remain identical, and decoding uses temperature 0.
- Protocol and aggregate results: 31% mismatched-token error inflation raises mean RA-MPJPE from 0.272 to 0.357 across 48 held-out segments.Force-GT is only marginally better than Free: 0.272 against 0.281.
- Protocol and aggregate results: Force-GT < Force-Shuf holds on 33/48 segments, indicating that mismatched injected actions produce worse generated pose than ground-truth actions.The reported metric is hunter root-aligned MPJPE at the fixed evaluation horizon.
- Per-class compliance: Wrong-token error inflation is 2.3× for stationary actions, 1.8× for in-place attacks, and 1.3× for locomotion.Locomotion shows the smallest inflation because pose shape varies least across action ids.
- Per-class compliance: Token switching at 5 seconds produces the corresponding divergent pose within a few frames while preserving identical prefixes.The three rollouts share identical seeds and differ only in the injected token after the switch.
G Baseline Scope
The comparison restricts quantitative head-to-head evaluation to a task-matched pixel-autoregressive baseline and uses commercial generators as qualitative references for control fidelity.
- Baseline scope: Quantitative comparison is limited to an end-to-end pixel-autoregressive model trained on the same data and conditioning.Many other interactive video models use protocols that differ in camera, controllable entities, or control inputs.
- Baseline scope: Commercial video generators are evaluated qualitatively because they lack structured, per-frame entity control.They receive the same first frame and an explicit timestamped action schedule.
- Qualitative comparison: Figure 6 compares token schedules with shared seeds and fixed camera, making frames pixel-identical until the 5-second intervention.The held token and two replacement action ids produce distinct poses within a few frames.
- Qualitative comparison: Seedance preserves scene and character fidelity but never attacks, while Grok attacks roughly three seconds late and swaps both characters’ appearance.The decoupled model executes the token switch at the commanded frame.
- Implementation and assets: The observation model uses Wan2.2-Fun-5B-Control, while the training and inference code extends VideoX-Fun.The listed external assets and code are released under Apache License 2.0 where specified.
- Implementation and assets: Monster Hunter Wilds footage is used solely for non-commercial academic research, under the stated rights-holder and release restrictions.The game is commercial and is not distributed under an open licence.