State World Models: Execute the Rules, Generate the Pixels

• Responsive or navigable video does not ensure that action outcomes satisfy world rules or persist across interactions when relevant state is hidden from view.The same action may need to open a chest only when the player has a key, even if the initial views are identical.Source: §1 · pp. 1–2
• STATEFUL HARNESS resolves actions against evolving state and fixed rules, renders the resulting event with a frozen video model, and uses clip observations to update state for subsequent turns.An editable 3D scene supplies spatial and camera guidance, while a vision-language model proposes updates that the runtime checks against the rules.Source: §1 · p. 2
• With MiniMax-H3, PAIRFOLLOW rises from 32% under Rule-only instructions and 64% with explicit ending instructions to 92% with the full harness; PAIRSTRICT reaches 90%.Among eleven unwrapped video backends, the highest Rule-only PAIRFOLLOW is 32%.Source: §1 · p. 2
• WorldState-Bench tests 50 counterfactual pairs that share an initial frame, action, and rule program but differ in hidden values requiring distinct outcomes.PAIRFOLLOW requires both clips to show their required endings; PAIRSTRICT additionally requires forbidden endings to be absent from both clips.Source: §1 · p. 2

• Action-conditioned world models and systems for memory or visual consistency provide a foundation, while STATEFUL HARNESS resolves actions against explicit state and executable rules before rendering [Micheli et al., 2022, Hafner et al., 2025, Ha & Schmidhuber, 2018].The related work also describes latent actions, promptable environments, and open-world interaction [Bruce et al., 2024, Parker-Holder et al., 2024, Parker-Holder & Fruchter, 2025, Decart & Etched, 2024].Source: §2 · p. 2
• Camera-conditioned methods inject trajectories through pose or motion representations, while other approaches use reconstructed geometry, boxes, source videos, or reference clips [Wang et al., 2024, Yang et al., 2024b, He et al., 2025a, Xu et al., 2024, Bahmani et al., 2025].STATEFUL HARNESS carries an editable scene across turns and uses rule execution to determine the spatial edits and semantic events sent to the renderer [Hou & Chen, 2024, Lin et al., 2023, Bai et al., 2025, Jiang et al., 2025, Zhang et al., 2026b].Source: §2 · pp. 2–3
• Prior systems combine explicit state or rules with engines, code-maintained state, state recovery, or selected state variables [Huang et al., 2026, Chen et al., 2026b, Deng et al., 2026, Lin et al., 2026, Meng et al., 2026, Zhu et al., 2026a].STATEFUL HARNESS combines a persistent editable scene, semantic state ledger, and frozen renderer; new rule-governed worlds use a small typed program without an engine or fine-tuning [Li et al., 2025a, Cheng et al., 2025].Source: §2 · p. 3
• Most existing benchmarks measure visual quality, prompt adherence, navigation, actions, editing, or camera and geometry consistency, rather than distinct rule-mandated outcomes under changed hidden state [Ying et al., 2026, Xu et al., 2026b,a, Zhao et al., 2026, Ding et al., 2026, Fang et al., 2026, Yang et al., 2026].Executable-game benchmarks can compare behavior with engine state but require an existing game or simulator; WorldState-Bench instead tests an explicit rule program in generated video [Hafner, 2022, Bard et al., 2020, Light et al., 2023, Ouyang et al., 2026, La et al., 2026, Luo et al., 2026].Source: §2 · p. 3

3 METHOD: THE STATEFUL HARNESS

• The harness separates rule execution, visual realization, and observation by resolving each action against current state and rules before rendering, then incorporating compatible clip-derived updates into the next state.The game definition remains fixed throughout an episode, and the runtime checks proposed semantic and spatial updates against those rules.Source: §3 · p. 3

3.1 STAGE 0: WORLD CONSTRUCTION

• A coding agent creates a typed game definition of state, actions, rules, events, goals, and visual semantics; the definition is validated and frozen before interaction.Each rule has a guard over the current state and action and an effect specifying ledger updates, spatial edits, and an event.Source: §3.1 · p. 4, Figure 3 · p. 5
• An image is lifted into instance boxes that initialize an untextured, editable 3D clay model for object edits and procedural camera previews.The preview represents layout and cameras rather than final appearance.Source: §3.1 · p. 4, §3.2 · p. 5
• Twin instances share a first frame and spatial layout but receive different hidden semantic values, so the same action and rules require different visible outcomes.The semantic ledger records typed fields such as resources, entity attributes, progress, attitudes, and terminal status; spatial state tracks editable layout, scale, and camera path.Source: §3 · p. 3, §3.1 · p. 4

3.2 STAGE 1: STATE EVOLUTION AND VIDEO GENERATION

• At each turn, the runtime settles the rule-governed outcome and spatial edits, compiles the event into an instruction, and renders it with a frozen video model using a preview and visual anchor.A vision-language model extracts visible events and proposes state updates; the runtime admits only well-formed updates compatible with the fixed rules, while contradictory apparent events cannot overturn the settled outcome.Source: §3.2 · pp. 5–6
• The rule-settled event determines the preview’s spatial layout and camera guidance, while VisualScript carries semantic facts that an oriented-box stream cannot express.The instruction identifies the preview as a spatial reference to discourage the renderer from reproducing its clay appearance.Source: §3.2 · p. 5

• WorldState-Bench uses counterfactual pairs with identical visible inputs, different hidden values, and rule-determined outcomes to test whether both required outcomes appear in generated video.The benchmark targets worlds initialized from a description and image without a predefined engine state.Source: §4 · p. 6
• The Change-to-Comply protocol distinguishes whether an action produces a discernible event from whether that event satisfies the rule for the hidden state.Clips without a discernible event or with unjudgeable content are labeled UNCLEAR and receive no credit.Source: §4 · p. 6
• Each pair shares a first frame and action script but differs in one hidden value that requires different visible endings; compliance requires both clips to show their required endings.The stricter criterion also requires the forbidden ending to be absent throughout both clips.Source: §4 · p. 6
• The benchmark comprises 50 cases drawn from 49 hidden variables and 21 scenario themes, with each hidden value unobservable in the shared first frame and selecting visibly different rule outcomes.Authors audit generated first frames for visual cues that might reveal hidden values and specify typed programs determining initial states, required endings, and judging questions.Source: §4 · p. 6
• Judges assess sampled frames with fixed-choice questions about the ending and forbidden outcomes, without receiving the hidden variable or rule text.PAIRFOLLOW requires both clips to satisfy ENDFOLLOW, while PAIRSTRICT requires both to satisfy STRICT; TWINCONTRAST records whether judged endings differ.Source: §4 · p. 7

5.1 EXPERIMENT SETUP

• The benchmark evaluates eleven untrained backends on 50 counterfactual pairs, each sharing an initial frame and action but differing in hidden state and required outcome.Each backend generates one clip per action; pair-level evaluation tests both twins jointly.Source: §5.1 · p. 7

5.2 RESULTS ON WORLDSTATE-BENCH

• Under Rule-only prompting, MiniMax-H3 achieves 66% ENDFOLLOW and 32% PAIRFOLLOW, while Wan2.2 achieves 65% ENDFOLLOW and 32% PAIRFOLLOW.Wan2.2 has the highest unwrapped PAIRSTRICT score at 26%; MiniMax-H3 reaches 4% PAIRSTRICT. The harness resolves each action before generation and does not select among repeated samples using compliance judgments.Source: §5.2 · p. 7
• With MiniMax-H3 weights held fixed, the harness reaches 92% PAIRFOLLOW, 90% PAIRSTRICT, and 96% TWINCONTRAST; human raters prefer it in 98% of comparisons against four unwrapped baselines.The harness reaches distinct required outcomes in illustrated counterfactual pairs where baselines repeat an ending across twins or include forbidden content.Source: §5.2 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8
• A component-chain decomposition attributes gains to outcome specification (+32 points) and executed state with compiled references (+28 points), with both increments significant under paired exact McNemar tests.Source: §5.2 · p. 8

5.3 QUALITATIVE RESULTS

• Three real harness episodes span 15, 29, and 11 turns across town exploration, dating narrative, and turn-based duel, using the same frozen weights and runtime.Each genre is instantiated training-free as a new executable game definition.Source: §5.3 · p. 8, Figure 6 · p. 9

5.4 ADDITIONAL EXPERIMENTS

• 30–86 points: With the same frozen weights, the full harness improves every WorldState-Bench metric over its track’s raw baseline: ENDFOLLOW 66→96, PAIRFOLLOW 32→92, TWINCONTRAST 32→96, and PAIRSTRICT 4→90.On WBench, setting adherence rises from 71.1 under the official text protocol to 87.2 with the harness; video quality is 78.0 versus the text-driven class average of 77.4, and interaction is 77.7, the highest reported in the comparison.Source: §5.4 · pp. 8–9, Table 2

6 CONCLUSION

• With MiniMax-H3 weights held fixed, PAIRFOLLOW rises from 32% under Rule-only instructions and 64% with explicit ending instructions to 92% with the full harness.The harness resolves actions under explicit state and executable rules, renders events with a frozen video model, and uses generated clips to inform subsequent state; it also sustains episodes of 11 to 29 turns and extends to spatial triggers.Source: §6 · p. 9

A WORLDSTATE-BENCH: CASE CONSTRUCTION DETAILS

• Each WorldState-Bench case comprises a theme and scene description, first frame, hidden-variable declaration, action script, and rules specifying distinct endings; A/B twins share theme, frame, action, and rules but differ in hidden value.The suite spans 21 themes and 49 distinct hidden variables.Source: A WORLDSTATE-BENCH: CASE CONSTRUCTION… · p. 16
• Hidden values select mechanisms in 42 cases, character responses in 6, and forms or scene sides in 2; all are invisible in the frame by construction and audit.Source: A WORLDSTATE-BENCH: CASE CONSTRUCTION… · p. 16
• Cases use cast, gesture, mechanism, and zone triggers; zone cases additionally require the render to make the region’s state visible.The rule text specifies both branch endings so expected outcomes can be derived directly.Source: A WORLDSTATE-BENCH: CASE CONSTRUCTION… · p. 16

B JUDGING PROTOCOL AND METRIC DEFINITIONS

• The judge answers fixed-choice questions identifying the ending and whether forbidden content appears, with answers outside the choices or missing answers scored incorrect.The case program fixes expected outcomes before generation; the judge does not receive the hidden value, rule program, or the other twin’s expected outcome.Source: B JUDGING PROTOCOL AND METRIC… · p. 18
• Appendix C reports agreement between human ratings and VLM judgments, with blind human audits independently assessing clip correctness and pairwise preference, validating a fixed, versioned judge protocol on the audited subset.Source: B JUDGING PROTOCOL AND METRIC… · p. 18

B.1 METRIC DEFINITIONS

• STRICT requires the prescribed ending in the final frames and the forbidden outcome to remain absent throughout the clip.UNCLEAR outputs are counted separately and never credited.Source: §B.1 · p. 18
• PAIRFOLLOW requires both twins to show their required endings, while PAIRSTRICT requires both twins to satisfy STRICT.TWINCONTRAST records whether the judged endings differ between instances A and B.Source: §B.1 · p. 18
• ∆STRICT is STRICT under Rule+end minus STRICT under Rule-only, measuring the effect of specifying the ending in addition to the rule instruction.Source: §B.1 · p. 18

C HUMAN AUDIT PLATFORM AND PROTOCOLS

• For clip verification, 102 clips receive three independent blinded ratings, using the benchmark’s fixed-choice questions about required endings and forbidden outcomes.The audit covers 17 cases under the full harness, frozen MiniMax-H3 baseline, and Wan2.2.Source: C HUMAN AUDIT PLATFORM AND PROTOCOLS · p. 19
• Battle comparisons use 25 held-out pairs, with the harness compared against four Rule-only baselines in forced-choice presentation.Each comparison receives two independent ratings, with system positions randomized and no tie option.Source: C HUMAN AUDIT PLATFORM AND PROTOCOLS · p. 19
• The VLM judge agrees with human majority on 91.2% of verification clips for Q1 and 86.3% for Q2.The corresponding agreement coefficient is κ = 0.72/0.72.Source: C HUMAN AUDIT PLATFORM AND PROTOCOLS · p. 19
• Under forced choice, human raters preferred the full harness in 98% of pairwise comparisons across baselines (96%–100% per baseline); no comparison favored a baseline under the aggregate decision rule.At the individual-vote level, 99.0% of votes favored the harness.Source: C HUMAN AUDIT PLATFORM AND PROTOCOLS · p. 19, Table 3 · p. 19

D ADDITIONAL RESULTS

• 32% of pairs are jointly correct under Rule-only prompting, rising to 64% with required endings and 92% with the full harness using the same frozen weights.The respective gains of 32 and 28 points are significant under paired exact McNemar tests.Source: D ADDITIONAL RESULTS · p. 19, Table 4 · p. 21
• On the external WBench full split, the raw-text configuration scores 74.2 overall versus 77.9 for the full harness.The reported comparison is part of the channel ablation with frozen MiniMax-H3 weights.Source: D ADDITIONAL RESULTS · p. 20
• Across ten backends other than MiniMax-H3, Rule+end changes STRICT by a mean of +2.3 points: six improve and four decline.The reported diagnostics indicate that explicit endings do not reliably produce distinct outcomes across hidden states or exclude contradictory content.Source: D ADDITIONAL RESULTS · pp. 21–22, Figure 11 · p. 22
• With the deployed first-frame reference, the harness reduces copy-paste from 30% to 2% and raises TWINCONTRAST from 46% to 96%.The text-only configuration has no copy-paste errors but higher direction error (6% versus 4%) and leakage (5% versus 1%).Source: D ADDITIONAL RESULTS · p. 22, Table 6 · p. 22
• The unwrapped MiniMax-H3 baseline’s PAIRSTRICT remains 4% under Rule+end, as direction errors fall from 34% to 18% while leakage rises from 40% to 53%.Direction, leakage, and double-error diagnostics are not mutually exclusive and should not be summed.Source: D ADDITIONAL RESULTS · p. 23, Figure 12 · p. 23
• Across trigger types, the harness reports STRICT / PAIRSTRICT scores of 100.0% / 100.0% for cast, 95.2% / 90.5% for gesture, and 92.3% / 84.6% for mechanism.Zone-triggered clips reach 91.7% STRICT and 83.3% PAIRSTRICT; this family has the lowest text-only STRICT score and the largest improvement from the first-frame reference.Source: D ADDITIONAL RESULTS · p. 23

D.1.1 CHANNEL ABLATION DETAILS

• The ablation holds MiniMax-H3 weights fixed across the official text protocol, first-frame anchoring, spatial-preview harness, and raw text with the same preview.The interaction column on this split is the navigation-trajectory metric.Source: §D.1.1 · p. 23, Table 9 · p. 25
• First-frame anchoring raises setting adherence from 72.7 to 91.5 and overall navigation to 76.8; adding the control video raises overall navigation to 79.0.With the control video, camera trajectory rises from 69.0 to 77.7 and consistency improves from 75.0 to 79.2.Source: §D.1.1 · p. 24, Table 8 · p. 24
• Raw text with the control video achieves the highest camera-trajectory score, 79.3, but an overall navigation score of 65.7.Its setting adherence, consistency, and physical plausibility are all below the raw-text configuration without the preview.Source: §D.1.1 · p. 24, Table 8 · p. 24
• This ablation isolates reference-channel effects within a bundled harness; a complementary decomposition varies instruction and state components.The complementary decomposition is reported in Table 1 and Appendix Table 4.Source: §D.1.1 · p. 24

E ADDITIONAL QUALITATIVE EXAMPLES

• The harness depicts only required endings in all sixteen clips shown under Rule-only and Rule+end probes, while baselines sometimes ignore hidden state or show forbidden content.Examples include baselines producing the same ending for both hidden states and MiniMax-H3 activating a pillar in both cases; sign text is also malformed in baseline clips.Source: E ADDITIONAL QUALITATIVE EXAMPLES · p. 25, Figure 13 · p. 26, Figure 14 · p. 27
• Across three episodes, the HUD tracks stamina recovery and forward-only quest progress, affection accumulation before confession acceptance, and boss-duel damage culminating in victory.The episodes contain fifteen acts, twenty-nine turns, and eleven turns, respectively.Source: E ADDITIONAL QUALITATIVE EXAMPLES · p. 25, Figure 15 · p. 28, Figure 16 · p. 29

F.1 FORMAL FRAMEWORK AND CROSS-CHANNEL GROUNDING

• Rule settlement determines the provisional state and event before rendering; generated-video observations can update later state only after validation against fixed rules.If an observation contradicts the settled event, it is treated as a rendering inconsistency; when no update is accepted, the provisional state remains.Source: §F.1 · p. 31
• Text instructions specify actors, actions, and outcomes, while the clay preview supplies approximate layout and camera guidance; neither channel fully specifies the turn alone.The clay model supplies both the camera description and rendered trajectory, aligning their intended paths at the input level.Source: §F.1 · p. 31

F.2 TURN LIFECYCLE AND DOMAIN ADAPTATION

• Each turn runs through action selection, rule evaluation, compilation, rendering, final-frame anchoring, validated observation feedback, and recording; clay previews use reconstructed boxes and procedural camera programs.The domain definition supplies scenario-specific rules and content while the shared runtime interfaces remain in place.Source: §F.2 · pp. 32–33, Table 10 · p. 33

F.3 DSL AND IR SPECIFICATIONS

• Case programs derive expected outcomes from executable semantics, while typed VisualScript records describe settled turns and undergo validation before deterministic compilation into backend-specific instructions.Case checks enforce mutually exclusive twin branches and hidden-state isolation; separating the game definition from instruction format supports different backend input grammars.Source: §F.3 · pp. 33–34

G POSITIONING AGAINST CONCURRENT EXPLICIT-STATE SYSTEMS

• Table 12 organizes concurrent explicit-state systems by state representation, control interface, spatial-state handling, and evaluation protocol.The caption defines “Native ref.” as a frozen model’s standard reference inputs.Source: G POSITIONING AGAINST CONCURRENT… · p. 35, Table 12 · p. 35

H REPRODUCIBILITY DETAILS

• Independent reproduction requires the 50 case programs, first frames, audit records, rules, compiler and adapters, generation settings, sampled clips, judge prompts, and per-clip verdicts.The case programs determine expected outcomes, and deterministic compilation makes reported numbers traceable through per-clip verdicts.Source: H REPRODUCIBILITY DETAILS · p. 35
• Full-episode replay requires the same accepted observations and state updates, while pixel-level replay additionally depends on deterministic backend behavior and serving conditions.The proposal kernel, video backend, and observer may vary across runs; frozen weights alone do not ensure pixel-level replay.Source: H REPRODUCIBILITY DETAILS · p. 35
• The reported comparison modifies no backend weights; MiniMax-H3 receives text instructions, an image anchor, and, in the full harness, a preview clip.Evaluations used a single 8-GPU node, with harness logic running on CPUs.Source: H REPRODUCIBILITY DETAILS · p. 35

I LIMITATIONS AND FUTURE WORK

• The box-based clay model provides coarse layout; instance-recovery errors can propagate, and the renderer may reproduce clay appearance despite the instruction.The video model determines articulated pose and fine appearance, and the spatial pipeline operates offline.Source: I LIMITATIONS AND FUTURE WORK · p. 35
• Future work targets richer temporal and quantified rules, more tracked entities and relations, additional backends, and broader settings including longer episodes and multi-agent interaction.The stated scope also includes open-ended environments where goals and rules evolve over time.Source: I LIMITATIONS AND FUTURE WORK · p. 35