PaperlayerView exact revision on arXiv

Source-linked AI summary

StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation

Zijun Lin, Zeqing Wang, Cheston Tan, Bihan Wen, Yeying Jin

arXiv:2607.26754v1cs.CV

TL;DR

Existing game world models can produce visually plausible rollouts while violating state-dependent mechanics. StatePlay jointly predicts internal game states and visual content, improving mechanics fidelity by 18.6% over stateless models.

  • Problem

    Existing game world models overlook internal state dynamics, limiting their ability to generate rollouts that follow state-dependent game rules.

  • Method

    StatePlay jointly predicts game states and visual content using a state–frame–action dataset and a mixture-of-transformers architecture with specialized branches.

  • Results

    18.6% improvement in mechanics fidelity over stateless game world models, with average normalized L1 state-prediction distance below 0.06.

  • Takeaways & Limitations

    Explicit state modeling supports visually plausible game rollouts that better preserve state-dependent events and underlying mechanics.

  • Takeaways & Limitations

    Initial evaluation focuses on Street Fighter 3 because publicly available synchronized state–frame–action datasets remain limited.

Abstract

from arXiv · show

Recent game world models can generate visually realistic and interactive environments conditioned on player actions. However, games are not defined by pixels alone; they are governed by explicit mechanics, namely state-dependent rules that control health reduction, skill activation, and game termination. These mechanics depend on precise internal states, such as health points, skill meters, and timers, which are tightly coupled with visual observations and determine how gameplay evolves. Without modeling these state dynamics, existing game world models may generate visually plausible rollouts but violate the underlying game rules. In this paper, we propose StatePlay, a novel state-aware game world model that jointly predicts visual content and game states to promote mechanics-consistent generation. StatePlay adopts a mixture-of-transformers (MoT)-style architecture that preserves specialized visual and state representations while enabling cross-modal interaction, allowing predicted states to guide frame generation. Each branch is further optimized with a distinct objective suited to its modality. Experiments show that StatePlay achieves an average normalized L1 distance below 0.06 for state prediction. Furthermore, compared with models without explicit state modeling, our method improves mechanics fidelity in generated game rollouts by 18.6%. Overall, our work highlights the importance of state-aware game world modeling and advances beyond pixel-level realism toward complete and mechanically faithful game generation.

1 Introduction

StatePlay addresses the gap between visually plausible game-world rollouts and mechanics-consistent gameplay by jointly modeling internal states and visual content. It introduces state-aware data and architecture, achieving accurate state prediction and improved mechanics fidelity.

  • Motivation: Games rely on health points, skill meters, and timers to regulate valid actions, visual transitions, and progression.For example, special attacks require a full skill meter, and gameplay should end when either player’s health reaches zero.
  • Problem: Existing game world models focus on pixel-space prediction, producing visually plausible rollouts that can violate game rules.Observed failures include continuing after game termination or activating super arts without full skill meters.
  • Approach: StatePlay jointly predicts game states and visual content to generate mechanics-consistent gameplay.Its state-aware formulation explicitly addresses the causal role of internal state dynamics in shaping future observations.
  • Data: StatePlay constructs a synchronized SF3 dataset containing frames, player actions, and internal states.The recorded states include health points, skill meters, and timers, enabling investigation of explicit state modeling.
  • Results: 18.6% improvement in mechanics fidelity over the best-performing baseline accompanies an average normalized L1 distance below 0.06 across key state variables.StatePlay also better preserves skill activation, health reduction, and game termination while maintaining visual quality and action controllability.

2 Related Work

Recent game world models increasingly support interactive, real-time, long-horizon, and high-resolution generation, but largely overlook explicit game-state prediction. StatePlay addresses this gap by jointly predicting states and video frames that follow game rules.

  • Game World Models: Game world models generate interactive content conditioned on player actions, beginning with Genie’s action-controllable environments learned from unlabeled videos.Genie demonstrates that video generative models can produce playable environments from action-controllable dynamics.
  • Game World Models: ReactiveGWM, Incantation, and MultiWorld extend interactivity from free exploration to interactions with environments, NPCs, and other players.These works broaden the forms of interaction supported by game world models.
  • Game World Models: Lyra 2.0, HY-World 1.5, and Matrix-Game 3.0 target real-time interaction, long-horizon consistency, and high-resolution generation.These efforts focus on improving generation quality and temporal or interactive behavior.
  • Game State Modeling: Despite this progress, state prediction remains largely overlooked, although games depend on task-specific variables such as ammunition, nitro, health points, and skill meters.The passage identifies state modeling as a missing component because game states are not always directly extractable or reliably inferable from generated content.
  • Game State Modeling: StatePlay fills this gap by simultaneously predicting game states and generating video frames that follow underlying game rules.The approach connects explicit state prediction with mechanics-consistent visual generation.

3 StatePlay

StatePlay constructs a balanced, state-aware SF3 dataset and combines specialized visual and state branches with bidirectional cross-modal attention. It jointly trains visual generation and state prediction to support mechanics-consistent game frames.

  • Dataset Construction: StatePlay uses Street Fighter 3 because its explicit health, skill-meter, and termination rules connect internal states directly to gameplay progression.The dataset is collected in three stages: gameplay recording, mechanics distribution balancing, and NPC strategy annotation.
  • Dataset Construction: Gameplay episodes are programmatically collected through a Gymnasium-compatible SF3 interface, with agents sampling actions from an 11-dimensional space spanning movement, normal attacks, and super art.The action space contains 4 movement actions, 6 normal attacks, and 1 super art.
  • Dataset Construction: The dataset balances five mechanics-related categories—result win, result lose, macro success, macro fail, and normal—using Gemini detections together with recorded game states.A macro success requires a detected super art, the corresponding command, and a skill meter above threshold; macro failure involves insufficient skill meter.
  • Dataset Construction: The balanced training set contains 10,000 clips, with 40% state-critical cases and 60% normal cases, alongside video, actions, states, and NPC descriptions.Result win, result lose, macro success, and macro fail each contribute 10% of the dataset.
  • Model Architecture: StatePlay adds a 0.76B state branch to a 5B visual branch, preserving modality-specific representations while joint attention exchanges mechanics-related state and visual dynamics.Visual tokens attend to state tokens, while state tokens attend to visual tokens, enabling bidirectional information exchange.
  • Training Objectives: The visual branch uses flow matching, whereas the state branch uses Smooth L1 regression, and both objectives are combined through a weighted joint optimization.Regression is selected for the low-dimensional, mechanics-constrained nature of game states.

4 Experiments

Experiments evaluate visual quality, action control, state alignment, and mechanics fidelity on generated gameplay. StatePlay outperforms existing models in mechanics-consistent generation by explicitly coupling internal-state prediction with frame generation.

  • Evaluation: The benchmark evaluates generated clips on visual quality, action control, state alignment, and mechanics fidelity using an additional test set of 100 samples.State alignment covers timer, player HP, opponent HP, and both skill meters; mechanics fidelity is assessed by visual judges.
  • Zero-shot comparison: Zero-shot baselines generate visually plausible gameplay but preserve game rules poorly, with ReactiveGWM remaining below 50% mechanics fidelity under both Gemini and GPT evaluation.Baselines use their original released architectures and pretrained weights, and only visual quality and mechanics fidelity are reported.
  • State-aware comparison: 0.947 state alignment, 82.3% Gemini mechanics fidelity, and 78.3% GPT mechanics fidelity are achieved by StatePlay after state-aware training.StatePlay consistently outperforms all baselines across three runs and both evaluators.
  • Architecture analysis: StatePlay’s MoT-style backbone preserves specialized visual and state representations while maintaining interaction between frame generation and state prediction.The comparison motivates modality-specific structure because visual dynamics and rule-sensitive state dynamics differ in representation and temporal behavior.
  • Qualitative mechanics: StatePlay correctly executes threshold-triggered skills, resets the skill meter, and produces the appropriate win or loss outcome when either character reaches zero health.ReactiveGWM fails these mechanics, including not triggering the super art, leaving a defeated NPC standing, and depicting the wrong winner.

5 Conclusions and Discussions

StatePlay integrates explicit internal-state prediction with game-frame generation, modeling timers, health points, and skill meters alongside visual dynamics. Its initial evaluation is limited by the scarcity of synchronized state, frame, and action datasets, motivating broader state-aware data collection.

  • Contributions: StatePlay incorporates explicit prediction of timers, health points, and skill meters into game frame generation.This couples state prediction with frame generation rather than modeling only visual dynamics conditioned on player actions.
  • Contributions: The resulting rollouts are described as visually plausible and more consistent with the game’s internal states.
  • Limitations: Initial evaluation focuses on Street Fighter 3 because publicly available datasets synchronizing states, frames, and actions remain limited.
  • Limitations: Broader validation across game genres requires larger and more diverse state-aware datasets.The paper encourages datasets with explicit state annotations across a wider range of games to support model scaling and generalization.

A Experimental Model Information

StatePlay uses a lightweight state branch within an MoT-style architecture and consistently outperforms game world model baselines in the experiments.

  • Model configuration: 0.76B parameters comprise StatePlay’s lightweight state branch added to the model configuration.The state branch is explicitly introduced to support state-aware game modeling.
  • Model configuration: StatePlay consistently outperforms the evaluated game world model baselines.The passage reports this comparison without specifying individual performance values.
  • Model configuration: Most evaluated models use base diffusion models, text encoders, and VAEs from the same model family, highlighting the effectiveness of StatePlay’s MoT-style architecture.The shared component family provides a common basis for the architectural comparison.

B State-aware Dataset Construction · B.1 Mechanics Distribution Balancing Criteria

StatePlay constructs a balanced training dataset by requiring clips to satisfy both visual and state conditions for category assignment. The dataset contains 10,000 samples spanning state-critical and normal gameplay scenarios.

  • B State-aware Dataset Construction: 10,000 training samples comprise the StatePlay dataset.The collection is constructed to ensure balanced coverage of diverse mechanics scenarios.
  • B.1 Mechanics Distribution Balancing Criteria: 1,000 clips are collected for each of four state-critical categories.These categories target mechanics scenarios where internal state is critical.
  • B.1 Mechanics Distribution Balancing Criteria: 6,000 clips represent normal gameplay in the training set.Normal gameplay complements the four state-critical categories.
  • B.1 Mechanics Distribution Balancing Criteria: A clip enters a category only when it satisfies both corresponding visual and state conditions.This criterion enforces consistency between observed gameplay and underlying mechanics.
  • B State-aware Dataset Construction: The distribution is designed to balance coverage across diverse mechanics scenarios.The construction uses four equally sized state-critical categories alongside normal gameplay.
  • B.1 Mechanics Distribution Balancing Criteria: The classification criteria are organized by dataset category in Table 4.Categories are assigned using paired visual and state requirements.

B.2 NPC Strategy Annotation · NPC Strategy Prompt · C Failure Cases

StatePlay annotates NPC behavior into offense, control, and defense using visible gameplay evidence, then stores these labels with clips, actions, and game states. Despite accurate internal-state prediction, visual indicators can remain inconsistent, especially when multiple mechanics overlap.

  • B.2 NPC Strategy Annotation: NPC behavior is classified into three strategies: offense, control, and defense, using Gemini-3.1-Pro and following ReactiveGWM.The resulting labels support strategically engaging NPCs in generated rollouts.
  • B.2 NPC Strategy Annotation: Each strategy label is added to the training dataset alongside its gameplay video clip, player actions, and game states.
  • NPC Strategy Prompt: The annotation prompt analyzes only an NPC’s visible behavior in a 5-second fighting-game clip and forbids inferring hidden intentions.It first records observable actions, movement toward the player, damage, and sustained effects.
  • NPC Strategy Prompt: The prompt requires factual observations of attacks, advancing toward the player, taking damage, and applying sustained effects before assigning a strategy.
  • C Failure Cases: StatePlay accurately predicts the internal states required for valid mechanics, but health bars and skill meters can still show occasional inconsistencies.
  • C Failure Cases: Visual quality can degrade when multiple mechanics occur simultaneously, such as a super art coinciding with the end of a match.The passage suggests stronger visual generation capabilities may help address these limitations.

D Details of Mechanics Fidelity Evaluation · Evaluation Prompt (Mechanics Fidelity) · E State Input Comparisons

The mechanics-fidelity evaluation classifies generated rollouts from visible outcomes using explicit rules, priorities, and reference images. For state-input training, noise-based inputs outperform broadcasting in state alignment and mechanics fidelity, likely because broadcasting hinders temporal-dynamics learning.

  • D Details of Mechanics Fidelity Evaluation: Mechanics fidelity is evaluated by Gemini-3.1-Pro and GPT-5.5 using explicit visual rules, fixed decision priority, and reference images.Win and loss require visible result screens, while macro success requires a clearly executed super art that hits the opponent.
  • Evaluation Prompt (Mechanics Fidelity): Generated Street Fighter 3 clips are assigned one of five visible-outcome categories: normal, macro_success, macro_fail, result_win, or result_lose.The evaluator must classify what is visible rather than relying on filenames or ground-truth labels.
  • Evaluation Prompt (Mechanics Fidelity): Decision priority assigns result_win when “YOU WIN” is visible, then result_lose when “YOU LOSE” is visible, followed by successful or failed macro execution, and otherwise normal.The prompt also supplies reference images for result_win, result_lose, and macro/super-art cases.
  • Evaluation Prompt (Mechanics Fidelity): The evaluator returns a JSON object containing the predicted state, ground-truth match, confidence, visible-evidence flags, and a short reason.Evidence fields include visibility of win, loss, and super art, plus whether the super art hits.
  • E State Input Comparisons: StatePlay compares broadcasting the initial clean state across timesteps against adding noise to the state sequence during regression-based state prediction training.These are the two state-input construction strategies evaluated in Tab. 5.
  • E State Input Comparisons: Noise-based input consistently outperforms the broadcasting baseline in both state alignment and mechanics fidelity.The paper hypothesizes that broadcasting creates an overly static initialization that hinders learning temporally evolving frame dynamics; training therefore perturbs video latents and raw states.

F More Qualitative Comparisons

Figure 7 qualitatively compares ReactiveGWM and StatePlay after identical 40,000-step fine-tuning on a state-aware dataset. ReactiveGWM’s visual-only modeling produces rollouts that violate underlying game mechanics.

  • ReactiveGWM and StatePlay are fine-tuned on the same state-aware dataset for 40,000 steps.
  • ReactiveGWM fails to execute a super art even when the skill meter is full.
  • ReactiveGWM omits the game result after NPC health reaches zero and produces blurred, distorted frames after displaying “You Lose.”
Loading 2607.26754v1…