PaperlayerView exact revision on arXiv

Source-linked AI summary

GameWAM: A World Action Model for Video Games

Yuncheng Guo, Zhanqiu Zhang, Yiwen Guo, Weijia Li

arXiv:2608.26200v1cs.AIcs.CVcs.LG

TL;DR

GameWAM addresses the limited integration of visual world dynamics and task-directed native control in rapidly changing video-game environments. It jointly generates visual futures and keyboard–mouse actions with mode-aware routing, block–cycle replanning, and hierarchical history. Across Minecraft and ViZDoom it achieves competitive closed-loop performance, using fewer executed native actions than compared agents in Minecraft, while exposing LASI as a source-sensitivity failure mode.

  • Problem

    Existing agents map visual and task context to actions while interactive world models predict visual futures from actions without serving as task policies; WAM behavior in open-ended video-game interaction remains less understood.

  • Method

    GameWAM jointly models future visual observations and executable keyboard–mouse trajectories using parallel Video and Action DiTs, block-causal conditioning, mode-specific action generation, and block–cycle control.

  • Results

    Across Minecraft and ViZDoom, GameWAM achieves competitive closed-loop performance, using fewer executed native actions than compared agents in Minecraft.

  • Takeaways & Limitations

    Joint world–action modeling is a promising direction for interactive game control while exposing source sensitivity as a failure mode of generative action policies.

  • Takeaways & Limitations

    The study is restricted to digital game environments and does not involve deployment in physical systems.

Abstract

from arXiv · show

Modern video games combine first-person perception, rapid visual changes, persistent world state, and heterogeneous native controls. Existing game agents map visual and task context directly to actions but lack explicit world dynamics modeling, whereas interactive game world models predict visual futures from supplied actions but do not serve as task policies. World-Action Models (WAMs) unify these objectives, but remain largely unexplored under the dynamics and open-ended interaction of video games. We introduce GameWAM, to our knowledge the first WAM for native closed-loop gameplay and GUI control. GameWAM jointly generates future visual observations and executable keyboard-mouse trajectories through parallel visual and action generative processes with block-causal conditioning and flow matching. To support joint world-action learning, we construct synchronized gameplay and GUI trajectories. To handle heterogeneous native control, GameWAM predicts a gameplay/GUI mode at each action step and generates actions with mode-specific prediction distributions and continuous-action normalization. For long-horizon interaction, block-cycle control predicts beyond the committed horizon, executes only a short action prefix, and replans from new observations, while fine-grained within-cycle context and hierarchical cross-cycle history preserve temporal continuity. Experiments demonstrate competitive task success with fewer executed native actions than the compared agents. We further uncover Low-Frequency Action Source Imprinting (LASI), in which low-frequency components of the sampled action source systematically steer coarse generated camera motion under fixed conditioning, revealing a source-sensitivity failure mode in generative control. Project page is available at https://yunncheng.github.io/GameWAM/.

1 INTRODUCTION

GameWAM addresses the gap between agents that map context to actions and world models that predict visual futures without serving as task policies. It jointly models visual observations and executable native actions for closed-loop gameplay and GUI control, with mechanisms for heterogeneous controls, long-horizon replanning, and source-sensitivity analysis.

  • Existing game agents map visual and task context to actions but do not explicitly model visual evolution under executed actions.
  • GameWAM is introduced as the first WAM for native closed-loop gameplay and GUI control, jointly generating future observations and keyboard–mouse trajectories.
  • Mode-specific action distributions and normalization address incompatible gameplay and GUI semantics within a shared native keyboard–mouse space.
  • Block–cycle control predicts P actions but commits only E actions, with E < P, enabling longer look-ahead while preserving frequent replanning.
  • GameWAM achieves competitive closed-loop performance with fewer executed native actions than compared agents, while LASI reveals persistent directional camera bias from reused low-frequency action sources.
  • Synchronized gameplay and GUI trajectories support joint world–action learning across the two interaction regimes.

2 RELATED WORK

Related work spans scalable game environments, multimodal and hierarchical game agents, action-conditioned game world models, and emerging world–action models. GameWAM extends this paradigm to native closed-loop gameplay and GUI control.

  • Video games provide scalable, controllable settings for native control and long-horizon interaction with diverse, repeatable environments.
  • Game agents increasingly emphasize multimodal reasoning, memory, hierarchical control, generalist pretraining, and executable native keyboard–mouse actions.
  • Game world models learn controllable visual dynamics through latent-action generation, diffusion modeling, neural simulation, and action-conditioned generation.
  • World–Action Models connect predictive world modeling with generative control through structured action prediction and visual prediction.
  • Recent WAMs jointly predict video and actions or adapt execution using realized observations, while GameWAM studies native gameplay and GUI control.

3 DATA CONSTRUCTION AND TRAINING SAMPLING

The data pipeline constructs synchronized, WAM-ready gameplay and GUI trajectories and samples event-centered clips to concentrate training on instruction-relevant interaction.

  • Minecraft training uses three complementary WAM-ready streams combining broad naturalistic trajectories with event-anchored, instruction-aligned interactions.
  • Event-anchored clip sampling separates dataset construction from temporal sampling density and emphasizes supervision near annotated interaction events.

4 METHOD

GameWAM is a block-causal world–action model that jointly generates visual futures and native actions using parallel diffusion branches, mode-specific routing, block–cycle control, and hierarchical history.

  • 4 METHOD: GameWAM models block-causal joint distributions over video latents and native action vectors for gameplay and GUI control.
  • 4.1 BLOCK-CAUSAL WORLD–ACTION MODELING: Joint flow matching generates continuous and discrete coordinates within one native action vector, converting discrete outputs into executable binary decisions after generation.
  • 4.1 BLOCK-CAUSAL WORLD–ACTION MODELING: Parallel Video and Action DiTs use shared causal visual prefixes while noisy video and action variables remain conditionally decoupled.
  • 4.1 BLOCK-CAUSAL WORLD–ACTION MODELING: A timestep-wise router selects gameplay- or GUI-specific action branches, with mode-specific continuous normalization and validity masks for unavailable controls.
  • 4.2 BLOCK-CYCLE CONTROL AND HIERARCHICAL HISTORY: Plans contain P actions but execute only an E-action prefix, then replan from realized observations while discarding unexecuted look-ahead.
  • 4.2 BLOCK-CYCLE CONTROL AND HIERARCHICAL HISTORY: Transient cycle-local K/V context is combined with persistent recent and long-term visual history across cycle boundaries.
  • 4.3 TRAINING OBJECTIVES: Training teacher-forces completed blocks, separately reduces continuous and discrete action losses, supervises routing, and regularizes persistent history to retain predictive visual information.

5 EXPERIMENTS

GameWAM is evaluated on Minecraft MCU and four-map ViZDoom, showing strong closed-loop performance and improved interaction efficiency. Ablations support future-video supervision, event-anchored sampling, mode-specific controls, and predicting beyond the committed horizon, while LASI exposes source-sensitive camera bias.

  • MCU results: GameWAM achieves the highest average success rates on both MCU task sets while requiring fewer steps per successful episode across all task categories.The interaction-efficiency gains are particularly pronounced on embodied and GUI tasks.
  • ViZDoom results: GameWAM consistently improves over Game-TARS and achieves competitive or leading average rewards across all four ViZDoom scenarios.Each map’s reported average reward is calculated over 50 episodes.
  • Ablations: Removing future-video supervision or using coarser temporal sampling produces the largest performance degradation in the MCU ablations.Event-anchored clip sampling also contributes substantially to performance.
  • Ablations: A unified gameplay/GUI action distribution underperforms mode-specific prediction and normalization for heterogeneous native controls.The ablation supports separating action distributions across gameplay and GUI regimes.
  • Ablations: Matching prediction and execution horizons, P = E, further degrades average success rate, supporting prediction beyond the committed horizon.GameWAM predicts a longer action horizon than it commits for execution.
  • LASI: Under fixed conditioning, low-frequency source components steer coarse generated camera motion, with source replacement and zeroing providing controlled evidence.Yaw DCT0 reaches r = 0.890; source replacement transfers the donor output in 94.8% of trials, while zeroing removes 99.25% of associated output variance.

6 CONCLUSION

GameWAM jointly models future visual observations and executable keyboard–mouse actions for native closed-loop gameplay and GUI control. Across Minecraft and ViZDoom, it achieves competitive performance while using fewer executed native actions in Minecraft, and it reveals LASI as a source-sensitivity failure mode.

  • Conclusion: GameWAM jointly models future visual observations and executable keyboard–mouse actions for native closed-loop gameplay and GUI control.The model is presented as a World–Action Model for these settings.
  • Conclusion: Across Minecraft and ViZDoom, GameWAM achieves competitive closed-loop performance while using fewer executed native actions than compared agents in Minecraft.The conclusion reports this pattern across the two evaluated environments.
  • Conclusion: LASI shows that sampled low-frequency action-source structure can induce persistent control bias when reused across replanning steps.The finding identifies a source-sensitivity failure mode in generative action policies.
  • Conclusion: The results support joint world–action modeling as a promising direction for interactive game control while exposing a generative-policy source-sensitivity failure mode.

AI USE STATEMENT

The authors used generative AI for language polishing and editorial assistance, while retaining authorship of the research and experimental content.

  • AI use: Generative AI was used for grammar, clarity, concision, academic phrasing, and presentation refinement of author-produced results.
  • AI use: The authors developed or produced the core research questions, method, model design, data construction, implementation, experiments, and numerical measurements.

ETHICS STATEMENT

The study is limited to learned closed-loop control in simulated video-game environments and does not include physical-system deployment or human-subject experiments.

  • Scope: The experiments are restricted to digital game environments and do not involve deployment in physical systems.
  • Scope: The study conducts no new human-subject experiments and collects no sensitive personal data.
  • Scope: Adapting general keyboard–mouse control beyond the studied environments should consider authorization, safety, and potential misuse.

REPRODUCIBILITY STATEMENT

The paper provides detailed methodological and experimental documentation, and commits to releasing resources intended to support reproduction.

  • The main text and appendices document the formulation, training objectives, rollout procedure, data construction, architecture, optimization, and evaluation protocols.The documentation also covers trajectory alignment, native action interfaces, sequence semantics, sampling, objective weights, and configuration settings.
  • The authors plan to release code, data-processing scripts, constructed datasets, trained weights, and configuration files.Release remains subject to applicable third-party licensing and redistribution requirements.
  • The appendix clarifies how block-causal information boundaries operate during teacher-forced training and online interaction.It also explains how realized context is organized within and across interaction cycles.

A.1 TEACHER-FORCED TRAINING AND ONLINE ROLLOUT

Training and online rollout share a causal information boundary while differing in how realized context is obtained. Prediction extends beyond execution, and hierarchical memory preserves temporal information across cycles without promoting unexecuted predictions into history.

  • Teacher-forced training and online rollout: Teacher-forced anchors use ground-truth causal prefixes, while future visual and action variables remain generation targets.Later trajectory observations are excluded from earlier anchors, and predicted futures are never reused as clean context.
  • Teacher-forced training and online rollout: At each decision, GameWAM predicts P actions, commits only E actions, then replans from the newly realized observation while discarding the unexecuted suffix.This preserves execution-grounded closed-loop context during online interaction.
  • Prediction–execution decomposition: The execution horizon E defines cycle boundaries, whereas the prediction horizon P determines future supervision distance.When P > E, neighboring targets overlap by P − E actions without changing execution-aligned anchors.
  • Prediction–execution decomposition: Increasing P extends forecast supervision and reuses overlapping transitions across anchors, but adds future targets and prediction losses.Longer-horizon supervision therefore does not make computation free or extend the autoregressive commitment chain.
  • Temporal context and hierarchical memory: The clean prefix combines persistent cross-cycle history Hc with realized within-cycle context Cc,j, while predicted unexecuted variables belong to neither.The online K/V cache is a transient processed representation of this prefix and is incrementally extended only by new observations.
  • Temporal context and hierarchical memory: Persistent history retains recent executed segments and summarizes older evicted segments using temporally encoded compressed visual tokens.Recent and long-term memory share compressed segment representations while using distinct temporal references and role-specific processing.

B DATA AND TRAINING CONSTRUCTION

GameWAM training data is standardized into a shared observation–state–action representation across gameplay, event-focused gameplay, GUI, and ViZDoom sources. Event-aware construction reduces redundancy and balances interaction types while scripted GUI data broadens interface coverage.

  • Shared trajectory representation: All sources use a common pre-action observation–state–action timeline retaining controls, continuous values, padding indicators, and validity masks.The pre-action convention also determines Minecraft gameplay versus GUI labeling.
  • Shared trajectory representation: VPT recordings are synchronized with mouse, keyboard, state, and video data, then filtered to remove redundant null controls while preserving alignment.The resulting episodes retain aligned observations, states, actions, and event metadata.
  • Minecraft data streams: Minecraft uses regular VPT, Event-Anchored VPT, and scripted MineStudio GUI trajectories to provide broad, event-conditioned, and interface-interaction coverage.Scripted GUI demonstrations use the same native action representation as VPT data.
  • Event-Anchored dataset: Event-Anchored dataset construction selects retained sub-trajectories, whereas event-anchored clip sampling controls training-window density afterward.The two procedures are separate stages.
  • ViZDoom data: ViZDoom expert trajectories preserve executed actions in a unified 9-D representation, including continuous horizontal turning on Battle maps.ViZDoom does not require a separate gameplay/GUI mode.
  • Event-Anchored dataset: The Event-Anchored dataset detects interaction events, extracts surrounding windows, filters temporal redundancy, and allocates capacity fairly across event types.Max–min allocation limits domination by frequent events while retaining distinct supervision.
  • Sampling motivation: Event-focused sampling addresses uneven supervision density because event-adjacent transitions contain changes and actions more directly related to instructions than distant trajectory regions.Uniform sampling can otherwise spend capacity on traversal, incidental camera motion, or weakly related behavior.

C.6 CLOSED-LOOP EVALUATION PROTOCOLS

The evaluation uses fixed post-training checkpoints and closed-loop task protocols that count executed native actions. Minecraft measures task success and successful-episode steps, ViZDoom measures episode reward, and inference frequency is evaluated separately.

  • Checkpoint and sampling protocol: All task-performance evaluations use the final checkpoint after the second training epoch without validation- or MCU-based checkpoint selection.The action source is independently resampled at each replanning step unless otherwise stated.
  • Minecraft MCU: Minecraft MCU evaluates 30 Mini tasks across mining, crafting, and combat, with success determined by environment task-completion signals.Native interaction steps exclude internal denoising evaluations and are averaged over successful episodes by category.
  • Reporting: Model scale and training-token consumption are summarized in Table 5 using each work’s reported token categories.For two-stage methods, modality-level counts are shown separately for Stages 1 and 2.
  • ViZDoom evaluation: ViZDoom evaluation uses four maps and average episode reward with the same P = 16, E = 8 overlapping-plan geometry as Minecraft.The protocol uses the action interface specified for ViZDoom.
  • Inference efficiency: Online execution frequency is measured separately on one NVIDIA H200 GPU over 10 closed-loop episodes per model variant.Timing includes model-side replanning computation but excludes environment simulation and execution time.

D.1 TRAINING SCALE AND TOKEN CONSUMPTION

Modality decoupling improves overall closed-loop control and inference efficiency, while cross-modal coupling can degrade visual prediction and is more consequential in fast-motion gameplay. Closed-loop rollouts nevertheless show temporal consistency, error recovery, and adaptation to changing targets and agent state.

  • Cross-modal masking: 50.7% MCU Mini ASR and 46.6% MCU All ASR follow modality decoupling, up from 46.3% and 39.6%, respectively.The broader MCU All evaluation shows the larger advantage, especially on Combat.
  • Inference efficiency: 12.51 Hz versus 8.12 Hz yields approximately 1.54× higher online execution frequency for modality-decoupled inference.The gain comes from removing iterative future-video denoising during online action generation.
  • Optimization dynamics: Faster action-objective fitting in the joint model does not produce stronger closed-loop control or sharper future-video predictions.The joint model fits the action objective faster, while its video optimization progresses more slowly and its predictions are generally less sharp.
  • Regime dependence: Cross-modal coupling has little GUI impact but larger Combat impact, where rapid camera motion, moving targets, and abrupt scene changes make visual prediction harder.Prediction blur is also more pronounced in fast-motion gameplay sequences, particularly for the joint model.
  • Closed-loop behavior: Repeated replanning preserves stable mining behavior, persistent goal pursuit, and rapid target switching across changing visual states.The rollouts include sustained mining, changing-target pursuit, and combat with repeated orientation changes.
  • Closed-loop behavior: Closed-loop replanning enables recovery from intermediate GUI errors because later actions condition on the realized interface state.The paper distinguishes this error recovery from explicit symbolic diagnosis or textual reasoning.

E.1 METHOD AND EVALUATION SCOPE

GameWAM is a low-level closed-loop controller whose scope excludes standalone high-level planning and explicit procedural representations. The evaluation also remains limited to digital games and exposes diagnostic, not mechanistically complete, evidence about LASI.

  • Method scope: GameWAM does not explicitly maintain symbolic task graphs, inventory plans, recipe representations, or long-horizon search.It is designed as a low-level closed-loop controller rather than a standalone high-level planner.
  • Method scope: Structured GUI procedures such as crafting and inventory manipulation remain outside separately represented planning capabilities.The model realizes these procedures through the same native-action policy, although scripted GUI data provides procedural supervision.
  • Method scope: Persistent history is predominantly visual and lacks direct supervision for retrieving historical information during action generation and task completion.Suggested extensions include task- or action-aware objectives, retrieval supervision, cross-cycle credit assignment, and multimodal memory.
  • Evaluation scope: Evaluation is restricted to digital game environments and does not establish transfer to physical control or isolate every Minecraft dataset-composition factor.The cited boundary includes differing physical sensing, actuation, safety, and contact dynamics, plus dependence on the available VPT/MineStudio mixture.
  • LASI interpretation: LASI evidence shows source interventions affect low-frequency action outputs, but the diagnostics do not identify a unique mechanism or explain complete closed-loop trajectories.The clearest effect appears in continuous camera control, where smooth low-frequency motion may provide a data-level explanation.
  • LASI interpretation: Source resampling is used as an effective deployment-time mitigation, while mechanism-level removal of LASI remains an open problem.Future objectives must avoid persistent low-frequency bias without suppressing smooth motion, action diversity, or closed-loop responsiveness.
Loading 2608.26200v1…