PaperlayerView exact revision on arXiv

Source-linked AI summary

Hunyuan-GameCraft-2: Instruction-following Interactive Game World Model

Junshu Tang, Jiacheng Liu, Jiaqi Li, Longhuang Wu, Haoyu Yang, Penghao Zhao, Siruis Gong, Xiang Yuan, Shuai Shao, Linfeng Zhang, Qinglin Lu

arXiv:2511.23429v2cs.CV

TL;DR

Interactive world models still face constrained action schemas and insufficient scalable interactive-video data. Hunyuan-GameCraft-2 combines free-form text and keyboard/mouse control with automated data pipelines and long-horizon generation methods. The authors report strong interaction performance and high visual fidelity and temporal coherence, while noting limits on very long sequences and multi-stage tasks.

  • Problem

    Interactive world models remain constrained by discrete control inputs, and suitable training videos are scarce because available sources do not jointly provide interactivity, scale, and broad diversity.

    Source: §1 · p. 2, §3 · p. 4

  • Method

    Hunyuan-GameCraft-2 unifies text and keyboard/mouse control, builds interactive-video data through automated pipelines, and uses long-video training and inference methods.

    Source: §1 · p. 2

  • Results

    Experiments report state-of-the-art interactive-video performance on InterBench while maintaining high visual fidelity and temporal coherence.

    Source: §1 · p. 3, §7 · p. 17

  • Takeaways & Limitations

    The framework supports free-form instruction and keyboard/mouse control over generated game videos, with InterBench assessing interaction completeness, effectiveness, causal coherence, and physical plausibility.

    Source: §1 · p. 3, §7 · p. 17

    Semantic drift may remain beyond 500 frames, and supported interactions center on single-step immediate-effect actions rather than multi-stage tasks requiring reasoning or planning.

    Source: §6 · p. 17

Abstract

from arXiv · show

Recent advances in generative world models have enabled remarkable progress in creating open-ended game environments, evolving from static scene synthesis toward dynamic, interactive simulation. However, current approaches remain limited by rigid action schemas and high annotation costs, restricting their ability to model diverse in-game interactions and player-driven dynamics. To address these challenges, we introduce Hunyuan-GameCraft-2, a new paradigm of instruction-driven interaction for generative game world modeling. Instead of relying on fixed keyboard inputs, our model allows users to control game video contents through natural language prompts, keyboard, or mouse signals, enabling flexible and semantically rich interaction within generated worlds. We formally defined the concept of interactive video data and developed an automated process to transform large-scale, unstructured text-video pairs into causally aligned interactive datasets. Built upon a 14B image-to-video Mixture-of-Experts(MoE) foundation model, our model incorporates a text-driven interaction injection mechanism for fine-grained control over camera motion, character behavior, and environment dynamics. We introduce an interaction-focused benchmark, InterBench, to evaluate interaction performance comprehensively. Extensive experiments demonstrate that our model generates temporally coherent and causally grounded interactive game videos that faithfully respond to diverse and free-form user instructions such as "open the door", "draw a torch", or "trigger an explosion".

1. Introduction

Video-based world models have advanced interactive simulation, but current approaches still face limited interaction and data-construction challenges. Hunyuan-GameCraft-2 combines free-form text and keyboard/mouse controls with automated interactive-data pipelines and long-horizon generation methods; experiments report strong interaction and video-quality performance.

1. Introduction

  • Video-based world models learn dynamics from video and respond to user inputs, while 3D-based approaches emphasize geometric consistency and physical accuracy but are often limited to scripted or static interactions.The paper situates this shift as moving from what a world looks like to how users interact with it.Source: §1 · p. 2
  • Existing approaches lack a scalable construction pipeline for interactive video data and face challenges in maintaining quality and interaction accuracy during multi-turn long-video generation.The paper defines interaction as explicit-agent actions that trigger causally related, physically or logically valid environmental state transitions.Source: §1 · p. 2
  • Hunyuan-GameCraft-2 unifies text instructions and keyboard/mouse signals for semantically grounded control, with automated pipelines converting unstructured text–video pairs into interactive datasets.Its training uses autoregressive distillation, randomized long-video tuning, and KV-recache for multi-turn inference; engineering optimizations reach 16 FPS.Source: §1 · pp. 2–3
  • The framework achieves state-of-the-art interactive-video performance while maintaining visual fidelity and temporal coherence.InterBench evaluates interaction completeness, action effectiveness, causal coherence, and physical plausibility.Source: §1 · p. 3

2. Related Works

Related work addresses long-video coherence, interactive world models, and text-guided video generation. The paper distinguishes continuous state-changing interaction from systems that generate or edit videos from static prompts or scripts.

2.1. Long Video Extension

  • Long-video methods address train-short-test-long semantic drift through self-conditioning, rolling windows, explicit memory, alternative generation paradigms, and inference-time adaptation.Examples include Self-Forcing, Rolling-Forcing, Memory-Forcing, StreamingT2V [20], DiffusionForcing [8], and LongLive.Source: §2.1 · p. 3

2.2. Interactive Video-based World Model

  • Interactive world models have progressed from game-specific discrete controls toward broader control signals, but prompts in recent systems mainly guide setup rather than directly controlling interaction.The paper argues that reliance on discrete physical input devices still constrains interaction richness.Source: §2.2 · p. 3

2.3. Text-guided Video Generation and Editing

  • Text-guided generation enriches prompts or executes plans, while video editing handles discrete text-directed transformations; these approaches lack continuous feedback and state transitions.Hunyuan-GameCraft-2 instead uses user prompts to drive the evolution of a dynamic world state.Source: §2.3 · pp. 3–4

3. Interactive Video Data Construction

The paper frames interactive-data construction around causally meaningful state changes and proposes automated methods for generating and curating such data. Its pipelines combine synthetic interaction videos with cleaned gameplay footage and captions that represent both scene appearance and transitions.

3. Interactive Video Data Construction

  • Available video sources trade off realism, controllability, scale, diversity, and annotation quality, leaving high-quality interactive data insufficient.Real-world capture is costly, simulation is expensive to scale across scenes, internet videos are noisy, and academic datasets are limited in scale and domain coverage.Source: §3 · p. 4

3.1. Definition of Interactive Video Data

  • Interactive Video Data records a causally driven transition from a clear initial state to a significantly different final state, including state changes, subject emergence or action, and scene evolution.The paper groups examples into Environmental Interactions, Actor Actions, and Entity and Object Appearances.Source: §3.1 · p. 4

3.2. Synthetic Data Construction

  • Synthetic videos begin with an initial frame and instruction-specific VLM prompt; stationary scenes use edited target frames, while dynamic actions generate from the first frame alone.Text-to-image synthesis supplies specific initial frames on demand when manual sourcing would be costly and inefficient.Source: §3.2 · p. 5, Figure 2 · p. 5

3.3. Game Scene Data Curation

  • The curation pipeline partitions footage, filters quality, creates structured captions, and annotates camera trajectories; interaction captions capture semantic changes between adjacent clips.The dataset draws on over 150 AAA games, and the two caption types support appearance-level perception and action-level reasoning.Source: §3.3 · pp. 5–6, Figure 3 · p. 6, Figure 4 · p. 6

4. Method

Hunyuan-GameCraft-2 extends a 14B image-to-video MoE model into a free-form instruction-controlled game video generator. Its method combines action and prompt conditioning with staged training for long-video generation and cache-based multi-turn interaction.

4.1. Model Architecture

  • A 14B image-to-video MoE foundation model combines camera-conditioned keyboard/mouse actions with prompt-based interaction in one generator.An MLLM extracts and injects interaction information to strengthen guidance for fine-grained interactive behavior.Source: §4 · p. 6, §4.1 · pp. 6–7

4.2.3. Autoregressive Generator Distillation

  • Self-Forcing distills the bidirectional MoE generator into a few-step causal model, retaining the initial frame as a sink token and using block-sparse local attention.Distinct expert learning rates and noise-boundary-aligned denoising steps keep teacher and student selections of high- or low-noise experts consistent.Source: §4.2 · p. 7, §4.2.3 · p. 8, Algorithm 1 Randomized Extended… · p. 8

4.2.1. Action-Injected Training

  • Action-injected training adapts the pretrained model to scene dynamics and longer-term consistency through sequential 45-, 81-, and 149-frame 480p video phases.Training also mixes long and short captions and concatenates interactive captions for interaction learning.Source: §4.2.1 · p. 7

4.2.2. Instruction-Oriented Supervised Fine-Tuning

  • Instruction-oriented fine-tuning uses 150K samples augmented with synthetic videos, then freezes the camera encoder and fine-tunes the MoE experts.The stage is designed to refine alignment with semantic control cues.Source: §4.2.2 · p. 7

4.2.4. Randomized Extended Long-Video Tuning

  • Randomized extension tuning rolls out videos to varying lengths, samples contiguous windows, and interleaves self-forcing with teacher-forcing at arbitrary trajectory states.The student conditions on predicted history while the teacher conditions on ground truth, exposing the model to varied states for recovery and temporal stability.Source: Algorithm 1 Randomized Extended… · p. 8, §4.2.4 · p. 9

4.3. Multi-turn Interactive Inference

  • For a new interaction prompt, recaching recomputes the last autoregressive block and updates self- and cross-attention KV caches for subsequent generation.The rolling cache retains sink tokens alongside a local window of preceding frames.Source: §4.3 · p. 9

4.4. Real-time Interaction Acceleration

  • System-level optimizations bring inference to 16 FPS for real-time interactive video generation with stable quality and low latency.The listed optimizations include FP8 quantization, parallelized VAE decoding, SageAttention, and sequence parallelism.Source: §4.4 · p. 9

5. Experiments

The experiments evaluate controllable video generation with general video-quality and camera-control metrics alongside InterBench’s action-level interaction measures. Hunyuan-GameCraft-2 scores highly on interaction initiation and fidelity, shows qualitative strengths across interaction categories, and is also examined for generalization and long-video behavior.

5.1. Model and Dataset Configurations.

  • The test suite covers three interaction categories across 100 diverse images, with 20 additional closed-door images for open-door evaluation; videos use 832×448 resolution and 93 frames.The compared foundation-model baselines are HunyuanVideo, Wan2.2 A14B, and LongCatVideo.Source: §5.1 · pp. 9–10

5.2.1. General Metrics

  • General metrics assess video fidelity, aesthetics, temporal consistency, motion intensity, and camera trajectory accuracy, complementing interaction-specific evaluation.Camera control uses translational and rotational Relative Pose Error after Sim3 Umeyama alignment.Source: §5.2.1 · p. 10, §5.2 · p. 10

5.2.2. InterBench: Benchmarking Action-Level Interaction in Video Generation

  • InterBench uses a VLM evaluator and six dimensions to assess interaction triggering, prompt alignment, fluency, scope, end-state consistency, and object physics.Trigger Rate is binary; the other five dimensions receive ordinal scores that are averaged by category and aggregated into a global score.Source: §5.2.2 · pp. 11–12
  • Prompt design concatenates an interaction prompt with a base prompt, enabling instruction changes while keeping the visual context constant across models.This supports controlled comparisons with identical instructions for each model.Source: §5.2.2 · pp. 12–13

5.3. Interaction Evaluation

  • Trigger scores reach 0.962 for Environmental Interactions and 0.983 for Actor Actions; Physics margins are 0.683 for Environmental Interactions and over 0.52 for Entity & Object Appearances.For Actor Actions, Fluency and EndState improve by +0.70 and +0.63, respectively.Source: §5.3 · p. 13, Table 5 · p. 15
  • Qualitative comparisons report that GameCraft-2 produces more coherent actions, more stable object behavior, and more physically plausible environmental effects than baselines.Examples include global snowfall accumulation, stable grasping and object manipulation, and objects integrated with scene lighting and perspective.Source: §5.3 · pp. 13–14, Figure 11 · p. 14
  • The model handles previously unseen subjects and objects with coherent, physically plausible state transitions, including examples absent from the training data.The reported examples include a dragon emerging, a man appearing, and an actor taking out a phone.Source: §5.3 · pp. 15–16, Figure 15 · p. 17
  • Randomized extended long-video tuning improves fidelity and motion consistency beyond the 450th frame, while larger sink-token and local-attention settings enrich detail but increase artifacts.The qualitative comparison holds input prompts and camera parameters constant across settings.Source: §5.3 · pp. 16–17, Figure 16 · p. 17

6. Limitation and Future Work

The framework retains several limitations in long-horizon generation, task scope, and deployment. The authors identify semantic drift beyond 500 frames, a focus on immediate single-step actions, and latency and hardware-accessibility needs.

6. Limitation and Future Work

  • Semantic drift may still occur beyond 500 frames because the model lacks explicit long-term memory and relies on finite KV-cache capacity.Randomized long-video tuning alleviates, but does not eliminate, autoregressive error accumulation.Source: §6 · p. 17
  • Supported interactions currently center on single-step, immediate-effect actions; multi-stage tasks requiring reasoning or planning remain a future challenge.Source: §6 · p. 17
  • Although generation reaches 16 FPS, further latency reduction is needed for highly reactive gameplay and deployment on more accessible hardware.Source: §6 · p. 17

7. Conclusion

Hunyuan-GameCraft-2 combines free-form text and keyboard/mouse control with automated interactive-video data pipelines, long-video tuning, and KV-recache. The conclusion reports stronger interaction fidelity, visual quality, and temporal coherence than existing state-of-the-art models.

7. Conclusion

  • Hunyuan-GameCraft-2 generates controllable video from free-form text instructions and keyboard or mouse actions.Source: §7 · p. 17
  • The work defines interactive video data and proposes automated curation and synthesis pipelines to address a data bottleneck.Source: §7 · p. 17
  • The conclusion reports significant outperformance over existing state-of-the-art models across interaction fidelity, visual quality, and temporal coherence.It attributes stable, long-horizon, real-time generation to randomized long-video tuning and efficient inference mechanisms including KV-recache.Source: §7 · p. 17

A. Illustrative Examples of Interactive Video Data

The section presents representative examples to clarify the boundary between interactive and non-interactive video data.

A. Illustrative Examples of Interactive Video Data

A.1. Positive Examples: Interactive Video Data

Interactive video data requires causal, agent-driven state transitions rather than motion alone. The appendix groups interactions into three categories and illustrates the sources and scope of the constructed data.

A.1. Positive Examples: Interactive Video Data

  • Interactive examples show agent- or event-driven transitions, including vehicles or objects appearing, doors opening, weapons firing, weather changing, and action-driven scene shifts.Each example is framed through initial, transition, and final states.Source: §A.1 · p. 21

A.2. Negative Examples: Non-Interactive Video Data

  • Continuous snowfall, busy traffic, and ambient motion are non-interactive examples when they lack a coherent, agent-driven macroscopic state transition.Source: §A.2 · p. 21

A.3. Interaction Categories

  • The dataset structures interactions as environmental changes, actor actions, and entity or object appearances, with complex cases demanding coherent effects, body–object coordination, or stable placement and identity.Examples include explosions, opening doors, and introducing animals or vehicles.Source: §A.3 · pp. 21–22

B. Dataset Showcase

  • The appendix’s showcase draws on curated gameplay footage and synthetically generated interactive videos.Source: B. Dataset Showcase · p. 22

B.1. Curated Gameplay Data

The curated gameplay collection spans more than 150 AAA games and diverse viewpoints and environments. Its coverage includes natural and urban scenes under varied conditions.

B.1. Curated Gameplay Data

  • The curated gameplay data comes from over 150 AAA games and includes both first-person and third-person viewpoints.Source: §B.1 · p. 22
  • The collection spans natural and urban environments with varied lighting, weather, and terrain conditions.Source: §B.1 · p. 22
  • The passage describes this diversity as important for training robust and generalizable world models.Source: §B.1 · p. 22

B.2. Synthetic Interaction Data

The synthetic pipeline covers environmental interactions, actor actions, and entity or object appearances. Qualitative comparisons report greater temporal stability, action coherence, and object-emergence accuracy than baseline models.

B.2. Synthetic Interaction Data

  • Synthetic examples cover weather changes and explosions, complex body–object actions, and new subjects introduced with high fidelity.The examples include snowfall, rainfall, lightning, drawing or firing weapons, opening doors, animal intrusion, and vehicle entry.Source: §B.2 · p. 22, Figure 19 · p. 25, Figure 20 · p. 26

C. Detailed Comparison Across Interaction Dimensions

D. InterBench: A Detailed Protocol for Benchmarking Action-Level Interaction

InterBench addresses the gap between conventional video-quality metrics and evaluating whether an action is correctly executed. It breaks interaction quality into distinct dimensions covering initiation, alignment, motion, scope, and physical and final-state correctness.

D. InterBench: A Detailed Protocol for Benchmarking Action-Level Interaction

Loading 2511.23429v2…