PaperlayerView exact revision on arXiv

Source-linked AI summary

GameFactory: Creating New Games with Generative Interactive Videos

Jiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, Xihui Liu

arXiv:2501.08325v4cs.CV

TL;DR

A central challenge in generative game videos is achieving action control that generalizes across scenes without binding it to a specific game style. GameFactory combines action-annotated game data with decoupled style and control training; experiments report open-domain action-controllable video generation.

  • Problem

    A central challenge is generating action-controlled game videos that generalize across scenes without tying control to a fixed game style.

    Source: §1 · p. 2, §5 · p. 5

  • Method

    GameFactory combines action-annotated GF-Minecraft data and keyboard/mouse controls with a domain adapter and multi-phase training that separates style learning from action control.

    Source: §1 · p. 2

  • Results

    Multi-phase training achieves better action following than one-phase training, performs better on CLIP, FID, and FVD, and keeps the model domain closer to the original.

    Source: §6.3 · p. 8

  • Takeaways & Limitations

    GameFactory represents a first effort toward a fully capable generative game engine.

    Source: §7 · p. 8

    Challenges remain in diverse level and gameplay design, player feedback, in-game object manipulation, long-context memory, and real-time game generation.

    Source: §7 · p. 8

Abstract

from arXiv · show

Generative videos have the potential to revolutionize game development by autonomously creating new content. In this paper, we present GameFactory, a framework for action-controlled scene-generalizable game video generation. We first address the fundamental challenge of action controllability by introducing GF-Minecraft, an action-annotated game video dataset without human bias, and developing an action control module that enables precise control over both keyboard and mouse inputs. We further extend to support autoregressive generation for unlimited-length interactive videos. More importantly, GameFactory tackles the critical challenge of scene-generalizable action control, which most existing methods fail to address. To enable the creation of entirely new and diverse games beyond fixed styles and scenes, we leverage the open-domain generative priors from pre-trained video diffusion models. To bridge the domain gap between open-domain priors and small-scale game datasets, we propose a multi-phase training strategy with a domain adapter that decouples game style learning from action control. This decoupling ensures that action control learning is no longer bound to specific game styles, thereby achieving scene-generalizable action control. Experimental results demonstrate that GameFactory effectively generates open-domain action-controllable game videos, representing a significant step forward in AI-driven game generation.

1. Introduction

GameFactory targets action-controlled, scene-generalizable game video generation, addressing the limitations of game-specific systems and the difficulty of obtaining open-domain action annotations. It combines GF-Minecraft and action-control mechanisms with autoregressive generation and a domain adapter trained to separate game style from action control.

Introduction

  • Video diffusion models show promise as generative game engines that could reduce manual content creation and provide players with unlimited game content to explore [50] [52].Source: §1 · p. 2
  • Existing action-controllable game video systems focus on particular games or limited game-specific datasets, restricting scene generalization and generation beyond existing games [2] [7] [42] [47].Source: §1 · p. 2
  • Open-domain action annotations are prohibitively expensive to collect, while pretrained video models offer abundant-data generative priors; direct fine-tuning on small game datasets risks collapsing scene generalizability to that game domain.Source: §1 · p. 2
  • GameFactory combines action-controlled, scene-generalizable game video generation with GF-Minecraft, keyboard and mouse control mechanisms, and autoregressive long-video generation.The autoregressive approach varies noise levels across frames and conditions on previously generated frames to create new ones.Source: §1 · p. 2
  • A domain adapter and multiphase decoupled training strategy separate Minecraft style learning from action control to preserve open-domain generation with controllable actions.The approach is motivated by the risk that directly fine-tuning on Minecraft data embeds its style and compromises open-domain generalization.Source: §1 · p. 2

2. Related Work

Prior work advances video diffusion and introduces additional controls, while game-generation methods still face limited scene generalization. GameFactory uses pretrained video-model priors and GF-Minecraft to address that gap across diverse scenarios and a complex action space.

2.1. Video Diffusion Models

  • Video diffusion models shifted from U-Nets to Transformers, enabling realistic, longer videos and motivating their potential as world models [33] [45].Diffusion models have advanced image generation [10] [34] [38] and video generation [5] [50] [52]; their ability to simulate physical rules suggests applications in autonomous driving and robotics.Source: §2.1 · p. 3

2.2. Controllable Video Generation

  • Additional control signals address ambiguous text-to-video outputs, with image controls [30] and camera-control methods [43] [17] enabling more nuanced control.Direct-a-Video uses three camera parameters for basic movements such as left panning; MotionCtrl [43] and CameraCtrl [17] offer more nuanced camera-pose control.Source: §2.2 · p. 3

2.3. Game Video Generation

  • Video-based game generators [2] [6] [7] [14] [42] [47] advance game generation, but many overfit specific games or datasets and broader scene generalization remains challenging.Genie 2 demonstrates control generalization using extensive action-labeled data, while Matrix [14] shows promise in racing games; broader scenes and more diverse action controls remain open challenges.Source: §2.3 · p. 3
  • GameFactory targets scene generalization with pretrained video-model priors and low-cost GF-Minecraft, validated across diverse open-world scenarios and a more complex action space.The action space includes forward/backward/left/right movement, jumping, and acceleration/deceleration.Source: §2.3 · p. 3

3. Preliminaries

GameFactory adopts a transformer-based latent video diffusion backbone, compressing videos into latent frames for action-conditioned training and decoding generated latents back into videos. Actions correspond to frame transitions, and inference samples a clean latent from noise.

3. Preliminaries

  • The backbone is a transformer-based latent video diffusion model [33], with an encoder compressing each (1 + rn)-frame video into (1 + n) latent frames.The latent sequence is Z = E(X), where r is the temporal compression ratio.Source: §3 · p. 3
  • Each action a_i transfers the video from frame x_{i−1} to x_i, and training minimizes noise-prediction error conditioned on the noisy latent, prompt, action sequence, and timestep.Noise is added to the clean latent during training; the action sequence spans the frame transitions.Source: §3 · p. 3
  • At inference, the model samples a clean latent Z_0 from noisy latent Z_T, then decodes it into video X.The decoder D maps the predicted latent back to video: X = D(Z_0).Source: §3 · p. 3

4. Action-Controlled Video Generation

Section 4 establishes the action-controlled generation foundation for Sec. 5 through GF-Minecraft, action-control modules, and autoregressive long-video generation.

4.1. GF-Minecraft Dataset

  • GF-Minecraft addresses human-biased, text-free VPT [3] gameplay data with 70 hours of Minecraft videos, balanced atomic actions, randomized durations, and diverse text-described scenes.The scenes span different weather conditions and times of day, with textual descriptions annotated using MiniCPM [51].Source: §4.1 · pp. 3–4

4.2. Action Control Module

  • Sliding-window action grouping aligns action sequences with temporally compressed video features and captures delayed effects such as a jump influencing subsequent frames.For each feature, the method groups actions within a preceding window and pads out-of-range indices with boundary actions.Source: §4.2 · p. 4, Figure 4 · p. 4

4.3. Autoregressive Generation of Long Action-Controllable Game Videos

  • Autoregressive generation conditions each multi-frame prediction on recent history, enabling repeated extension to infinite-length videos rather than fixed-length outputs.Training conditions on the first k+1 clean latents and computes noise-prediction loss only for the remaining N−k frames; inference reuses the latest k+1 history latents. Generating multiple frames per step reduces the time required for long videos.Source: Figure 5 · p. 5, §4.3 · p. 5

4.2. Action Control Module

  • Continuous mouse inputs are concatenated with transformer features for MLP and temporal-attention processing, while discrete keyboard embeddings serve as keys and values in cross-attention.The action-control module is integrated into the video diffusion transformer's blocks.Source: §4.2 · pp. 4–5, Figure 3 · p. 4

5. Open-Domain Game Scene Generalization

GameFactory separates game-style adaptation from action-control learning so control can generalize beyond the styles of its training videos. Its multi-phase strategy preserves open-domain generation priors while enabling action-controlled generation across open-domain scenes.

5. Open-Domain Game Scene Generalization

  • Directly fine-tuning open-domain video models on game data risks inheriting training-set style bias, so GameFactory decouples style adaptation from action control.The framework uses a Style-Action Decoupled Model with a domain adapter and a multi-phase training strategy.Source: §5 · p. 5

5.1. Style-Action Decoupling with Domain Adapter

  • An independent LoRA domain adapter learns game-specific visual styles, while the action control module learns controllability without binding it to those styles.The LoRA adapter can be plugged in or out without affecting the original model’s open-domain generation priors.Source: §5.1 · p. 5

5.2. Multi-Phase Training Strategy

  • Training first adapts the model to game videos with LoRA, then freezes those weights while training action control, and removes LoRA at open-domain inference.The phases separate style learning from control learning, preserving open-domain capabilities from pretraining for style-independent action-controlled generation.Source: §5.2 · pp. 5–6, Figure 6 · p. 6

6. Experiments

Experiments show that multi-phase training improves action following, text alignment, and generation quality across generalized scenes while preserving the pretrained model’s domain. Ablations support different control mechanisms for discrete and continuous actions, and evaluations examine the GF-Minecraft dataset and long-video generation.

6.1. Implementation Details

  • Experiments use a distilled internal 1B text-to-video model at 360×640 resolution, with a held-out 5% test set and metrics for action following, camera motion, text alignment, and video quality.Training or finetuning phases take about two to four days on 8 A100 GPUs with batch size 64; sampling uses DDIM with 50 steps.Source: §6.1 · pp. 6–7

6.2. Action Controllability

  • Cross-attention works better for discrete keyboard control, while concatenation works better for continuous mouse movement.Mouse movements affect visual output more than keyboard inputs, especially in the mouse-large test set; CLIP, FID, and FVD show little difference between methods.Source: Table 2 · p. 7, Figure 7 · p. 7, §6.2 · p. 7

6.3. Scene Generalization

  • Multi-phase training improves scene-generalized action following, text alignment, and generation quality over one-phase training while keeping the generated domain closer to the original model.The one-phase approach produces noticeable artifacts, whereas multi-phase training preserves the original generation domain without style leakage.Source: Table 3 · p. 7, Figure 8 · p. 7, §6.3 · p. 8

6.4. Evaluation for GF-Minecraft Dataset

  • GF-Minecraft uses a uniform distribution of atomic actions, enabling response to rarely performed actions, unlike the highly skewed keyboard-input distribution in VPT gameplay recordings.The comparison uses clips from VPT’s Find Cave dataset, selected because it closely matches the setting by primarily excluding inventory management and block modification.Source: §6.4 · p. 8

6.5. Evaluation for Long Video Generation

  • Computing training losses only on frames requiring prediction performs better than computing them across all frames and supports generation of videos exceeding 300 frames.The paper attributes the improvement to avoiding noise from previously generated frames, which is irrelevant to future video generation.Source: §6.5 · p. 8

7. Conclusion

GameFactory is a framework for creating new games with generative interactive videos, particularly addressing scene generalization. The paper presents it as a first effort toward a fully capable generative game engine while identifying substantial challenges that remain.

7. Conclusion

  • GameFactory uses generative interactive videos to create new games and addresses gaps in prior research, particularly in scene generalization.The paper characterizes this as its first effort in the field.Source: §7 · p. 8
  • Remaining challenges include designing diverse levels and gameplay and developing player feedback systems.Source: §7 · p. 8
  • Further challenges include in-game object manipulation, long-context memory, and real-time game generation; the authors aim to progress toward a fully capable generative game engine.Source: §7 · p. 8

Appendix A. Details of Minecraft Dataset · A.1. Basic Information

GameFactory’s Minecraft dataset uses MineDojo snapshots and simulated, randomly combined action sequences to reduce bias from human play habits. It contains 2,000 action-annotated clips generated across configured biome, weather, and time-of-day conditions, with 2,000 frames per clip.

A.1. Basic Information

  • MineDojo [13] provides Minecraft snapshots with details of the agent’s surroundings for data collection.Source: §A.1 · p. 12
  • Action frequencies and durations were sampled uniformly to mitigate biases from human player habits.Source: §A.1 · p. 12
  • Sampled actions were randomly combined, including simultaneous key presses and keyboard-plus-mouse operations, to enhance action generalization.Source: §A.1 · p. 12
  • The action sequences were executed in the MineDojo simulation environment, where corresponding observations were collected.Source: §A.1 · p. 12
  • Scene generation covered three biomes—forest, plains, and desert—and three weather conditions—clear, rain, and thunder.Source: §A.1 · p. 12
  • The configured conditions yielded 2,000 action-annotated video clips, each containing 2,000 frames.Source: §A.1 · p. 12
  • The scene configurations also included six different times of day, from the start of day through sunrise.The listed times are starting of a day, noon, sunset, beginning of night, midnight, and beginning of sunrise.Source: §A.1 · p. 12

A.2. Data PreProcessing · A.3. Details of Action Space

The appendix describes video preprocessing through sampled clips and text annotations, then specifies the keypress and mouse-movement controls used from MineDojo’s action space.

A.2. Data PreProcessing

  • Preprocessing randomly samples consecutive k-frame sequences from each full video until the sampled-frame total reaches n times the original frame count.In the experimental setup, k = 81 and n = 3.Source: §A.2 · p. 12
  • The experimental preprocessing uses 81-frame clips and samples until the total sampled frames reach three times the original video’s frame count.The clip length is slightly larger than the video diffusion model’s generation length of 77 frames.Source: §A.2 · p. 12
  • Each sliced clip receives text annotations from the open-source multimodal model MiniCPM-V [51].Figure 11 shows an annotation example with scene- and object-describing words highlighted in red and bold.Source: §A.2 · p. 12, Figure 11 · p. 12

A.3. Details of Action Space

  • The action space uses a MineDojo [13] subset encompassing nearly all actions available to human players, including keypresses, mouse movements, and clicks.The selected control signals are keypresses and mouse movements.Source: §A.3 · p. 12
  • The setup’s binary actions are grouped across five interfaces, with mutually exclusive actions such as forward and backward assigned to the same interface.The specific binary actions are listed in Table 7.Source: §A.3 · p. 12
  • Mouse input is represented by the pointer’s offset from the game-region center, accumulated relative to the first frame and supplied as an absolute offset.Source: §A.3 · p. 12

Appendix B. Supplementary Experimental Results · B.1. Interaction in generative Minecraft videos. · B.2. More Inspiration from Examples of Racing Games

GameFactory exhibits collision-responsive behavior in generative Minecraft videos and transfers learned mouse yaw control to steering in a racing-game example. The examples also raise the possibility of broader game-control and autonomous-driving applications.

B.1. Interaction in generative Minecraft videos.

  • Randomly generated scenes naturally provide numerous collision examples, a critical physical interaction for navigation agents.Source: §B.1 · p. 12
  • During collisions, agents should ideally remain stationary despite action inputs, but such corner cases can hinder action-control learning when data is limited.Source: §B.1 · pp. 12–13
  • At inference, the model showed collision detection and appropriate interaction feedback, as demonstrated in Figure 12.Source: §B.1 · p. 13, Figure 12 · p. 13

B.2. More Inspiration from Examples of Racing Games

  • Because the collected Minecraft data is first-person, the learned action space primarily generalizes to first-person scenes.Source: §B.2 · p. 13
  • In a prompted racing-game scenario, learned mouse yaw control generalized to steering, while backward and sideways controls were diminished in line with typical racing-game needs.Source: §B.2 · p. 13
  • The racing example raises questions about a broader action space for varied game controls and whether simulation-trained models could generate unlimited open-domain autonomous-driving data.Source: §B.2 · p. 13

Appendix C. Potential of Generalizable World Model

GameFactory is presented as a generalizable world model that transfers physical knowledge from small labeled datasets to open-domain scenarios. It can produce action-annotated data and simulate diverse real-world-like conditions for training agents.

As a data producer

  • As a data producer, GameFactory transfers knowledge from small labeled datasets to open-domain scenarios to generate diverse, unlimited action-annotated data approximating real-world complexity.The paper identifies autonomous driving [15] [20] and embodied AI [4] [35] [46] [53] as areas facing a shortage of large-scale action-labeled datasets.Source: Appendix C · p. 13

Generalization to a different game type

  • In a racing game, Minecraft-learned yaw control transfers to steering, while unrelated movement and pitch controls automatically diminish.Figure 13 demonstrates generalization to a different game type.Source: Figure 13 · p. 14

As a simulator

  • As a simulator, GameFactory generates controlled, diverse scenarios—including extreme situations—to train agents for real-world tasks [1] [22] [36] and support robustness, generalizability, and sim-to-real transfer.The paper describes exposing policy models to a wide range of environments and interactions as a way to address sim-to-real transfer challenges.Source: Appendix C · p. 14

Related papers

Loading 2501.08325v4…