Source-linked AI summary
GameFactory: Creating New Games with Generative Interactive Videos
Jiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, Xihui Liu
TL;DR
Existing generative game systems often lack scene generalization, while collecting large-scale action-annotated open-domain videos is impractical. GameFactory combines GF-Minecraft, action-control modules, pretrained video priors, and decoupled style-action training to generate open-domain interactive game videos. The framework also supports autoregressive long-video generation, while the authors identify broader generative game-engine challenges that remain unresolved.
Problem
Existing generative game systems are mainly game-specific, and large-scale action annotation for open-domain videos is prohibitively expensive.
Method
GameFactory learns action control from GF-Minecraft, injects keyboard and mouse control into pretrained video diffusion models, and decouples game style through a domain adapter and multiphase training.
Results
GameFactory generates action-controllable game videos across diverse open-domain scenes while preserving the pretrained model’s generation domain.
Takeaways & Limitations
The framework enables diverse interactive game generation beyond existing games using small-scale annotated game data and open-domain generative priors.
Takeaways & Limitations
The study does not yet resolve diverse levels and gameplay, player feedback, object manipulation, long-context memory, or real-time game generation.
Abstract
from arXiv · showhide
Generative videos have the potential to revolutionize game development by autonomously creating new content. In this paper, we present GameFactory, a framework for action-controlled scene-generalizable game video generation. We first address the fundamental challenge of action controllability by introducing GF-Minecraft, an action-annotated game video dataset without human bias, and developing an action control module that enables precise control over both keyboard and mouse inputs. We further extend to support autoregressive generation for unlimited-length interactive videos. More importantly, GameFactory tackles the critical challenge of scene-generalizable action control, which most existing methods fail to address. To enable the creation of entirely new and diverse games beyond fixed styles and scenes, we leverage the open-domain generative priors from pre-trained video diffusion models. To bridge the domain gap between open-domain priors and small-scale game datasets, we propose a multi-phase training strategy with a domain adapter that decouples game style learning from action control. This decoupling ensures that action control learning is no longer bound to specific game styles, thereby achieving scene-generalizable action control. Experimental results demonstrate that GameFactory effectively generates open-domain action-controllable game videos, representing a significant step forward in AI-driven game generation.
1 The University of Hong Kong 2 Kuaishou Technology https://yujiwen.github.io/gamefactory/
GameFactory combines pretrained video generation with an action-control module learned from a small Minecraft dataset to create new games in open-domain scenes.
- GameFactory uses pretrained video models to generate new games.The framework leverages the pretrained model’s open-domain generative capabilities.
- The action-control module is learned from a small-scale first-person Minecraft dataset.This learned control is transferred to open-domain videos.
- The framework supports action control across diverse, newly generated open-domain scenes.Figure 1 illustrates keyboard and mouse interactions in these scenes.
1. Introduction
Existing generative game systems are largely tied to specific games and datasets, limiting scene generalization. GameFactory addresses this gap with action-controlled generation, GF-Minecraft, and decoupled style and action learning.
- Existing generative game research mainly targets specific games or limited game-specific datasets, restricting scene generalization.This limitation prevents content creation beyond existing games.
- Large-scale action-annotated open-domain video datasets are impractical to collect, motivating reuse of pretrained open-domain video priors.The paper instead combines abundant open-domain videos with small-scale annotated game data.
- GameFactory introduces GF-Minecraft and dedicated mechanisms for keyboard, mouse, and autoregressive long-video control.The framework injects action-control modules into pretrained video diffusion models.
- A domain adapter and multiphase training strategy disentangle game style from action control.This removes Minecraft style during open-domain generation while preserving action controllability.
- GameFactory targets diverse interactive game generation beyond existing games.The stated contributions combine scene-generalizable generation, annotated data, action control, and autoregressive video generation.
2. Related Work
Prior game-generation systems increasingly use video generation models but generally remain tied to particular games. GameFactory instead combines pretrained video priors with GF-Minecraft for scene-generalizable action control.
- The paper positions video diffusion models as promising candidates for generative game engines.Their video-generation capabilities motivate applications in game generation.
- Video-generation research has expanded control beyond text through image, camera, and motion signals.Existing methods provide varying degrees of camera and motion control.
- Recent game-generation systems apply diffusion-based world modeling to games including Atari, CS:GO, DOOM, and Minecraft.Earlier GAN-based approaches were limited by weaker generative capabilities.
- GameFactory uses pretrained video-model priors and the low-cost GF-Minecraft dataset to address scene generalization.Experiments cover diverse open-world scenarios and a complex action space.
3. Preliminaries
GameFactory uses a transformer-based latent video diffusion model that compresses videos into latent frames, predicts noise conditioned on prompts and actions, and decodes clean latents back into videos.
- The backbone is a transformer-based latent video diffusion model.An encoder compresses video spatially and temporally into latent representations.
- A video X is encoded into latent sequence Z, with temporal compression mapping (1 + rn) frames to (1 + n) latent frames.The latent sequence is represented as Z = [z0, z1, ..., zn].
- Actions A contain the inputs taken between successive video frames and condition the diffusion loss.The model predicts added noise from noisy latents, prompts, actions, and timestep t.
- During inference, the model samples a clean latent from a noisy latent and decodes it into video.The decoder D maps the predicted clean latent Z0 back to X.
4. Action-Controlled Video Generation
GameFactory builds action-controlled video generation around GF-Minecraft, separate mechanisms for mouse and keyboard inputs, and autoregressive prediction for long interactive videos.
- 4.1. GF-Minecraft Dataset: GF-Minecraft provides accessible, action-annotated gameplay with unbiased action distributions, diverse scenes, and textual descriptions.The dataset contains 70 hours of gameplay collected through Minecraft’s API and randomized atomic keyboard and mouse actions.
- 4.2. Action Control Module: The action control module is integrated into transformer blocks and separately handles continuous mouse movements and discrete keyboard inputs.Mouse actions are fused with features through reshaping, repetition, concatenation, an MLP, and temporal self-attention; keyboard actions use cross-attention.
- 4.2. Action Control Module: Sliding-window grouping aligns temporally compressed action sequences with latent features and captures delayed effects such as jumping across subsequent frames.The method uses temporal compression ratio r = 4 and a window size w = 3, with boundary-action padding for out-of-range indices.
- 4.3. Autoregressive Video Generation: The autoregressive model conditions on recent frames, predicts later frames, and repeatedly merges new outputs into history for unlimited-length video generation.During training, loss is computed only on predicted frames; inference generates N − k new frames from the latest k + 1 conditional frames.
- 4.3. Autoregressive Video Generation: Unlike conventional next-frame generation, the approach supports multi-frame generation at each autoregressive step.This design targets continuous video streams required for practical game applications.
5. Open-Domain Game Scene Generalization
GameFactory separates game-style adaptation from action-control learning so pretrained open-domain video priors can support action-controlled generation across diverse scenes.
- 5.1. Style-Action Decoupling with Domain Adapter: Directly fine-tuning pretrained video models on game data risks style bias and degraded scene generalizability.The framework addresses this domain gap by separating style adaptation from action control.
- 5.1. Style-Action Decoupling with Domain Adapter: A domain adapter implemented with LoRA learns game-specific visual styles independently from the action control module.LoRA can be plugged in or out without affecting the original model’s open-domain generation priors.
- 5.2. Multi-Phase Training Strategy: Phase #1 adapts the pretrained model to game videos, while Phase #2 freezes pretrained and LoRA parameters and trains only action control.This separation allows Phase #2 to focus primarily on action-control learning rather than style learning.
- 5.2. Multi-Phase Training Strategy: Table 1 positions scene-generalizable action control across complex action spaces and open-domain scenarios as GameFactory’s core contribution.The caption describes this evaluation scope as distinguishing the method from related technical work.
- 5.2. Multi-Phase Training Strategy: During Phase #3, removing LoRA weights enables action-controlled generation in open-domain scenes while preserving pretrained generative capabilities.The multi-phase design aims to make action control independent of specific game styles.
6. Experiments
Experiments evaluate implementation settings, action-control mechanisms, scene generalization, dataset bias, and long-video generation. The reported results favor multi-phase training and show successful generation beyond 300 frames.
- Evaluation Setting: Evaluation retains 5% of the segmented dataset as a test set and uses only-key, mouse-small, and mouse-large subsets for ablation.Qualitative experiments also support custom combinations of input actions.
- Action-Control Ablation: Cross-attention performs better for discrete keyboard control, whereas concatenation is more effective for continuous mouse movement.The comparison covers the two typical action-control mechanisms evaluated in Table 2 and Figure 7.
- Scene Generalization: Multi-phase training achieves action following closer to the in-domain baseline than one-phase training and performs better on CLIP, FID, and FVD.The comparison uses Cam and Flow for action following and also evaluates text alignment and generation quality.
- Dataset Bias: GF-Minecraft is compared with the human-biased VPT dataset on actions that human players use less commonly.The comparison tests whether the dataset’s unbiased action distribution supports responses to rarely performed actions.
- Long Video Generation: Losses computed only on frames requiring prediction outperform losses computed across all frames, and the model generates videos exceeding 300 frames.The selective loss scope avoids noise from previously generated frames during long-video training.
7. Conclusion
The conclusion presents GameFactory as an effort toward generative interactive videos for new games, while identifying major capabilities still needed for a fully capable generative game engine.
- Conclusion: GameFactory addresses gaps in generative game-engine research, particularly scene generalization, through generative interactive videos for creating new games.The conclusion frames this as the paper’s first effort in the field.
- Remaining Challenges: The study identifies diverse levels and gameplay, player feedback systems, in-game object manipulation, long-context memory, and real-time generation as remaining challenges.These challenges define scope boundaries for the current work.
A.2. Data PreProcessing
The preprocessing pipeline samples fixed-length clips from collected videos and adds textual annotations, while the control space covers keyboard, mouse, and click-related inputs.
- Slicing: Preprocessing randomly samples consecutive sequences of k frames until the sampled-frame total reaches n times the original video’s frame count.The experimental setup uses k = 81 and n = 3.
- Text Annotation: Text annotation is applied after slicing to describe scenes and objects in the sampled video clips.The annotation example highlights descriptive words in red and bold.
- Control Signals: The control space uses keypresses and mouse movements from MineDojo’s human-player action space, with interface mappings defined for actionable commands.The broader space includes clicks, while the setup uses keypresses and mouse movements as control signals.
Appendix B. Supplementary Experimental Results
Supplementary results examine collision feedback, action-space mappings, and transfer to racing-game scenes. They show appropriate collision responses and suggest that mouse yaw control can transfer to steering.
- Collision Response: Collision examples test a common navigation interaction where the agent should remain stationary despite receiving action inputs.During inference, the model developed collision detection and provided appropriate interaction feedback.
- Action Space: The action-space table maps raw control signals to MineDojo action interfaces.It distinguishes the training inputs from the corresponding actionable commands.
- Racing-Game Transfer: In a racing-game scenario, learned mouse yaw control generalized to steering, while backward and sideways controls were diminished.The observed adaptation aligns with controls typically needed in racing games.
Appendix C. Potential of Generalizable World Model
GameFactory is presented as a generalizable world model with two applications: producing diverse action-annotated data and simulating varied environments for agent training. Its learned control also transfers from Minecraft to racing-game steering while unrelated controls diminish.
- As a data producer: GameFactory transfers knowledge from small labeled datasets to open-domain scenarios, enabling diverse, unlimited action-annotated data approximating real-world complexity.
- Cross-game transfer: Yaw control learned in Minecraft transfers to steering control in a racing game, while unrelated movement and pitch controls automatically diminish.
- As a simulator: As a simulator, GameFactory generates controlled and diverse scenarios, including extreme situations that are difficult to capture through real-world data collection.
- As a simulator: The simulator exposes policy models to varied environments and interactions, supporting robustness and generalizability.