Source-linked AI summary
Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Condition
Jiaqi Li, Junshu Tang, Zhiyong Xu, Longhuang Wu, Yuan Zhou, Shuai Shao, Tianbao Yu, Zhiguo Cao, Qinglin Lu
TL;DR
Interactive game-video generation remains limited in dynamic fidelity, generality, long-term consistency, and efficiency. Hunyuan-GameCraft unifies keyboard and mouse actions in camera space, uses hybrid history conditioning and distillation, and reports substantial improvements over existing methods. Its scope remains mainly focused on open-world exploration rather than broader game-specific actions.
Problem
Interactive video-generation methods remain limited in dynamics, generality, long-term consistency, and efficiency for playable game scenarios.
Method
Hunyuan-GameCraft unifies keyboard and mouse inputs into continuous camera representations, autoregressively extends videos with hybrid history conditioning, and uses model distillation.
Results
Hunyuan-GameCraft obtains a significant lead over current models across curated game scenes and general styles, with 55% fewer interaction errors in crossdomain tests.
Takeaways & Limitations
The framework supports fine-grained control, efficient inference, and scalable long-video synthesis for immersive gaming environments.
Takeaways & Limitations
The current action space mainly targets open-world exploration and lacks game-specific actions such as shooting, throwing, and explosions.
Abstract
from arXiv · showhide
Recent advances in diffusion-based and controllable video generation have enabled high-quality and temporally coherent video synthesis, laying the groundwork for immersive interactive gaming experiences. However, current methods face limitations in dynamics, generality, long-term consistency, and efficiency, which limit the ability to create various gameplay videos. To address these gaps, we introduce Hunyuan-GameCraft, a novel framework for high-dynamic interactive video generation in game environments. To achieve fine-grained action control, we unify standard keyboard and mouse inputs into a shared camera representation space, facilitating smooth interpolation between various camera and movement operations. Then we propose a hybrid history-conditioned training strategy that extends video sequences autoregressively while preserving game scene information. Additionally, to enhance inference efficiency and playability, we achieve model distillation to reduce computational overhead while maintaining consistency across long temporal sequences, making it suitable for real-time deployment in complex interactive environments. The model is trained on a large-scale dataset comprising over one million gameplay recordings across over 100 AAA games, ensuring broad coverage and diversity, then fine-tuned on a carefully annotated synthetic dataset to enhance precision and control. The curated game scene data significantly improves the visual fidelity, realism and action controllability. Extensive experiments demonstrate that Hunyuan-GameCraft significantly outperforms existing models, advancing the realism and playability of interactive game video generation.
1 Tencent Hunyuan 2 Huazhong University of Science and Technology https://hunyuan-gamecraft.github.io/
The paper presents Hunyuan-GameCraft as an interactive game-video generator whose outputs respond to simulated movement and view-angle inputs. Figures show multi-action control and generated sequences from a single image and prompt.
- Hunyuan-GameCraft generates interactive game video from a single image, prompt, and simulated action signals.
- W, A, S, D control transition movement, while the arrow keys change view angles.
- The figures depict key moments generated in response to different inputs and additional multi-action control results.
1. Introduction
Existing interactive video methods remain limited in dynamic fidelity, generality, long-sequence consistency, and efficiency. Hunyuan-GameCraft addresses these gaps through unified action control, hybrid history conditioning, and distillation, and reports a significant lead over current models.
- Interactive video approaches still struggle with real-time dynamic scene fidelity, long-sequence consistency, and computational efficiency.
- Hunyuan-GameCraft maps keyboard and mouse inputs into a shared camera space for smooth interpolation between movement and camera operations.
- Hybrid history-conditioned training autoregressively extends sequences while preserving scene information and using a mask indicator to address error accumulation.
- Model distillation reduces computational overhead while maintaining consistency across long temporal sequences for real-time deployment.
- Hunyuan-GameCraft obtains a significant lead over current models on curated game scenes and general styles.
2. Related Work
Related work spans interactive game world models, controllable video generation, and long-video extension. Hunyuan-GameCraft is positioned around continuous action conditioning, temporal dynamics, and historical scene preservation.
- Interactive game models: WorldDreamer, GameGen-X, and Genie explore multimodal interaction, open-world game video generation, and 3D world synthesis.
- Camera-controlled video generation: MotionCtrl and CameraCtrl use unified motion or camera representations to control generated-video perspectives and object movement.
- Hunyuan-GameCraft: Hunyuan-GameCraft is described as generating infinitely long game videos from continuous action signals while preserving historical scene information.
- Long video generation: Long-video methods address temporal consistency and scalability through memory blocks, streaming generation, or historical segments.
3. Dataset Construction
The dataset pipeline combines curated gameplay, interaction annotations, and synthetic motion sequences. Its architecture converts discrete inputs into camera trajectories and separates history from predicted frames during long-video extension.
- Game scene data curation: The dataset covers over 100 AAA titles and uses 23 hours of gameplay recordings segmented into coherent 6-second clips.
- Game scene data curation: The curation pipeline uses scene- and action-aware partitioning, data filtering, interaction annotation, and structured captioning.
- Interaction annotation: Interaction annotation reconstructs 6-DoF camera trajectories with frame-by-frame position and orientation data.
- Synthetic data construction: About 3,000 rendered motion sequences provide diverse camera trajectories and improve motion prediction accuracy and temporal coherence during viewpoint transitions.
- Synthetic data construction: The architecture transforms keyboard or mouse signals into continuous camera space, encodes trajectories, and uses a mask to distinguish history from predicted frames.
4. Method
Hunyuan-GameCraft unifies keyboard and mouse inputs in a continuous camera-action space and injects them into an MM-DiT backbone for fine-grained control. Its hybrid history-conditioned extension balances long-term consistency with responsiveness, while distillation improves inference efficiency.
- Continuous Action Space and Injection: Keyboard and mouse inputs are unified into a shared camera representation space for fine-grained, smoothly interpolated control.The representation uses translation and rotation directions with bounded speed parameters, while removing roll freedom and retaining velocity control.
- Continuous Action Space and Injection: Dual lightweight tokenizers inject camera-pose controls into the MM-DiT backbone through token addition with adaptive feature fusion.The design achieves interactive performance with significant encoder parameter reduction and negligible additional computational overhead.
- Hybrid history conditioned Long Video Extension: Hybrid history conditioning addresses the quality collapse of single-image extrapolation and the control degradation of history-clip conditioning.The method mixes multiple extension modes during training to jointly optimize interactive capability and generation consistency.
- Hybrid history conditioned Long Video Extension: Each autoregressive step denoises a noisy chunk latent using a clean head latent, interactive signals, concatenation, and a binary mask.The clean historical condition guides flow-matching denoising to generate temporally consistent video segments for subsequent iterations.
- Hybrid history conditioned Long Video Extension: The hybrid paradigm unifies initial-frame generation and video extension without architectural modifications, supporting flexible control and coherent long-term generation.It is designed to balance the competing objectives of action responsiveness and historical consistency for practical deployment.
5. Experiment
Experiments compare Hunyuan-GameCraft with interactive game and camera-controlled baselines, showing improvements in control, dynamics, quality, temporal consistency, and interaction reliability. Ablations indicate that combining gameplay and synthetic data balances dynamic generation and interaction accuracy.
- Quantitative Comparison: Hunyuan-GameCraft improves generation quality, dynamic capability, control accuracy, and temporal consistency over Matrix-Game under identical gaming scenarios.The comparison uses the same base model and reports the strongest dynamic performance among the compared methods.
- Quantitative Comparison: 55% fewer interaction errors are achieved in crossdomain tests compared with Matrix-Game.The authors attribute this improvement to optimized training and conditional injection.
- Quantitative Comparison: Nearly real-time inference is achieved, with a slight reduction in dynamic and visual quality to better support game-scene interaction.The method is evaluated for inference speed alongside generation quality and control accuracy.
- Qualitative Comparison: Hybrid history conditioning improves historical information retention during continuous left-right rotations in qualitative comparisons.The comparison includes sequential single-action scenarios in the Minecraft environment used to train Matrix-Game.
- User Study: User rankings from 30 evaluators give Hunyuan-GameCraft the highest scores across multiple dimensions.The study addresses the lack of comprehensive benchmarks for interactive video generation in gaming and general scenarios.
- Data Distribution: Training exclusively on synthetic data improves interaction accuracy but degrades dynamic generation, while gameplay data shows the opposite pattern.The combined training distribution achieves balanced results; synthetic data omits highlighted dynamic objects because generating dynamic scenes is costly and complex.
6. Generalization on Real Worlds
Hunyuan-GameCraft generalizes beyond curated game scenes to real-world images and long-form video generation. It produces camera-conditioned real-world videos while preserving dynamics and supports minute-level clips with maintained visual quality.
- Long Video Extension: Minute-level video clips can be generated while maintaining visual quality.
- Real-World Video Generation: Figure 10 presents high-fidelity, high-dynamic real-world generation with accurate camera control.
- Real-World Video Generation: Hunyuan-GameCraft generates reasonable real-world videos from images with conditioned camera movement while preserving dynamics.The authors attribute this generalization capability to integrating a pre-trained video foundation model.
7. Limitations and Future Work
The current action space is designed mainly for open-world exploration and does not cover a broader range of game-specific actions.
- Limitations and Future Work: The action space lacks game-specific actions such as shooting, throwing, and explosions.Future work will expand the dataset with more diverse gameplay elements and target more physical, playable interactions.
8. Conclusion
Hunyuan-GameCraft combines unified action representation, hybrid history-conditioned training, and model distillation for controllable, efficient, long-video interactive generation. The framework improves realism, responsiveness, and temporal coherence relative to existing methods and is positioned for real-time immersive gaming environments.
- Conclusion: Unified action representation, hybrid history-conditioned training, and model distillation provide fine-grained control, efficient inference, and scalable long-video synthesis.
- Conclusion: The framework delivers enhanced realism, responsiveness, and temporal coherence while demonstrating substantial improvements over existing methods.