Source-linked AI summary
HunyuanVideo: A Systematic Framework For Large Video Generative Models
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, Kathrina Wu, Qin Lin, Junkun Yuan, Yanxin Long, Aladdin Wang, Andong Wang, Changlin Li, Duojun Huang, Fang Yang, Hao Tan, Hongmei Wang, Jacob Song, Jiawang Bai, Jianbing Wu, Jinbao Xue, Joey Wang, Kai Wang, Mengyang Liu, Pengyu Li, Shuai Li, Weiyan Wang, Wenqing Yu, Xinchi Deng, Yang Li, Yi Chen, Yutao Cui, Yuanbo Peng, Zhentao Yu, Zhiyu He, Zhiyong Xu, Zixiang Zhou, Zunnan Xu, Yangyu Tao, Qinglin Lu, Songtao Liu, Dax Zhou, Hongfa Wang, Yong Yang, Di Wang, Yuhong Liu, Jie Jiang, Caesar Zhong
TL;DR
Open-source video generation trails closed-source systems, partly because robust open-source foundation models remain scarce. HunyuanVideo addresses this gap with a systematically scaled video foundation model and related generation applications, achieving strong overall performance and text alignment while supporting audio, image-to-video, and avatar animation.
Problem
Video generation lacks robust open-source foundation models, leaving a gap between public and closed-source capabilities.
Method
HunyuanVideo combines data curation, architecture optimization, infrastructure, and model scaling, including timestep shifting to reduce inference steps.
Results
HunyuanVideo achieved the best overall performance among compared models, particularly in motion quality, while demonstrating robust text alignment and a selected 13B-parameter scale.
Takeaways & Limitations
The foundation model extends to synchronized video-to-audio generation, image-to-video generation, and controllable avatar animation under multimodal conditions.
Takeaways & Limitations
Data acquisition is governed by GDPR principles and relies on data synthesis and privacy computing for compliance.
Abstract
from arXiv · showhide
Recent advancements in video generation have significantly impacted daily life for both individuals and industries. However, the leading video generation models remain closed-source, resulting in a notable performance gap between industry capabilities and those available to the public. In this report, we introduce HunyuanVideo, an innovative open-source video foundation model that demonstrates performance in video generation comparable to, or even surpassing, that of leading closed-source models. HunyuanVideo encompasses a comprehensive framework that integrates several key elements, including data curation, advanced architectural design, progressive model scaling and training, and an efficient infrastructure tailored for large-scale model training and inference. As a result, we successfully trained a video generative model with over 13 billion parameters, making it the largest among all open-source models. We conducted extensive experiments and implemented a series of targeted designs to ensure high visual quality, motion dynamics, text-video alignment, and advanced filming techniques. According to evaluations by professionals, HunyuanVideo outperforms previous state-of-the-art models, including Runway Gen-3, Luma 1.6, and three top-performing Chinese video generative models. By releasing the code for the foundation model and its applications, we aim to bridge the gap between closed-source and open-source communities. This initiative will empower individuals within the community to experiment with their ideas, fostering a more dynamic and vibrant video generation ecosystem. The code is publicly available at https://github.com/Tencent/HunyuanVideo.
1 Introduction
HunyuanVideo addresses the gap between closed-source and open-source video generation by presenting an open-source foundation model and a systematic framework for training it. The framework reduces resource requirements through an effective scaling strategy.
- Closed-source video models have outpaced publicly available alternatives, limiting public-community algorithmic innovation.
- HunyuanVideo combines training infrastructure, data curation, architecture optimization, and model training in an open-source foundation model.
- 5× lower computational-resource requirements were achieved through the reported scaling strategy while maintaining desired model performance.
2 Overview
HunyuanVideo is presented as a comprehensive video training system spanning data processing, model development, acceleration, evaluation, and applications.
- HunyuanVideo encompasses the workflow from data processing through model deployment.
- The report covers data preprocessing, component architecture, training and inference strategies, acceleration, and evaluation against open-source and proprietary models.
- The report also showcases applications built on the pretrained foundation model, including a video-to-audio generative model.
3 Data Pre-processing
HunyuanVideo curates multimodal training data through joint image-video training, hierarchical filtering, manual annotation, and structured captioning. The pipeline progressively tightens quality criteria across training stages.
- Image-video joint training uses five video groups and two image groups tailored to their respective training processes.
- Data acquisition follows GDPR principles and uses data synthesis and privacy computing for compliance.
- The data pipeline filters clips for aesthetics, clarity, motion, scene boundaries, text content, and other visual properties.
- Five training datasets progressively increase video resolution from 256 × 256 × 65 to 720×1280 × 129 as filter thresholds become stricter.
- The final fine-tuning dataset contains approximately 1M human-annotated samples selected for visual aesthetics and compelling content motion.
- Structured captions combine scene content, background, style, shot type, metadata, and camera movement information to improve caption diversity and control.
4 Model Architecture Design
HunyuanVideo uses a causal 3D VAE and diffusion-transformer architecture operating in a compressed spatiotemporal latent space. Its design combines reconstruction objectives, curriculum training, and tiled processing for high-resolution video.
- The model overview includes a Causal 3D VAE, diffusion backbone, and scaling-law experiments.
- The 3D VAE compresses videos and images into compact latent features, reducing diffusion-transformer token counts for original-resolution and frame-rate training.
- The VAE is trained from scratch with a 4:1 video-to-image data ratio and combines L1, perceptual, adversarial, and KL losses.
- Curriculum learning progresses from low-resolution short videos to high-resolution long videos, with frame intervals sampled from 1 ∼8 for high-motion reconstruction.
- Spatial-temporal tiling splits high-resolution long videos into overlapping tiles and stitches their separately encoded or decoded outputs.
- Tiling-aware fine-tuning reduces train-inference inconsistency by randomly enabling or disabling tiling during training.
- The VAE reports higher PSNR on video data than other video VAEs and surpasses both video and image VAEs on images.
4.2 Unified Image and Video Generative Architecture
HunyuanVideo unifies image and video generation through a Transformer architecture, multimodal text encoding, and scaling laws that guide model and training-resource choices.
- Architecture: Unified Full Attention processes image and video tokens together, supporting unified generation while leveraging LLM-oriented acceleration.The design is motivated by performance, scalability, and training/inference efficiency.
- Inputs and text conditioning: Video latents are patchified into token sequences, while text is encoded with an advanced LLM and supplemented by pooled CLIP guidance.Images are treated as single-frame videos for unified input processing.
- Architecture: A dual-stream-to-single-stream design first processes video and text independently, then concatenates their tokens for joint modeling.The dual-stream phase allows each modality to learn its own modulation mechanisms before fusion.
- Position embedding: Rotary Position Embedding supports multi-resolution, multi-aspect-ratio, and varying-duration generation by producing position-aware query and key embeddings.Feature channels are divided across temporal, height, and width coordinates.
- Scaling laws: DiT-T2X models range from 92M to 6.6B parameters and provide image-based scaling laws used to derive video-model scaling behavior.The framework models relationships among computation, parameters, and tokens for image and video generation.
- Scaling laws: 13B parameters was selected for the final model after considering video scaling-law results together with training consumption and inference cost.The reported video scaling coefficients are a1 = 0.0189, b1 = 0.3618, a2 = 0.0108, and b2 = 0.6289.
4.5 Model-pretraining
HunyuanVideo uses Flow Matching with progressive image pretraining and joint image-video curriculum learning to improve convergence while scaling toward longer, higher-resolution videos.
- Progressive pretraining: The training process begins with 256px and 512px image pretraining before joint image-video training from 256px to 960px.The stages are designed to warm up the model before video training.
- Flow Matching: Flow Matching trains the model to predict velocity fields that transport noisy latent samples toward image or video data.Inference uses a first-order Euler ODE solver to integrate the predicted velocity from noise to the final sample.
- Image pretraining: 256px multi-aspect image pretraining teaches broad aspect-ratio coverage while reducing crop-induced text-image misalignment.Low-resolution samples also provide more data for learning low-frequency concepts.
- Image pretraining: Mix-scale training combines multiple image resolutions to improve higher-resolution ability without severely degrading 256px generation.Direct 512px fine-tuning was observed to harm 256px image generation.
- Data organization: Duration and aspect-ratio bucketization assigns videos to BT × BAR buckets with bucket-specific maximum batch sizes.The scheme accommodates differing token counts while preventing out-of-memory errors.
- Video curriculum: Progressive video-image joint training increases video duration and resolution from low-resolution short videos to high-resolution long videos.The curriculum targets short-term consistency, longer temporal dynamics, and high-resolution detail while maintaining coherence.
- Video curriculum: Images are mixed into every training stage to compensate for scarce high-quality video data and prevent catastrophic forgetting of image-space semantics.Joint training broadens learned world knowledge while addressing image-video distribution differences.
4.6 Prompt Rewrite
HunyuanVideo rewrites user prompts into standardized, detailed forms using Hunyuan-Large, then refines them through self-revision and LoRA-based deployment.
- 4.6 Prompt Rewrite: Hunyuan-Large rewrites prompts to accommodate varied linguistic styles and lengths within a training-free prompt-rewrite framework.The module uses detailed instructions and in-context examples.
- 4.6 Prompt Rewrite: Multilingual input adaptation preserves meaning and context across user prompt languages.
- 4.6 Prompt Rewrite: Prompt standardization reshapes user wording into an information structure resembling training captions.
- 4.6 Prompt Rewrite: Terminology simplification converts complex wording into clearer expressions while maintaining the user’s original intent.
- 4.6 Prompt Rewrite: Self-revision compares original and rewritten prompts to improve accuracy and alignment with model capabilities.
- 4.6 Prompt Rewrite: A LoRA-fine-tuned Hunyuan-Large model accelerates and simplifies prompt rewriting for application use.Its training data comes from high-quality rewrite pairs produced by the training-free method.
- 4.7 High-performance Model Fine-tuning: High-performance fine-tuning selects four subsets from the large pretraining dataset to improve dynamic videos, continuous motion control, and character animation.The subsets undergo automated initial screening before fine-tuning.
5 Model Acceleration
HunyuanVideo reduces inference cost through time-step shifting and supports large-scale training with distributed infrastructure, parallelism, memory optimization, and fault tolerance.
- Inference Step Reduction: A larger shifting factor is used at lower inference-step counts to condition the flow model more strongly on early time steps.The factor is set to 7 for 50 steps and increased to 17 below 20 steps.
- Inference Step Reduction: Time-step shifting enables reduced-step generation to match results from numerous inference steps.
- Inference Step Reduction: At 10 inference steps, time-step shifting produces better visual quality than MovieGen’s linear-quadratic scheduler.
- Guidance efficiency: Classifier-free guidance improves sample quality and motion stability but adds unconditional outputs, increasing computational cost and inference latency.The burden is especially high for large models and high-resolution video generation.
- Training infrastructure: AngelPTM and the Tencent XingMai network provide the large-scale training framework and efficient inter-server communication.GPU scheduling is handled through the Tencent Angel machine-learning platform.
- Parallel training: HunyuanVideo combines tensor, sequence, context, and data parallelism with ZeroCache to distribute computation, support long sequences, and reduce memory redundancy.Ring Attention enables context parallelism across multiple GPUs.
- Optimization: FusedAttention accelerates attention computation as sequence length increases.
- Optimization: Recomputation and activation offload trade additional computation or host-memory transfers for lower GPU-memory use.Selected activations are released and recalculated, or moved from GPU memory to host memory.
6 Fundation Model Performance
HunyuanVideo demonstrates strong text alignment, visual quality, motion dynamics, concept generalization, action reasoning, and character writing. Professional evaluation reports the best overall performance among the compared models, especially in motion quality.
- Text Alignment: HunyuanVideo closely follows complex prompts, including scenes with multiple subjects and their relationships.Figure 12 illustrates this capability with a cat, its owner, and a child interacting in one scene.
- High-quality: HunyuanVideo produces ultra-detailed video content after spatial-quality fine-tuning.The paper presents high-quality generated videos in Figure 13.
- High-motion Dynamics: HunyuanVideo generates high-motion videos across diverse scenes and motion types.Figure 14 demonstrates the model’s dynamic-video capability.
- Concept Generalization: HunyuanVideo generalizes to novel scenes that combine concepts absent from its training data.The paper uses an astronaut floating on a gemstone-like lake as an example of an unseen scenario and concept combination.
- Action Reasoning and Planning: HunyuanVideo captures sequential actions photorealistically and generates scene text or gradually appearing handwritten text.Figures 16 and 17 illustrate action reasoning, planning, and text generation.
- Professional Evaluation: 60 professional evaluators found HunyuanVideo best overall across Text Alignment, Motion Quality, and Visual Quality, with particular strength in motion quality.The evaluation used 1,533 prompts, one inference per model under default settings, and consistent resolution.
7 Applications
HunyuanVideo supports applications spanning synchronized video-to-audio generation, image-to-video synthesis, and controllable avatar animation. These extensions combine multimodal conditioning with specialized modules for audio, images, poses, expressions, and hybrid control.
- Audio Generation: The video-to-audio module generates synchronized sound effects and contextually appropriate background music for generated videos.It targets cinematic-grade Foley generation and more cohesive multimedia experiences.
- Audio Generation: The video-to-audio system uses a flow-matching diffusion transformer with triple-stream processing followed by single-stream multimodal fusion.Visual, audio, and textual modalities are encoded in a mel-spectrogram latent framework.
- Image-to-Video: HunyuanVideo extends text-to-video to image-to-video by replacing the first-frame token and injecting semantic image tokens.The reference image latent is used as the first frame with timestep 0, while remaining frames follow text-to-video processing.
- Avatar Animation: The avatar system supports explicit speech, expression, and pose driving alongside implicit text-prompt control.Reference-image latents provide strong guidance for maintaining appearance consistency.
- Expression-Driven Animation: Expression control uses implicit representations injected through cross-attention and a face-region mask.The method uses VASA-derived expression features rather than facial landmarks for expression driving.
- Audio-Driven Animation: Audio-driven animation extends beyond portrait faces to upper-body characters and can model realistic background motion.The system also animates characters talking and gesturing vividly using audio alone.
- Hybrid-Driven Animation: Hybrid control separately drives body motion with skeletal poses and facial expression with implicit representations.The modules are jointly fine-tuned end-to-end to provide richer editability.
- Hybrid-Driven Animation: Hybrid avatar control supports full body and facial motion control, upper-body animation, and both real-human and CGI characters.The reported capabilities are framed as enabling fully controllable and editable avatars.
8 Related Works
Related video-generation work evolved from 3D U-Nets toward latent diffusion models with temporal attention mechanisms. These developments established progressively more efficient approaches to text-based video generation.
- Video Generation Models: VDM extended 2D image-diffusion U-Nets into 3D U-Nets for text-based video generation.It is described as one of the first such extensions.
- Video Generation Models: MagicVideo and Mindscope introduced 1D temporal attention mechanisms to reduce computation in latent diffusion-based video generation.Their designs build on latent diffusion models while adding temporal modeling.
Project Contributors
The report credits dedicated teams and contributors across sponsorship, leadership, infrastructure, data and recaptioning, modeling, architecture, pre-training, downstream tasks, and broader project support.
- Project Leadership: Project sponsors and leaders oversee the HunyuanVideo effort.The listed leaders include Caesar Zhong, Hongfa Wang, Dax Zhou, Songtao Liu, Qinglin Lu, and Yangyu Tao.
- Technical Contributions: Contributors are organized across infrastructure, data and recaptioning, VAE and model distillation, algorithm and architecture, pre-training, and downstream tasks.The report identifies separate contributor groups for these technical areas.
- Acknowledgments: Additional contributors supported the broader HunyuanVideo project.Their names are listed in the contributor acknowledgments.