Source-linked AI summary
HunyuanCustom: A Multimodal-Driven Architecture for Customized Video Generation
Teng Hu, Zhentao Yu, Zhengguang Zhou, Sen Liang, Yuan Zhou, Qin Lin, Qinglin Lu
TL;DR
Customized video generation must preserve specific subjects while following flexible conditions, but existing approaches struggle with identity consistency and limited modalities. HunyuanCustom builds on HunyuanVideo with multimodal identity-preserving conditioning, and experiments report superior performance across customization tasks and modalities.
Problem
Customized video generation seeks subject-specific videos under flexible user conditions, while existing methods struggle with identity consistency and limited input modalities.
Method
HunyuanCustom combines LLaVA-based text-image fusion and temporal image-ID enhancement with modality-specific audio and video condition injection built on HunyuanVideo.
Results
HunyuanCustom outperforms existing open- and closed-source methods across single- and multi-subject generation, with reported strengths in ID consistency, authenticity, and video-text alignment.
Takeaways & Limitations
The framework supports identity-centric customized video generation across image, audio, video, and text conditions, including audio- and video-driven tasks.
Abstract
from arXiv · showhide
Customized video generation aims to produce videos featuring specific subjects under flexible user-defined conditions, yet existing methods often struggle with identity consistency and limited input modalities. In this paper, we propose HunyuanCustom, a multi-modal customized video generation framework that emphasizes subject consistency while supporting image, audio, video, and text conditions. Built upon HunyuanVideo, our model first addresses the image-text conditioned generation task by introducing a text-image fusion module based on LLaVA for enhanced multi-modal understanding, along with an image ID enhancement module that leverages temporal concatenation to reinforce identity features across frames. To enable audio- and video-conditioned generation, we further propose modality-specific condition injection mechanisms: an AudioNet module that achieves hierarchical alignment via spatial cross-attention, and a video-driven injection module that integrates latent-compressed conditional video through a patchify-based feature-alignment network. Extensive experiments on single- and multi-subject scenarios demonstrate that HunyuanCustom significantly outperforms state-of-the-art open- and closed-source methods in terms of ID consistency, realism, and text-video alignment. Moreover, we validate its robustness across downstream tasks, including audio and video-driven customized video generation. Our results highlight the effectiveness of multi-modal conditioning and identity-preserving strategies in advancing controllable video generation. All the code and models are available at https://hunyuancustom.github.io.
1 Introduction
HunyuanCustom addresses limited controllability and subject inconsistency in customized video generation with a subject-centered framework supporting image, audio, video, and text conditions. Its experiments report stronger performance across single- and multi-subject settings and downstream audio- and video-driven customization.
- Customized video generation remains difficult because current models struggle to satisfy specific user requirements for fine-grained control.
- Existing methods face limitations including arbitrary-object handling, subject consistency, video quality, and reliance on single-modality image inputs.
- HunyuanCustom builds on HunyuanVideo to generate subject-consistent videos conditioned on images, audio, video, and text.
- Its image-text pipeline combines a LLaVA-based fusion module with temporal image-ID enhancement to integrate modalities and reinforce identity features across frames.
- Experiments cover single- and multi-subject consistency, key quality metrics, and audio- and video-driven customization, with HunyuanCustom outperforming existing methods.
2 Related Work
Video generation has progressed from image synthesis toward large-scale spatiotemporal modeling, while customization methods increasingly inject identity information into pretrained video models. The framework overview depicts HunyuanCustom's multimodal conditioning scope.
- Diffusion-based video generation has advanced from static image synthesis to dynamic spatiotemporal modeling through large-scale video-text frameworks.
- Instance-specific customization fine-tunes pretrained video models separately for each identity using several same-identity images.
- End-to-end customization trains additional conditioning networks so models can generalize to arbitrary identity images during inference.
- Figure 2 presents HunyuanCustom as a framework for identity-consistent videos conditioned on text, images, audio, and video.
3 Method
HunyuanCustom is a multi-modal customization framework built on HunyuanVideo that generates subject-consistent videos from text, images, audio, and video. Its method combines identity-aware conditioning with a quality-controlled, multi-subject-capable data pipeline.
- Multi-modal task: HunyuanCustom uses distinct condition-injection mechanisms for audio and video, disentangling these modalities from image-level identity injection.This design supports independent control over image, audio, and video conditions.
- Multi-modal task: HunyuanCustom supports text-driven generation plus image-, audio-, and video-driven customization, including human, non-human, and multiple identities.Image conditioning guides identity-specific video generation; audio conditioning synchronizes subject actions, while video conditioning supports object replacement or insertion.
- Multi-Modal Data Construction: The data pipeline filters and standardizes videos, extracts single or multiple subjects, augments masks, and enriches captions with structured annotations.Processing includes single-shot segmentation, text-region filtering, subject extraction, resolution standardization, mask dilation or bounding-box conversion, and detailed video annotation.
- Multi-Modal Data Construction: Training data quality is treated as important for subject consistency, video editing, and audio-driven generation, with audio-video synchronization and visual-quality filtering applied to audio data.Videos with synchronization confidence below 3 or hyperIQA scores below 40 are removed, and audio-video offsets are adjusted to zero.
- Image-driven Video Customization: A LLaVA-based image-text interaction module integrates visual identity information with textual descriptions for image-conditioned video generation.The module jointly models image and text inputs to improve understanding of the identity and accompanying description.
- Image-driven Video Customization: An identity enhancement module concatenates image and noisy video latents along the temporal sequence to propagate identity information across generated frames.The image is resized to the video-frame size, encoded with HunyuanVideo’s causal 3DVAE, and assigned a preceding temporal position using 3D-RoPE.
4 Multimodal subject-centric video generation
HunyuanCustom extends subject-consistent video customization across audio and video conditions while separating modality-specific injections from image identity conditioning. Audio features are temporally aligned and spatially injected, while conditioning videos are feature-aligned and added directly to video latents for efficient fusion.
- Multimodal subject-centric video generation: HunyuanCustom extends subject-centric customization to audio and video conditions using distinct injection mechanisms that remain independent of image-level identity injection.This multimodal design supports audio-driven and video-driven customized video generation while centering subject identity.
- Audio-driven video customization: AudioNet extracts multi-level audio features and aligns them with the temporally compressed video latent before conditioning generation.Audio features are padded and aggregated every four frames to account for the uncompressed identity frame and the VAE temporal compression ratio of 4.
- Audio-driven video customization: The ID-enhanced video latent contains f + 1 frames because the identity image is inserted as an uncompressed initial frame.This structure requires audio features to be padded and regrouped before injection.
- Audio-driven video customization: Spatial cross-attention injects aligned audio features into each video frame while decoupling temporal from spatial dimensions to reduce inter-frame interference.The attention operates along width and height rather than mixing audio and video information across frames.
- Video-driven video customization: For video conditioning, HunyuanCustom compresses the clean conditioning video, aligns its features with noisy video latents, and injects them without concatenating sequences.The strategy avoids the doubled sequence length and quadratic attention growth associated with temporal concatenation.
- Video-driven video customization: The addition-based video injection method preserves content more effectively than concatenation after feature alignment, enabling efficient and lightweight conditioning.Experiments report substantial information loss for concatenation-based injection, whereas direct addition facilitates fusion and information transfer.
5 Experiment
HunyuanCustom is evaluated across identity preservation, alignment, temporal consistency, and dynamic degree, including single-subject, multi-subject, audio-driven, video-driven, and application settings. Across these experiments, it achieves strong subject consistency, visual quality, prompt adherence, and editing integration.
- Evaluation metrics: Evaluation measures identity preservation, subject similarity, text-video alignment, temporal consistency, and object movement.The metrics use ArcFace, DINO-v2, CLIP-B, and VBench-based procedures.
- Baselines: The comparison includes commercial systems and open-source methods, with 100 human-identity and 100 nonhuman-identity videos generated for each model.The evaluation is designed to assess general customization ability across human and nonhuman subjects.
- Single-subject customization: HunyuanCustom provides high identity consistency alongside good generation quality and diversity, while competing methods exhibit issues including copy-paste artifacts, blur, subtitle errors, or incomplete body consistency.The qualitative comparison attributes video quality partly to the HunyuanVideo-13B base model.
- Single-subject customization: HunyuanCustom achieves the best ID consistency and subject consistency, with comparable prompt-following and temporal-consistency results.The table reports Face-Sim, DINO-Sim, CLIP-B-T, Temp-Consis, and DD as evaluation metrics.
- Multi-subject customization: In multi-subject customization, HunyuanCustom captures human and nonhuman identities, follows prompts, and maintains visual quality and stability.The comparison covers commercial software and open-source methods.
- Applications: HunyuanCustom generates virtual-human advertisements that preserve human identity and product details while producing natural interaction and close prompt adherence.Inputs include a human image and a product image.
- Audio-driven customization: Audio-driven customization supports diverse text-specified scenes and postures while preserving identity and enabling interaction with other subjects or humans.The paper identifies live streaming and advertising as application contexts.
- Audio-driven customization: Audio-driven multi-subject virtual try-on preserves target identities, applies specified attire, and synchronizes with the provided audio.This combines text prompts and audio for virtual try-on generation.
6 Conclusion
HunyuanCustom is presented as a multimodal framework for identity-centric customized video generation using image, audio, video, and text conditions. Extensive experiments report stronger performance across single-subject, multi-subject, audio-driven, and video-driven tasks, particularly for identity consistency, authenticity, and video-text alignment.
- Conclusion: HunyuanCustom combines image, audio, and video modalities with text conditioning to generate high-quality videos with identity consistency and flexible user control.Its design integrates text-image fusion, image ID enhancement, and audio/video feature injection.
- Conclusion: Extensive experiments report that HunyuanCustom outperforms existing methods across single-subject, multi-subject, audio-driven, and video-driven customization tasks.The reported advantages concern ID consistency, authenticity, and video-text alignment.
7 Contributors and Ackonwledgements
The section lists project leaders, core contributors, and acknowledgements for collaborators who provided valuable input.
- Contributors: Qinglin Lu, Qin Lin, and Yuan Zhou are listed as project leaders.The section separately identifies core contributors and acknowledged collaborators.
- Contributors: Zhentao Yu, Zhengguang Zhou, Teng Hu, and Sen Liang are listed as core contributors.Additional individuals are acknowledged for valuable inputs.
- Acknowledgements: The acknowledgements thank Yi Chen, Zixiang Zhou, Hongmei Wang, Yuanbo Peng, Zunnan Xu, Linqing Wang, Yifu Sun, and Sihuan Lin.They are credited for valuable inputs.