Source-linked AI summary
HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning
Liyang Chen, Tianxiang Ma, Jiawei Liu, Bingchuan Li, Zhuowei Chen, Lijie Liu, Xu He, Gen Li, Qian He, Zhiyong Wu
TL;DR
HCVG must coordinate text, image, and audio despite scarce paired triplet data and competing control objectives. HuMo combines a paired multimodal dataset, progressive task-specific training, and adaptive inference guidance, and surpasses specialized state-of-the-art methods across its sub-tasks.
Problem
HCVG lacks paired triplet data and struggles to jointly achieve text following, subject consistency, and audio-visual synchronization.
Method
HuMo combines paired multimodal data, progressive task-specific training, minimally invasive image injection, audio cross-attention, focus-by-predicting, and time-adaptive CFG.
Results
HuMo surpasses specialized state-of-the-art methods on subject preservation and audio-visual synchronization, with validation at 1.7B and 17B parameters.
Takeaways & Limitations
HuMo provides unified collaborative control across text, image, and audio for multimodal human-centric video generation.
Takeaways & Limitations
Realistic multimodal human-video synthesis raises risks of deepfakes, non-consensual content, manipulation, misinformation, and privacy harms.
Abstract
from arXiv · showhide
Human-Centric Video Generation (HCVG) methods seek to synthesize human videos from multimodal inputs, including text, image, and audio. Existing methods struggle to effectively coordinate these heterogeneous modalities due to two challenges: the scarcity of training data with paired triplet conditions and the difficulty of collaborating the sub-tasks of subject preservation and audio-visual sync with multimodal inputs. In this work, we present HuMo, a unified HCVG framework for collaborative multimodal control. For the first challenge, we construct a high-quality dataset with diverse and paired text, reference images, and audio. For the second challenge, we propose a two-stage progressive multimodal training paradigm with task-specific strategies. For the subject preservation task, to maintain the prompt following and visual generation abilities of the foundation model, we adopt the minimal-invasive image injection strategy. For the audio-visual sync task, besides the commonly adopted audio cross-attention layer, we propose a focus-by-predicting strategy that implicitly guides the model to associate audio with facial regions. For joint learning of controllabilities across multimodal inputs, building on previously acquired capabilities, we progressively incorporate the audio-visual sync task. During inference, for flexible and fine-grained multimodal control, we design a time-adaptive Classifier-Free Guidance strategy that dynamically adjusts guidance weights across denoising steps. Extensive experimental results demonstrate that HuMo surpasses specialized state-of-the-art methods in sub-tasks, establishing a unified framework for collaborative multimodal-conditioned HCVG. Project Page: https://phantom-video.github.io/HuMo.
1 Introduction
HuMo addresses data scarcity and conflicting multimodal-control objectives in HCVG with paired triplet data, progressive training, and time-adaptive inference guidance.
- 1 Introduction: HuMo targets collaborative control across text, reference images, and audio while supporting text-image, text-audio, and text-image-audio inputs.The framework is designed for flexible multimodal human-centric video generation.
- 1 Introduction: HuMo constructs a high-quality dataset with paired text, reference images, and audio through a multimodal data-processing pipeline.The pipeline retrieves semantically matched reference images and filters video data for audio synchronization.
- 1 Introduction: HuMo progressively learns subject preservation and audio-visual synchronization with task-specific strategies while preserving the foundation model’s text-following ability.Image injection is minimally invasive, while audio is incorporated through cross-attention and focus-by-predicting.
- 1 Introduction: Time-adaptive classifier-free guidance dynamically adjusts modality guidance strengths across denoising steps for flexible and fine-grained control.The strategy coordinates text following, subject preservation, and audio-visual synchronization during inference.
- 1 Introduction: HuMo surpasses specialized state-of-the-art methods on subject preservation and audio-visual synchronization and is validated at 1.7B and 17B parameters.The reported significance result covers both sub-tasks and two model scales.
2 Related Works
Prior HCVG work separately emphasizes audio-driven animation or subject-consistent generation, while HuMo combines these capabilities within a progressive multimodal framework.
- 2.1 Audio-Driven Human Animation: Audio-driven human animation generates lip movements matching speech but typically requires a subject-complete start frame with visible facial features.This requirement limits user creativity despite strong facial-animation and body-motion performance.
- 2.2 Subject-Consistent Video Generation: Subject-consistent video generation uses reference images and text prompts to preserve subject appearance while generating text-aligned videos.Existing approaches use semantic encoders, adapters, DiT cross-attention, or in-context reference-image conditioning.
- 2.2 Subject-Consistent Video Generation: HuMo progressively learns subject preservation and audio-visual synchronization from a DiT-based text-to-video backbone.Figure 3 presents the framework’s staged training and multimodal data-processing pipeline.
3 Methodology
HuMo extends a DiT-based text-to-video model into a unified framework for text, reference-image, and audio control through paired multimodal data, progressive training, and adaptive inference guidance.
- 3.1 Preliminaries: HuMo extends a DiT-based T2V backbone with text, image, and audio conditions for collaborative multimodal video generation.The model uses flow matching over noisy video latents and multimodal conditioning.
- 3.2 Multimodal Dataset: A staged data pipeline builds paired text-reference-image samples and adds tightly aligned audio-visual pairs for training.Reference images are cross-paired to vary visual attributes, while speech segments are filtered using lip-sync analysis.
- 3.3 Progressive Multimodal Training: Progressive training first establishes text-image controllability, then jointly learns audio-visual synchronization while retaining earlier subject-preservation ability.The two stages use corresponding dataset subsets and progressively incorporate the audio-visual sync task.
- 3.3 Progressive Multimodal Training: Minimal-invasive image injection preserves the backbone architecture by appending reference-image latents to noisy video latents and limiting parameter updates.Reference latents are placed at the end of the video sequence to avoid treating them as a start frame.
- 3.3 Progressive Multimodal Training: Focus-by-predicting adds a facial-region mask predictor after late audio cross-attention blocks, using binary cross-entropy supervision as a soft regularizer.The design guides audio attention toward facial regions without hard-gating full-body motion and interactions.
- 3.4 Inference Strategies: Time-adaptive CFG changes modality guidance during denoising, supporting text-image, text-audio, and text-image-audio combinations with null tokens for absent conditions.Early denoising emphasizes semantic layout, while later stages emphasize identity and audio-visual synchronization.
4 Experiment
HuMo is evaluated against specialized and multimodal baselines across subject preservation, audio-visual sync, ablations, and controllability settings. The experiments report strong qualitative and quantitative performance, while testing progressive training, inference guidance, and multimodal editing.
- Multimodal Conditioned Comparison: The audio-visual sync comparison uses the MoCha benchmark and includes S2V and audio-visual methods, with I2V baselines receiving subject-complete start frames.HunyuanCustom and HuMo instead use cropped faces from the start frame, while OmniHuman-1 lacks text input through its official API.
- Multimodal Conditioned Comparison: HuMo preserves reference identities while following text edits, including changes to clothing, accessories, makeup, scenes, and prompts involving multiple subjects.Qualitative results describe four-person identity preservation, text-driven appearance changes, and controllable scene generation.
- Multimodal Conditioned Comparison: HuMo surpasses specialized methods on subject preservation and audio-visual sync, with strong text following, subject consistency, and audio-visual correspondence.The supplied passages report highest or superior results across the relevant qualitative and quantitative comparisons, although the complete table values are not provided.
- Method Analysis: Removing progressive training degrades most metrics and identity similarity, while removing focus-by-predicting lowers Sync-C and misaligns lip movements.The ablations link progressive training to multimodal coordination and face-location prediction to audio-visual correspondence.
- Method Analysis: Time-adaptive CFG prioritizes text-driven layout early and identity preservation later, adapting guidance to changing denoising-stage roles.The strategy uses different guidance emphases across generation stages rather than a static configuration.
5 Conclusion
HuMo unifies text, image, and audio control for human-centric video generation through paired multimodal data, progressive training, and adaptive inference guidance. The framework supports text following, subject preservation, and audio-visual synchronization in short-video creation.
- 5 Conclusion: HuMo combines paired text-image-audio data, progressive multimodal training, and time-adaptive CFG into one collaborative generation framework.The framework integrates modality-specific capabilities and dynamically adjusts guidance during inference.
- 5 Conclusion: HuMo satisfies text prompt following, subject preservation, and audio-visual sync requirements for human-centric short video creation.The conclusion presents these as the framework’s supported capabilities across the three modalities.
6 Ethical Considerations
HuMo’s realistic multimodal human-video synthesis raises ethical concerns involving deepfakes, non-consensual content, manipulation, misinformation, likeness protection, consent, privacy, and transparency.
- 6 Ethical Considerations: Realistic multimodal human-video synthesis may enable deepfakes or non-consensual content, requiring consent and protection of individuals’ likenesses.The stated concern follows from HuMo’s ability to synthesize human videos from text, images, and audio.
- 6 Ethical Considerations: Fine-grained control over generated content calls for responsible-use guidelines addressing manipulation, misinformation, transparency, data privacy, and harm prevention.The passage explicitly identifies these safeguards as ethical standards for developers and users.