Source-linked AI summary
OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models
Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, Chao Liang
TL;DR
Existing human-animation methods struggle to scale training data because filtering limits their applicability. OmniHuman mixes motion-related conditions in a DiT framework, using complementary training principles and multimodal conditioning to generate realistic videos across varied inputs and portrait formats, while remaining subject to motion and distribution limitations.
Problem
Existing end-to-end human-animation methods rely on highly filtered data and limited input scenarios, preventing them from scaling training data like general video-generation models.
Method
OmniHuman combines a DiT-based multi-condition model with mixed training over text, audio, and pose, using stronger and weaker conditions to expand and balance training.
Results
OmniHuman produces highly realistic human videos from varied portrait types, aspect ratios, image styles, backgrounds, audio, pose, and combined driving signals.
Takeaways & Limitations
The framework supports flexible human animation from weak signals, especially audio, while improving gesture generation across varied scenarios.
Takeaways & Limitations
Weak audio-motion correlation can still produce uncoordinated or overly expressive movements and unrealistic object interactions, especially for inputs far from the training distribution.
Abstract
from arXiv · showhide
End-to-end human animation, such as audio-driven talking human generation, has undergone notable advancements in the recent few years. However, existing methods still struggle to scale up as large general video generation models, limiting their potential in real applications. In this paper, we propose OmniHuman, a Diffusion Transformer-based framework that scales up data by mixing motion-related conditions into the training phase. To this end, we introduce two training principles for these mixed conditions, along with the corresponding model architecture and inference strategy. These designs enable OmniHuman to fully leverage data-driven motion generation, ultimately achieving highly realistic human video generation. More importantly, OmniHuman supports various portrait contents (face close-up, portrait, half-body, full-body), supports both talking and singing, handles human-object interactions and challenging body poses, and accommodates different image styles. Compared to existing end-to-end audio-driven methods, OmniHuman not only produces more realistic videos, but also offers greater flexibility in inputs. It also supports multiple driving modalities (audio-driven, video-driven and combined driving signals). Video samples are provided on the ttfamily project page (https://omnihuman-lab.github.io)
1. Introduction
Existing end-to-end human animation models achieve realistic results but rely on heavily filtered data and remain limited in scene and input diversity. OmniHuman addresses this scaling challenge by mixing text, audio, and pose conditions during training to use more data and improve generalization.
- Large-scale video-text training helps DiT models learn diverse motion priors and generalize across objects and scenes.
- Existing end-to-end human animation models use highly filtered datasets, limiting them largely to front-facing facial or portrait images with static backgrounds.
- Audio-conditioned data requires filtering because audio correlates weakly with body pose, background motion, camera movement, and lighting.
- Mixing text, audio, and pose conditions lets otherwise discarded data contribute to training while stronger conditions provide guidance absent from audio alone.
- OmniHuman uses two principles: weaker-condition data expands training, while stronger conditions receive lower training ratios to preserve each modality’s learning value.
- OmniHuman generates realistic human motion from reference images and audio across portrait types, aspect ratios, image styles, backgrounds, and auxiliary pose inputs.
2. Related Works
Video generation methods improved with diffusion models, but limited video data constrained quality and human-animation systems remained smaller and less scalable than major generative-model domains.
- Diffusion-model advances made generative models practically usable and pushed research toward video generation.
- Early text-to-video methods adapted pretrained text-to-image models or added temporal layers, but limited video data often produced unsatisfactory quality.
- Early GAN-based human-animation methods used self-supervised video-driven learning on datasets containing tens of thousands of videos.
- Human-animation systems generally used fewer than a thousand hours and 2B-scale structures, without the scaling-law trends seen in LLMs, VLMs, and text/image-to-video tasks.
3. Method
OmniHuman is a multi-condition DiT framework that combines reference appearance with text, audio, and pose signals, then scales training through progressively mixed-condition data. Its architecture and training principles support flexible human-video generation from varied driving inputs.
- Overview: OmniHuman generates human videos from a reference image and text, audio, pose, or combined driving conditions, with audio as the primary focus.
- Overview: The model uses a pretrained MMDiT-based text-to-video backbone and integrates audio and skeleton-map features as driving signals.
- Driving Conditions: Audio tokens are extracted from multi-scale wav2vec features, aligned to video hidden size and frame rate, and injected through frame-wise cross-attention.
- Driving Conditions: Pose tokens are formed from encoded skeleton-map sequences and stacked with noisy latents for visual alignment and dynamic modeling.
- Appearance Conditions: Reference and noisy video latents share the DiT backbone through self-attention, while modified RoPE distinguishes reference tokens without extra parameters.
- Omni-Conditions Training: Mixed-condition training activates modalities according to available data, allowing image-text, image-text-audio, and image-text-audio-pose tasks to contribute.
- Omni-Conditions Training: Principle 1 reuses data excluded from stronger audio-conditioned tasks in weaker text- and image-conditioned tasks to expand training data.
- Omni-Conditions Training: Principle 2 assigns lower training ratios to stronger conditions so weaker conditions remain effective rather than being overshadowed.
4. Experiments
Experiments evaluate OmniHuman’s mixed-condition training through ablations, comparisons with specialized baselines, and visual applications. Results indicate that mixed-condition data improves motion generation and supports broader inputs, tasks, body proportions, and styles.
- Implementation Details: OmniHuman trains on 18.7K hours of human-related data, with 13% selected using lip-sync and pose-visibility criteria for audio and pose modalities.The data is filtered for aesthetics, image quality, and motion amplitude, then compositionally adjusted for omni-condition training.
- Ablation Studies: More text-conditioned data improves FVD, FID, lip-sync accuracy, and gesture richness and quality, while enhancing generalization to different input types.Insufficient text-conditioned data can reduce lip-sync performance, whereas increasing data improves joint modality modeling.
- Ablation Studies: Introducing pose too early harms high-quality motion generation, while training pose at a lower proportion than audio follows the second training principle.The reported proportions are A=50% and P=25%; reversing them to A=50% and P=75% degrades generation quality.
- Ablation Studies: Omni-condition training improves pretrained image-to-video generation by enhancing local human features, overall realism, and movement naturalness.The comparison specifically highlights improved gesture quality relative to the pretrained I2V model.
- Comparisons with Existing Methods: OmniHuman outperforms specialized methods across portrait and body animation tasks while supporting varied input sizes, aspect ratios, and body proportions with one model.It also performs satisfactorily on pose-only driving tasks despite being a unified model for audio- and pose-driven generation.
- Extended Visual Results and Applications: Visual results show compatibility with stylized humanoids, 2D cartoons, nonhuman images, object interactions, singing, and speaker selection through CFG-region adjustment.The model preserves input motion styles, including characteristic mouth movements in anime.
5. Conclusion
OmniHuman is an end-to-end multimodality-conditioned framework that generates realistic human videos from a single image and motion signals across varied body proportions and scenarios.
- OmniHuman generates human videos from a single image conditioned on audio, pose, or combined motion signals.
- Its mixed-data training strategy addresses the scarcity of high-quality data affecting previous human animation methods.
- The framework produces highly realistic results from weak signals, especially audio, across portraits, half-body, and full-body images.
- OmniHuman supports images with any aspect ratio and delivers lifelike results across varied scenarios.
Limitation
OmniHuman reduces audio-variation overfitting through mixed-condition training, but weak audio–motion correlation and distribution shifts can still reduce motion naturalness and interaction realism.
- Mixed-condition training, especially pose conditioning, mitigates overfitting to audio variations.
- Weak correlation between audio and motion can produce uncoordinated or overly expressive movements.
- Object interactions may appear unrealistic, particularly when input images differ substantially from the training distribution.
- A relatively high CFG scale is needed for synthesis stability but can cause some overfitting.
- Future work proposes richer motion conditions, including styles, intensities, and intentions, to improve motion naturalness.