Source-linked AI summary
DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation
Xu Guo, Fulong Ye, Qichao Sun, Liyang Chen, Bingchuan Li, Pengze Zhang, Jiawei Liu, Songtao Zhao, Qian He, Xiangwang Hou
TL;DR
Existing human-centric audio-video methods generally isolate generation, editing, and animation, while multi-person identity-timbre binding remains difficult. DreamID-Omni unifies these tasks with symmetric conditioning, dual-level disentanglement, and progressive multi-task training, and reports state-of-the-art performance across video, audio, and audio-visual consistency.
Problem
Existing approaches largely isolate R2AV, RV2AV, and RA2V, while multi-person generation suffers identity-timbre mismatch and attribute-content misattribution.
Method
DreamID-Omni combines a Symmetric Conditional DiT, Syn-RoPE and Structured Captions, and progressive training across weakly and strongly constrained tasks.
Results
DreamID-Omni achieves state-of-the-art performance across multiple tasks and across video, audio, and audio-visual consistency on IDBench-Omni.
Takeaways & Limitations
The framework provides a single paradigm for controllable human-centric audio-video generation spanning reference-based generation, editing, and animation.
Abstract
from arXiv · showhide
Recent advancements in foundation models have revolutionized joint audio-video generation. However, existing approaches typically treat human-centric tasks including reference-based audio-video generation (R2AV), video editing (RV2AV) and audio-driven video animation (RA2V) as isolated objectives. Furthermore, achieving precise, disentangled control over multiple character identities and voice timbres within a single framework remains an open challenge. In this paper, we propose DreamID-Omni, a unified framework for controllable human-centric audio-video generation. Specifically, we design a Symmetric Conditional Diffusion Transformer that integrates heterogeneous conditioning signals via a symmetric conditional injection scheme. To resolve the pervasive identity-timbre binding failures and speaker confusion in multi-person scenarios, we introduce a Dual-Level Disentanglement strategy: Synchronized RoPE at the signal level to ensure rigid attention-space binding, and Structured Captions at the semantic level to establish explicit attribute-subject mappings. Furthermore, we devise a Multi-Task Progressive Training scheme that leverages weakly-constrained generative priors to regularize strongly-constrained tasks, preventing overfitting and harmonizing disparate objectives. Extensive experiments demonstrate that DreamID-Omni achieves comprehensive state-of-the-art performance across video, audio, and audio-visual consistency, even outperforming leading proprietary commercial models. We will release our code to bridge the gap between academic research and commercial-grade applications.
1 Introduction
DreamID-Omni targets the fragmented landscape of controllable human-centric audio-video generation by unifying generation, editing, and animation while addressing multi-person identity-timbre confusion and task conflicts.
- Motivation: Existing human-centric approaches use references or stronger inputs for separate generation, editing, and animation tasks, but largely treat these capabilities in isolation.The paper identifies a missing unified perspective for joint audio-video generation.
- Framework: The framework integrates R2AV, RV2AV, and RA2V through a Symmetric Conditional DiT that shares heterogeneous conditioning signals across tasks.Conditioning includes reference images, voice timbres, source videos, and driving audio, enabling task switching without architectural changes.
- Disentanglement: Dual-Level Disentanglement addresses multi-person identity-timbre binding and speaker confusion at both signal and semantic levels.Syn-RoPE binds identities to voice timbres in attention space, while Structured Captions map subjects to attributes and speech content.
- Training: Multi-Task Progressive Training begins with weakly constrained R2AV learning before jointly introducing strongly constrained RV2AV and RA2V tasks.This ordering is designed to harmonize tasks with different constraint strengths and prevent overfitting to strongly constrained objectives.
- Results: Extensive experiments report comprehensive state-of-the-art performance across video, audio, and audio-visual consistency, including comparisons with proprietary commercial models.The contribution summary presents this as the framework’s overall empirical result.
2 Related Work
Related work spans joint audio-video synthesis, reference-based generation, video editing, and audio-driven animation, but the reviewed approaches remain specialized or incomplete for unified human-centric audio-video control.
- Joint Audio-Video Generation: Joint audio-video systems use coupled U-Nets, dual-stream DiTs with fusion layers, or unified DiTs with joint self-attention for multimodal alignment.The passage contrasts architectural approaches used for synchronized audio-video synthesis.
- Reference-based Generation: Reference-based generation has expanded from single-identity to multi-subject video control, but these works typically do not support audio generation.Their central goal is maintaining identity consistency by integrating reference features into diffusion.
- Video Editing and Animation: Video editing modifies identity attributes in source videos, whereas audio-driven animation generates lip movements from reference images and speech signals; both remain task-specific.The related approaches provide temporal control but are not presented as a unified model.
3 Methodology
DreamID-Omni formulates controllable human-centric audio-video generation as a unified framework covering reference-based generation, editing, and animation. It combines symmetric conditioning, dual-level disentanglement, and progressive multi-task training to coordinate heterogeneous controls and reduce multi-person confusion.
- Problem Formulation: DreamID-Omni models synchronized video-audio generation conditioned on text, reference identities, voice timbres, and optional source-video or driving-audio inputs.Selective structural conditions support reference-based generation, editing, and animation within one conditional distribution.
- Symmetric Conditional DiT: The dual-stream DiT uses bidirectional cross-attention and symmetric conditional sequences to integrate identity-specific and structural guidance.Reference features are concatenated with noisy latents, while structural conditions enter through element-wise addition, separating identity preservation from structural guidance.
- Dual-Level Disentanglement: Syn-RoPE binds each reference identity to its voice timbre in attention space by synchronizing audio-video positions and reserving positional margins for identity slots.The method scales audio RoPE frequencies by γ = L_v/L_a and partitions temporal positional indices among target and reference sequences.
- Dual-Level Disentanglement: Structured Captions use anchor tokens and separate video, audio, and joint fields to map subjects explicitly to their attributes, motions, and speech content.Consistent subject tokens ⟨sub_k⟩ across fields address semantic ambiguity in multi-person scenarios.
- Multi-Task Progressive Training: The three-stage curriculum begins with R2AV in-pair reconstruction, adds cross-pair disentanglement, and finally jointly fine-tunes R2AV, RV2AV, and RA2V.Masked reconstruction computes loss on unmasked regions, while cross-pair training separates abstract identity and timbre concepts from source-specific correlations.
- Multi-Task Progressive Training: Progressive training uses a weakly constrained R2AV generative prior as the foundation for high-fidelity strongly constrained editing and animation.The stated goal is to learn conditional control without sacrificing generative quality or overfitting to strongly constrained tasks.
4 Experiments
Experiments evaluate DreamID-Omni on unified generation, editing, and animation using IDBench-Omni and task-specific comparisons. The results report strong cross-modal performance, while ablations support dual-level disentanglement and progressive training.
- 4.1 Setup: IDBench-Omni contains 200 instances across generation, video editing, and audio-driven animation, including multi-person dialogues and in-the-wild conditions.It comprises 100 identity-timbre-caption triplets, 50 masked videos, and 50 driving audios.
- 4.1 Setup: DreamID-Omni evaluates video quality, text following, identity, audio quality and fidelity, lip synchronization, and multi-person speaker confusion.Metrics include AES, ViCLIP, ID-Sim., WER, T-Sim., Sync-C, Sync-D, and Spk-Conf.
- 4.2 Comparison: On R2AV, DreamID-Omni achieves superior or comparable video, audio, and audio-visual consistency, with correct identity-timbre binding in qualitative multi-person cases.It is reported as more realistic than Wan2.6 in one case and more identity-consistent than Ovi and LTX-2 in another.
- 4.2 Comparison: On RV2AV, DreamID-Omni achieves state-of-the-art video-centric performance while also producing strong audio quality, fidelity, and synchronization.Compared methods do not support audio generation, so audio metrics are reported only for DreamID-Omni.
- 4.2 Comparison: On RA2V, DreamID-Omni matches Humo on lip-sync accuracy, leads video-related metrics, and follows structured captions to animate the correct subject.Humo and HunyuanCustom frequently misattribute speakers in multi-subject scenarios.
- 4.3 Ablation Studies: Removing Structured Captions triples speaker confusion from 0.08 to 0.26, while removing Syn-RoPE sharply degrades timbre preservation and lip-sync accuracy.The ablation supports complementary semantic subject-attribute mapping and signal-level identity-timbre binding.
- 4.3 Ablation Studies: Training only with In-pair Reconstruction causes copy-paste behavior, while training only with Cross-pair Disentanglement produces very low identity and timbre similarity.Joint training from scratch without Omni-Task Fine-tuning also underperforms on R2AV text following.
- 4.3 Ablation Studies: Without progressive training, the model overfits easier strongly-constrained tasks and fails to generalize to more complex weakly-constrained generation tasks.The experiments support establishing a strong R2AV generative prior before introducing RV2AV and RA2V.
5 Conclusion
DreamID-Omni unifies reference-based generation, editing, and animation for controllable human-centric audio-video generation. Its disentanglement and progressive-training strategies address multi-person confusion and task harmonization, with state-of-the-art performance reported across multiple tasks.
- DreamID-Omni integrates reference-based generation, video editing, and audio-driven animation into one human-centric audio-video framework.
- Syn-RoPE provides signal-level identity-timbre binding, while Structured Captioning provides semantic-level disentanglement for multi-person generation.
- Experiments on IDBench-Omni demonstrate state-of-the-art performance across multiple tasks.
A Appendix
The appendix organizes supplementary material covering qualitative comparisons, data construction, judging prompts, user-study details, and additional qualitative results.
- Supplementary Section A.1 provides qualitative comparisons with baselines on RV2AV and RA2V.
- Supplementary Section A.2 details the data construction pipeline.
- Supplementary Sections A.3 and A.4 provide the MLLM-based judge prompt and additional user-study details.
- Supplementary Section A.5 provides more qualitative results for R2AV, RV2AV, and RA2V.
A.1 Comparison Results on RV2AV and RA2V
Figures 4 and 5 qualitatively compare DreamID-Omni with state-of-the-art methods on RV2AV and RA2V, respectively, and demonstrate superior performance.
- Figures 4 and 5 compare DreamID-Omni with state-of-the-art methods on RV2AV and RA2V, respectively.
- The qualitative results demonstrate superior performance for DreamID-Omni across both tasks.
A.2 Data Construction Details
The dataset contains approximately 1M high-quality audio-video pairs, constructed through a pipeline with in-pair data construction as one primary stage.
- The full dataset contains approximately 1M high-quality audio-video pairs.
- The data construction pipeline is categorized into two primary stages.
- In-pair data construction: In-pair construction extracts internal references from each video clip.Speaker diarization produces reference voice-timbre timestamps, while DWPose detects and crops face regions for reference identities.
A.3 MLLM-Based Judge
The evaluation uses Gemini-2.5-Pro as an MLLM-based judge for speaker confusion, with prompts that combine reference images and user instructions for joint audio-video generation.
- Gemini-2.5-Pro serves as an MLLM-based judge for Speaker Confusion.
- Structured Caption: The structured-caption prompt combines visual understanding of reference images with user prompts for joint video-audio generation.User prompts may include visual descriptions, audio descriptions, and quoted speech content.
- Input Data: The prompt accepts one or more reference images that may contain one or more persons.
A.4 User Study
The user study evaluates IDBench-Omni with 30 professional video creators, blinded ratings across seven dimensions, and averaged 1–5 scores; Table 7 reports strong performance across multiple dimensions.
- Thirty professional video creators evaluated the videos in the IDBench-Omni user study.
- Evaluators rated each video on seven dimensions using a 1–5 scale, with averaged ratings producing final scores.
- The study was conducted in a blinded setting.
- Table 7 indicates strong performance across multiple dimensions.
A.5 More Visual Results
This section presents additional qualitative results from DreamID-Omni across the R2AV, RV2AV, and RA2V tasks, organized in Figures 10–13.
- Figures 10 and 11 show additional qualitative results for the R2AV task.
- Figure 12 shows additional qualitative results for the RV2AV task.
- Figure 13 shows additional qualitative results for the RA2V task.