DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation

Motivation

• Although human-centric generation has advanced, existing approaches treat generation, editing, and animation largely as isolated tasks despite their shared mapping of static identity anchors onto dynamic spatio-temporal content.The introduction argues that these tasks can therefore be unified around a consistent data source.Source: §1 · pp. 1–2

Contributions

• DreamID-Omni integrates R2AV, RV2AV, and RA2V in one framework using a Symmetric Conditional DiT that accepts heterogeneous conditioning signals without architectural changes.The signals include reference images, voice timbres, source videos, and driving audio.Source: §1 · pp. 2–3
• Syn-RoPE binds reference identities to voice timbres in attention space, while Structured Captions map specific subjects to attributes and speech content.Together, these signal-level and semantic-level mechanisms target identity–timbre binding failures and speaker confusion in multi-person generation.Source: §1 · pp. 2–3
• Training first focuses on weakly constrained R2AV with in-pair reconstruction and cross-pair disentanglement, then jointly trains R2AV, RV2AV, and RA2V.The staged strategy is intended to prevent overfitting to strongly constrained tasks while preserving R2AV performance.Source: §1 · pp. 2–3
• The paper reports state-of-the-art performance across video, audio, and audio-visual consistency, including comparisons with leading proprietary commercial models.This is the paper’s reported experimental conclusion in the introduction.Source: §1 · p. 3

2.1 Joint Audio-Video Generation

• DiT-based audio-video systems use dual-stream fusion or unified joint self-attention, but generally lack external identity and voice-timbre conditioning for persistent consistency.Earlier work used coupled U-Net backbones; current DiT approaches focus mainly on text-to-audio-video or first-frame-conditioned synthesis.Source: §2.1 · p. 3

2.2 Controllable Video Generation Model

• Reference-based video generation expanded from single-identity methods to multi-subject settings, but remains video-centric and does not support audio generation.The cited single-identity efforts are [18] [39] [62], and later multi-subject work includes [1] [10] [21] [23] [29] [35] [65].Source: §2.2 · p. 3
• Video editing changes identity attributes in source videos, while audio-driven animation generates lip movements matching speech from reference images; both remain task-specific, with no model unifying them and reference-based generation.The cited editing frameworks are [5] [7] [25] [37] [52] [58], and animation methods are [2] [31] [51].Source: §2.2 · p. 3

3.1 Problem Formulation

• The framework models synchronized video-audio output conditioned on text, reference identities, and voice timbres, with optional source-video and driving-audio inputs for editing and animation.Selecting which conditions are supplied enables reference-based generation (R2AV), editing (RV2AV), and animation (RA2V).Source: §3.1 · pp. 3–4

3.2 Framework

• DreamID-Omni uses parallel video and audio DiT backbones linked by bidirectional cross-attention for temporal synchronization and semantic alignment.The video stream synthesizes visuals and the audio stream synthesizes acoustics.Source: §3.2 · p. 4

3.2.1 Symmetric Conditional DiT

• The Symmetric Conditional DiT concatenates identity and voice-reference features with noisy target latents, while adding source-video and driving-audio conditions as structural guidance.Null structural inputs enable switching among R2AV, RV2AV, and RA2V with a unified parameter set.Source: §3.2.1 · pp. 4–5

3.2.2 Dual-Level Disentanglement

• Dual-Level Disentanglement addresses identity-timbre mismatches and subject-attribute misattribution with signal-level Syn-RoPE and semantically structured captions.Syn-RoPE assigns each identity a separate temporal position segment shared across its image and audio features; captions use unique ⟨subk⟩ anchors across video, audio, and joint descriptions.Source: §3.2.2 · pp. 5–6

3.3 Multi-Task Progressive Training

• The three-stage curriculum progresses from masked in-pair R2AV reconstruction to unmasked cross-pair disentanglement, then fine-tunes on mixed R2AV, RV2AV, and RA2V tasks.The initial weakly constrained R2AV stage builds a generative prior intended to support high-fidelity control in the strongly constrained editing and animation tasks without sacrificing generative quality.Source: §3.3 · p. 6

3.4 Inference Pipeline

• At inference, multi-condition Classifier-Free Guidance [20] is applied independently to video and audio, chaining text guidance with stream-specific identity or timbre guidance.The stream-specific condition is identity for video and voice timbre for audio; this text-aligned chaining is intended to stabilize and cohere results.Source: §3.4 · pp. 6–7

4.1 Setup

• IDBench-Omni evaluates unified generation, editing, and animation on 200 instances across three specialized test sets.The sets include 100 identity-timbre-caption triplets, 50 masked videos, and 50 driving audios, spanning multi-person dialogue, identity and timbre variation, and in-the-wild recording. Evaluation covers video fidelity and text following, audio quality and semantics, and audio-visual synchronization and speaker attribution.Source: §4.1 · p. 8

4.2 Comparison

• Across tasks, DreamID-Omni is superior or comparable on R2AV, achieves SOTA video metrics plus strong audio metrics on RV2AV, and leads video metrics with Humo-comparable lip-sync on RA2V.On RA2V, the model animates the correct subject by following structured captions, while Humo and HunyuanCustom frequently misattribute speakers.Source: §4.2 · pp. 8–9, Table 3 · p. 8, Table 4 · p. 8

4.3 Ablation Studies

• Without structured captions, Spk-Conf. more than triples (0.08 to 0.26) and ViCLIP is lowest; removing Syn-RoPE sharply degrades T-Sim. and Sync-C/D.The ablations also show visual-attribute and content mismatches without structured captions, and identity-timbre swaps without Syn-RoPE.Source: §4.3 · p. 10, Table 5 · p. 10, Figure 6 · p. 10
• Only IR yields copy-paste despite high ID-Sim. and T-Sim.; Only CD starts too difficultly, while joint training without OFT weakens R2AV text-following.The findings support establishing a generative prior on weakly constrained R2AV before adding strongly constrained RV2AV and RA2V tasks; otherwise, training can overfit easier tasks and fail to generalize.Source: §4.3 · pp. 10–11, Table 6 · p. 10, Figure 6 · p. 10

5 Conclusion

• DreamID-Omni integrates reference-based generation, editing, and animation into one paradigm for controllable human-centric audio-video generation.The unified framework addresses limitations of previous task-specific models.Source: §5 · p. 11
• DreamID-Omni achieves state-of-the-art performance across multiple tasks on the new IDBench-Omni benchmark.The conclusion reports the overall result without specifying task-level metrics or comparisons.Source: §5 · p. 11
• Syn-RoPE binds identity and timbre at the signal level, while Structured Captioning provides semantic-level disentanglement to tackle multi-person confusion.The conclusion also states that Multi-Task Progressive Training harmonizes disparate objectives.Source: §5 · p. 11

A.1

• The appendix presents qualitative baseline comparisons for RV2AV and RA2V.These comparisons are in Sec. A.1.Source: A Appendix · p. 16

A.2

• The appendix details the data construction pipeline.These details are in Sec. A.2.Source: A Appendix · p. 16

A.3

• The appendix provides the MLLM-based judge prompt.The prompt is in Sec. A.3.Source: A Appendix · p. 16

A.4

• The appendix provides further details about the user study.These details are in Sec. A.4.Source: A Appendix · p. 16

A.5

• The appendix includes additional qualitative results for R2AV, RV2AV, and RA2V.These results are in Sec. A.5.Source: A Appendix · p. 16

A.1 Comparison Results on RV2AV and RA2V

• On RV2AV, qualitative results show DreamID-Omni outperforming state-of-the-art methods.Figure 4 presents the comparison.Source: §A.1 · p. 16
• On RA2V, qualitative results show DreamID-Omni outperforming state-of-the-art methods.Figure 5 presents the comparison.Source: §A.1 · p. 16

• The full dataset consists of approximately 1M high-quality audio-video pairs.Source: §A.2 · p. 16
• In-pair construction extracts reference timbres using DiariZen diarization and reference identities using DWPose face detection and keyframe crops.Source: §A.2 · p. 16
• Cross-pair construction labels speaker segments with DiariZen and Gemini, clones and denoises each speaker’s voice with CosyVoice and ClearerVoice, and builds video identities using Phantom-Data.Source: §A.2 · p. 16

A.3 MLLM-Based Judge

• Gemini-2.5-Pro serves as the MLLM-based judge for Speaker Confusion, with its system prompt presented in Fig. 9.Source: §A.3 · p. 16

A.4 User Study

• The user study reports strong performance across multiple dimensions in R2AV comparisons with state-of-the-art methods.Table 7 identifies the comparison scope but the supplied passage does not provide individual scores or establish a winner.Source: §A.4 · p. 17, Table 7 · p. 18
• Thirty professional video creators evaluated videos in a blinded user study on IDBench-Omni.The study was conducted as part of the evaluation on IDBench-Omni.Source: §A.4 · p. 17
• Evaluators rated each video on seven dimensions using a 1–5 scale, and the ratings were averaged into final scores.Source: §A.4 · p. 17

• Figures 10 and 11 show additional qualitative results for DreamID-Omni on R2AV.Source: §A.5 · p. 18, Figure 10 · p. 19, Figure 11 · p. 20
• Figure 12 presents additional qualitative results for DreamID-Omni on RV2AV.Source: §A.5 · p. 18, Figure 12 · p. 21
• Figure 13 presents additional qualitative results for DreamID-Omni on RA2V.Source: §A.5 · p. 18, Figure 13 · p. 21