Source-linked AI summary
DreamVideo-Omni: Omni-Motion Controlled Multi-Subject Video Customization with Latent Identity Reinforcement Learning
Yujie Wei, Xinyu Liu, Shiwei Zhang, Hangjie Yuan, Jinbo Xing, Zhekai Chen, Xiang Wang, Haonan Qiu, Rui Zhao, Yutong Feng, Ruihang Chu, Yingya Zhang, Yike Guo, Xihui Liu, Hongming Shan
TL;DR
DreamVideo-Omni targets the unresolved problem of jointly preserving multi-subject identity and controlling motion at multiple granularities. It introduces a progressive unified framework with structured conditioning, identity-aware embeddings, and latent reward feedback, and reports superior identity preservation and motion control across benchmarks.
Problem
Existing unified video-generation methods provide limited motion granularity, ambiguous multi-subject control, and degraded identity fidelity during large-amplitude motion.
Method
DreamVideo-Omni uses a two-stage unified DiT framework with heterogeneous motion and appearance conditioning, condition-aware 3D RoPE, hierarchical injection, group and role embeddings, and a VDM-based latent identity reward model.
Results
DreamVideo-Omni outperforms baselines across single- and multi-subject scenarios, including a 0.395 mIoU increase and nearly 70% lower EPE than Tora.
Takeaways & Limitations
The framework simultaneously supports high-fidelity subject customization and precise control of global, local, and camera motion in zero-shot settings.
Abstract
from arXiv · showhide
While large-scale diffusion models have revolutionized video synthesis, achieving precise control over both multi-subject identity and multi-granularity motion remains a significant challenge. Recent attempts to bridge this gap often suffer from limited motion granularity, control ambiguity, and identity degradation, leading to suboptimal performance on identity preservation and motion control. In this work, we present DreamVideo-Omni, a unified framework enabling harmonious multi-subject customization with omni-motion control via a progressive two-stage training paradigm. In the first stage, we integrate comprehensive control signals for joint training, encompassing subject appearances, global motion, local dynamics, and camera movements. To ensure robust and precise controllability, we introduce a condition-aware 3D rotary positional embedding to coordinate heterogeneous inputs and a hierarchical motion injection strategy to enhance global motion guidance. Furthermore, to resolve multi-subject ambiguity, we introduce group and role embeddings to explicitly anchor motion signals to specific identities, effectively disentangling complex scenes into independent controllable instances. In the second stage, to mitigate identity degradation, we design a latent identity reward feedback learning paradigm by training a latent identity reward model upon a pretrained video diffusion backbone. This provides motion-aware identity rewards in the latent space, prioritizing identity preservation aligned with human preferences. Supported by our curated large-scale dataset and the comprehensive DreamOmni Bench for multi-subject and omni-motion control evaluation, DreamVideo-Omni demonstrates superior performance in generating high-quality videos with precise controllability.
I. INTRODUCTION
DreamVideo-Omni addresses the challenge of preserving multiple subject identities while controlling global, local, and camera motion. It combines unified conditioning, ambiguity-resolving embeddings, latent identity rewards, and dedicated training and evaluation resources.
- Multi-subject video generation requires preserving several identities while controlling global object motion, local limb dynamics, and camera movements.
- Existing methods separately emphasize appearance injection or motion control, often lacking precise spatial control or identity preservation.
- DreamVideo-Omni uses a progressive two-stage paradigm combining reference subjects, global boxes, local trajectories, and camera controls in one DiT.
- Group and role embeddings bind motion signals to corresponding identities, while condition-aware 3D RoPE and hierarchical motion injection support heterogeneous, fine-grained control.
- Experiments report superior identity preservation and motion control, while the framework also unlocks zero-shot image-to-video generation and first-frame-conditioned trajectory control.
- The framework trains on 2M video clips and introduces DreamOmni Bench with 1,027 annotated real-world videos for unified zero-shot evaluation.
II. RELATED WORK
Related work spans customized video generation and controllable motion generation, but the cited approaches address these capabilities through distinct lines of research.
- Customized video generation: Customized video generation studies identity consistency from subject or facial images, including challenging multi-subject video customization.
- Motion-controlled video generation: Motion-control methods condition video generation on trajectories, coordinates, object masks, or bounding boxes for camera and object-motion control.
III. OUR MEHTOD: DREAMVIDEO-OMNI
DreamVideo-Omni adapts a pretrained text-to-video DiT into a unified system that jointly conditions subject appearance, object motion, and camera movement. Its progressive training and hierarchical signal injection support compositional generation without test-time fine-tuning.
- Model Architecture and Task Design: The unified video diffusion transformer conditions jointly on reference appearances, global and local object motions, and camera movements.
- Model Architecture and Task Design: Stage 1 trains an all-in-one video DiT on single- and multi-subject customization, global and local motion control, and camera movement control.
- Conditioning Signals: The conditioning signals include segmented subject reference images and scene-anchored bounding boxes that encode position, scale, aspect ratio, and relative depth.
- Progressive Training: Stage 2 applies latent identity reward feedback learning to improve identity fidelity during dynamic motion generation.
- Conditioning Signal Injection: Hierarchical motion injection adds bounding-box latents to noisy input latents and DiT-block outputs through learnable zero-convolutions.
3) Specialized Architectural Components :
The architecture coordinates heterogeneous tokens with condition-aware temporal indexing and binds each subject’s reference, box, and trajectory controls through group and role embeddings.
- Condition-aware 3D RoPE: Condition-aware 3D RoPE preserves spatial indexing while assigning distinct temporal indices to video, reference-image, padding, and trajectory tokens.
- Condition-aware 3D RoPE: Video and trajectory tokens share frame-aligned temporal indices, while reference tokens receive a shared static-condition index and padding tokens receive an invalid index.
- Group and Role Embeddings: Group embeddings assign each ⟨Reference Subject, Global Box, Local Trajectory⟩ unit a shared identity association across its corresponding latent regions.
- Group and Role Embeddings: Role embeddings distinguish the functional roles of heterogeneous inputs, complementing group embeddings to reduce multi-subject control ambiguity.
- Training Objective: The training objective conditions denoising on text, reference-image, box, and trajectory features while balancing its terms with λ1.
B. Latent Identity Reinforcement Learning
DreamVideo-Omni introduces latent identity reinforcement learning to improve identity fidelity during motion generation. Its Latent Identity Reward Model evaluates identity consistency in latent video representations using spatiotemporal information and human preference data.
- Latent Identity Reward Model: The latent identity reinforcement learning stage supplements supervised fine-tuning to improve identity fidelity while aligning video customization with human preferences.The method trains a latent identity reward model for reward feedback learning.
- Latent Identity Reward Model: LIRM evaluates videos directly in latent space rather than RGB space, reducing computational overhead for reward feedback learning.Its architecture includes a video diffusion model backbone, identity cross-attention, and a reward prediction head.
- Motion-Aware Reward Modeling: LIRM leverages a video diffusion model’s spatiotemporal priors to assess identity consistency together with motion dynamics.This design is intended to penalize copy-paste artifacts and preserve identity under large motion.
- Preference Optimization: The reward model is trained on approximately 27,500 training videos and 500 testing videos with human-labeled video preference pairs and reference images.Labels indicate whether each video aligns with the identity defined by its reference image.
- Preference Optimization: To reduce computation, the model uses the first eight blocks of the video diffusion model as its backbone and jointly updates its reward-model components.The jointly updated components are the VDM backbone, identity cross-attention layer, and reward prediction head.
2) Latent Identity Reward Feedback Learning:
Latent identity reward feedback learning evaluates intermediate denoising latents with a frozen identity reward model and optimizes the generator toward higher identity fidelity. A supervised objective regularizes this process to prevent reward hacking and preserve motion control and diversity.
- Reward Feedback Learning: Reward feedback learning operates in latent space, avoiding repeated VAE decoding and incorporating information from intermediate diffusion states.The approach addresses computational bottlenecks and the limitation of evaluating only the final denoised result.
- Intermediate-Latent Evaluation: The procedure denoises from Gaussian noise without gradients until a sampled intermediate timestep, then performs one gradient-enabled step to obtain the predicted latent.The target timestep is sampled uniformly from 0 to T−1.
- Intermediate-Latent Evaluation: The frozen LIRM evaluates the predicted intermediate latent and supplies an identity reward used to maximize expected identity fidelity.The reward conditions on the latent, timestep, context, and reference subject.
- Objective Design: The final objective combines the first-stage supervised fine-tuning loss with the latent identity reinforcement loss to prevent reward hacking.The supervised term regularizes optimization against overemphasizing identity scores at the expense of visual quality or diversity.
- Objective Design: The balanced objective aligns identity preferences while preserving the precise motion control and generative diversity established during supervised fine-tuning.The reward-feedback strength is controlled by λ2.
C. Dataset Construction Pipeline
The dataset construction pipeline produces densely annotated videos for jointly training subject identity and multi-granularity motion control. Its annotations cover multi-subject references, spatial structure, and motion information needed for customization and controllable generation.
- Motion-Based Filtering: The pipeline filters videos for significant temporal dynamics before constructing training annotations.Dense optical flow is estimated with RAFT, and videos with small average motion magnitude are discarded.
- Semantic Processing: Automated processing extracts semantic tags, refines them to retain significant moving subjects, and generates detailed video captions.The passage attributes these steps to RAM++, Qwen3 Max, and Qwen3-VL, respectively.
- Spatiotemporal Annotation: The dataset uniquely supports multi-subject customization with comprehensive motion annotations, including segmentation masks, bounding boxes, and trajectories.These annotations provide a shared basis for video customization and motion-control tasks.
- Dataset Scope: DreamOmni’s dataset is presented as distinct from prior video customization and controllable-generation datasets because it combines multi-subject customization with comprehensive motion-control annotations.The comparison is stated in the dataset table caption.
- Evaluation Annotations: DreamOmni Bench samples include multiple reference images, detailed captions, bounding boxes, motion trajectories, and subject masks for fine-grained evaluation.These annotations provide precise spatial-temporal ground truths for each test sample.
D. DreamOmni Bench
DreamOmni Bench addresses the absence of a unified evaluation protocol for multi-subject identity preservation and comprehensive motion control. It uses independently sourced real-world videos with dense annotations and evaluates identity and motion precision together.
- Benchmark Motivation: Existing benchmarks separately emphasize personalization or motion control and therefore do not jointly measure multi-subject consistency, motion controllability, and identity preservation.Personalization benchmarks are mainly single-subject, while motion benchmarks may focus on limited motion signals.
- Benchmark Construction: DreamOmni Bench is built from high-quality real-world videos sourced independently of the training dataset to enable zero-shot evaluation.Manual filtering removes static videos and frames containing text overlays or watermarks.
- Benchmark Construction: The benchmark uses dense captions and comprehensive annotations including subject masks, bounding boxes, and trajectories.These annotations are generated through the dataset-processing pipeline after video filtering.
- Evaluation Scope: DreamOmni Bench evaluates identity preservation for generic objects and human faces alongside bounding-box and trajectory accuracy.The benchmark is designed to quantify both identity and motion-control performance.
- Baselines: The experiments benchmark DreamVideo-Omni against prior methods from three categories on DreamOmni Bench and MSRVTT-Personalization Bench.The comparison includes DreamVideo-2, VACE, and Phantom among the listed baselines.
B. Main Results
DreamVideo-Omni consistently improves joint subject customization and motion control across qualitative comparisons and multiple benchmarks, including single- and multi-subject settings.
- DreamVideo-Omni preserves subject identity while accurately following complex spatial trajectories in joint customization and motion-control comparisons.
- DreamVideo-Omni significantly outperforms DreamVideo-2 across all metrics on DreamOmni Bench and generalizes to MSRVTT-Personalization Bench.On MSRVTT-Personalization, it achieves the highest R-DINO and best EPE in Subject Mode, while Face-S is comparable to Tora2 and EPE is significantly better.
- DreamVideo-Omni achieves state-of-the-art subject customization, with the highest R-DINO and R-CLIP in single-subject settings and superior R-DINO, Face-S, and CLIP-T in multi-subject settings.
- Compared with Tora, DreamVideo-Omni increases mIoU by 0.395 and reduces EPE by nearly 70% for motion precision.
- Despite having 1.3B parameters, DreamVideo-Omni surpasses the 14B-parameter Wan-Move across all metrics in both single- and multi-subject motion-control settings.
- Human evaluation reports the highest user preference for DreamVideo-Omni across diverse settings.
C. Emergent Capabilities
DreamVideo-Omni develops emergent generation capabilities beyond its text-to-video base model, including image-to-video generation and first-frame-conditioned trajectory control without task-specific fine-tuning.
- DreamVideo-Omni enables zero-shot Image-to-Video generation and first-frame-conditioned trajectory control despite using a text-to-video base model.These capabilities emerge from the multi-task training paradigm without task-specific fine-tuning.
- The ablation study evaluates DreamVideo-Omni components under both single-subject and multi-subject settings on DreamOmni Bench.
D. Ablation Studies
Ablations identify condition-aware 3D RoPE, group and role embeddings, hierarchical motion injection, and LIReFL as important components for controllability and identity preservation.
- Removing condition-aware 3D RoPE causes catastrophic metric degradation and training collapse with severe artifacts or meaningless noise.
- LIReFL outperforms standard SFT across most metrics, particularly in multi-subject scenarios, while improving subject customization and maintaining motion control.
- Removing group and role embeddings creates control ambiguity, weakening subject disentanglement and motion binding, especially in multi-subject settings.
- Removing hierarchical bounding-box injection causes subjects to fail to follow bounding boxes or trajectories, demonstrating its role in motion control.
- The latent identity reward model achieves higher average pairwise classification accuracy with BCE loss than with the Bradley-Terry model.The reported averages are 0.720 for BCE and 0.656 for Bradley-Terry.
- The final model adopts λ2 = 0.10 because excessively strong reward feedback disrupts motion coherence.
V. CONCLUSION
DreamVideo-Omni unifies multi-subject customization with global, local, and camera-motion control through progressive training and specialized conditioning, while using latent identity rewards to preserve identity.
- DreamVideo-Omni combines multi-subject customization with omni-motion control covering global motion, local object motion, and camera movement.
- Condition-aware 3D RoPE, hierarchical motion injection, and group and role embeddings coordinate heterogeneous inputs and bind motion signals to identities.
- Latent identity reward feedback learning uses a VDM-based reward model to prioritize motion-aware identity preservation aligned with human preferences.
- DreamOmni Bench and an automated data-construction pipeline support comprehensive evaluation of multi-subject and omni-motion control.