Source-linked AI summary
UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generation
Jiehui Huang, Yuechen Zhang, Xu He, Yuan Gao, Zhi Cen, Bin Xia, Yan Zhou, Xin Tao, Pengfei Wan, Jiaya Jia
TL;DR
RGB-centered video learning leaves open whether unifying visual modalities can strengthen world understanding. UnityVideo jointly trains across modalities and tasks using dynamic noise scheduling and modality-aware learning. The authors report competitive results across tasks, including best text-to-video metrics for depth-RGB joint generation and improvements over single-modality models in segmentation and depth estimation.
Problem
Video models have largely scaled RGB alone, leaving unclear whether unifying visual submodalities can strengthen physical-world understanding and whether unified training improves cross-modal interaction, convergence, and generalization.
Source: §1 · p. 2Method
UnityVideo jointly learns across modalities and training paradigms using dynamic noise scheduling, an in-context learner, and a modality-adaptive switcher.
Source: §1 · p. 2, §3.2 · p. 4Results
UnityVideo reports competitive results across tasks, best metrics for depth-RGB joint text-to-video generation, and improvements over single-modality models in segmentation and depth estimation.
Source: §4.2 · p. 6Takeaways & Limitations
The authors present unified multimodal modeling as a promising step toward comprehensive world models across diverse visual representations.
Source: §6 · p. 8The current VAE occasionally introduces reconstruction artifacts.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Recent video generation models demonstrate impressive synthesis capabilities but remain limited by single-modality conditioning, constraining their holistic world understanding. This stems from insufficient cross-modal interaction and limited modal diversity for comprehensive world knowledge representation. To address these limitations, we introduce UnityVideo, a unified framework for world-aware video generation that jointly learns across multiple modalities (segmentation masks, human skeletons, DensePose, optical flow, and depth maps) and training paradigms. Our approach features two core components: (1) dynamic noising to unify heterogeneous training paradigms, and (2) a modality switcher with an in-context learner that enables unified processing via modular parameters and contextual learning. We contribute a large-scale unified dataset with 1.3M samples. Through joint optimization, UnityVideo accelerates convergence and significantly enhances zero-shot generalization to unseen data. We demonstrate that UnityVideo achieves superior video quality, consistency, and improved alignment with physical world constraints. Code and data can be found at: https://github.com/dvlab-research/UnityVideo
1. Introduction
The introduction asks whether unifying visual modalities can improve world understanding, convergence, and generalization beyond RGB-centered video learning. UnityVideo combines modalities and training paradigms, and the authors report improved convergence and performance across tasks.
1. Introduction
- The motivation is that video foundation models have largely scaled RGB alone, leaving open whether depth, flow, segmentation, skeleton, and DensePose can strengthen physical-world understanding.Source: §1 · p. 2
- The introduction identifies complementary modality cues: segmentation separates categories, DensePose distinguishes body parts, and skeletons encode fine-grained motion.Source: §1 · p. 2
- UnityVideo is a unified framework for multimodal video generation, estimation, and joint modeling, integrating modalities and training paradigms to improve convergence and zero-shot generalization.Source: §1 · p. 2
- Unified multi-modal and multi-task training achieves the lowest final loss on RGB video generation, outperforming single-modality joint training and RGB finetuning.Source: Figure 2 · p. 2
- The framework combines a modality-adaptive learner, an in-context learner, and dynamic noise scheduling across joint generation, estimation, and controllable generation objectives.Source: §1 · p. 2
- The authors introduce OpenUni with 1.3M multimodal video samples and UniBench, a 30K-sample benchmark with rendered ground-truth depth and optical flow.Source: §1 · p. 2
2. Releated Work
Related work describes progress in video generation and reconstruction, while identifying incomplete unification of modalities and tasks as a remaining gap.
2.1. Video Generation
- Prior video-generation work models multiple modalities jointly or conditions generation on multimodal inputs, but most studies isolate a single architecture or modality, limiting cross-task synergy.Source: §2.1 · p. 3
2.2. Video Reconstruction
- Video-reconstruction systems demonstrate bidirectional mappings across video and geometric, skeletal, audio, or text signals, but diverse modalities and in-context abilities remain insufficiently unified.Source: §2.2 · p. 3
3. Method
UnityVideo unifies video generation and multimodal understanding in one diffusion transformer, using dynamic task routing and modality-aware components. A staged training curriculum and three mode-specific objectives support joint learning across tasks and modalities.
3. Method
- The shared diffusion transformer processes RGB video, text conditions, and auxiliary modalities, with dynamic noise schedules and modality-aware components supporting simultaneous convergence.Source: §3 · p. 3
3.1. Unifying Multiple Tasks
- UnityVideo trains conditional RGB generation, auxiliary-modality estimation from RGB, and joint generation from noise within one architecture.Source: §3.1 · p. 3
- Probabilistic task routing selects a noise schedule for each training iteration, with probabilities ordered by learning difficulty to train the three distributions concurrently.Source: §3.1 · p. 3
3.2. Unifying Multiple Modalities
- The in-context learner uses modality-type prompts for auxiliary features, while the Modality-Adaptive Switcher generates modality-specific AdaLN scale, shift, and gate parameters for plug-and-play inference.Source: §3.2 · pp. 3–4
3.3. Training Strategy
- Training progresses from pixel-aligned modalities on curated single-person data to all five modalities and diverse scenes; OpenUni contains 1.3M clips sampled in balanced modality-and-source groups.Source: §3.3 · p. 5
3.4. Training Objective
- The three losses train conditional generation, modality estimation, and joint generation by predicting their respective RGB and modality velocity targets; each batch sample is assigned one mode.Source: §3.4 · p. 5
4. Experiment
UnityVideo is evaluated on video generation and modality estimation against existing models, with ablations testing multimodal and multi-task training and architectural choices. The reported results cover benchmark performance, qualitative world understanding, generalization, and a user study.
4.1. Experimental Setup
- Experiments compare UnityVideo with generation, estimation, and segmentation baselines on VBench and UniBench using video-quality, depth-estimation, and segmentation metrics.Training uses a 10B-parameter DiT backbone in two stages, first with 500K clips and then with 1.3M clips.Source: §4.1 · pp. 5–6
4.2. Main Results
- Table 1 reports superior or competitive performance across tasks; depth-RGB joint generation achieves the best text-to-video results on all metrics.For controllable generation, UnityVideo excels in background consistency, overall consistency, and dynamic degree while maintaining competitive aesthetic quality; it also outperforms single-modality models on video segmentation and depth estimation.Source: Table 1 · p. 6, §4.2 · p. 6
- Qualitative comparisons report stronger adherence to physical principles, faithful depth guidance, and finer modality-estimation details, including on unseen data.Examples include more accurate refraction, fewer background-flickering and subject-distortion issues, finer edges, wider fields of view, and accurate 3D point clouds.Source: §4.2 · p. 6, Figure 5 · p. 7
4.3. Abalation Study
- Joint depth and optical-flow training improves performance across metrics over the baseline, with multimodal training adding gains particularly in image quality and overall consistency.The ablation attributes these gains to complementary supervision and mutual enhancement between modalities.Source: §4.3 · p. 7, Table 2 · p. 7
- Unified training across Joint Generation and Controllable Generation recovers and surpasses baseline performance after ControlGen-only training degrades it.Compared with Joint Generation alone, unified training slightly reduces subject and background consistency while still outperforming the baseline overall.Source: §4.3 · p. 7, Table 3 · p. 7
- The In-Context Learner and Modality Switcher each improve performance, and combining them yields additional gains under depth-guided text-to-video evaluation.The ablation describes the two strategies as complementary.Source: §4.3 · p. 7, Table 4 · p. 8
4.4. Model Analyze
- The In-Context Learner generalizes a fixed two-person segmentation task to unseen two-object scenarios, while the Modality Switcher alone does not.As additional modalities such as depth are learned, the authors also report improved motion understanding and more accurate text responses in RGB videos.Source: §4.4 · p. 8, Figure 6 · p. 8
- In a 70-response user study, UnityVideo achieves the best performance across human evaluations and automatic metrics.Human ratings cover physical, semantic, and overall quality; automatic evaluation uses dynamism and aesthetics.Source: §4.4 · p. 8, Table 5 · p. 8
5. Limitation and Future Work
The authors identify VAE reconstruction artifacts as a current limitation and suggest fine-tuning or improved autoencoders as possible remedies.
5. Limitation and Future Work
- The current VAE occasionally introduces reconstruction artifacts, which may be addressed through fine-tuning or improved autoencoder architectures.The authors also identify scaling to larger backbones and incorporating more visual modalities as future directions for enhancing emergent world understanding.Source: §5 · p. 8
- Scaling to larger backbones and incorporating more visual modalities are proposed as future directions, not established capabilities of the current model.Source: §5 · p. 8
- The authors characterize UnityVideo as a foundation for unified multimodal video understanding despite these limitations.Source: §5 · p. 8
6. Conclusion
UnityVideo unifies visual modalities and tasks in a diffusion transformer, supporting bidirectional learning between RGB video and auxiliary modalities. The authors report strong zero-shot generalization and contribute OpenUni and UniBench for training and evaluation.
6. Conclusion
- UnityVideo models multiple visual modalities and tasks within a single diffusion transformer.Source: §6 · p. 8
- Modal-adaptive learning supports bidirectional learning between RGB video and depth, optical flow, segmentation, skeleton, and DensePose, with mutual enhancement across tasks.Source: §6 · p. 8
- The authors report state-of-the-art performance across diverse benchmarks and strong zero-shot generalization to unseen modality combinations.Source: §6 · p. 8
- OpenUni contains 1.3M synchronized samples, while UniBench provides ground-truth annotations for evaluation.Source: §6 · p. 8
A.1. Modal Interaction Analysis
UnityVideo develops stronger cross-modal interactions during training, while modality-specific output layers address occasional confusion between requested outputs. Unified multimodal training also outperforms standalone and single-modality alternatives across evaluation dimensions.
A.1. Modal Interaction Analysis
- As joint training progresses, RGB–auxiliary attention interactions strengthen alongside richer geometric representations and improved text following.The attention map divides self-modality regions (RGB, RGB; Flow, Flow) from cross-modality regions (RGB, Flow; Flow, RGB).Source: §A.1 · p. 12, Figure 1 · p. 13
A.2. Modality-Specific Output Layers
- A shared output layer can occasionally confuse modalities, such as producing skeletons when segmentation masks are requested; dedicated output layers eliminate this confusion while maintaining comparable performance.The design retains a shared input layer for cross-modal information sharing.Source: §A.2 · p. 12, Table 1 · p. 12
B.1. Compare with T2V
- Unified multimodal joint generation improves over both the baseline and T2V-only training, with better overall balance than single-modality joint training.The comparison is summarized in the Averaged column.Source: Table 2 · p. 12, §B.1 · p. 13
B.2. Scalability with Increasing Modalities
Adding modalities yields consistent performance gains in joint and controllable generation, while UnityVideo also shows stronger physical-law understanding than baseline methods. The reported physical examples include refraction, collisions, and plausible motion.
B.2. Scalability with Increasing Modalities
- Performance improves monotonically across evaluation criteria as training expands from three modalities to five, without reported negative interference.The five-modality configuration adds skeleton and segmentation to depth, optical flow, and DensePose.Source: §B.2 · p. 13, Table 3 · p. 13
B.3. The influence of different modalities
- Unified multimodal training consistently outperforms single-modality training on ControlGeneration.The comparison uses identical training data and iteration budgets.Source: §B.3 · p. 13, Table 4 · p. 14
B.4. World perception comparison
- UnityVideo shows superior understanding of physical laws to baseline methods, with accurate refraction, collision responses, and physically plausible motion trajectories.The evaluation uses physics-focused prompts involving refraction, collision dynamics, and other physical phenomena.Source: §B.4 · p. 14, Figure 2 · p. 14
C.1. OpenUni
OpenUni supplies a large multimodal video corpus with synchronized annotations across five modalities. UniBench pairs synthetic data with rendered ground-truth depth and optical flow for estimation with annotated real-world samples for generation evaluation.
C.1. OpenUni
- OpenUni contains approximately 1.3M video pairs, each with synchronized annotations across five modalities.The modalities are optical flow, depth, DensePose, skeleton, and segmentation.Source: §C.1 · pp. 14–15
C.2. UniBench
- UniBench evaluates estimation with Unreal Engine synthetic data and generation with curated test-split videos carrying multimodal annotations.The synthetic data provides ground-truth depth and optical flow; the benchmark’s two components are shown in Figure 3.Source: §C.2 · p. 15, Figure 3 · p. 15
D. More Visuals and Applications
UnityVideo supports controllable generation, video estimation, and joint generation across varied scenes and subjects. Reported zero-shot transfer includes multi-person scenes, animal motion, and out-of-distribution objects and scenes.
More Visuals and Applications
- The model accepts arbitrary modality inputs for controllable generation and supports flexible modality estimation across diverse subjects and scenarios.Representative outputs include coherent RGB video aligned with segmentation, DensePose, optical flow, skeleton, and depth.Source: D. More Visuals and Applications · p. 16, Figure 5 · p. 16, Figure 4 · p. 16
- Zero-shot transfer extends from primarily single-person training to multi-person estimation, from human skeletons to animal motion, and to out-of-distribution objects and scenes.The reported depth and segmentation estimates for out-of-distribution objects and scenes require no additional fine-tuning.Source: D. More Visuals and Applications · p. 16