Source-linked AI summary
OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning
Kaihang Pan, Qi Tian, Jianwei Zhang, Weijie Kong, Jiangfeng Xiong, Yanxin Long, Shixue Zhang, Haiyi Qiu, Tan Wang, Zheqi Lv, Yue Wu, Liefeng Bo, Siliang Tang, Zhao Zhong
TL;DR
Open-source unified video generation remains fragmented and trails proprietary systems in handling diverse tasks and inputs. OmniWeaving addresses this gap with a unified multimodal composition and reasoning framework trained on broad scenario data and evaluated with IntelligentVBench. It achieves state-of-the-art performance among open-source unified frameworks, while remaining behind proprietary counterparts.
Problem
Open-source unified video generation remains fragmented, with existing models struggling to integrate diverse tasks and benchmarks limited in complexity and input formats.
Method
OmniWeaving combines visual understanding and generation with large-scale composition and reasoning-augmented training for free-form multimodal video inputs.
Results
OmniWeaving achieves state-of-the-art performance across existing open-source unified frameworks and even surpasses specialized models.
Takeaways & Limitations
OmniWeaving provides an open-source reference for unified video generation with multimodal composition and reasoning-informed capabilities.
Takeaways & Limitations
A substantial performance and input-flexibility gap remains between OmniWeaving and proprietary systems such as Seedance-2.0.
Abstract
from arXiv · showhide
While proprietary systems such as Seedance-2.0 have achieved remarkable success in omni-capable video generation, open-source alternatives significantly lag behind. Most academic models remain heavily fragmented, and the few existing efforts toward unified video generation still struggle to seamlessly integrate diverse tasks within a single framework. To bridge this gap, we propose OmniWeaving, an omni-level video generation model featuring powerful multimodal composition and reasoning-informed capabilities. By leveraging a massive-scale pretraining dataset that encompasses diverse compositional and reasoning-augmented scenarios, OmniWeaving learns to temporally bind interleaved text, multi-image, and video inputs while acting as an intelligent agent to infer complex user intentions for sophisticated video creation. Furthermore, we introduce IntelligentVBench, the first comprehensive benchmark designed to rigorously assess next-level intelligent unified video generation. Extensive experiments demonstrate that OmniWeaving achieves SoTA performance among open-source unified models. The codes and model have already been publicly available. Project Page: https://omniweaving.github.io.
1 Introduction
Unified video generation remains fragmented and behind proprietary systems, motivating OmniWeaving’s framework for multimodal composition, reasoning, and broader evaluation.
- Open-source unified video models lag behind proprietary systems and remain fragmented across text-to-video, image-to-video, and video-to-video tasks.
- OmniWeaving integrates visual understanding and generation to support free-form text, image, and video inputs.
- The framework is presented as an omni-level generator that weaves interleaved text, image, and video inputs into coherent spatio-temporal narratives.
- IntelligentVBench evaluates multimodal composition and abstract reasoning in unified video generation across four distinct tasks.
- Its training dataset spans diverse generative scenarios, including multimodal composition and reasoning-related tasks.
2 Related Work
Prior work is divided between proprietary omni-capable systems and open-source or academic benchmarks that generally remain task-specific and limited in complexity.
- Proprietary systems have approached omni-capable intelligent video generation, but their underlying techniques remain undisclosed.
- Open-source video generation remains dominated by fragmented models tailored to text-to-video, image-to-video, or video-to-video synthesis.
- Existing benchmarks commonly use rigid, task-specific input formats and focus on foundational single-shot generation scenarios.
3 Training Data
OmniWeaving’s training data combines real and synthetic videos with interleaved multimodal and reasoning-augmented tasks, constructed through complementary pipelines.
- Text-video paired data alone is insufficient for complex in-context reasoning and composition over interleaved text, image, and video inputs.
- The corpus combines real-world videos for natural appearances and motion with synthetic data for sparse or noisy highly conditioned tasks.
- Training tasks cover foundational generation and editing, multimodal composition, and reasoning-augmented video generation.
- A dual output-first and input-first pipeline uses videos, extracted images, generated descriptions, and auxiliary models to construct task-specific inputs.
- Multimodal composition data includes interleaved text-and-multi-image-to-video and text-image-video-to-video tasks.
- Reasoning-augmented examples pair standard inputs and target videos with traces that bridge the input to the intended output.
4 Model: OmniWeaving
OmniWeaving unifies multimodal understanding and video generation through an MLLM, MMDiT, and VAE, enhanced with thinking mode, DeepStacking, and progressive training.
- Model Architecture: The architecture combines an MLLM for multimodal understanding, an MMDiT for generation, and a VAE for visual tokenization.
- Model Architecture: The MLLM parses free-form multimodal inputs into semantic representations, while the MMDiT combines conditioning semantics and VAE latents with noise.
- Model Architecture: Thinking mode generates intermediate reasoning steps and an enhanced prompt that conditions the MMDiT alongside original MLLM features.
- Model Architecture: DeepStacking projects intermediate MLLM features into the MMDiT embedding space and injects multi-granular guidance into its first three conditioning layers.
- Training Strategy: Training progresses from modality alignment, through multi-task free-form pretraining, to reasoning-augmented fine-tuning.
- Training Strategy: The final stage unfreezes the MLLM, jointly optimizes it with the MMDiT, and adds next-token prediction on reasoning traces.
5 Benchmark: IntelligentVBench
IntelligentVBench is designed to evaluate unified video generation on heterogeneous, free-form inputs requiring both multimodal composition and abstract reasoning. It organizes these capabilities into four tasks and uses automated VLM judging across three metrics.
- Benchmark motivation: IntelligentVBench addresses the limited complexity and comprehensiveness of existing video-generation benchmarks by evaluating free-form composition and abstract reasoning.The benchmark targets heterogeneous inputs and complex interactions rather than isolated, single-shot rendering.
- Benchmark design: The benchmark combines interleaved text, multiple images, and reference videos, with images serving roles such as spatial anchors or compositional elements.This design is intended to reflect the complexity of real-world human queries.
- Task structure: The four tasks are Implicit I2V, Interpolative DI2V, Compositional MI2V, and TIV2V; the first two assess reasoning, while the latter two assess composition.The tasks cover implicit temporal inference, boundary-frame interpolation, multi-image composition, and cross-modal video editing.
- Dataset: IntelligentVBench contains 1,030 test cases spanning challenging motion, large visual variance, multiple subjects, and multimodal editing scenarios.The benchmark includes 250 Implicit I2V cases and 250 Interpolative DI2V cases; the supplied passage begins the Compositional MI2V count at 320.
- Evaluation metrics: Evaluation uses a VLM-as-a-Judge protocol with Gemini2.5-Pro and scores Instruction Following, Condition Preserving, and Overall Visual Quality.These metrics assess semantic intent, fidelity to visual references, and aesthetic, temporal, motion, and physical quality.
6 Experiments
Experiments evaluate OmniWeaving against unified and task-specific models on IntelligentVBench and established T2V and V2V benchmarks. The results report strong cross-task performance, improved reasoning with thinking mode, and broad input-format versatility, while also identifying remaining weaknesses in existing unified systems.
- Main results: Existing unified models show substantial subtask imbalance, while specialized models remain stronger on reasoning-intensive Implicit I2V and Interpolative DI2V scenarios.Unified models instead perform better on Compositional MI2V, indicating uneven capability across reasoning and composition.
- Main results: Across all four IntelligentVBench tasks, OmniWeaving achieves SoTA performance under both MIN and AVG overall metrics among the compared open-source models.The authors attribute this to synergistic multi-task integration that mitigates intertask competition and mutual suppression.
- Main results: OmniWeaving outperforms baselines on complex multi-subject and tri-modal inputs, demonstrating stronger multimodal compositional flexibility.The reported comparison covers image, video, and text combinations within IntelligentVBench.
- Existing benchmarks: On VBench T2V, OmniWeaving outperforms existing unified frameworks and performs comparably to specialized HunyuanVideo despite T2V data comprising less than 10% of training.The reported result indicates preservation of foundational generation capabilities without significant degradation.
- Existing benchmarks: OmniWeaving achieves an average score of 3.15 across six OpenVE-Bench V2V subtasks, surpassing specialized and unified counterparts.Performance is balanced across global, local, and text-based editing, with a relative dip on local-add.
- Qualitative comparisons: Qualitative comparisons show OmniWeaving better preserves reference-frame alignment, integrates specified visual elements, and avoids unintended edits to unedited video regions.The comparison includes Intent-Driven I2V, Interpolative DI2V, Compositional MI2V, and TIV2V.
- In-depth analysis: After Reasoning-Augmented Fine-Tuning, enabling thinking mode significantly boosts AVG performance on Implicit I2V and Interpolative DI2V.Before the final training stage, thinking mode severely degraded performance, whereas after fine-tuning it improved results.
- In-depth analysis: A three-layer DeepStacking strategy using broadly separated layers [8, 16, 24] outperforms shallow, deep, two-layer, four-layer, and non-DeepStacking alternatives on Compositional MI2V.The results support combining semantic features across low-level to high-level representations.
7 Conclusion and Future Work
OmniWeaving unifies free-form multimodal video generation with visual comprehension and reasoning-informed generation, while IntelligentVBench evaluates intelligent video generation. The authors report strong open-source performance but acknowledge a persistent gap with proprietary systems and plan broader multimodal support.
- OmniWeaving supports free-form interleaved multimodal inputs and uses visual understanding to guide video generation within one framework.
- IntelligentVBench evaluates abstract reasoning and compositional capabilities for next-level intelligent video generation.
- OmniWeaving achieves state-of-the-art performance among open-source unified frameworks and surpasses specialized models.
- A substantial performance, modality, and format-flexibility gap remains between OmniWeaving and proprietary systems such as Seedance-2.0.The authors attribute this disparity largely to proprietary models’ greater computational resources and training data.
- Future work targets interleaved multiple-image-and-video inputs and additional audio input and output for audio-visually synchronized video generation.
A Model Details
The model instantiates OmniWeaving with Qwen2.5-VL as its multimodal language model and HunyuanVideo-1.5 as its generative engine. Distinct task prompts help the multimodal model communicate task requirements to the video generator, with specialized reasoning guidance for reasoning-augmented tasks.
- OmniWeaving is instantiated with Qwen2.5-VL as the MLLM and HunyuanVideo-1.5 as the core generative engine.
- HunyuanVideo-1.5’s Glyph-ByT5 and SigLIP encoders provide cross-lingual text rendering and vision-text semantic alignment.
- Distinct MLLM system prompts differentiate video-generation tasks so the MMDiT can identify which task to perform.
- Reasoning-augmented tasks append specialized reasoning guidance to the user input.
B More Experimental Results
Additional qualitative results show OmniWeaving handling multiple video-generation scenarios within a single model, spanning synthesis, editing, multimodal composition, and reasoning-augmented generation.
- Qualitative results cover Text-to-Video, First-Frame-to-Video, Key-Frames-to-Video, Video-to-Video editing, multi-image composition, text-image-video inputs, and reasoning-augmented generation.
- The results illustrate that OmniWeaving integrates diverse video-generation capabilities within one model.
C Benchmark Evaluation Prompts
The appendix provides task-specific system prompts, reasoning examples, qualitative outputs, and IntelligentVBench evaluation templates. The benchmark uses Gemini 2.5 Pro to score task-specific capabilities on three 1–5 metrics.
- IntelligentVBench uses Gemini 2.5 Pro as an automated evaluator, scoring three metrics per task on a 1–5 scale.
- System prompts specify inputs and generation behavior for Text-to-Video and First-Frame-to-Video tasks.The prompts identify user images and text as conditioning inputs.
- Key-Frames-to-Video prompts require analyzing visual trajectories and connecting key frames with temporally consistent motion.
- Interleaved text-and-multi-image and text-image-video prompts support multimodal inputs involving text, images, and video.
- Reasoning-augmented prompts ask the model to describe image features and explain how text instructions should alter motion over time.
- Qualitative appendix figures report results across text-to-video, frame-conditioned, editing, compositional, and reasoning-augmented tasks.
- The evaluation templates define criteria for implicit I2V, interpolative DI2V, compositional MI2V, and local-object video editing tasks.