Source-linked AI summary
Omni-Video 2: Scaling MLLM-Conditioned Diffusion for Unified Video Generation and Editing
Hao Yang, Zhiyu Tan, Jia Gong, Luozheng Qin, Hesen Chen, Xiaomeng Yang, Yuqing Sun, Yuetan Lin, Mengping Yang, Hao Li
TL;DR
Unified video generation and editing must preserve pretrained generation quality while handling complex instructions, multimodal references, and temporal consistency. Omni-Video 2 combines MLLM-produced target captions with a lightweight adapter over a pretrained diffusion model, achieving strong editing and generation performance. On FiVE, it reaches a FiVE-Acc of 73.53 versus UniVideo’s 62.53, while generation quality remains competitive or superior.
Problem
Unified video systems must handle new multimodal editing conditions and complex temporal edits without disrupting pretrained text-to-video behavior.
Method
Omni-Video 2 uses MLLM reasoning to produce explicit target captions and a lightweight adapter to inject multimodal conditions into a pretrained text-to-video diffusion model.
Results
FiVE-Acc reaches 73.53 versus UniVideo’s 62.53, while video generation remains competitive or superior on standard quality, alignment, and temporal-coherence metrics.
Takeaways & Limitations
A single diffusion backbone can support strong video generation and instruction-driven editing without task-specific architectures or architectural branching.
Abstract
from arXiv · showhide
We present Omni-Video 2, a scalable and computationally efficient model that connects pretrained multimodal large-language models (MLLMs) with video diffusion models for unified video generation and editing. Our key idea is to exploit the understanding and reasoning capabilities of MLLMs to produce explicit target captions to interpret user instructions. In this way, the rich contextual representations from the understanding model are directly used to guide the generative process, thereby improving performance on complex and compositional editing. Moreover, a lightweight adapter is developed to inject multimodal conditional tokens into pretrained text-to-video diffusion models, allowing maximum reuse of their powerful generative priors in a parameter-efficient manner. Benefiting from these designs, we scale up Omni-Video 2 to a 14B video diffusion model on meticulously curated training data with quality, supporting high quality text-to-video generation and various video editing tasks such as object removal, addition, background change, complex motion editing, \emph{etc.} We evaluate the performance of Omni-Video 2 on the FiVE benchmark for fine-grained video editing and the VBench benchmark for text-to-video generation. The results demonstrate its superior ability to follow complex compositional instructions in video editing, while also achieving competitive or superior quality in video generation tasks.
1. Introduction
Omni-Video 2 addresses the difficulty of unified video generation and editing by combining MLLM reasoning with a pretrained text-to-video diffusion model. It uses explicit target captions and a lightweight adapter to support complex, temporally coherent edits while preserving pretrained generative capabilities.
- Unified video modeling remains difficult because video systems require substantial computation and strict temporal consistency.
- New source-video, multimodal, and editing conditions can interfere with pretrained conditioning and degrade original text-to-video performance.
- An MLLM-based prompt reasoner converts user instructions into explicit target captions that guide complex and compositional edits through the pretrained text-to-video interface.
- A lightweight multimodal adapter injects additional controls without modifying the diffusion backbone or retraining the model end-to-end.
- The 14B model supports text-to-video generation and diverse editing tasks, with FiVE and VBench experiments showing strong instruction following and competitive or superior generation quality.
2. Scaling Omni-Video 2
Omni-Video 2 unifies generation and editing through an MLLM understanding branch and a diffusion generation branch connected by explicit semantic conditioning. The system predicts target captions and combines multiple conditional signals through a shared, parameter-efficient cross-attention interface.
- The framework integrates multimodal perception and instruction reasoning in an MLLM branch with visual synthesis in a diffusion decoder.
- Given a source video and edit instruction, the MLLM predicts a target caption and extracts cross-modal interaction features for diffusion conditioning.
- The diffusion model jointly receives target-caption, edit-instruction, MLLM interaction, and source-reference conditions.
- The Editing Prompt Reasoner makes ambiguous or underspecified user edits generation-friendly by describing the intended output explicitly.
- A shared cross-attention interface concatenates projected multimodal, text, instruction, and reference tokens without task-specific diffusion heads.
- Randomized dropout masks auxiliary conditions while retaining the source reference, improving robustness to missing controls and preserving the caption pathway.
- A standardized four-condition interface allows the same diffusion decoder to support generation and multiple editing tasks.
3. Data Curation
Omni-Video 2 builds a unified corpus for generation and editing from real-world and synthetic data, using staged cleaning and a shared optional-field schema. The resulting dataset spans more than one million instances across four task families.
- The unified corpus contains over one million instances covering text-to-image, text-to-video, image editing, and video editing.
- Data Sources: Real-world data supplies diverse visual content and natural descriptions for learning appearance, motion, scene dynamics, and text-to-video quality.
- Data Sources: Synthetic paired source-target data strengthens editing supervision by aligning instructions, target captions, and controlled visual transformations.
- Instruction Categories: Editing instructions are organized into local add, local remove, local replace, global edit, attribute change, and complex edit categories.
- Unified Schema: Each example uses optional task, source, instruction, and target-caption fields, keeping all tasks anchored to caption-style semantic guidance.
- Data Cleaning: A four-stage cleaning pipeline removes invalid files, poor visual content, text–visual mismatches, and editing pairs with unreliable transformations.
- Unified Schema: Samples are bucketed by spatial resolution and temporal duration to support variable-resolution and variable-length video training.
4. Training
Training preserves the pretrained text-to-video prior by introducing new capabilities through conditioning and lightweight adapters while leaving the diffusion backbone structurally unchanged. Scaling to large models requires sequence-parallel training because heterogeneous conditioning makes cross-attention computationally significant.
- Omni-Video 2 introduces unified generation and editing through conditioning and a lightweight adapter rather than altering the core diffusion architecture.
- The model uses two 14B-parameter diffusion transformers for low-noise and high-noise regimes.
- Large batch sizes increased optimization-step time, limiting progress to approximately 4,000 steps per day even with 1600 GPUs.
- Ulysses-style sequence parallelism addresses long heterogeneous conditioning sequences in which cross-attention forms a substantial share of computation.
5. Evaluation
Omni-Video 2 is evaluated on video editing and generation, achieving state-of-the-art FiVE editing performance while retaining competitive VBench generation quality. Qualitative results further cover object manipulation, attribute changes, global edits, motion-sensitive edits, and complex instructions.
- Video Editing Performance: 73.53 FiVE-Acc surpasses UniVideo’s 62.53 by 11.0 absolute points, establishing state-of-the-art performance on fine-grained video editing.The gains span all evaluation metrics and are attributed to instruction-faithful, temporally consistent editing without task-specific architectures or fine-tuning.
- Video Generation Performance: Omni-Video 2 achieves competitive or superior VBench performance across semantic alignment, temporal coherence, and overall visual quality.The model shows no degradation relative to strong text-to-video baselines despite supporting instruction-driven editing.
- Unified Evaluation: The unified model supports strong performance on both video generation and editing without architectural branching or task-specific optimization.Results demonstrate that generation and editing can be handled within a single diffusion backbone.
- Qualitative Editing Results: Qualitative examples show accurate local object addition while preserving temporal consistency and producing realistic, well-integrated objects.These examples complement the quantitative editing results with evidence of temporal stability and scene integration.
- Qualitative Editing Results: Qualitative results show successful object removal, local and portrait attribute changes, and global style or background edits while preserving temporal consistency.The evaluation also includes complex motion dynamics and multi-part instructions, with edits maintaining original motion patterns or temporal coherence.