PaperlayerView exact revision on arXiv

Source-linked AI summary

Omni-Video: Democratizing Unified Video Understanding and Generation

Zhiyu Tan, Hao Yang, Luozheng Qin, Jia Gong, Mengping Yang, Hao Li

arXiv:2507.06119v4cs.CV

TL;DR

Unified understanding and generation models have largely focused on images, leaving video understanding and generation less addressed. Omni-Video connects MLLMs to diffusion decoders through continuous visual tokens and a staged training approach, and reports results across understanding, generation, and editing. Its reported VBench-Long performance is competitive overall, with advantages over a named baseline in several categories.

  • Problem

    Existing unified understanding-and-generation models predominantly focus on images, while video modeling requires temporal dynamics, motion coherence, and frame-wise consistency.

    Source: §1 · p. 3

  • Method

    Omni-Video trains existing MLLMs to produce continuous visual tokens and uses a lightweight adapter to condition diffusion decoders, with multi-stage training and extensions for editing.

    Source: §1 · p. 3

  • Results

    Omni-Video reports favorable results across understanding, generation, and editing; on VBench-Long, it is competitive overall with Wan2.1-T2V-1.3B and surpasses it in 10 of 16 categories.

    Source: §1 · pp. 3–4, §5.2 · p. 12

  • Takeaways & Limitations

    The report concludes that one framework can address video understanding, generation, and editing, and that think mode yields consistent improvements in output quality.

    Source: §1 · p. 3

    High-fidelity, long-duration synthesis is constrained by memory demands and limited training data, while decoupled MLLM and diffusion components can misalign representations for discriminative video reasoning.

    Source: §6 · p. 17

Abstract

from arXiv · show

Notable breakthroughs in unified understanding and generation modeling have led to remarkable advancements in image understanding, reasoning, production and editing, yet current foundational models predominantly focus on processing images, creating a gap in the development of unified models for video understanding and generation. This report presents Omni-Video, an efficient and effective unified framework for video understanding, generation, as well as instruction-based editing. Our key insight is to teach existing multimodal large language models (MLLMs) to produce continuous visual clues that are used as the input of diffusion decoders, which produce high-quality videos conditioned on these visual clues. To fully unlock the potential of our system for unified video modeling, we integrate several technical improvements: 1) a lightweight architectural design that respectively attaches a vision head on the top of MLLMs and a adapter before the input of diffusion decoders, the former produce visual tokens for the latter, which adapts these visual tokens to the conditional space of diffusion decoders; and 2) an efficient multi-stage training scheme that facilitates a fast connection between MLLMs and diffusion decoders with limited data and computational resources. We empirically demonstrate that our model exhibits satisfactory generalization abilities across video generation, editing and understanding tasks.

1 Introduction

Existing unified models largely focus on static images, while video modeling also demands temporal dynamics, motion coherence, and frame-wise consistency. Omni-Video connects video-understanding MLLMs to diffusion generation and extends the framework to editing, reporting favorable results across tasks.

1 Introduction

  • Prior unified models largely target static images, while video requires temporal-dynamics modeling, motion coherence, frame-wise consistency, and greater data and computational scalability.The report frames unified video modeling as important for immersive-media understanding, production, and interaction.Source: §1 · pp. 2–3
  • Omni-Video teaches existing video-understanding MLLMs to produce continuous visual tokens, then adapts them into text-to-video diffusion conditioning with a lightweight adapter.A vision head produces the tokens, and an adapter maps them to the diffusion model’s conditional space.Source: §1 · p. 3
  • Its multi-stage training aligns MLLM visual tokens with pretrained vision representations, adapts them to T2V conditioning, then fine-tunes selected components and the diffusion model.The contribution describes the scheme as resource-friendly and not requiring massive computation.Source: §1 · p. 3
  • For editing, pretrained VAE priors provide low-level visual information to the diffusion model during denoising; think mode adds logical inference and contextual planning for generation and editing.The authors motivate the VAE approach by the diffusion model’s ability to recover clean videos from corrupted signals.Source: §1 · p. 3
  • Omni-Video reports favorable performance across image/video understanding, text-to-image/video generation, and editing; think mode improves instruction understanding and output quality.The report also describes editing as enhancing the framework’s flexibility.Source: §1 · pp. 3–4

2 Related Work

Related work spans multimodal understanding systems, diffusion and autoregressive visual generation, and unified architectures combining understanding with generation. The report positions Omni-Video as addressing the limited video capability and resource demands of prior unified approaches.

2.1 Multimodal Understanding

  • Multimodal understanding models connect visual encoders to LLMs for multimodal perception and reasoning, but primarily produce text rather than other modalities.Video-centric MLLMs encode videos as sequential visual tokens compatible with LLMs.Source: §2.1 · p. 4

2.2 Multimodal Generation

  • Diffusion and autoregressive models are established visual-generation approaches: diffusion denoises toward data distributions, while AR models predict sequences of visual tokens.The report identifies Wan2.1 as the open-source video-generation model integrated into Omni-Video.Source: §2.2 · p. 4

2.3 Unified Multimodal Understanding and Generation

  • Unified systems include pure AR and diffusion-AR hybrids, but many train from scratch or focus on static images; Omni-Video instead targets unified video understanding and generation.Discrete visual representations can struggle to preserve fine-grained details, while the report identifies MetaMorph as the closest related work and notes its limited video capability.Source: §2.3 · p. 5

3 Model Architecture

Omni-Video connects an autoregressive MLLM for understanding to a diffusion model for generation by training the MLLM to produce visual tokens that the diffusion model can use. Separate text and vision heads, an adapter, and source-video embeddings support understanding, generation, and editing.

3.1 Overview

  • The architecture connects an AR-based MLLM to a DiT-based diffusion model, using lightweight adapters and a three-stage training scheme to align visual tokens with the diffusion model’s conditioning space.The approach reuses pretrained components rather than training the whole system from scratch.Source: §3.1 · p. 6

3.2 Producing Visual Tokens with MLLMs

  • Separate text and vision heads produce textual outputs for understanding and continuous visual tokens for generation, with the vision head trained to align its tokens to embeddings from pretrained visual encoders.The text head uses next-token prediction, while the vision head’s alignment is trained against visual embeddings.Source: §3.2 · p. 6

3.3 Making Visual Tokens Understandable for DMs

  • A lightweight adapter projects vision-head tokens into the diffusion decoder’s conditioning space, which is aligned to textual embeddings and trained with paired text and visual data using the decoder’s objective.Only the adapter is updated during this training step.Source: §3.3 · pp. 6–7

3.4 Unifying Understanding, Generation and Editing

  • Generation conditions the adapter on vision-head latents and text, while editing also supplies source-video embeddings extracted by a 3D-causal-VAE.The combined conditional sequence is projected by the adapter and injected into each DiT block through cross-attention.Source: §3.4 · p. 7

4 Data and Training

Omni-Video is trained on a mixture of image- and video-related understanding, generation, and editing data using a staged procedure. Training includes component-specific stages, multi-task acceleration strategies, and a reasoning-based generation mode.

4.1 Data

  • The training mixture contains 17.2 million image-understanding, 1.98 million video-understanding, 15 million image-generation, 8.6 million image-editing, 1.8 million video-generation, and 0.8 million video-editing samples.The authors describe the mixture as image-centric while retaining resources for video-related research; text-to-image and text-to-video data undergo quality and content filtering.Source: §4.1 · pp. 7–8

4.3 Training

  • Training first attaches a vision head to the MLLM, trains an adapter to align its visual tokens with the diffusion decoder’s conditioning space, then jointly fine-tunes understanding and generation components.Figure 4 marks trainable modules with flames and frozen parameters with blocks.Source: §4.3 · p. 8, Figure 4 · p. 10
  • The model initializes its MLLM from VILA and its diffusion component from the 1.3B Wan 2.1 text-to-video model; memory strategies include ZeRO Stage-1, sequence parallelism, and gradient checkpointing.The authors note that the original Wan 2.1-1.3B model does not natively support visual editing.Source: §4.3 · pp. 8, 10, 11
  • Stage 1 freezes the MLLM and diffusion model while training the text and vision heads, aligning visual tokens to pretrained visual embeddings and preserving MLLM understanding capabilities.Understanding and generation data are both used, and generated visual tokens are marked by special tokens.Source: §4.3 · p. 9
  • Stage 2 freezes understanding-related parameters and trains the adapter, first aligning it to the diffusion decoder’s textual embedding space and then using a flow-matching objective across four tasks.The tasks are text-to-image, text-to-video, image editing, and video editing; the authors say this broadens capabilities beyond text-to-video synthesis.Source: §4.3 · p. 9
  • Stage 3 jointly fine-tunes the vision head, adapter, and diffusion decoder on higher-quality data, increasing video sampling from 8 to 12 fps, particularly for videos with substantial motion.The authors report that this stage does not significantly increase overall training cost despite more trainable parameters and 50% more target frames.Source: §4.3 · p. 9
  • Multi-task training uses text-to-image and text-to-video first, then adds editing tasks; hybrid batching groups image and video tasks by resolution and frame count to sustain throughput and increase update frequency.The authors state that these strategies facilitate faster convergence.Source: §4.3 · p. 10
  • Think mode has the MLLM rewrite the prompt and adjust the vision-head output to match, allowing the diffusion model to generate from the rewritten prompt without retraining the diffusion model.Source: §4.3 · p. 10

5 Experiments

Experiments evaluate Omni-Video across generation, editing, and understanding tasks. On VBench-Long, it leads in four metrics and surpasses Wan2.1-T2V-1.3B in 10 of 16 categories, alongside qualitative results across video tasks.

5.1 Experiment Settings

  • Generation uses UniPC, CFG (5.0/50 steps for T2I; 3.0/40 for T2V), and a universal negative prompt; understanding uses top-p 0.6, temperature 0.2, and a 1,024-token cap.The appendix provides the evaluation prompts.Source: §5.1 · p. 11

5.2 Experiment Results

  • On GenEval, performance is comparable to most state-of-the-art unified models despite short test prompts versus detailed training prompts; qualitative T2I outputs show fidelity across portraits, animals, and landscapes.No special fine-tuning was performed to address the prompt-domain gap.Source: Table 2 · p. 12, §5.2 · pp. 12–13
  • On VBench-Long, Omni-Video is state of the art in Subject Consistency (98.39%), Temporal Flickering (99.87%), Object Class recognition (93.54%), and Temporal Style consistency (25.81%), surpassing Wan2.1-T2V-1.3B in 10/16 categories.Its overall performance is reported as competitive with Wan2.1-T2V-1.3B, which has the highest quality score (85.23%).Source: Table 3 · p. 12, §5.2 · p. 12
  • Text-to-video examples show coherent motion in facial micro-expressions, rabbit gallops, and urban splash and reflection sequences.The examples describe subject framing, locomotion, and silhouette continuity across changing movement.Source: Figure 6 · p. 13, §5.2 · p. 13
  • Image edits preserve structure and texture across botanical insertions, style transfers, sky transitions, and perspective-aware lake expansion.Figure 7 presents the source image, ground truth, and model output in each row.Source: §5.2 · p. 13, Figure 7 · p. 14
  • Video editing spans object removal, background replacement, and attribute addition while maintaining cross-frame consistency and detail without flicker.Examples include changing a street background to an ocean, removing a person, and adding a hat.Source: Figure 8 · p. 15, §5.2 · p. 16
  • Video-understanding examples combine object counts and attributes with motion, seasonal context, and affect, including puppy movements partly obscured by occlusion.The examples cover both a Pleistocene scene and a winter scene with golden retriever puppies.Source: Figure 9 · p. 16, §5.2 · p. 16
  • Think mode uses long chain-of-thought to parse ambiguous instructions into visual-token choices, illustrated by material, architectural-proportion, and environmental details.The mausoleum example interprets descriptors such as “elegant marble” and “Indian architectural brilliance.”Source: §5.2 · p. 16, Figure 10 · p. 17

6 Limitations

The framework remains constrained in high-fidelity, long-duration video synthesis by memory demands and limited training data, while its decoupled components can misalign representations.

6 Limitations

  • High-fidelity, long-duration synthesis is constrained by memory demands and limited training data for complex interactions and distribution modeling.The decoupled diffusion-decoder and MLLM design can misalign latent representations, particularly for frame-by-frame causal reasoning in video understanding.Source: §6 · p. 17

A Prompts Used in the Main Text

This section lists prompts used in the main-text figures, ordered to match each figure’s visual layout from top to bottom and left to right.

Figure 1

Figure 5

Figure 6

Figures 7–8

  • Figure 7 and Figure 8 editing prompts request object additions or removal, style and color changes, and scene transformations.Examples include adding a flower or hat, changing clouds or water color, and turning a park into a beach.Source: A Prompts Used in the Main Text · p. 25
Loading 2507.06119v4…