PaperlayerView exact revision on arXiv

Source-linked AI summary

UniVideo: Unified Understanding, Generation, and Editing for Videos

Cong Wei, Quande Liu, Zixuan Ye, Qiulin Wang, Xintao Wang, Pengfei Wan, Kun Gai, Wenhu Chen

arXiv:2510.08377v4cs.CV

TL;DR

UniVideo addresses the limited extension of unified multimodal modeling from images to video and introduces a dual-stream MLLM–MMDiT framework for understanding and generation. Joint training across video generation and editing tasks yields competitive performance and generalization to unseen edits and task compositions, while retaining multimodal reasoning and text generation.

  • Problem

    Unified multimodal understanding, generation, and editing remained largely limited to text and images, while video systems struggled to handle diverse multimodal instructions and tasks in one model.

  • Method

    UniVideo combines a frozen MLLM for multimodal instruction understanding with an MMDiT and VAE-based visual conditioning for video generation, jointly training across diverse tasks.

  • Results

    UniVideo matches or surpasses task-specific baselines across text/image-to-video generation, video editing, and in-context generation, while generalizing to unseen edits and task compositions.

  • Takeaways & Limitations

    A unified instruction-driven model can combine video generation and editing capabilities and transfer editing ability from image data to free-form video editing.

  • Takeaways & Limitations

    UniVideo sometimes over-edits unrelated regions, struggles to preserve original-video motion, and has lower free-form video-editing success than image-editing success.

Abstract

from arXiv · show

Unified multimodal models have shown promising results in multimodal content generation and editing but remain largely limited to the image domain. In this work, we present UniVideo, a versatile framework that extends unified modeling to the video domain. UniVideo adopts a dual-stream design, combining a Multimodal Large Language Model (MLLM) for instruction understanding with a Multimodal DiT (MMDiT) for video generation. This design preserves the MLLM's original text generation capabilities, enables accurate interpretation of complex multimodal instructions, and maintains visual consistency in the generated content. Built on this architecture, UniVideo unifies diverse video generation and editing tasks under a single multimodal instruction paradigm and is jointly trained across them. Extensive experiments demonstrate that UniVideo matches or surpasses state-of-the-art task-specific baselines in text/image-to-video generation, in-context video generation and in-context video editing. Notably, the unified design of UniVideo enables two forms of generalization. First, UniVideo supports task composition, such as combining editing with style transfer, by integrating multiple capabilities within a single instruction. Second, even without explicit training on free-form video editing, UniVideo transfers its editing capability from large-scale image editing data to this setting, handling unseen instructions such as changing the environment or altering materials within a video. Beyond these core capabilities, UniVideo also supports visual-prompt-based video generation, where the MLLM interprets visual prompts and guides the MMDiT during synthesis. To foster future research, we released our model and code.

1 INTRODUCTION

UniVideo extends unified multimodal understanding, generation, and editing from images to videos through a single instruction-driven framework. Its dual-stream design supports diverse tasks, multimodal reasoning, and generalization to unseen edits and task compositions.

  • Motivation: Unified multimodal models had largely remained limited to text and images, leaving multimodal video understanding and unified video editing underexplored.Existing video generators mainly target text-to-video, while editing systems commonly use task-specific pipelines.
  • Approach: UniVideo combines an MLLM for multimodal instruction understanding with an MMDiT for visual generation, extending a design previously explored in images to video.The MLLM supports semantic reasoning, while the MMDiT supplies generation capability and visual consistency.
  • Unified training: UniVideo is jointly trained across text-to-image, text-to-video, image-to-video, in-context generation, in-context editing, and image editing.Natural-language multimodal instructions unify these tasks within one framework.
  • Results: UniVideo matches or surpasses task-specific baselines across text/image-to-video generation, video editing, and in-context generation.The conclusion reports this pattern across the evaluated task families.
  • Generalization: UniVideo generalizes to unseen free-form video edits and novel task compositions without explicit training on those settings.Examples include changing materials or weather and combining editing with style transfer or multiple identity edits.
  • Multimodal reasoning: The frozen MLLM preserves visual understanding and text generation, enabling interpretation of ambiguous multimodal instructions and visual prompts.UniVideo is therefore characterized as a post-trained unified multimodal generative system rather than a model trained from scratch.

2 METHOD

UniVideo uses an MLLM–MMDiT dual stream to process multimodal instructions and generate videos. Training aligns the streams before progressively fine-tuning generation and broadening to multiple tasks.

  • Model architecture: The MLLM handles visual-textual understanding, while the MMDiT combines semantic features from the MLLM with fine-grained visual signals from a VAE.A trainable connector aligns MLLM hidden states with the MMDiT understanding stream.
  • Task conditioning: Text-to-video and image-to-video inputs are routed through the MLLM for understanding and the MMDiT for generation, with visual inputs additionally encoded for the generation branch.The same instruction paradigm extends to in-context video generation and editing.
  • Task conditioning: Multiple visual conditions are VAE-encoded, padded to a uniform shape, concatenated along the temporal axis, and processed with self-attention.This supports reference images and videos in in-context generation and editing.
  • Visual prompting: The MLLM can translate annotated visual prompts into structured plans and dense prompt tokens that directly guide MMDiT video generation.This avoids invoking multiple downstream generators through an agent-based design.
  • Training procedure: Training proceeds from connector alignment, to T2I/T2V fine-tuning, to multi-task training while keeping the MLLM frozen.The final stage adds in-context generation, video editing, image editing, and image-to-video tasks.

3 EXPERIMENTS

UniVideo is evaluated as a unified model across visual understanding, image and video generation, image editing, and in-context video tasks. Experiments report competitive or superior performance, including generalization to unseen editing instructions and task compositions.

  • Experimental scope: UniVideo is benchmarked across visual understanding, text-to-image, text-to-video, image-to-video, image editing, in-context video generation, and in-context video editing.The evaluation spans both generalist capabilities and task-specific comparisons.
  • Visual understanding and generation: 83.5 on MMBench, 58.6 on MMMU, and 66.6 on MM-Vet demonstrate competitive visual-understanding scores while UniVideo supports I2V and T2V in one model.Understanding results use the MLLM backbone, while generation performance is comparable to the HunyuanVideo backbone on VBench.
  • Image generation: UniVideo retains competitive text-to-image performance despite being an image–video generalist model trained with only a small proportion of T2I samples in Stage 3.It outperforms many specialized text-to-image and unified text–image models on GenEval.
  • Image editing: UniVideo achieves strong image-editing performance and surpasses several specialized image-editing models on ImgEdit-Bench and GEdit-Bench-EN.Semantic Consistency is reported as the primary metric.
  • In-context video generation: UniVideo achieves superior or competitive performance across all in-context generation metrics and preserves identity when following complex instructions involving multiple identities.The evaluation includes Subject Consistency, Prompt Following, Overall Video Quality, Smoothness, and Aesthetics; baselines often struggle with four reference images.
  • In-context video editing: UniVideo achieves superior or competitive in-context editing performance under the more challenging mask-free setting while preserving reference-image identity.Quantitative comparisons are reported in Table 5.
  • Zero-shot generalization: Without explicit training on general free-form video editing, UniVideo handles instructions such as changing materials or environments and composes multiple edits with style transfer.Examples include deleting one identity while adding another.
  • Ablation study: Multi-task learning benefits editing, while removing visual inputs from the MMDiT causes a dramatic drop in identity preservation.The ablation compares UniVideo with single-task models and a variant feeding visual inputs only to the MLLM.

4 RELATED WORK

Unified multimodal generation has advanced mainly in text and image domains, while video systems remain dominated by single-task frameworks. Related work therefore motivates unified modeling for video understanding, generation, and editing.

  • Unified multimodal understanding and generation: Recent unified multimodal systems jointly optimize understanding and generation, but their strongest advances remain concentrated in the text and image domains.The literature includes autoregressive, diffusion–autoregression hybrid, and LLM-based regression approaches.
  • Image/video generation and editing: Video generation and editing remain dominated by single-task or task-specific frameworks, and existing unification attempts still require task-specific pipelines.This contrasts with unified image editing systems and motivates a unified video model.

5 CONCLUSION

UniVideo unifies multimodal understanding, video generation, and editing through an MLLM–MMDiT design. It matches or outperforms task-specific baselines, generalizes to unseen tasks and compositions, and supports visual-prompt-based generation.

  • UniVideo integrates an MLLM for multimodal reasoning with an MMDiT for visually consistent generation across images and videos.
  • UniVideo matches or outperforms task-specific baselines across text/image-to-video, video editing, and in-context generation.
  • The unified model generalizes to unseen tasks and novel task compositions that specialized pipelines struggle to achieve.
  • UniVideo uses the MLLM stream to interpret complex multimodal prompts, including visual prompts, before generation.
  • The work positions UniVideo as a step toward assistants communicating naturally through language, images, and video.

D.1 MODEL DESIGN

The model-design study compares connector and diffusion-generator architectures for aligning a frozen MLLM with generation. UniVideo’s MMDiT alignment provides robust text following while query conditioning trades training memory for inference efficiency.

  • MODEL DESIGN: The study asks which alignment approach most effectively connects a pretrained MLLM to a diffusion generator during Stage 1 training.
  • MODEL DESIGN: Cross-attention DiT variants replace the original text encoder with an MLP connector and test standard or learnable-query conditioning.
  • MODEL DESIGN: UniVideo uses MMDiT self-attention for joint text–video interaction, with only the MLP connector trained while the MLLM and MMDiT remain frozen.
  • MODEL DESIGN: Cross-attention variants require an unfrozen DiT for effective alignment, yet still show limited compositional text following compared with UniVideo.
  • MODEL DESIGN: UniVideo (query) uses a fixed number of learnable queries, reducing conditioning tokens and improving inference efficiency for video-dominated tasks such as editing.
  • MODEL DESIGN: The query variant incurs additional training memory because optimizing its learnable queries requires backpropagation through the MLLM.

E TRAINING DETAILS

UniVideo combines pretrained Qwen2.5-VL and HunyuanVideo-T2V backbones, using Qwen2.5-VL as a unified multimodal embedder for the MMDiT.

  • UniVideo uses qwen2.5VL-7B as its MLLM backbone and HunyuanVideo-T2V-13B as its MMDiT backbone.

F LIMITATION AND FUTURE WORK

UniVideo’s multimodal training supports diverse tasks but does not always follow editing instructions precisely or preserve original video motion. The authors identify stronger video backbones and larger video-editing datasets as future needs.

  • UniVideo sometimes over-edits unrelated regions instead of strictly following editing instructions.
  • The model struggles to fully preserve motion in original videos because of backbone limitations.
  • Although UniVideo generalizes to free-form video editing, its success rate is lower than in image editing.
  • Future work includes large-scale video-editing datasets, improved backbones, and native end-to-end multimodal video models.

G TRAINING DATASET CONSTRUCTION

This section introduces the construction of UniVideo’s datasets.

  • The section details the construction of UniVideo’s datasets.

G.1 ID-RELATED TASKS

UniVideo constructs and filters data for identity-related video tasks, stylization, and image editing. The resulting resources include approximately 500K high-quality edited image samples and a 20-case in-context video generation test set.

  • ID-related tasks: In-context video data uses keyframes and Qwen2.5-VL-7B to identify semantically meaningful primary subjects while ignoring irrelevant backgrounds.
  • ID-related tasks: Object-swap training pairs combine an inpainter-generated video and subject image as input with the original video as target.
  • ID-related tasks: Object removal and addition use surrounding visual context for inpainting, with task-specific input-target pair construction.
  • ID-related tasks: Human filtering evaluates edited videos for quality, instruction following, and consistency with the source video, rejecting artifacts, partial edits, and temporal flicker.
  • ID-related tasks: Stylized-to-real conversion uses a grayscale, downsampled-and-upsampled tile signal that preserves spatial layout while suppressing style and texture.
  • ID-related tasks: Approximately 500K high-quality edited image samples remain after filtering samples that fall below threshold on any evaluation dimension.
  • ID-related tasks: The in-context video generation test set contains 20 cases, evenly split between single-ID and multi-ID scenarios, with diverse subjects including humans, animals, objects, and cartoons.

H.2 IN-CONTEXT VIDEO EDITING

UniVideo evaluates instruction-based in-context video editing on four UNICBench tasks: ID insertion, ID swap, ID deletion, and stylization. The setting replaces mask-based editing inputs with direct instructions while preserving target video content or motion as specified.

  • H.2 IN-CONTEXT VIDEO EDITING: UNICBench evaluation covers ID insertion, ID swap, ID deletion, and stylization using instruction-based inputs rather than masks.
  • ID Insertion: ID insertion integrates objects or elements from a reference identity into the target video using a direct instruction.
  • ID Swap: ID swap replaces specified target-video elements with corresponding reference-identity elements while preserving the original context and motion.
  • ID Deletion: ID deletion removes specified video elements and fills their regions with appropriate background content while maintaining visual consistency.
  • Stylization: Stylization transforms the target video toward a reference image’s artistic style while preserving its content and motion dynamics.
Loading 2510.08377v4…