Source-linked AI summary
Uni-ViGU: Towards Unified Video Generation and Understanding via A Diffusion-Based Video Generator
Luozheng Qin, Jia Gong, Qian Qiao, Tianjiao Li, Li Xu, Haoyu Pan, Chao Qu, Zhiyu Tan, Hao Li
TL;DR
Visual generation, especially video, is far more computationally costly than understanding, motivating a generation-centric unified model. Uni-ViGU combines continuous video and discrete text flow matching with shared attention, modality-specific FFNs, and bidirectional training, achieving high-quality joint video-text generation while supporting both generation and understanding.
Problem
Visual generation requires substantially more computation than understanding, making unified video generation and understanding difficult to implement efficiently.
Method
Uni-ViGU extends a video generator with unified continuous video and discrete text flow matching, shared attention, modality-specific FFN layers, and two-stage bidirectional training.
Results
Uni-ViGU produces high-quality videos paired with detailed captions that are more descriptive and faithful to visual content than the original prompts.
Takeaways & Limitations
The framework supports joint video-text generation and repurposes text-to-video correspondences for unified video understanding and generation.
Abstract
from arXiv · showhide
Unified multimodal models integrating visual understanding and generation face a fundamental challenge: visual generation incurs substantially higher computational costs than understanding, particularly for video. This imbalance motivates us to invert the conventional paradigm: rather than extending understanding-centric MLLMs to support generation, we propose Uni-ViGU, a framework that unifies video generation and understanding by extending a video generator as the foundation. We introduce a unified flow method that performs continuous flow matching for video and discrete flow matching for text within a single process, enabling coherent multimodal generation. We further propose a modality-driven MoE-based framework that augments Transformer blocks with lightweight layers for text generation while preserving generative priors. To repurpose generation knowledge for understanding, we design a bidirectional training mechanism with two stages: Knowledge Recall reconstructs input prompts to leverage learned text-video correspondences, while Capability Refinement fine-tunes on detailed captions to establish discriminative shared representations. Experiments demonstrate that Uni-ViGU achieves competitive performance on both video generation and understanding, validating generation-centric architectures as a scalable path toward unified multimodal intelligence. Project Page and Code: https://fr0zencrane.github.io/uni-vigu-page/.
1 Introduction
Uni-ViGU addresses the difficulty of unifying visual understanding and generation by using a video generator as the foundation. It combines unified flow matching, modality-specific adaptations, and bidirectional training to repurpose generation knowledge for understanding.
- Unified multimodal models seek to combine visual understanding and generation, supporting shared representations and streamlined model design.
- Autoregressive visual generation can limit fidelity, while diffusion-based MLLM approaches often decouple generation from understanding.
- Video generation is substantially more expensive than understanding because iterative denoising expands token consumption across frames.A 5-second 720P video may require millions of tokens, compared with thousands for a single image.
- Uni-ViGU extends a diffusion-based video generator to unify video generation and understanding within one foundation.
- Uni-ViGU performs continuous flow matching for video and discrete flow matching for text within a single generative process.This design addresses the mismatch between continuous video latents and discrete text tokens.
- A modality-driven MoE framework preserves pretrained generative priors while separating video and text generation through different FFN layers.Knowledge Recall reconstructs dropped-out input prompts, followed by Capability Refinement on detailed captions for fine-grained semantics.
2 Preliminary
WAN2.1 provides the latent-diffusion video-generation foundation underlying Uni-ViGU. Its flow-matching process transports Gaussian noise to video latents while conditioning velocity prediction on text.
- Uni-ViGU builds on WAN2.1, an efficient text-to-video generator using a standard latent diffusion paradigm.The authors state that the approach can extend naturally to other video generators with similar architectures.
- Video generators iteratively denoise Gaussian noise into videos in latent space, where a VAE compresses pixel information to reduce computational cost.
- WAN2.1 encodes a video into a latent representation and constructs intermediate latents between that representation and Gaussian noise.
- Flow matching defines a transport path from noise to data with constant velocity u = z1−z0.
- A neural network predicts the target velocity from the text prompt, intermediate latent, and diffusion time step.
- WAN2.1 uses spatial-temporal modeling for video features and cross-attention to inject text semantics.
3 Method
Uni-ViGU extends a pretrained video generator into a unified model for video generation and understanding. It combines continuous and discrete flow matching with shared attention, modality-specific experts, and bidirectional training.
- Uni-Flow: Uni-flow performs continuous flow matching for video and discrete flow matching for text within one generative process.Video uses continuous latent-space paths, while text is modeled in token-embedding space.
- Uni-Flow: Independently sampled modality time variables let video and text progress through different denoising stages, learning cross-modal dependencies across noise levels.The model jointly predicts both modality velocity fields under a combined objective.
- Uni-Flow: Setting τv = 1, τt = 0 enables video understanding, whereas τv = 0, τt = 1 enables video generation.The same unified flow supports clean-video/noisy-text and noisy-video/clean-text configurations.
- Modality-Driven MoE: The modality-driven MoE shares attention across concatenated video and text tokens while separating FFN experts by modality.Deterministic routing preserves shared relational reasoning and accommodates modality-specific generation.
- Modality-Driven MoE: The video expert retains pretrained weights, while a newly initialized text expert adds text-generation capacity with minimal parameter overhead.The asymmetric initialization is intended to preserve generative priors and facilitate rapid convergence.
- Bidirectional Training: Bidirectional training first recalls conditioning prompts, then refines video understanding through a second capability-development stage.Knowledge Recall leverages pretrained prompt-to-video correspondences for reverse video-to-text mapping; the two stages establish bidirectional mapping in one model.
- Bidirectional Training: Token-count normalization assigns λv = 1.0 and λt = |zv|/|zt| so video and text receive balanced per-token supervision.Video contributes approximately 30K tokens, compared with 256 text tokens, and video generation is already well modeled during pretraining.
4 Experiment
Uni-ViGU is trained through curated video-text data and a two-stage bidirectional procedure, then evaluated on joint video-text generation. The evaluation reports simultaneous denoising and mutually refined outputs, including captions that are more descriptive and faithful than conditioning prompts.
- Uni-ViGU is trained on meticulously curated video-text pairs using the two-stage bidirectional training framework.
- Stage 1 uses 10K video-prompt pairs for Knowledge Recall, while Stage 2 adds 10K video-prompt-detailed caption triples for Capability Refinement.Stage 1 reconstructs conditioning prompts; Stage 2 uses detailed captions as text-generation targets.
- The training data combines prompts, videos synthesized by state-of-the-art video generators, and LLM-produced detailed captions covering the videos and prompts.The paired data also enforces token-length constraints on conditioning prompts and captions.
- The model is initialized from the pretrained Wan2.1 video generator, retaining pretrained video-expert FFN weights while newly initializing the text-expert FFN.Stage 1 and Stage 2 use 40K and 60K training steps, respectively.
- Joint generation simultaneously denoises video and text from Gaussian noise, with shared attention enabling mutual refinement between modalities.The resulting videos are paired with captions described as more descriptive and faithful to visual content than the original conditioning prompts.
5 Related Work
Related work progresses from autoregressive and diffusion-based multimodal generation toward tighter integration, while Uni-ViGU instead extends video generators for understanding. Its unified flow formulation combines continuous video and discrete text flow matching in one process.
- Early unified multimodal approaches cast image generation as autoregressive prediction in MLLMs, while later methods use continuous diffusion modules to preserve visual fidelity.The latter methods commonly freeze a pretrained MLLM and add learnable query tokens as an interface to diffusion generation.
- Dual-tower frameworks couple separate understanding and generation branches through cross-attention for tighter multimodal integration.These systems train an MLLM for understanding alongside a duplicated MLLM serving as a generator.
- Video generation has shifted from 3D U-Net architectures toward scalable Diffusion Transformers that model complex temporal dynamics.Wan, CogVideoX, and OpenSora are cited as DiT-based systems for high-fidelity, long-form video synthesis.
- Video generators acquire text-to-video correspondences during large-scale pretraining, but this semantic knowledge is rarely used for explicit language-level comprehension.Uni-ViGU repurposes these learned correspondences bidirectionally.
- Uni-ViGU bridges isolated diffusion language modeling and visual synthesis by applying continuous flow matching to video and discrete flow matching to text within one generative process.The shared objective is described as enabling coherent joint optimization of both modalities.
6 Conclusion
The conclusion presents Uni-ViGU as a generation-centric framework that extends pretrained video generators to video generation and understanding. It combines unified flow matching, modality-specific FFN layers with shared attention, and bidirectional knowledge transfer.
- Uni-ViGU extends pretrained video generators to support both video generation and understanding.
- The framework repurposes visual-semantic correspondences learned during text-to-video pretraining for video understanding by treating understanding as the inverse of generation.
- Its contributions include uni-flow, modality-driven MoE blocks with shared attention and separated FFN layers, and bidirectional training that progressively activates cross-modal transfer.
- The paper positions the generation-centric paradigm as a principled and scalable alternative for unified multimodal intelligence.