PaperlayerView exact revision on arXiv

Source-linked AI summary

UniVid: The Open-Source Unified Video Model

Jiabin Luo, Junhui Lin, Zeyu Zhang, Biao Wu, Meng Fang, Ling Chen, Hao Tang

arXiv:2509.24200v2cs.CV

TL;DR

UniVid addresses challenges in faithful video-generation conditioning and efficient extension of image-centric MLLMs to video understanding. It couples an MLLM with a diffusion decoder, using timestep-aware attention and query-driven keyframe selection. The paper reports state-of-the-art or competitive results across VBench-Long and video-QA benchmarks.

  • Problem

    Video generation must maintain semantic faithfulness across flow steps, while extending image-centric MLLMs to video raises temporal-computation costs and makes relevant evidence selection difficult.

    Source: §1 · p. 2

  • Method

    UniVid couples an MLLM and diffusion video decoder with a lightweight adapter, using Temperature Modality Alignment for generation and Pyramid Reflection for query-driven keyframe selection.

    Source: §1 · p. 2, §3.4 · p. 6

  • Results

    UniVid reports state-of-the-art or competitive results on VBench-Long and video-QA benchmarks, including an overall VBench-Long score of 85.27 and 7B-scale state of the art on MSVD-QA and ActivityNet-QA.

    Source: §4.3 · pp. 8–9, §5 · p. 9

  • Takeaways & Limitations

    UniVid demonstrates a unified approach to video understanding and generation while avoiding costly retraining of image-centric backbones.

    Source: §5 · p. 9

    The keyframe-driven understanding pipeline works well on short clips but struggles with long videos and fine motion; native video encoders would require substantial training.

    Source: §A.7 · p. 24

Abstract

from arXiv · show

Unified video modeling that combines generation and understanding capabilities is increasingly important but faces two key challenges: maintaining semantic faithfulness during flow-based generation due to text-visual token imbalance and the limitations of uniform cross-modal attention across the flow trajectory, and efficiently extending image-centric MLLMs to video without costly retraining. We present UniVid, a unified architecture that couples an MLLM with a diffusion decoder through a lightweight adapter, enabling both video understanding and generation. We introduce Temperature Modality Alignment to improve prompt adherence and Pyramid Reflection for efficient temporal reasoning via dynamic keyframe selection. Extensive experiments on standard benchmarks demonstrate state-of-the-art performance, achieving a 2.2% improvement on VBench-Long total score compared to EasyAnimateV5.1, and 1.0% and 3.3% accuracy gains on MSVD-QA and ActivityNet-QA, respectively, compared with the best prior 7B baselines. Code: https://github.com/AIGeeksGroup/UniVid. Website: https://aigeeksgroup.github.io/UniVid.

1 INTRODUCTION

UniVid addresses the challenge of combining faithful video generation with efficient video understanding. It couples an MLLM and diffusion decoder, using timestep-aware attention and adaptive keyframe selection to support both tasks.

1 INTRODUCTION

  • The paper adopts a hybrid diffusion–AR approach to retain high-quality rendering while using an MLLM for semantic control and interpretability.The alternative AR-centric route projects modalities into shared discrete tokens for next-token prediction.Source: §1 · p. 2
  • Faithful conditioning across flow steps and efficient video understanding are central challenges, including text–visual token imbalance and selecting relevant temporal evidence.Uniform frame processing can be inefficient and noisy, while fixed sampling may miss evidence.Source: §1 · p. 2
  • UniVid couples an MLLM to a diffusion video decoder through a lightweight adapter, with semantic tokens supporting reasoning and text/image-to-video generation.The architecture also includes Temperature Modality Alignment and Pyramid Reflection for generation conditioning and temporal evidence selection.Source: §1 · p. 2, Figure 2 · p. 3
  • Experiments cover four video-QA datasets and VBench, with the authors reporting competitive performance and efficiency and ablations that verify component contributions.The introduction names MSVD-QA, MSRVTT-QA, TGIF-QA, ActivityNet-QA, and VBench.Source: §1 · p. 3

2 RELATED WORK

Related work spans video generation, video understanding, and unified multimodal models. The paper situates its approach among efforts to combine understanding and generation, including extensions from image-centric systems to video.

Video generation

  • Diffusion and flow models and GANs have advanced video generation, while long-term temporal consistency in extended sequences remains challenging.Examples include Video Diffusion Models (Ho et al., 2022b), Imagen Video (Ho et al., 2022a), VideoCrafter2 (Chen et al., 2024a), and Stable Video Diffusion (Blattmann et al., 2023b).Source: §2 · p. 3

Video understanding

  • Transformer-based and self-supervised methods model spatiotemporal relationships and learn transferable video representations, including through masked modeling and contrastive learning.The passage notes that self-supervised approaches reduce dependence on costly annotations.Source: §2 · pp. 3–4

Unified multimodal models

  • Unified multimodal systems have expanded from vision–language pretraining toward joint understanding and generation, with recent work extending such pipelines from images to video.Omni-Video adapts MLLM-produced continuous visual tokens for a diffusion video decoder.Source: §2 · p. 4

3 THE PROPOSED METHOD

UniVid uses a three-stage hierarchical training recipe to couple an MLLM with a video generator and support both generation and understanding. Its generation branch adapts MLLM conditions across flow steps, while Pyramid Reflection iteratively selects evidence for video question answering.

3.1 OVERVIEW

  • A three-stage hierarchical training recipe aligns MLLM conditioning with the generator, adds temporal cues through Pyramid Reflection, and then co-adapts both branches end-to-end.This overview frames the method as a unified model for video generation and understanding.Source: §3.1 · p. 4

3.2 ARCHITECTURE

  • The MLLM processes text and visual features into unified multimodal representations, conditioning Wan 2.2 for latent-space video generation and answering understanding tasks with iteratively selected keyframes.Generation uses ViT and VAE image encoding, while understanding uses ViT encoding; Pyramid Reflection selects frames through SigLIP2 and an Actor–Evaluator–Reflector loop without modifying the backbone.Source: §3.2 · p. 4

3.3 CONDITIONAL GENERATION WITH TEMPERATURE MODALITY ALIGNMENT

  • Temperature Modality Alignment uses a lightweight adapter to map MLLM outputs into time-indexed conditions and schedules attention to emphasize text early and visual refinement later.The DiT predicts a velocity field under cross-attention, and the resulting latent trajectory is decoded into video frames.Source: §3.3 · p. 5

3.4 PYRAMID REFLECTION FOR UNDERSTANDING

  • Pyramid Reflection treats video question answering as test-time decision-making over ordered frame evidence, expanding or pruning frames and verbally refining search text from evaluator feedback.Cached embeddings and relevance/diversity-based routing support static and dynamic questions, while confidence can trigger early stopping.Source: §3.4 · p. 6

4 EXPERIMENTS

UniVid is evaluated on video-generation and video-QA benchmarks, using a three-stage training recipe and task-specific inference settings. It reports strong generation results and state-of-the-art 7B-scale performance on two QA benchmarks, alongside qualitative examples of generation and understanding.

4.1 DATASET AND METRICS

  • Generation training uses curated OpenVid-1M samples and evaluation uses VBench; understanding training uses 20k ActivityNet-QA samples and evaluation spans four video-QA benchmarks.The benchmarks cover clips from short animated sequences to multi-minute activity videos; evaluation reports average accuracy and dataset scores for understanding, and fine-grained dimensions for generation.Source: §4.1 · pp. 6–7

4.2 IMPLEMENTATION DETAILS

  • Training proceeds from generation alignment to video-QA adaptation and then joint co-training, initializing from public checkpoints and updating selected components rather than training the unified model from scratch.Generation uses a textual adapter and DiT LoRA; understanding tunes the connector and last two ViT blocks with the LLM frozen, and joint training alternates the tasks.Source: §4.2 · p. 7
  • Generation uses 1280 × 704, 121-frame videos at 24 fps and 50 inference steps; understanding selects keyframes dynamically or on a static schedule from a cached 64-frame pool.The generation guidance scale is 5.0, with text gain scheduled from 1.3 to 1.0 over the first 40% of denoising steps; dynamic understanding uses a 64 →32 →16 keyframe schedule with MMR down-selection.Source: §4.2 · p. 7

4.3 MAIN RESULTS

  • 85.27 overall on VBench-Long establishes a new state of the art; UniVid also scores 80.58 in semantic alignment, 99.88 in temporal consistency, 99.25 in motion consistency, and 73.03 in imaging.The paper reports the best imaging score and describes the temporal and motion scores as near-perfect.Source: §4.3 · p. 8
  • 77.45 multi-object reasoning, 92.10 color faithfulness, and 80.70 spatial grounding are leading semantic-fidelity results, while action depiction and appearance consistency remain competitive.The paper attributes these results to design choices including hierarchical scene representation and dynamic frame alignment.Source: §4.3 · p. 8
  • In qualitative generation comparisons, competing models often omit basketballs or distort cars, while UniVid produces coherent jump shots and realistic racing scenes with stable dynamics and faithful semantics.The comparison includes Wan2.1-T2V-1.3B, MiniMax-Video-01, HunyuanVideo, and EasyAnimateV5.1.Source: §4.3 · p. 8
  • UniVid reaches the 7B-scale state of the art on MSVD-QA and ActivityNet-QA and remains competitive on MSRVTT-QA and TGIF-QA.The paper notes a smaller post-training set and no test-time ensembling; it associates joint finetuning with Pyramid Reflection with improved action–entity binding, grounding, temporal reasoning, and long-range evidence retrieval.Source: §4.3 · p. 9
  • Video-QA examples show baselines giving plausible but incomplete answers, while UniVid is described as producing precise answers through stronger action–entity binding, temporal reasoning, and resistance to distractor frames.The paper also describes Pyramid Reflection steering attention toward relevant keyframes when initial evidence is insufficient.Source: §4.3 · p. 9

5 CONCLUSION

UniVid combines an MLLM and diffusion decoder to support both video understanding and generation. Temperature Modality Alignment and Pyramid Reflection underpin its reported benchmark performance and efficient temporal reasoning.

5 CONCLUSION

  • UniVid couples an MLLM with a diffusion decoder through a lightweight adapter to support both video understanding and generation.The conclusion identifies Temperature Modality Alignment for scheduling cross-modal attention across flow steps and Pyramid Reflection for query-driven keyframe selection.Source: §5 · p. 9
  • The paper reports state-of-the-art or competitive results on VBench-Long and multiple video-QA benchmarks while avoiding costly retraining of image-centric backbones.It presents the released model as support for research on practical, controllable, unified video intelligence.Source: §5 · p. 9

A.1 LLM USE DECLARATION

The authors state that language models were used only to improve the clarity and fluency of the paper’s English.

A.1 LLM USE DECLARATION

  • ChatGPT was used only for English-writing clarity and fluency, not for research ideation, experimental design, data analysis, or interpretation.The authors take responsibility for the paper’s content.Source: §A.1 · p. 17

A.2 HIERARCHICAL POST TRAINING

UniVid’s post-training procedure starts from pretrained components, aligns the generation branch, adapts the model for video QA, and then co-trains both tasks. Training data and parameter updates differ by stage.

A.2 HIERARCHICAL POST TRAINING

  • Initialization combines Qwen2, SigLIP2, and a frozen FLUX VAE, interleaving text, visual, and VAE tokens through generalized causal attention.The authors say checkpoint initialization and tuning only small parameter subsets avoid the prohibitive cost of training a unified video model from scratch.Source: §A.2 · p. 17
  • Understanding examples use ActivityNet-QA converted to two-turn dialog records, while generation uses curated OpenVid-1M text/video and image/video pairs.Videos are uniformly subsampled and preprocessed identically to inference.Source: §A.2 · p. 17
  • Stage I aligns MLLM-produced tokens with Wan 2.2 using a textual adapter and DiT cross-attention LoRA, while freezing other DiT and MLLM weights.Training optimizes the context projector and LoRA parameters with a flow-matching objective and classifier-free guidance dropout on text.Source: §A.2 · p. 17
  • Stage II adapts the model for ActivityNet-QA using 20k samples, instruction supervision, and updates to only the last two ViT blocks and connector for four epochs.The LLM remains frozen, and loss is computed on assistant turns while user tokens are masked.Source: §A.2 · p. 17
  • Stage III alternates generation and understanding minibatches, updating primarily the connector and textual adapter with the sum of generation velocity loss and masked autoregressive loss.The paper describes this joint stage as optimizing the MLLM to improve both tasks.Source: §A.2 · p. 17

A.3 MORE EXAMPLES OF VIDEO GENERATION AND UNDERSTANDING.

Figures 5–7 provide qualitative examples of video understanding and generation, with Figure 5 distinguishing static from dynamic questions.

A.3 MORE EXAMPLES OF VIDEO GENERATION AND UNDERSTANDING.

  • Figure 5 presents qualitative video-understanding examples, using blue for static questions and green for dynamic questions.The colors identify question types; the passage does not report comparative outcomes.Source: Figure 5 · p. 18
  • Figure 6 presents qualitative results for text-to-video (T2V) and text-image-to-video (TI2V) generation.Source: Figure 6 · p. 19
  • Figure 7 shows the video-understanding pipeline.Source: Figure 7 · p. 20

A.4 TEXT PROMPTS USED IN THE UNDERSTANDING

The prompts define a staged understanding workflow: classify question type, analyze frames or a global caption, evaluate answers, and refine retrieval when needed.

A.4 TEXT PROMPTS USED IN THE UNDERSTANDING

  • The question classifier labels questions static when a small set of unordered frames suffices and dynamic when temporal reasoning is required.Examples of dynamic cues include counting, repetition, order, and change over time.Source: §A.4 · p. 20
  • The frame-summarization prompt turns chronologically ordered frame notes into a concise, factual global caption without inventing facts.It asks the caption to connect entities and actions across time and include temporal hints only when evidenced.Source: §A.4 · p. 20
  • The evaluator scores an answer from 0 to 1 and accepts it at 0.7 or above; a rejected answer can trigger a concise retrieval refinement.The reflector is instructed to return an empty refinement when confidence is already good or the answer is accepted.Source: §A.4 · p. 21
  • The single-frame prompt requests concise descriptions of entities, actions, salient objects, object states, and scene context.Source: §A.4 · p. 21
  • The answer prompt uses the question and global caption, requiring an evidence-limited response or “Not enough evidence from global caption.”Source: §A.4 · p. 21

A.5 PYRAMID REFLECTION AS TEST-TIME RL

Pyramid Reflection iteratively adjusts an ordered set of video frames using search text and evaluator feedback. Ablations report competitive results across four benchmarks, while the authors note compute and long-video constraints.

A.5 PYRAMID REFLECTION AS TEST-TIME RL

  • Pyramid Reflection treats video question answering as test-time reinforcement learning over ordered frame evidence, updating search text and frame selection from answer-evaluator feedback.Cached frame features let later rounds update indices and similarity or diversity scores rather than recomputing embeddings.Source: §A.5 · pp. 21–23

A.6 ABLATION STUDY

  • Generation ablations report reduced semantic fidelity without multi-level language modeling and degraded motion smoothness and temporal coherence without Temperature Modality Alignment.Figure 8 illustrates unnatural fingers, distorted poses, and implausible ball trajectories without TMA; Table 3 covers VBench-Long ablations.Source: Table 3 · p. 22, §A.6 · p. 23, Figure 8 · p. 23
  • Understanding ablations report gains from ActivityNet-QA-style finetuning over the base model and further accuracy gains when Pyramid Reflection is enabled.Table 4 compares the base, no-finetuning, no-reflection, and full variants across four video-QA benchmarks.Source: §A.6 · p. 23, Table 4 · p. 23
  • The full system combines data-efficient tuning with iterative reasoning to deliver competitive results across all four benchmarks.Source: §A.6 · p. 24

A.7 LIMITATION AND FUTURE WORK

  • The AR MLLM and DiT-based decoder increase compute and memory, slowing generation; keyframe-driven understanding works well on short clips but struggles with long videos and fine motion.The authors propose reducing coupling cost and integrating native video encoders as future work, while noting substantial training costs.Source: §A.7 · p. 24
Loading 2509.24200v2…