Source-linked AI summary
UniVBench: Towards Unified Evaluation for Video Foundation Models
Jianhui Wei, Xiaotian Zhang, Yichen Li, Yuan Wang, Yan Zhang, Ziyi Chen, Zhihang Tang, Wei Xu, Zuozhu Liu
TL;DR
Existing benchmarks do not capture the integrated capabilities of video foundation models because they are fragmented across tasks, metrics, and simple video settings. UniVBench addresses this with 200 annotated multi-shot videos spanning unified tasks and UniV-Eval’s standardized agentic scoring; results show specialization rather than one model excelling across the full spectrum.
Problem
Existing benchmarks are task-specific, use fragmented metrics, and often lack multi-shot content needed to evaluate unified video capabilities.
Method
UniVBench evaluates understanding, generation, editing, and reconstruction using 200 annotated multi-shot videos, while UniV-Eval standardizes prompting, parsing, and multidimensional scoring.
Results
Current models show distinct task specialization, and no single model excels across understanding, generation, and editing.
Takeaways & Limitations
UniVBench provides a unified baseline for comparing video models and attributing strengths and failures across the perception–generation spectrum.
Takeaways & Limitations
The benchmark’s primary limitation is its current dataset scale, motivating substantial expansion in future work.
Abstract
from arXiv · showhide
Video foundation models aim to integrate video understanding, generation, editing, and instruction following within a single framework, making them a central direction for next-generation multimodal systems. However, existing evaluation benchmarks remain fragmented and limited in scope, as they each target a single task, rely on task-specific metrics, and typically use short or simple video clips. As a result, they do not capture the unified capabilities that these models are designed to deliver. To address this gap, we introduce UniVBench, a benchmark purpose-built for evaluating video foundation models across four core abilities: video understanding, video generation, video editing, and a newly proposed task, video reconstruction, which assesses how faithfully a model can reproduce video content it has encountered. Our benchmark substantially expands the complexity of evaluation by incorporating 200 high-quality, diverse and multi-shot videos, each paired with detailed captions, multi-format editing instructions, and reference images. All videos are human-created and carefully validated, offering richer cinematic information than prior benchmarks. In addition, we develop a unified agentic evaluation system (UniV-Eval) that standardizes prompting, instruction parsing, and scoring across all tasks, enabling fair, scalable, and reproducible comparisons of unified video models. By grounding evaluation in instruction-based multi-shot video tasks, UniVBench provides the first framework for measuring the integrated capabilities that video foundation models aim to achieve. Extensive human annotations ensure our evaluation aligns with human judgment, enabling rigorous assessment and accelerating progress toward robust video intelligence.
1. Introduction
Video foundation models seek to unify understanding and generation, but existing benchmarks cannot evaluate this integration because they are task-specific, metric-fragmented, and limited in cinematic coverage. UniVBench addresses these gaps with a multi-task, multi-shot benchmark and UniV-Eval’s standardized, interpretable assessment.
- Existing video models remain divided between generation systems that synthesize videos and understanding systems that perceive content.
- Current benchmarks are task-specific, often use copyrighted or single-shot videos, and lack systematic coverage of cinematic dimensions.
- Existing metrics use fragmented task-dependent designs and single scalar scores that limit cross-task comparison and actionable error diagnosis.
- UniVBench introduces 200 high-quality multi-shot videos with detailed captions, editing instructions, and reference images for unified evaluation.The content is human-created and copyright-free.
- UniV-Eval standardizes prompting, instruction parsing, and multidimensional scoring to support interpretable cross-task comparisons and perception–generation error attribution.
2. Related Work
Prior video-model research developed specialized generation, understanding, and emerging unified architectures, while benchmarks and metrics remained largely separated by task. UniVBench responds with copyright-free, multi-task, multi-shot evaluation and finer-grained diagnostic assessment.
- Video Foundation Models: Early video systems specialized in synthesis or temporal understanding, while newer architectures seek bidirectional reasoning across text, image, and video.
- Video Benchmark: Understanding benchmarks expanded from single-shot captioning toward larger and more temporal datasets, but commonly rely on web-scraped content.
- Video Benchmark: Generation benchmarks broadened quality, semantic, temporal, and subjective evaluation, whereas editing benchmarks added instruction-following, reference-image, and adversarial scenarios.
- Video Benchmark: Existing benchmarks remain task-specific, restricted to single-shot scenarios, or exposed to copyright and contamination concerns.
- Evaluation Metrics: Traditional metrics operate at video or dataset level, lack fine-grained attribution and multi-shot handling, and impede cross-task comparison.
- Evaluation Metrics: LLM-as-a-Judge methods handle diverse inputs but typically produce single overall scores without multidimensional analysis, limiting diagnostic value.
3. UniVBench
UniVBench constructs richly annotated multi-shot videos across cinematic dimensions and introduces video reconstruction to test understanding–generation coupling. UniV-Eval then decomposes inputs and outputs into shot-level, multidimensional checklists and aggregates them into quantitative scores.
- Dataset Construction: The benchmark organizes video synthesis evaluation across eight cinematic dimensions and 21 fine-grained sub-dimensions.These include style, subject, action, background, camera, lighting, color, and spatial relationships.
- Dataset Construction: Fifteen trained video-production experts write shot-by-shot scripts that combine dimension categories while maintaining narrative coherence.Scripts undergo peer review for dimension coverage and coherence.
- Dataset Construction: Videos undergo automated filtering, unanimous reviewer validation against scripts, and specialist inspection for artifacts, unnatural motion, and temporal inconsistencies.
- UniV-Eval: UniV-Eval dynamically adapts evaluation by accepting arbitrary task inputs, planning and decomposing them, and returning fine-grained checklists.
- Dataset Construction: Captions are dimension-complete and shot-level, then independently checked for completeness and temporal accuracy with additional automated verification.
- Video Reconstruction: Video reconstruction requires a model to caption a source video and regenerate it from those captions, directly testing coupled understanding and generation.Failure in either stage increases discrepancy between the reconstruction and the original.
- UniV-Eval: The system segments videos into shots, aligns optional reference images and instructions, evaluates 21 shot-level subcategories, and aggregates weaknesses into final scores.
4. Experiments
UniVBench evaluates models under a unified protocol across six tasks and eight dimensions, revealing specialization, cross-dimensional weaknesses, and reconstruction-specific information loss. UniV-Eval is also compared with existing metrics and validated against human judgments.
- Main Results: Models show distinct task specialization, with performance differing substantially between systems designed for single tasks and unified video tasks.Table 4 summarizes results across six tasks and eight dimensions.
- Main Results: 54.1% average V2T performance makes Gemini 2.5 Pro the understanding leader, while Seedance-1.0-Pro leads T2V at 77.9% and Wan2.1-VACE-14B leads V2V at 62.7%.Showo-2 scores 16.3% in V2T, illustrating weaker perceptual reasoning among the cited unified models.
- Main Results: Action is frequently the weakest dimension, whereas generative models often score highest on Color, Lighting, and Video Style.The results identify temporal dynamics as a persistent challenge while stylistic attributes are more controllable.
- Main Results: No single model excels across understanding, generation, and editing, exposing a measurable unification gap.The benchmark maps strengths and weaknesses across existing architectures.
- Case Studies: V2V reconstruction shows more pronounced inconsistencies than T2V generation, indicating information transmission loss in the V2T → T2V pipeline.Reconstruction uses model-derived understanding text, unlike T2V, which uses ground-truth video text.
- Case Studies: UniV-Eval provides fine-grained, dynamically adaptive assessment beyond BLEU and conventional LLM-as-a-Judge metrics, which can distort or limit evaluation.BLEU can be affected by differing caption lengths, while conventional LLM judges typically cover limited dimensions.
- Case Studies: UniV-Eval achieves nearly 85% average agreement with human judgments in a threefold cross-validation study on a randomly sampled 10% of the data.Human experts reviewed samples against reference annotations.
5. Conclusion
UniVBench provides unified evaluation across understanding, generation, editing, and reconstruction, combining multi-shot cinematic coverage with interpretable cross-task scoring. Its agentic evaluator supports shot-level analysis and attribution of failures to perception or generation.
- Conclusion: UniVBench evaluates understanding, generation, editing, and reconstruction using 200 multi-shot videos annotated with captions, editing instructions, and reference images.The benchmark covers eight cinematic dimensions and 21 fine-grained sub-dimensions.
- Conclusion: UniV-Eval standardizes six-task assessment with multidimensional, shot-level scoring for interpretable cross-task comparison and perception-versus-generation failure attribution.This addresses limitations of existing single-scalar metrics.
6. Limitation and Future Works
The primary limitation is UniVBench’s dataset scale: its 200 richly annotated videos support comprehensive evaluation, but future work should expand the dataset substantially.
- Limitation and Future Works: UniVBench’s primary limitation is its current dataset scale despite the comprehensive evaluation enabled by 200 richly annotated videos.The authors identify substantial dataset expansion as a key future direction.
A. Evaluation Cases
Qualitative cases show that current video generation models often fail to satisfy detailed text requirements across actions, events, and cross-shot identity. The examples therefore expose remaining weaknesses in multi-shot generation.
- Evaluation Cases: Current video generation models struggle to satisfy detailed text requirements, including specified actions and events across multi-shot examples.CogVideoX and OmniVideo miss described animal actions and the dinosaur-bed event.
- Evaluation Cases: Seedance-Lite exhibits serious subject identity shift when the referenced subject cuts to the next shot.The case appears in the reference-image-to-video generation examples.
B.1. Captioning Meta Data Distribution
The benchmark’s metadata distribution is presented as semantically rich and diverse, with captions generated through structured extraction and script construction. UniV-Eval uses task-specific prompts and templates across six task categories.
- The dataset’s video content distribution spans multiple sub-dimensions, indicating semantically rich and diverse metadata.
- Caption generation first extracts video attributes such as subjects, actions, backgrounds, camera information, color, lighting, and style.
- The extracted attributes are merged into a coherent, structured video script that separates fixed content from other descriptions.
- UniV-Eval provides prompts for six task categories: V2T, T2V, R2V, TV2V, RV2V, and V2V.
- V2T evaluation combines its prompt with a predefined template, while other tasks use comparison rules for generic objects.
D. Evaluation Cost
The evaluation system is designed to remain relatively inexpensive while separating evaluatee and evaluator models to address self-preference bias. One task costs less than 10 US dollars to evaluate.
- Less than 10 US dollars is required to evaluate one task.The reported average cost is provided in Table D1.
- Self-preference bias is addressed by using video generation models as evaluatees and vision-language models as evaluators.The two model types differ substantially in architecture and training data.
F. Evaluation cases
The evaluation cases illustrate both the detailed prompt and template materials used by the benchmark and a discrepancy between human and LLM-as-Judge assessments. The examples cover generation outputs, metadata, captioning, and six evaluation tasks.
- Evaluation case: Human evaluators identify a salient cucumber-cutting error that the LLM judge overlooks in its detailed assessment.The LLM judge focuses on surface texture and white dots, whereas humans identify a conflict between the sideways cut and board slices.
- Generation examples: Figures A1 and A2 show baseline generation failures to capture required animal actions and a dinosaur-shaped pet bed’s opening event.The cited examples concern CogVideoX and OmniVideo.
- Captioning prompts: Captioning prompts extract attributes including style, camera information, subjects, actions, background, color, and lighting before producing structured descriptions.The script format and field explanations support detailed video-caption construction.
- Evaluation prompts: UniV-Eval’s evaluation materials include prompts for V2T, R2V, TV2V, RV2V, and V2V, plus JSON templates for evaluation outputs.