PaperlayerView exact revision on arXiv

Source-linked AI summary

MSVBench: Towards Human-Level Evaluation of Multi-Shot Video Generation

Haoyuan Shi, Yunxin Li, Nanhao Deng, Zhenran Xu, Xinyu Chen, Longyue Wang, Baotian Hu, Min Zhang

arXiv:2602.23969v1cs.MMcs.CV

TL;DR

Multi-shot video generation lacks benchmarks with complete story assets and cross-shot metrics, limiting evaluation of long-form coherence. MSVBench addresses this gap with hierarchical scripts and reference images plus a hybrid LMM–expert-model protocol, and its evaluation correlates 94.4% with human judgments while supporting human-aligned lightweight evaluation. The benchmark also reveals persistent long-form consistency limitations in current systems.

  • Problem

    Existing benchmarks remain centered on single-shot videos and lack comprehensive story assets and cross-shot metrics for evaluating multi-shot narrative coherence.

  • Method

    MSVBench organizes stories hierarchically and combines LMM semantic reasoning with domain-specific expert models for perceptual evaluation.

  • Results

    94.4% Spearman’s rank correlation with human judgments validates the benchmark, while evaluation of 20 systems finds current models behave primarily as visual interpolators rather than true world models.

  • Takeaways & Limitations

    Evaluation traces can provide scalable supervision: a fine-tuned lightweight Qwen3-VL-4B evaluator achieves human-aligned performance surpassing commercial models such as Gemini-2.5-Flash.

  • Takeaways & Limitations

    The benchmark contains relatively few stories, constraining training data for its lightweight evaluator and potentially limiting automated-evaluation effectiveness.

Abstract

from arXiv · show

The evolution of video generation toward complex, multi-shot narratives has exposed a critical deficit in current evaluation methods. Existing benchmarks remain anchored to single-shot paradigms, lacking the comprehensive story assets and cross-shot metrics required to assess long-form coherence and appeal. To bridge this gap, we introduce MSVBench, the first comprehensive benchmark featuring hierarchical scripts and reference images tailored for Multi-Shot Video generation. We propose a hybrid evaluation framework that synergizes the high-level semantic reasoning of Large Multimodal Models (LMMs) with the fine-grained perceptual rigor of domain-specific expert models. Evaluating 20 video generation methods across diverse paradigms, we find that current models--despite strong visual fidelity--primarily behave as visual interpolators rather than true world models. We further validate the reliability of our benchmark by demonstrating a state-of-the-art Spearman's rank correlation of 94.4% with human judgments. Finally, MSVBench extends beyond evaluation by providing a scalable supervisory signal. Fine-tuning a lightweight model on its pipeline-refined reasoning traces yields human-aligned performance comparable to commercial models like Gemini-2.5-Flash.

1 Introduction

MSVBench addresses the mismatch between single-shot evaluation and emerging multi-shot video generation by combining hierarchical story assets with hybrid semantic and perceptual assessment. Across 20 systems, it reports strong human alignment and finds that current models remain visual interpolators rather than true world models.

  • Motivation: Existing benchmarks rely on single-shot prompt–video pairs and lack complete scripts, per-shot reference images, and metrics for cross-shot properties.These limitations leave multi-shot evaluation unable to replace costly human assessment.
  • MSVBench: MSVBench introduces hierarchical data organization and a hybrid evaluation framework for multi-shot video generation.The framework combines LMM semantic reasoning with domain-specific expert-model perceptual fidelity.
  • Evaluation: 94.4% Spearman’s rank correlation with human judgments validates MSVBench’s reliability across 20 diverse systems.The evaluation spans commercial, open-source, and agent-based generation paradigms.
  • Findings: Current systems struggle with long-form stability and cross-shot consistency despite high prompt alignment, behaving more like visual interpolators than true world models.This result identifies a fundamental limitation in current video generation frameworks.
  • Beyond evaluation: MSVBench converts evaluation traces into supervision data for fine-tuning a lightweight evaluator.The resulting model achieves human-aligned performance surpassing commercial models such as Gemini-2.5-Flash.

2 Related Work

Video-generation evaluation has progressed from specialized models for basic quality dimensions toward LMM-based semantic assessment and story-oriented benchmarks. However, existing approaches span distinct generation paradigms and benchmark designs without the unified multi-shot comparison introduced by MSVBench.

  • Generation paradigms: Storyboard-driven synthesis separates narrative keyframe generation from subsequent image-to-video animation.StoryDiffusion and StoryAdapter exemplify this two-stage approach.
  • Benchmark comparison: Table 1 compares benchmarks by reference-image support, evaluated aspects, evaluator backbone, and human-alignment correlation.The listed dimensions include visual quality, story or prompt alignment, video consistency, and motion quality.
  • Generation paradigms: Agent-based frameworks use LLM orchestration for planning, structural reasoning, or generate-and-select quality assurance.Examples include MovieAgent, MM-StoryAgent, VideoGen-of-Thought, AnimDirector, and AniMaker.
  • Evaluation benchmarks: Earlier video benchmarks used specialized small models for basic quality dimensions, while Video-Bench shifted toward LMMs for semantic reasoning.Story-oriented benchmarks subsequently added reference-image integration for narrative generation.

3 MSVBench

MSVBench structures multi-shot video evaluation around hierarchical story assets and combines semantic LMM reasoning with domain-specific perceptual models. Its 20 metrics span visual quality, story alignment, consistency, and motion quality.

  • 3.1 Hierarchical Dataset Schema: MSVBench organizes each narrative into global characters and environments, scene-level segments, and shot-level multimodal conditions.Shot conditions include visual context, descriptions of states and actions, and cinematography instructions.
  • 3.2 Dataset Construction: The dataset pipeline synthesizes reference frames, refines prompts to align static states with dynamic actions, and enriches cinematography with explicit camera-motion instructions.These stages establish consistent visual narratives and translate static shot specifications into dynamic instructions.
  • 3.3 Evaluation Framework: MSVBench combines domain-specific expert models for low-level perceptual fidelity with Gemini-2.5-Flash for high-level semantic reasoning.The hybrid framework addresses the limited semantic understanding of specialized small models and the lack of domain-specific precision in pure LMM approaches.
  • 3.3 Evaluation Framework: The framework contains 20 sub-metrics across Visual Quality, Story Video Alignment, Video Consistency, and Motion Quality.The metric suite includes perceptual measures such as DOVER and MusIQ, attribute and style consistency, semantic alignment, identity and background consistency, and motion-related measures.
  • 3.3 Evaluation Framework: Story and shot-level metrics evaluate semantic alignment, required entities, perspective, state transitions, script consistency, identity, clothing, relative size, and actions.Additional measures assess action strength, camera control, and physical plausibility.
  • 3.3.4 Motion Quality: Motion metrics quantify optical-flow intensity, camera-trajectory execution, and adherence to expected physical dynamics.Action strength uses average flow magnitude, camera control compares recovered 6-DoF trajectories with instructions, and physical plausibility uses five-point grading.

4 Experiments

MSVBench evaluates 20 multi-shot video generation methods across four dimensions, combining quantitative benchmarking with human-alignment analysis and critical examination of current model limitations. The results show strong commercial performance, narrowing open-source gaps, and persistent failures in physical interaction, cross-shot consistency, and geometric modeling.

  • Main Results: AniMaker achieves a well-balanced performance profile across all evaluation dimensions.
  • Critical Insights: Current models act primarily as local visual interpolators rather than holistic world models, despite strong single-shot interpretation.Low Physical Interaction Accuracy and declining Character Consistency and Clothing consistency expose failures in causal dynamics and persistent character attributes.
  • Critical Insights: Aggressive motion and camera control remain entangled with content stability, producing structural distortion and character-identity disruption.Even Sora2 and Veo3.1 score below 3.0 / 5 on Physical Interaction Accuracy, while extreme motion can yield high aggregate motion scores alongside negligible interaction accuracy.
  • Critical Insights: Reference images improve Video Consistency but constrain physical plausibility because static 2D anchors omit depth and kinematic potential.Wan2.2-T2V reaches 3.86 Physical Plausibility versus 3.44 for Wan2.2-I2V, motivating richer geometric inputs such as 3D meshes or depth priors.
  • Human Preference Alignment: 94.4% Spearman correlation and 83.6% Kendall correlation establish MSVBench’s strong alignment with human judgments, outperforming VBench and ViStoryBench.The benchmark’s aggregate correlations exceed those of individual dimensions, supporting complementary metric coverage.
  • From Benchmark to Supervisor: A Qwen3-VL-4B model trained on over 1,000 synthesized supervision samples reaches 83.6% Spearman correlation and 66.3% Kendall correlation, surpassing Gemini-2.5-Flash.The resulting data transfers broad preference patterns while retaining human-like variance.

5 Conclusion

MSVBench unifies multi-shot video evaluation by combining perceptual expert models with LMM-based semantic reasoning. It achieves state-of-the-art human alignment, diagnoses current systems’ world-model limitations, and converts reasoning traces into supervision for more human-aligned lightweight evaluation.

  • MSVBench bridges low-level visual fidelity and high-level narrative consistency through a hybrid protocol using expert models and LMMs.
  • A lightweight Qwen3-VL-4B model trained on MSVBench reasoning traces achieves more human-aligned video evaluation while the benchmark identifies current systems’ world-model limitations.

Limitations

MSVBench identifies three remaining limitations: incomplete modality coverage, limited story-data scale, and alignment difficulties for continuous-generation models.

  • Audio-Visual Coverage: The framework is predominantly visual and lacks metrics for audio-visual synchronization and generated-audio quality.These omitted modalities are described as important for immersive video generation.
  • Story-Data Scale: The benchmark contains relatively few stories, constraining training data for the lightweight evaluator’s data-reconstruction pipeline.The authors state that this data-constrained regime may prevent the automated evaluator from reaching its full potential.
  • Continuous Generation: Continuous-generation models without explicit shot boundaries create alignment difficulties between videos, text prompts, and reference images.As a result, some shot-level metrics become inapplicable or difficult to compute.

C Quantitative Results

MSVBench visualizations compare 20 methods across four dimensions, while case studies show that its metrics distinguish missing elements, continuity violations, camera-control failures, and consistency errors.

  • Visualization Results: Visualizations of two stories show five sampled shots per story from all 20 evaluated methods across four evaluation dimensions.The comparisons cover Visual Quality, Story Video Alignment, Video Consistency, and Motion Quality.
  • Metric Case Studies: Case studies demonstrate that the metrics inspect script-required entities, quantities, and camera language rather than ranking models only relative to one another.For example, StoryAdapter receives Score 1 for missing required characters, Sora2 receives Score 5 for recalling mandated elements, and Self-Forcing receives Score 1 for an incorrect full shot.
  • Metric Case Studies: Case studies identify cross-shot clothing and color violations while recognizing consistent outputs, including VideoGen-of-Thought at Score 3, StoryAdapter at Score 1, and LTXV at Score 5.The Relative Size metric also evaluates logical spatial consistency across shots.
  • Metric Case Studies: State Shift & Persistence and Camera Control separately assess narrative-state continuity and prompted camera movement.The examples assign Self-Forcing Score 1 for losing an injury state, AniMaker Score 5 for preserving it, CogVideoX Score 1 for a static shot, and validate Wan2.2-TI2V’s tracking shot.

D.4 Motion Quality and Physics

MSVBench evaluates motion quality through physical plausibility and interaction accuracy, then uses objective script- and physics-based scoring to identify violations without relative model ranking.

  • Physics Evaluation: Physical Plausibility and Physical Interaction Accuracy assess whether generated motion follows real-world physics.The metrics target physical adherence for each generation.
  • Objective Scoring: MSVBench provides fine-grained objective scores by evaluating generated content against the input script and physical principles rather than relative rankings.The framework is designed to penalize hallucinations, continuity errors, and physical violations.
  • Human Alignment: The human-alignment analysis computes correlations between aggregated objective metric scores and human ratings using Spearman’s and Kendall’s coefficients.Metric aggregation precedes the statistical correlation analysis.

E.1 Metric Aggregation

MSVBench aggregates heterogeneous sub-metrics by converting them to ranks and negatively averaging those ranks so higher-quality models receive higher dimension-level scores.

  • Rank Normalization: Each raw sub-metric value is converted into a rank from 1 to N, with rank 1 representing the best performance.Tied values receive the average rank of the tied group.
  • Dimension Aggregation: For each dimension, the method aggregates K sub-metric ranks into a model score through negative averaging.The aggregation operates after rank normalization across sub-metrics with different scales and units.
  • Score Direction: The negative sign makes smaller ranks produce larger aggregated scores, aligning the metric direction with human evaluation scores.This preserves the interpretation that higher scores indicate better quality.

E.2 Statistical Analysis

MSVBench uses Spearman’s and Kendall’s rank correlations to measure alignment between benchmark scores and human ratings, with Kendall’s formulation accounting for ties. The leaderboard comparison reports strong overall agreement with human perception.

  • Correlation Measures: Spearman’s rank correlation measures global monotonic consistency between MSVBench and human rankings.It uses centered ranks for the two score lists.
  • Correlation Measures: Kendall’s rank correlation measures pairwise ordering consistency and remains robust when ties occur.The formulation accounts for concordant and discordant pairs plus ties in benchmark and human scores.

E.3 Overall Score Calculation

MSVBench aggregates four dimension-level rank scores into a unified model score called Average Rank. The resulting overall ranking shows strong alignment with human perception.

  • Overall Score: MSVBench defines each model’s overall score as the mean of its four dimension-level rank scores.The overall score is also called the model’s Average Rank.
  • Overall Score: The derived overall ranking demonstrates strong alignment with human perception.The leaderboard presents the overall average ranking of evaluated methods.
Loading 2602.23969v1…