PaperlayerView exact revision on arXiv

Source-linked AI summary

MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs

Tianhao Peng, Haochen Wang, Yuanxing Zhang, Zekun Wang, Zili Wang, Gavin Chang, Jian Yang, Shihao Li, Yanghai Wang, Xintao Wang, Houyi Li, Wei Ji, Pengfei Wan, Steven Huang, Zhaoxiang Zhang, Jiaheng Liu

arXiv:2511.07250v2cs.CVcs.AI

TL;DR

Most existing video-understanding benchmarks focus on single-video inputs, despite real-world scenarios requiring information across multiple videos. MVU-Eval evaluates eight multi-video perception and reasoning competencies, and experiments report substantial model limitations and performance differences. The authors identify short video duration and the lack of audio evaluation as limitations.

  • Problem

    Most video-understanding benchmarks use single-video inputs, while real-world scenarios such as multi-camera driving and cross-angle sports analytics require understanding across multiple videos.

    Source: §1 · p. 1

  • Method

    MVU-Eval assesses eight multi-video perception and reasoning competencies through 1,824 curated question-answer pairs spanning 4,959 videos.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    Experiments find substantial room for improvement in multi-video understanding, with Gemini 2.5 Pro at 56.6% accuracy and most open-source MLLMs outside the Qwen2.5-VL series below 50%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments highlight the need for improved architectures and training-data strategies for complex multi-video scenarios in practical applications.

    Source: §5 · p. 9

    The benchmark uses relatively short videos, limiting assessment of long-range temporal reasoning and narrative comprehension over extended sequences.

    Source: §A.2 · p. 14

Abstract

from arXiv · show

The advent of Multimodal Large Language Models (MLLMs) has expanded AI capabilities to visual modalities, yet existing evaluation benchmarks remain limited to single-video understanding, overlooking the critical need for multi-video understanding in real-world scenarios (e.g., sports analytics and autonomous driving). To address this significant gap, we introduce MVU-Eval, the first comprehensive benchmark for evaluating Multi-Video Understanding for MLLMs. Specifically, our MVU-Eval mainly assesses eight core competencies through 1,824 meticulously curated question-answer pairs spanning 4,959 videos from diverse domains, addressing both fundamental perception tasks and high-order reasoning tasks. These capabilities are rigorously aligned with real-world applications such as multi-sensor synthesis in autonomous systems and cross-angle sports analytics. Through extensive evaluation of state-of-the-art open-source and closed-source models, we reveal significant performance discrepancies and limitations in current MLLMs' ability to perform understanding across multiple videos. The benchmark will be made publicly available to foster future research.

1 Introduction

Existing video benchmarks largely evaluate one video at a time, despite real-world tasks that require combining multiple videos. MVU-Eval addresses this gap with an eight-competency benchmark and experiments that expose current models’ limitations.

1 Introduction

  • Most existing video-understanding benchmarks take a single video as input, unlike real-world tasks such as multi-camera driving and cross-angle sports analytics.The introduction also cites summarization over multiple retrieved videos as an example.Source: §1 · p. 1
  • MVU-Eval evaluates eight perception and reasoning competencies using 1,824 question-answer pairs spanning 4,959 videos from varied domains.Its perception tasks are Object Recognition, Spatial Understanding, Counting, and Comparison; its reasoning tasks are Knowledge-Intensive Reasoning, In-Context Learning, Retrieval-Augmented Generation, and Temporal Reasoning.Source: §1 · p. 1
  • The paper evaluates open- and closed-source models on MVU-Eval and identifies performance discrepancies and limitations in multi-video understanding.The authors describe these experiments as revealing challenges and potential directions for handling and reasoning over multiple videos.Source: §1 · pp. 1–2
  • 56.6% accuracy is achieved by Gemini 2.5 Pro, while most open-source MLLMs other than the Qwen2.5-VL series score below 50%.The paper reports substantial room for improvement in multi-video understanding.Source: §1 · p. 2
  • Models’ strongest results vary across perception tasks, while scaling Qwen2.5-VL and InternVL models and increasing VideoLLaMA3-7B’s frames and resolution improve performance.The highest accuracies on OR, SU, and Counting come from Qwen2.5-VL-72B, Qwen2.5-VL-32B, and Gemini 1.5 Pro, respectively.Source: §1 · p. 2

2 Related Works

Prior work includes multimodal models for video and benchmarks for single-video understanding, including long-video evaluation. The related-work section identifies multi-video understanding as a gap and presents MVU-Eval as the first benchmark addressing it.

2 Related Works

  • Prior video MLLMs include Video-LLaMA, which uses dual-path ViT and Q-Former encoding, and Mavors, which extracts multi-granularity video representations.The cited related work describes their respective aims as spatiotemporal modeling and retaining spatial fidelity and temporal coherence.Source: §2 · p. 2
  • Existing benchmarks cover concise video QA, long-video understanding, durations from seconds to an hour, and multidisciplinary lecture videos.The section names MVBench, MLVU, LongVideoBench, Video-MME, and Video-MMLU as examples.Source: §2 · p. 2
  • The related-work section says existing video benchmarks usually use a single video as input and presents MVU-Eval as the first multi-video benchmark.Table 1 is described as comparing MVU-Eval with related video benchmarks.Source: §2 · p. 3, Table 1 · p. 3

3 MVU-Eval

MVU-Eval is a benchmark for multi-video perception and reasoning, designed to evaluate cross-video integration across real-world tasks and diverse video sources. It combines automated construction with human quality checks and contains 1,824 questions spanning 4,959 videos.

3.1 Overview

  • MVU-Eval evaluates whether models can aggregate, correlate, and reason across multiple video sources, unlike benchmarks focused on single-video analysis.It includes 1,824 questions and 4,959 videos across diverse sources and real-world tasks.Source: §3.1 · p. 3

3.2 Evaluation Tasks

  • Perception tasks cover object recognition across clips, spatial layouts from complementary camera angles, counting across asynchronous videos, and fine-grained cross-video comparison.Comparison includes detecting replacements, removals, and additions.Source: §3.2 · p. 3
  • Reasoning tasks test domain-knowledge integration, transfer from examples, relevant-evidence retrieval, and chronological alignment across videos.Temporal reasoning includes ordering clips, grounding descriptions in segments, and filling missing events.Source: §3.2 · p. 4
  • Knowledge-intensive reasoning includes classifying sports actions, judging their difficulty, and assessing athletic performance from the inferred actions.The tasks use relevant sports knowledge and multi-video evidence to resolve ambiguities.Source: §3.2 · p. 4

3.3 Data Collection

  • Construction samples task-specific video pairs, generates questions automatically or from templates, removes leakage and easy questions, then uses human verification.Pairing examples include similar-caption videos for retrieval and videos containing the same objects for counting.Source: §3.3 · pp. 4–5, Figure 2 · p. 4

3.4 Quality Control

  • Quality control removes answer leakage and format cues, filters questions that several strong models can answer, and has annotators verify answerability and correctness.After review, 1,824 samples remain—about 46% of the original generated data.Source: §3.4 · p. 5

3.5 Dataset Statistics

  • The dataset has 1,824 questions across eight relatively balanced topics, with an average of 4.7 videos per question and rare cases reaching 13.Answer options are relatively balanced; the benchmark contains 4,959 videos from diverse sources.Source: §3.5 · p. 5, Figure 3 · p. 5, Figure 4 · p. 5

4 Experiments

The experiments evaluate open- and closed-source MLLMs under a zero-shot protocol and use further analyses to examine visual input, video quantity, resolution, and input format. Results show sensitivity to the amount and form of visual information, while failure cases expose perception and reasoning challenges.

4 Experiments

  • The evaluation tests 22 open-source and three closed-source MLLMs using zero-shot accuracy, generally sampling 32 frames per video and resizing the longer side to 720 pixels.Some model families use the stated 32-frame, 720-pixel setting.Source: §4 · pp. 6–7

4.2 Further Analysis

  • VideoLLaMA3-7B performance declines as visual information is reduced; textual descriptions outperform single-frame-per-video input, while some video-disabled settings fall below random guessing.The authors attribute below-chance results in some settings to models refusing to answer when important visual information is missing.Source: §4.2 · p. 7, Table 4 · p. 7
  • More frames generally improve VideoLLaMA3 performance until 64 frames, while higher resolution helps up to 720 pixels before performance degrades at 960.The passages attribute these degradations to excessive input tokens overwhelming model processing capacity.Source: §4.2 · p. 7, Figure 6 · p. 8, Figure 7 · p. 8
  • Input format significantly affects multi-video performance, and the Qwen2.5-VL series outperforms similarly sized models, possibly because it supports native multiple-video inputs.The study compares per-video frame sampling and concatenating videos with black frames between segments.Source: §4.2 · p. 8, Table 5 · p. 8
  • Failure cases include difficulty interpreting object status and cross-view spatial relations, filtering irrelevant information, using domain knowledge, and reasoning about temporal or causal links.A counting example involved a longer clip with visually similar tables; a spatial example suggested retained temporal reasoning but weak spatial understanding.Source: §4.2 · pp. 8–9, Figure 8 · p. 9

5 Conclusion

MVU-Eval is presented as the first comprehensive benchmark for evaluating MLLMs on multi-video understanding, spanning eight competencies in perception and reasoning. Experiments highlight the need for improved architectures and training data strategies for complex multi-video scenarios.

5 Conclusion

  • MVU-Eval evaluates MLLMs on multi-video understanding across eight core competencies spanning basic perception and advanced reasoning.The paper reports extensive experiments on multiple MLLMs and identifies needs for improved architectures and training data strategies in complex multi-video scenarios.Source: §5 · p. 9
  • The experiments highlight the need for improved architectures and training data strategies to tackle complex multi-video scenarios in practical applications.Source: §5 · p. 9
  • The paper presents MVU-Eval as the first comprehensive benchmark for evaluating MLLMs on multi-video understanding.Source: §5 · p. 9

6 Future Works

Future work targets temporal and spatial alignment across videos, scalable fusion, and robustness to diverse video sources. The work reports no direct ethical concerns and says its materials contain no personally identifiable information or sensitive content.

6 Future Works

  • Future directions include aligning asynchronous videos, identifying shared objects for cross-video spatial understanding, and reasoning about temporal relations across unaligned streams.The benchmark's Spatial Understanding tasks require geometric comprehension beyond a single viewpoint, while temporal reasoning becomes more challenging as video count increases.Source: §6 · p. 9
  • Scalable fusion is needed to handle more videos without token-limit overruns, since excessive video, frame, or resolution tokens can degrade current MLLMs' performance.Source: §6 · p. 9
  • Future work also targets robustness across indoor, outdoor, gaming, AIGC, and movie and TV sources for unseen real-world scenarios.Source: §6 · p. 9

A.1 Ethics Statement

  • The authors report no direct ethical concerns; videos and annotations are synthetic or from public datasets, contain no personal or sensitive information, and involve no human participants.Source: §A.1 · p. 14

A.2 Limitations

MVU-Eval uses short videos and visual inputs only, limiting evaluation of long-range temporal reasoning, narrative comprehension, and audio-based understanding. The benchmark is research-focused rather than tied directly to specific applications or deployments.

A.2 Limitations

  • Short video samples constrain assessment of long-range temporal reasoning and narrative comprehension over movie-length sequences, while visual-only inputs exclude audio-based evaluation.The authors say future extensions will aim to incorporate audio-based assessments.Source: §A.2 · p. 14

A.3 Broader Impacts

  • MVU-Eval is a research dataset focused on core technical capabilities, without direct ties to specific applications or deployments.The paper describes its contribution as supporting development of robust and generalizable models for multi-video understanding tasks.Source: §A.3 · p. 14

B.1 Data Source

Table 6 shows the data sources for MVU-Eval's eight tasks.

B.1 Data Source

  • Table 6 presents the data sources for MVU-Eval's eight tasks.Source: §B.1 · p. 14

B.2 Evaluation Settings

The evaluation settings standardize video inputs where possible, establish expert-verified answers and zero-shot scoring, and specify prompts for video formats and task types.

B.2.1 Frames and Resolution

  • Most models and closed-source systems use 32 frames per video at a maximum 720-pixel longer side; InternVL uses 448 × 448 and InternVideo2.5 728 × 728.The InternVL settings follow model-specific requirements, while the InternVideo2.5 setting aims to minimize the impact of varying experimental settings.Source: §B.2.1 · pp. 14–15

B.2.2 Human Evaluation

  • Five experts independently answered all eight subtasks; disagreements triggered joint review, with unresolved cases settled by a three-expert majority and flagged as difficult.Experts first completed the benchmark without communicating; during review they discussed video details and task ambiguities.Source: §B.2.2 · p. 15

B.2.3 Evaluation Protocol

  • Zero-shot accuracy counts exact answer matches; regex workflows extract option candidates, defaulting to the response’s first letter when none is found.Some models generate intermediate reasoning despite being prompted to answer with an option letter.Source: §B.2.3 · p. 15

B.2.4 Prompt Templates for Video Information

  • Table 5 experiments compare per-video multi-image input against concatenated videos separated by black delimiter frames, using format-specific prompts and letter-only answer instructions.The multi-image prompts present frames by video; the merged-video prompt describes videos concatenated in order.Source: §B.2.4 · p. 15

B.2.5 Prompt Templates for Tasks

  • Task templates include questions on gestures, temporal ordering, athlete and traffic comparisons, object counting, knowledge-intensive difficulty, and retrieval-augmented inference.The examples include inferring a question from answers to three videos and then answering it for a fourth.Source: §B.2.5 · pp. 15–17

C Further Analysis

Long-video MLLMs show limited gains on MVU-Eval, including on temporal reasoning, while some models also fail to follow answer-format instructions. The analysis identifies cross-video fusion and alignment as additional needs beyond temporal modeling.

C.1 Instruction-Following Behavior

  • Over 98% of LLaVA-Video-7B responses were free-form paragraphs despite prompts requesting single-letter answers; generated text could be relevant yet ignore the task question.The authors identify this as an instruction-following and output-controllability limitation.Source: §C.1 · p. 17

C.2 Performance of Long-Video MLLMs

  • Long-video MLLMs remain modest on MVU-Eval: stronger temporal modeling yields limited temporal-reasoning gains and can underperform on spatial subtasks.The analysis suggests cross-video reasoning also requires effective inter-video fusion and alignment mechanisms.Source: §C.2 · pp. 17–18

Related papers

Loading 2511.07250v2…