PaperlayerView exact revision on arXiv

Source-linked AI summary

MVBench: A Comprehensive Multi-modal Video Understanding Benchmark

Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, Yu Qiao

arXiv:2311.17005v4cs.CV

TL;DR

Existing MLLM benchmarks largely emphasize static-image spatial understanding rather than temporal video comprehension. MVBench addresses this gap with 20 statically derived dynamic tasks and automatic multiple-choice QA from public annotations, while VideoChat2 achieves over 15% higher accuracy than leading models on MVBench.

  • Problem

    Existing MLLM benchmarks predominantly assess spatial understanding in static images, providing limited comprehensive evaluation of temporal understanding in videos.

  • Method

    MVBench defines 20 temporal tasks by adapting static image tasks with dynamic evolution and automatically converts annotations from 11 public video benchmarks into multiple-choice QA.

  • Results

    VideoChat2 surpasses leading models by over 15% accuracy on MVBench, while existing MLLMs remain far from satisfactory in temporal perception and cognition.

  • Takeaways & Limitations

    MVBench provides a comprehensive, ground-truth-based evaluation of temporal understanding, and its analyses guide MLLM design for this capability.

Abstract

from arXiv · show

With the rapid development of Multi-modal Large Language Models (MLLMs), a number of diagnostic benchmarks have recently emerged to evaluate the comprehension capabilities of these models. However, most benchmarks predominantly assess spatial understanding in the static image tasks, while overlooking temporal understanding in the dynamic video tasks. To alleviate this issue, we introduce a comprehensive Multi-modal Video understanding Benchmark, namely MVBench, which covers 20 challenging video tasks that cannot be effectively solved with a single frame. Specifically, we first introduce a novel static-to-dynamic method to define these temporal-related tasks. By transforming various static tasks into dynamic ones, we enable the systematic generation of video tasks that require a broad spectrum of temporal skills, ranging from perception to cognition. Then, guided by the task definition, we automatically convert public video annotations into multiple-choice QA to evaluate each task. On one hand, such a distinct paradigm allows us to build MVBench efficiently, without much manual intervention. On the other hand, it guarantees evaluation fairness with ground-truth video annotations, avoiding the biased scoring of LLMs. Moreover, we further develop a robust video MLLM baseline, i.e., VideoChat2, by progressive multi-modal training with diverse instruction-tuning data. The extensive results on our MVBench reveal that, the existing MLLMs are far from satisfactory in temporal understanding, while our VideoChat2 largely surpasses these leading models by over 15% on MVBench. All models and data are available at https://github.com/OpenGVLab/Ask-Anything.

1. Introduction

MVBench addresses the limited temporal coverage of existing MLLM benchmarks by defining 20 video tasks through static-to-dynamic transformations. It automatically builds fair multiple-choice evaluations from public annotations and introduces VideoChat2, which substantially improves performance on MVBench.

  • Motivation: Existing benchmarks mainly test spatial perception in static images, limiting comprehensive evaluation of temporal understanding in videos.Prior video benchmarks are described as basic, domain-specific, scene-restricted, or labor-intensive to annotate.
  • MVBench: MVBench defines 20 temporal video-understanding tasks by adapting static image tasks with dynamic evolution.For example, an image position question becomes a video moving-direction question requiring reasoning over the whole video.
  • MVBench: An automatic annotation paradigm converts 11 public video benchmarks into multiple-choice questions, reducing manual annotation costs while covering diverse domains and scenes.The resulting ground-truth annotations support fair and accurate evaluation without biased LLM scoring.
  • Results: The evaluation finds that existing image and video MLLMs remain far from satisfactory on temporal perception and cognition.This result motivates development of a stronger video MLLM baseline.
  • Results: VideoChat2 surpasses the leading VideoChat model by over 15% accuracy on MVBench.It is trained progressively with diverse multimodal instructions and also achieves state-of-the-art results on video conversation and zero-shot QA benchmarks.

2. Related Works

The related work positions MVBench among multimodal language-model research and prior video benchmarks. Its distinction is broad temporal-task coverage, temporally sensitive videos, public annotations, and comprehensive MLLM evaluation.

  • MLLMs: MLLM research combines pretrained language models with visual inputs to improve multimodal understanding and generation.The related work cites systems such as Flamingo and PaLM-E as prominent examples.
  • MVBench: MVBench task examples use videos collected from 11 public datasets, with QA-generation tasks marked by “‡”.The listed sources span datasets including STAR, PAXION, Moments in Time V1, FunQA, CLEVRER, Perception Test, Charades-STA, MoVQA, NTU RGB+D, VLN-CE, and TVQA.
  • Video benchmarks: MVBench differs from prior benchmarks by covering a wide range of temporal tasks and emphasizing temporally sensitive videos.It also highlights efficient use of public annotations and comprehensive evaluation of temporal understanding.

3. MVBench

MVBench defines 20 temporal video-understanding tasks by adapting static image tasks to dynamic video contexts, then automatically converts public annotations into multiple-choice QA. Its pipeline emphasizes diverse, temporally sensitive videos and robust, fair answer evaluation.

  • Temporal Task Definition: MVBench adapts static image tasks with dynamic goals to define 20 temporal tasks spanning perception to cognition.The resulting tasks require comprehensive video understanding and cannot be effectively solved with a single image.
  • Temporal Task Definition: The benchmark covers action, object, position, scene, count, and attribute understanding through tasks such as action prediction, localization, and counting.The listed tasks include retrieving events around actions, inferring subsequent events, tracking object movement, identifying scene transitions, and counting actions or moving objects.
  • Automatic QA Generation: MVBench automatically converts open-sourced video annotations into multiple-choice questions for efficient MLLM evaluation.The pipeline selects videos from existing benchmarks and uses LLMs to generate questions and answer options from task definitions and annotations.
  • Automatic QA Generation: Eleven video datasets provide diverse domains, perspectives, and scenes, while filtering favors temporally sensitive clips lasting primarily 5s to 35s.The selection removes clips with negligible motion and extremely long videos with overly complicated contexts.
  • Automatic QA Generation: Each temporal understanding task ultimately receives 200 multiple-choice QA pairs, with randomized options and length balancing to improve evaluation robustness.The benchmark uses multiple-choice QA to avoid LLM- or user-study-based scoring and reduce evaluation bias or manual intervention.
  • Automatic QA Generation: A dedicated answer prompt encloses options in parentheses and achieves a 100% option-extraction rate, enabling accuracy as a reliable evaluation metric.The prompt guides MLLMs to produce answers in a format that can be extracted without the more complex matching procedure described for MMBench.

4. VideoChat2

VideoChat2 addresses weak temporal understanding and limited instruction-data diversity with broad image-video training and progressive multimodal alignment. Its training bridges visual representations and language through staged alignment, connection, and instruction tuning.

  • VideoChat2: Existing image and video MLLMs perform far from satisfactorily on temporal understanding, motivating the VideoChat2 baseline.The authors develop VideoChat2 to address this gap in video MLLM capability.
  • Instruction-Tuning Data: VideoChat2 uses 2M instruction-tuning samples from 34 distinct sources and includes both image and video data.The dataset is designed to address limited diversity in existing instruction-tuning data.
  • Instruction-Tuning Data: ChatGPT generates instructions from dataset descriptions, task descriptions, and examples, while samples are organized around vision data and question-answer instructions.The uniform format uses image-or-video and QA keys, with instruction, question, and answer components.
  • Instruction-Tuning Data: The instruction-tuning data spans conversation, simple caption, and detailed caption categories to support conversational, basic-description, and visual-detail capabilities.Conversation data comes from LLaVA and VideoChat, while caption sources include COCO Caption, WebVid, and YouCook2.
  • Progressive Multi-Modal Training: VideoChat2 adopts progressive multimodal training to bridge the semantic gap between visual and linguistic representations.The paradigm proceeds through vision-language alignment, vision-language connection, and instruction tuning.
  • Progressive Multi-Modal Training: Stage 1 freezes the visual encoder and trains QFormer to compress visual tokens and align them with text using multimodal losses.The visual encoder is UMT-L, selected for spatial-temporal representation learning, and QFormer training uses image and video captions.
  • Progressive Multi-Modal Training: Stage 3 applies low-rank adaptation during instruction tuning and integrates instructions into QFormer to extract instruction-relevant visual tokens.The visual encoder, QFormer, and adapted LLM are tuned with a vision-grounded text-generation loss.

5. Experiments

Experiments show that existing MLLMs remain weak on temporal video understanding, while VideoChat2 substantially improves MVBench and related video evaluations. Ablations attribute gains to diverse instruction data, the visual encoder, progressive unfreezing, and prompt design.

  • Results on MVBench: Over 15% accuracy separates VideoChat2 from the leading model on MVBench.
  • Results on MVBench: 35.5% versus 34.7% average accuracy shows VideoChat only marginally surpasses VideoChat2text, despite the latter receiving blank videos.
  • Results on MVBench: VideoChat2 surpasses GPT-4V by 16.9% accuracy on MVBench.
  • Additional evaluations: VideoChat2 achieves robust performance on complex reasoning benchmarks including NExT-QA, STAR, TVQA, EgoSchema, and IntentQA.
  • Ablations of VideoChat2: 50.5% versus 42.1% accuracy indicates video data contributes more than image data as instruction diversity and quantity increase.
  • Ablations of VideoChat2: Compressing simple captions minimally changes performance, from 51.1% to 50.7%, while accelerating tuning by 1.7×.
  • Ablations of VideoChat2: UMT-L improves accuracy by an additional 6.2%, while unfreezing QFormer produces an 8.5% performance boost.
  • Prompt design: A comprehensive system prompt improves task completion, and a simple answer prompt enhances option extraction and response precision.

6. Conclusion

The paper presents MVBench for temporal understanding and VideoChat2 as a strong baseline, with analyses guiding MLLM design for temporal video tasks.

  • MVBench evaluates MLLMs’ temporal understanding, while VideoChat2 outperforms leading models by over 15% on MVBench.

A. Training Hyperparameters

Training uses staged video-frame schedules and specified hyperparameters across three progressive stages.

  • Stage3 incorporates LoRA modules with rank 16, alpha 32, and dropout 0.1.

B. More Ablations

Ablations show that VideoChat2 benefits from more QFormer queries and video frames, while higher resolution does not help. Instruction-data overlap and prompt design materially affect evaluation outcomes.

  • QFormer: 64 QFormer queries are adopted by default because adding more queries in Stages 2 and 3 improves performance.Instructions without a question also steer responses more accurately than longer instruction-plus-question contexts.
  • Resolution & Frame: Increasing resolution does not improve performance, whereas increasing the number of frames enhances MVBench outcomes.The authors interpret this as evidence that MVBench relies primarily on temporal rather than spatial understanding.
  • Instruction data: CLEVRER instruction data has a more pronounced influence than SthSthV2 data because CLEVRER overlaps with MVBench question types.Action Antonym videos also originate from SthSthV2, but removing SthSthV2 has less effect.
  • Question prompt: Prompt tactics intended to suppress explanations and encourage step-by-step reasoning produce negative consequences in the ablation results.The prompts were designed to prevent detailed option explanations and included “Let’s think step by step.”
  • Data generation: MVBench data generation uses strategies tailored to different source datasets and ChatGPT-generated questions to increase task difficulty and diversity.For datasets requiring question generation, ChatGPT produces three to five questions from the task definitions.

D. Results on Challenging Video QA

VideoChat2 remains competitive with current state-of-the-art methods across challenging long-video reasoning benchmarks. The evaluation follows MVBench’s direct-option output protocol rather than likelihood comparison.

  • Cross-benchmark results: VideoChat2 holds its own against current state-of-the-art methods on NExT-QA, STAR, TVQA, EgoSchema, and IntentQA.These results underscore its effectiveness and robustness, especially for long videos.

E. Leaderboards and Analyses

VideoChat2 ranks highly across MVBench, but performance varies by task: it is strong on action, pose, and scene transitions while struggling with position, counting, and cognition.

  • Overall leaderboard: VideoChat2 achieves the highest rank across 15 MVBench tasks.The leaderboard covers results available through 2023/11/28.
  • Action & Pose: VideoChat2 and VideoChat tend to outperform VideoChatGPT on action- and pose-related tasks.The analysis links this pattern to the importance of elaborate video backbones for action and pose recognition.
  • Object & Attribute: LLaVA compares favorably with VideoChat2 on object-related tasks, consistent with strong attribute-recognition capabilities.VideoChatGPT, tuned from LLaVA, achieves similar results on these tasks.
  • Position & Count & Character: Position-related tasks remain near random guessing, while VideoChat2 performs similarly to or worse than VideoChat2text on counting and character tasks.The authors hypothesize that limited related tuning data hinders localization and counting generalization.
  • Scene: VideoChat2 excels at scene-transition tasks, reflecting sensitivity to background changes and camera movements.The analysis connects this capability to examples shown in Fig. 7.
  • Cognition: VideoChat2 struggles with complex egocentric navigation and episode reasoning, suggesting that visual information alone may be insufficient.The authors identify depth and audio as potentially beneficial additional modalities.

F. Qualitative Results

Qualitative results show VideoChat2 performing well across diverse MVBench tasks and video types. It identifies moving-object properties, recognizes unexpected actions, predicts future movements, and describes video details.

  • Qualitative comparisons: VideoChat2 performs admirably across a range of MVBench tasks compared with VideoChat and VideoChatGPT.The qualitative evaluation includes both real and generated videos.
  • Temporal capabilities: VideoChat2 identifies properties of moving objects, recognizes unforeseen actions, and predicts future movements from video context.These capabilities illustrate temporal understanding beyond single-frame description.
  • Figures: The qualitative figures provide additional comparisons and descriptive examples of VideoChat2’s handling of diverse video tasks and details.The examples include videos with varied actions, viewpoints, and content.
Loading 2311.17005v4…