PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models

Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, Li Yuan

arXiv:2311.16103v2cs.CVcs.AI

TL;DR

Existing benchmarks do not comprehensively evaluate video understanding, despite Video-LLMs being expected to combine video perception, prior knowledge, and decision-making. Video-Bench offers a three-level benchmark and evaluation toolkit; its evaluation finds current models remain far from fully understanding and analyzing real-world videos.

  • Problem

    Existing benchmarks are not comprehensive for video: some focus on image understanding, while SEED-Bench covers only temporal understanding.

    Source: §1 · p. 1

  • Method

    Video-Bench evaluates ten tasks across three capability levels and provides a toolkit that maps model outputs to answers and calculates scores.

    Source: §1 · p. 2

  • Results

    Current Video-LLMs remain far from fully understanding video visual and audio content and helping people summarize, explain, or make decisions.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The paper identifies temporal awareness, domain-specific prior-knowledge pretraining, and long-video understanding as improvement directions.

    Source: §6 · p. 8

    Long-video understanding is highly neglected by existing Video-LLMs.

    Source: §6 · p. 8

Abstract

from arXiv · show

Video-based large language models (Video-LLMs) have been recently introduced, targeting both fundamental improvements in perception and comprehension, and a diverse range of user inquiries. In pursuit of the ultimate goal of achieving artificial general intelligence, a truly intelligent Video-LLM model should not only see and understand the surroundings, but also possess human-level commonsense, and make well-informed decisions for the users. To guide the development of such a model, the establishment of a robust and comprehensive evaluation system becomes crucial. To this end, this paper proposes \textit{Video-Bench}, a new comprehensive benchmark along with a toolkit specifically designed for evaluating Video-LLMs. The benchmark comprises 10 meticulously crafted tasks, evaluating the capabilities of Video-LLMs across three distinct levels: Video-exclusive Understanding, Prior Knowledge-based Question-Answering, and Comprehension and Decision-making. In addition, we introduce an automatic toolkit tailored to process model outputs for various tasks, facilitating the calculation of metrics and generating convenient final scores. We evaluate 8 representative Video-LLMs using \textit{Video-Bench}. The findings reveal that current Video-LLMs still fall considerably short of achieving human-like comprehension and analysis of real-world videos, offering valuable insights for future research directions. The benchmark and toolkit are available at: \url{https://github.com/PKU-YuanGroup/Video-Bench}.

1. Introduction

Video-Bench addresses the lack of comprehensive video-focused evaluation with a benchmark spanning three human-like capability levels and an automatic evaluation toolkit. Results on eight Video-LLMs show limitations in temporal and detail understanding, domain-specific prior knowledge, and complex comprehension and decision-making.

1. Introduction

  • Video-Bench evaluates video-exclusive understanding, prior knowledge-based question-answering, and comprehension and decision-making.The proposed levels include video-derived answers, questions requiring knowledge beyond the video, and scenario comprehension with predictions or decisions.Source: §1 · pp. 1–2
  • Existing benchmarks either focus on image understanding or cover only temporal video understanding, motivating a more comprehensive Video-LLM evaluation.MMBench and LVLM-eHub focus on images, while SEED-Bench covers only the first of the proposed capability levels.Source: §1 · p. 1
  • The benchmark includes tasks for summarization, abnormal detection, crowd counting, TV dramas, music videos, NBA videos, 3D scenes, and autonomous driving.Its video-exclusive tasks assess temporal and contextual aspects, while the other levels add prior-knowledge questions and decision-making tasks.Source: §1 · p. 2
  • The accompanying toolkit maps long model outputs to answer choices using probability selection or LLM-based semantic understanding, then calculates accuracy and final scores.It is intended to streamline the evaluation workflow.Source: §1 · p. 2
  • Evaluations of eight representative Video-LLMs found strengths in summarizing main video content but weaknesses in detecting details, temporal information, and domain-specific questions.The reported findings also describe limited comprehension and decision-making in complex scenarios, attributed to multimodal extraction constraints and weakened 7B or 13B LLM backends.Source: §1 · p. 2

2. Related Work

Prior work covers video-model architectures and task-specific datasets, while existing multimodal benchmarks are not tailored specifically to video. Video-Bench is positioned as a benchmark for evaluating Video-LLMs across capabilities and scenarios.

Video-LLMs

  • Video-LLMs build on open-source LLMs or image-language models, with varied visual, temporal, audio, and adaptation components.Examples include Q-Former mapping, average pooling, frame embeddings, ImageBind, and visual-token clustering.Source: §2 · pp. 2–3

Video Datasets

  • Video datasets span action recognition and localization, anomaly detection, object tracking, captioning and question-answering, and scenario-specific tasks.The paper notes that such datasets often focus on specific tasks and lack the complexity to measure Video-LLMs' comprehensive abilities effectively.Source: §2 · p. 3

Vision Language Evaluation Benchmarks

  • Multimodal benchmarks assess varied abilities, but the cited vision-language benchmarks are not tailored specifically for videos.The discussion includes MMBench, SEED-Bench, LVLM-eHub, and ELEVATER, with differing evaluation approaches and scopes.Source: §2 · p. 3
  • Video-Bench draws inspiration from HELM to measure Video-LLMs across human-like abilities, capabilities, and scenarios.Figure 2 presents its three-level evaluation and a ranking of existing Video-LLMs.Source: §2 · p. 3, Figure 2 · p. 3

3. Video-Bench

Video-Bench organizes ten tasks into three capability levels and pairs them with an automatic toolkit that converts model outputs into evaluated answer choices. Its tasks cover video-derived understanding, prior-knowledge questions, and comprehension and decision-making.

3. Video-Bench

  • Video-Bench evaluates three capability levels through ten tasks, and Figure 2 presents the benchmark structure alongside results for existing Video-LLMs.The levels are video-exclusive understanding, prior knowledge-based question-answering, and comprehension and decision-making.Source: §3 · p. 3, Table 2 · p. 6

3.1. Video-exclusive Summarization

  • Video-exclusive tasks assess information from the video itself, including objects, actions, attributes, and temporal connections, without external prior knowledge or complex inference.Basic understanding uses Activitynet-QA, MSVD-QA, MSRVTT-QA, and TGIF-QA; YouCook2 assesses longer-video cooking procedures and audio; UCF-Crime questions assess temporal anomaly understanding; MOT-based questions assess locating and counting people across frames.Source: §3.1 · pp. 3–4

3.2. Prior Knowledge-based Question-answering

  • Prior knowledge-based tasks assess questions about TV content, music videos, and NBA plays by combining video information with relevant outside knowledge.TV-QA incorporates video, audio, and subtitles; MV-QA uses QA pairs based on authoritative wiki sources; NBA-QA uses manually annotated teams, players, and technical actions.Source: §3.2 · pp. 4–5

3.3. Comprehension and Decision-making

  • Comprehension and decision-making tasks cover 3D indoor scene understanding, driver’s license exam animations, and decisions about complex traffic situations and accidents.The driving videos are manually annotated for scene analysis and accident causes, with the intended task of understanding their origins and making decisions to prevent them.Source: §3.3 · p. 5

3.4. Automatic Evaluation Toolkit

  • The automatic toolkit maps model outputs to predefined answer choices using probability, T5 textual similarity, or GPT-3.5-based formatting, then calculates scores.Table 2 reports task accuracy using the GPT-3.5-based metric; tasks with fewer videos receive half weight in the final result.Source: §3.4 · p. 5, Table 2 · p. 6

4. Experiment and Result

Video-Bench evaluates Video-LLMs across video-exclusive understanding, prior-knowledge QA, and comprehension and decision-making. Results show weaknesses in temporal understanding, domain knowledge, and complex decision-making; the paper also compares answer-evaluation metrics.

Experiment and Result

  • Video-LLMs often handle traditional QA but struggle to summarize event order, answer timestamp-related questions, and count people in crowds.The authors suggest weak precise localization and temporal association may contribute to crowd-counting failures.Source: §4 · p. 6
  • Models struggle to connect video evidence with prior knowledge and show limited audio understanding on music-video questions.Otter performs best on NBA-QA, while PandaGPT matches Otter on MV-QA; the authors suggest an audio encoder might improve this problem.Source: §4 · p. 7
  • Video-ChatGPT performs best on 3D-scene understanding and driving decision-making among the tested models.The authors attribute its performance to robust video instruction tuning.Source: §4 · p. 7
  • Probability-based answer mapping performs poorly overall, and the authors recommend GPT-based evaluation, especially for smaller models with unstable outputs.The comparison concerns Video-ChatGPT across datasets; the reported results use the GPT-based metric by default.Source: §4 · pp. 6–7, Figure 5 · p. 7

5. Visualization and Multi-Dimension Analysis

The analysis illustrates response failures and compares model modules and training-data sizes. The authors report that module additions have not produced significant gains under current settings, while video instruction-tuning data is associated with better performance.

Visualization

  • Only Video-ChatGPT gives the correct answer in the illustrated example; other models discuss the video but fail to make the correct judgment.The authors use the example to highlight that video-related responses may lack trustworthy reference value.Source: §5 · p. 7, Figure 6 · p. 8

Multi-Dimension Analysis

  • Adding the tested modules has not yielded significant improvements under the current data and training settings.The comparison covers configurations with a Temporal Module, an Audio Encoder, or both.Source: §5 · p. 7, Figure 7 · p. 8
  • Video instruction-tuning data is associated with better overall performance, and performance is positively correlated with its amount.The authors report that models trained on videos generally outperform those trained on images; they say pretraining dataset size may not be decisive.Source: §5 · pp. 7–8

6. Discussion and Conclusion

The authors conclude that current Video-LLMs remain far from fully understanding video content and supporting precise summaries, explanations, and decisions. They identify temporal awareness, domain-specific prior knowledge, long-video understanding, and stronger evaluation metrics as areas for improvement.

Discussion and Conclusion

  • Current Video-LLMs remain far from fully understanding video and audio content or providing precise summaries, explanations, and decisions.The conclusion frames this as a gap from the paper’s stated goal of a truly intelligent Video-LLM.Source: §6 · p. 8
  • The authors identify temporal awareness, domain-specific prior-knowledge pretraining, and long-video understanding as improvement directions.They note that frame-clip processing can miss temporal information and that long-video processing is constrained by memory and computation.Source: §6 · p. 8
  • The authors call for more robust metrics to evaluate Video-LLMs’ long-text responses and note that GPT or similar LLMs could help.Source: §6 · p. 8

7. T5 evaluation

The paper compares T5-based and GPT-based answer-evaluation metrics. It presents T5 as a cost-effective, offline-deployable auxiliary option with comparable performance in answer evaluation tasks.

T5 evaluation

  • T5-based evaluation provides an auxiliary alternative to GPT-based evaluation for answer assessment.Source: §7 · p. 12
  • T5 avoids ChatGPT API use and can be deployed offline on personal servers.Source: §7 · p. 12
  • T5-based results are reported as comparable to GPT-based results in answer evaluation tasks.Source: §7 · p. 12

8. Visualization Samples

This section presents generated-response examples from Video-LLMs across Video-Bench datasets and reports stronger performance on simple QA than on tasks requiring temporal, domain-specific, or complex-scene understanding.

8.1. Video-exclusive Understanding

  • Video-LLMs perform well on simple Activitynet-QA, MSVD-QA, MSRVTT-QA, and TGIF-QA tasks, with TGIF-QA results indicating understanding of simple GIFs.Source: §8.1 · p. 12
  • YouCook2 and UCF-Crime results indicate limited temporal awareness, while MOT results show inaccurate counting.Source: §8.1 · p. 12

8.2. Prior Knowledge-based Question-Answering

  • TV-QA and MV-QA remain difficult, while NBA-QA exposes difficulty connecting LLM knowledge to visual content without domain-specific vision-language pre-training.The authors suggest missing prior knowledge and audio or subtitle understanding may contribute to TV-QA and MV-QA difficulty.Source: §8.2 · p. 12

8.3. Comprehension and Decision-Making

  • The tested models show limited scene understanding and decision-making on the Driver’s License Examination, struggle with real driving environments, and handle only simple spatial scenes.Source: §8.3 · p. 12

Related papers

Loading 2311.16103v2…