PaperlayerView exact revision on arXiv

Source-linked AI summary

Lost in Time: A New Temporal Benchmark for VideoLLMs

Daniel Cores, Michael Dorkenwald, Manuel Mucientes, Cees G. M. Snoek, Yuki M. Asano

arXiv:2410.07752v3cs.CV

TL;DR

Video QA benchmarks are widely used to evaluate video-language models, but it is uncertain how well they measure video understanding. The paper analyzes these benchmarks and introduces TVBench, designed to require temporal reasoning and reduce text-only shortcuts. Most recent state-of-the-art models perform close to random chance on TVBench, with only a few exceeding the random baseline.

  • Problem

    It is unclear how reliably existing video QA benchmarks assess video understanding, despite their importance for measuring progress in a rapidly developing field.

    Source: §1 · p. 1

  • Method

    The authors analyze existing benchmarks and introduce TVBench, using temporally challenging answer choices and task-specific questions that resist text-only and prior-knowledge solutions.

    Source: §1 · p. 2, §5.1 · p. 6

  • Results

    Most recent state-of-the-art models perform close to random chance on TVBench, and only five models achieve above 50% accuracy.

    Source: §6 · p. 7

  • Takeaways & Limitations

    TVBench amplifies performance gaps between models with stronger and weaker temporal understanding and provides a yardstick for evaluating future advancements.

    Source: §6 · pp. 8–9

    LLM-evaluated open-ended benchmarks are unreliable, costly to access through closed-source evaluators, and may become unreproducible as evaluator versions change.

    Source: §4 · p. 6

Abstract

from arXiv · show

Large language models have demonstrated impressive performance when integrated with vision models even enabling video understanding. However, evaluating video models presents its own unique challenges, for which several benchmarks have been proposed. In this paper, we show that the currently most used video-language benchmarks can be solved without requiring much temporal reasoning. We identified three main issues in existing datasets: (i) static information from single frames is often sufficient to solve the tasks (ii) the text of the questions and candidate answers is overly informative, allowing models to answer correctly without relying on any visual input (iii) world knowledge alone can answer many of the questions, making the benchmarks a test of knowledge replication rather than video reasoning. In addition, we found that open-ended question-answering benchmarks for video understanding suffer from similar issues while the automatic evaluation process with LLMs is unreliable, making it an unsuitable alternative. As a solution, we propose TVBench, a novel open-source video multiple-choice question-answering benchmark, and demonstrate through extensive evaluations that it requires a high level of temporal understanding. Surprisingly, we find that most recent state-of-the-art video-language models perform similarly to random performance on TVBench, with only a few models such as Qwen2-VL, and Tarsier clearly surpassing this baseline.

1. Introduction

The introduction questions whether existing video question-answering benchmarks reliably measure visual and temporal understanding. It presents TVBench as a benchmark designed to require temporal information and resist text-only shortcuts.

1. Introduction

  • Existing benchmarks often remain solvable from single-frame information, informative text, or prior knowledge, limiting their assessment of video understanding.A leading model’s performance on existing benchmarks hardly drops when input videos are shuffled, while TVBench accuracy falls to chance (Figure 1).Source: Figure 1 · p. 1, §1 · p. 2
  • The authors analyze widely used video QA benchmarks and report that open-ended evaluation by LLMs yields unreliable, discrepant results across models on the same task.They also identify overly informative questions and answer choices that can permit text-only answers in multiple-choice tasks.Source: §1 · p. 2
  • TVBench uses temporally challenging answer choices, concise task-specific question templates, and questions requiring video content rather than prior world knowledge.These design choices aim to require temporal information while reducing text-only and knowledge-based solutions.Source: §1 · p. 2
  • Most recent state-of-the-art video-language models perform close to random chance on TVBench, while only a few, including Qwen2-VL and Tarsier, outperform the random baseline.Shuffling or reversing videos also causes significant performance drops for these models, unlike on prior benchmarks.Source: §1 · p. 2

2. Related Work

Related work covers task-specific video benchmarks and newer video-language QA benchmarks in open-ended and multiple-choice formats. It notes concerns about LLM-based open-ended evaluation and whether existing multiple-choice tasks effectively measure temporal understanding.

2. Related Work

  • Open-ended video QA uses LLM-based evaluators, but the authors report that these evaluations can hallucinate and produce unreliable conclusions; multiple-choice QA offers accuracy-based evaluation.Traditional metrics including ROUGE, METEOR, and CIDEr are described as inadequate for evaluating discrepancies in complex answers.Source: §2 · p. 2
  • The reviewed multiple-choice benchmarks target varied settings, but the authors say many MVBench tasks are spatially and textually biased rather than effectively testing temporal understanding.They position their proposed benchmark as requiring spatiotemporal understanding across tasks.Source: §2 · p. 2

3. Problems in Video MCQA Benchmarks

The authors find that MVBench and NextQA can reward spatial or textual shortcuts rather than temporal and visual reasoning. Their analyses identify frame and shuffle robustness, text-only success, and question, answer, or world-knowledge cues as key problems.

3.1. Does Time Matter?

  • 62.8%: GPT-4o's random-frame average across four MVBench tasks nearly matches its 65.8% video performance, exposing spatial bias.Shuffling has minimal impact on MVBench models, with an average difference of 2.3%; Tarsier achieves the same NextQA performance on shuffled and original video.Source: §3.1 · p. 3, Table 1 · p. 3

3.2. Does Vision Matter?

  • 62.3%: Gemini 1.5 Pro's text-only average across four MVBench tasks approaches Tarsier-34B's video score of 67.4%, showing textual shortcuts.Across all 20 MVBench tasks, Gemini 1.5 Pro reaches 38.2% with text-only, above the 27.3% random-chance baseline.Source: §3.2 · p. 4, Table 2 · p. 4
  • Textual shortcuts include unrealistic or overly informative generated QA, highly imbalanced answer frequencies, and questions answerable using prior knowledge of TV-show characters.For Action Count, the answer “3” is correct for 90 of 200 questions, and GPT-4o predicts it for 88 samples.Source: §3.2 · p. 4

4. Open-ended QA to the rescue?

Open-ended QA avoids predefined answer choices, but it does not remove textual or spatial biases and brings its own evaluation reliability problems. Different LLM evaluators can give substantially different judgments, including accepting answers unrelated to the video.

Evaluator reliability

  • Evaluator choice changes open-ended benchmark results by more than 20 points for the same method and dataset.Llama3 raises scores for text-only and single-image models while giving video models similar or lower results than GPT-3.5.Source: §4 · p. 5, Table 4 · p. 5

Biases remain

  • Text-only models can answer many open-ended questions, while single-frame models approach video-model performance and shuffling has little effect.Tarsier-34B drops less than 3% on average when videos are shuffled; examples also rely on prior knowledge or informative wording.Source: §4 · p. 5

Evaluator reliability

  • GPT-3.5 gives the highest matching score to an unrelated text-only answer about desert plants, seemingly disregarding the ground-truth answer.Other examples receive contradictory scores or appear affected by a shared context bias.Source: §4 · pp. 5–6

Limitations

  • Open-ended benchmarks remain costly and hard to reproduce because they rely on closed-source LLM evaluators, besides retaining spatial and textual bias.Source: §4 · p. 6

5. TVBench: A temporal VQA Benchmark

TVBench uses temporally demanding tasks and concise, task-specific question templates to evaluate video understanding. Its dataset spans varied sources and settings, and evaluation results show that performance depends on temporal information.

5. TVBench: A temporal VQA Benchmark

  • TVBench adopts multiple-choice QA to avoid open-ended VQA evaluation problems and tests its design using random-frame, shuffled-video, and reversed-video inputs.Source: §5 · p. 6

5.1. Designing TVBench

  • Ten tasks target counting, moving-object properties, temporal localization, event ordering, or distinguishing similar actions, with answer candidates grounded in original annotations.For Scene Transition and Action Sequence, candidates correspond to scenes or actions actually present in the video.Source: §5.1 · p. 6
  • Concise task-specific templates limit text-only clues; balanced candidate sets and removal of world-knowledge-heavy tasks further reduce textual shortcuts.The design removes Episodic Reasoning tasks based on television shows or movies.Source: §5.1 · pp. 6–7

5.2. Dataset Source

  • TVBench comprises 2,654 QA pairs across 10 tasks, drawing on eight video sources and spanning first- and third-person views, indoor and outdoor scenes, and real and synthetic data.QA pairs are generated from the original annotations using task-specific templates.Source: §5.2 · p. 7

5.3. TVBench Evaluation

  • 33.9% above random on shuffled MVBench videos versus 4.7% above random on shuffled TVBench videos for Tarsier-34B.On TVBench, a single image gives Gemini 1.5 Pro only a 3.0% gain over random, compared with 21.2% on MVBench; reversing videos can push top models below random.Source: §5.3 · p. 7

6. Discussion

On TVBench, many popular video-language models remain near random, while a small group performs above 50% accuracy. The benchmark distinguishes models with stronger temporal understanding and offers a yardstick for future progress.

6. Discussion

  • Only Qwen2-VL, LLaVA-Video, IXC-2.5, Aria, and Tarsier exceed 50% accuracy; several popular models are near random, with the largest gain only +8.9%.Source: §6 · p. 7
  • TVBench amplifies performance gaps between models with stronger and weaker temporal understanding.Source: §6 · p. 8
  • Text-only and visual models lacking temporal reasoning perform at random levels, while a handful of models achieve moderately high scores.Source: §6 · p. 9
  • The authors present TVBench as a reliable yardstick for evaluating future advances in video-language models.Source: §6 · p. 9

A.2.1. Benchmark Creation

TVBench constructs task-specific question-answer pairs from existing video datasets and annotations, with balanced candidate sets and negative choices designed to require temporal distinctions.

A.2.1. Benchmark Creation

  • For Action and Object Count, Object Shuffle, Action Localization, Unexpected Action, and Moving Direction, the candidate sets contain four options, each correct 25% of the time.Moving-direction options use four directional choices; Object Count questions can distinguish counts at the start and end of a video.Source: §A.2.1 · p. 12
  • Action Sequence and Scene Transition candidates are built from actions and scenes in the video, while Action Antonym uses similar actions that are difficult to distinguish without temporal understanding.Source: §A.2.1 · p. 12
  • Action Antonym draws videos from the NTU [20] RGB+D 120 dataset, retaining only 16 selected categories; for each QA, the labeled action is correct and its opposite is the alternative.Source: §A.2.1 · p. 12
  • Action Sequence uses two actions from STAR and asks which came first; Egocentric Sequence uses CSV annotations and creates three negatives by swapping tasks in the correct order.Source: §A.2.1 · p. 12
  • Scene Transition retains the original question and correct answer but creates a hard negative by reversing the sequence in that answer.Unexpected Action uses FunQA start and end times and four fixed temporal-location options; the correct option follows the original annotations.Source: §A.2.1 · p. 12

A.2.2. Human Baseline Study

The study estimates human performance on TVBench from 400 annotated videos and reports an average human accuracy of 94.8%. It also gives a conservative statistical-error estimate for that baseline.

A.2.2. Human Baseline Study

  • 94.8% was the average human accuracy across 400 annotated TVBench videos.Fourteen annotators each labeled 40 different videos; each video was annotated by one annotator.Source: §A.2.2 · p. 12
  • The estimate samples 400 videos from the benchmark’s population of 2,654 videos.Each video received one annotation, and the estimated average accuracy is based on the sampled videos.Source: §A.2.2 · p. 12
  • ±4.51% was the estimated 95% margin of error, with a standard error of approximately 0.0230.The estimate assumes p = 0.5, which the authors use as a conservative assumption in the absence of prior knowledge of true average accuracy.Source: §A.2.2 · pp. 12–13

A.3. Extended Analysis of Video MCQA Benchmarks

The extended analysis finds strong spatial and textual biases in NextQA, while the qualitative examples show that TVBench questions require temporal information and that MVBench examples exhibit benchmark biases.

A.3. Extended Analysis of Video MCQA Benchmarks

  • On NextQA, image-only reached 71.3% versus 79.0% for video; shuffling and reversing reduced performance by 0.5% and 1.4%, respectively.Text-only performance was 47.6%, 27.6% above the random-chance baseline.Source: §A.3 · pp. 13–14

A.4. TVBench qualitative examples

  • TVBench’s qualitative examples span Figures 7–14 and show that temporal information from the input video is indispensable for correctly answering the given questions.Source: §A.4 · p. 14

A.5. Spatial and Textual Bias in MVBench

  • MVBench examples show questions answerable from a single frame, including cases where one frame suffices to eliminate incorrect candidates.Figures 15–22 illustrate these spatial-bias cases.Source: §A.5 · p. 14, Figure 15 · p. 23, Figure 16 · p. 24
  • MVBench examples also show textual shortcuts, including implausible or artifact-ridden candidates, world-knowledge answers, and questions with only one candidate.Figures 23–29 provide examples of these textual-bias patterns.Source: §A.5 · p. 14, Figure 23 · p. 31, Figure 25 · p. 32

Related papers

Loading 2410.07752v3…