PaperlayerView exact revision on arXiv

Source-linked AI summary

TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models

Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, Yao Dou, Jaden Park, Jianfeng Gao, Yong Jae Lee, Jianwei Yang

arXiv:2410.10818v2cs.CVcs.AIcs.CLcs.LG

TL;DR

Existing video benchmarks often lack fine-grained temporal annotations, limiting evaluation of temporal understanding. TemporalBench builds a benchmark from detailed human annotations and evaluates diverse temporal-understanding abilities; its results show a substantial gap between current models and humans.

  • Problem

    Existing video benchmarks often use coarse annotations that do not adequately evaluate models’ fine-grained temporal understanding.

    Source: §1 · p. 1, §3 · p. 4

  • Method

    TemporalBench curates ∼10K question-answer pairs from ∼2K human-annotated captions and uses LLM-generated, filtered negative captions to probe fine-grained activity details.

    Source: §1 · p. 2, §3.2 · p. 6

  • Results

    GPT-4o achieves 38.5% average short-video QA accuracy under MBA, compared with 67.9% for humans, and models perform worse on long videos.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The detailed annotations can also support tasks such as spatio-temporal localization and text-to-video generation with fine-grained prompts.

    Source: §6 · p. 13

    The authors cannot fully analyze the behavior of proprietary models because they lack access to GPT-4o, Gemini-1.5-Pro, and Claude 3.5 Sonnet.

    Source: §6 · p. 13

Abstract

from arXiv · show

Understanding fine-grained temporal dynamics is crucial for multimodal video comprehension and generation. Due to the lack of fine-grained temporal annotations, existing video benchmarks mostly resemble static image benchmarks and are incompetent at evaluating models for temporal understanding. In this paper, we introduce TemporalBench, a new benchmark dedicated to evaluating fine-grained temporal understanding in videos. TemporalBench consists of ~10K video question-answer pairs, derived from ~2K high-quality human annotations detailing the temporal dynamics in video clips. As a result, our benchmark provides a unique testbed for evaluating various temporal understanding and reasoning abilities such as action frequency, motion magnitude, event order, etc. Moreover, it enables evaluations on various tasks like both video question answering and captioning, both short and long video understanding, as well as different models such as multimodal video embedding models and text generation models. Results show that state-of-the-art models like GPT-4o achieve only 38.5% question answering accuracy on TemporalBench, demonstrating a significant gap (~30%) between humans and AI in temporal understanding. Furthermore, we notice a critical pitfall for multi-choice QA where LLMs can detect the subtle changes in negative captions and find a centralized description as a cue for its prediction, where we propose Multiple Binary Accuracy (MBA) to correct such bias. We hope that TemporalBench can foster research on improving models' temporal reasoning capabilities. Both dataset and evaluation code will be made available.

1 INTRODUCTION

Existing video benchmarks often rely on coarse annotations that do not test fine-grained temporal reasoning. TemporalBench uses detailed human-annotated captions to support diverse video-understanding evaluations, and its results expose both a model–human performance gap and a multiple-choice QA bias.

1 INTRODUCTION

  • Existing benchmarks’ coarse annotations often fail to test temporal sequences, allowing single-frame or text-only approaches to solve coarse video questions.The introduction also notes language-prior and single-frame biases in existing video-understanding benchmarks.Source: §1 · pp. 1–2
  • TemporalBench contains ∼10K question-answer pairs curated from ∼2K high-quality human-annotated captions with rich activity details.Clips span diverse domains; AMT workers and authors produce and refine positive captions, while LLMs generate negative captions that are filtered by the authors.Source: §1 · p. 2
  • The benchmark targets fine-grained action distinctions and supports short- and long-video understanding, captioning, grounding, and evaluation of embedding and generative models.Its design emphasizes long-range dependencies, fine-grained visual observations, and event progression.Source: §1 · pp. 2–3
  • MBA addresses multiple-choice QA bias in which models exploit subtle edits among negative choices to identify a centralized description.Negative captions include word-level replacements and reversed event order.Source: §1 · p. 3
  • GPT-4o reaches 38.5% average short-video QA accuracy under MBA, versus humans’ 67.9%; models perform worse on long videos.The authors also report that current models remain limited in reasoning about fine-grained temporal relationships.Source: §1 · p. 3

2 RELATED WORK

Related work covers the development of multimodal models and benchmarks for visual and video understanding. TemporalBench shares their evaluation goals but distinguishes itself through human-curated, detailed annotations focused on temporal dynamics.

2 RELATED WORK

  • Recent multimodal models build on language models and image encoders, with work extending to region-level, 3D, and video models.Source: §2 · p. 4
  • Visual benchmarks evaluate general question answering, text-oriented understanding, integrated problem solving, scientific reasoning, mathematical reasoning, and hallucination.Source: §2 · p. 4
  • Video benchmarks address varied goals, including egocentric understanding, temporal ordering, and broader video-understanding or world-modeling evaluation.Source: §2 · p. 4
  • TemporalBench curates videos across domains and focuses on human actions and human-object interactions, unlike ShareGPT4Video’s entirely GPT-4o-generated long captions.Source: §2 · p. 4

3 TemporalBench

TemporalBench targets video questions that coarse descriptions can leave answerable without temporal reasoning, using detailed activity descriptions to test fine-grained distinctions across video tasks. It also identifies a shortcut in multi-choice QA and proposes binary-query evaluation to remove the centralized-option cue.

3 TemporalBench

  • The benchmark targets a gap left by coarse captions, which can let models distinguish answers without reasoning about temporal dynamics.Detailed descriptions and temporal changes such as changing three slices to two are used to examine temporal state transitions.Source: §3 · pp. 4–5
  • Detailed captions support fine-grained video QA, captioning, long-video understanding, and dense video-text matching and retrieval.Long-video descriptions can be formed by concatenating clip descriptions; captions can also test embedding models' ability to distinguish subtle text differences.Source: §3 · p. 5
  • Detailed text descriptions also support moment localization, while fine-grained prompts can evaluate whether generated videos reflect the described action details.Source: §3 · p. 5

3.1 VIDEO COLLECTION

  • The short-clip collection excludes mostly static clips, clips with multiple scene transitions, clips longer than 20 seconds, and audio during annotation.The authors say longer clips make it difficult for annotators to provide detailed action descriptions.Source: §3.1 · p. 6

3.2 VIDEO CAPTION ANNOTATION PROCESS

  • Positive captions use a two-stage process: AMT workers draft detailed descriptions, then authors correct mistakes and add missing action details.Negative captions target word or phrase changes and event-order reversals; three LLMs generate candidates for author review into the final dataset.Source: §3.2 · pp. 6–7

3.3 A PITFALL IN MULTI-CHOICE QUESTION ANSWERING

  • Text-only GPT-4o chose the derived negative N(C) in 66.4% of constructed four-option sets, versus 6.4% for the positive caption C.The setup makes N1(C) the centralized description among the answer options.Source: §3.3 · p. 7
  • Multiple binary QA decomposes one multi-choice question into M binary questions, eliminating the centralized option and making evaluation harder according to the authors.Source: §3.3 · p. 7

4 EXPERIMENTS

TemporalBench evaluates fine-grained temporal understanding across short- and long-video QA and captioning, with human performance substantially exceeding current models. Model performance improves only modestly with more frames, and long-video understanding remains especially difficult.

4.1 EXPERIMENT SETUP

  • The experiments compare generative and embedding models while varying frame counts, alongside single-image and text-only baselines.Human performance is evaluated with AMT workers who did not annotate the positive captions.Source: §4.1 · pp. 7–8, §4.2 · p. 8

4.3 FINE-GRAINED VIDEO QUESTION ANSWERING ON SHORT VIDEOS

  • Humans averaged 67.9% performance, significantly exceeding the best models, GPT-4o and Qwen2VL-72B, by approximately 30%, indicating a large human–model performance gap.The human evaluation used standard AMT workers and excluded positive-caption annotators to avoid data contamination.Source: §4.3 · p. 8, §4.2 · p. 8
  • More frames usually improve video-model performance, but gains generally saturate around 8–16 frames.The comparison varies the number of frames supplied to the models.Source: §4.3 · p. 8, Figure 6 · p. 8
  • 75.7% GPT-4o binary QA accuracy falls to 38.5% under multiple binary QA, which tests each answer choice separately.The metric splits one M + 1-way question into M binary questions to reduce answer-choice cue exploitation.Source: §4.3 · p. 8, Table 2 · p. 9
  • Video embedding models show near-random-chance performance, while single-image and text-only models perform poorly on the benchmark.The authors suggest embedding sizes around 768–2048 may be insufficient to capture fine-grained temporal details.Source: §4.3 · p. 9

4.4 VIDEO CAPTIONING

  • GPT-4o leads the zero-shot captioning evaluation, where sentence-transformer embedding similarity aligns more closely with QA results than classical captioning metrics.Classical metrics are inconsistent; GPT-4o’s BLEU-1 scores are 24.1 with one frame and 25.1 with 64 frames.Source: §4.4 · p. 10, Table 3 · p. 10

4.5 LONG VIDEO UNDERSTANDING

  • Long-video QA performance drops significantly for every evaluated multimodal video model compared with short-video understanding.Models perform better on relatively shorter videos such as Charades than on longer videos such as FineGym; the study sampled 1,574 videos lasting up to 20 minutes.Source: §4.5 · pp. 10–11

5 IN-DEPTH ANALYSIS

The analysis shows how answer-choice patterns can inflate standard multiple-choice QA performance and identifies action frequency as a particular challenge for models.

5.1 WHY MULTIPLE BINARY QA INSTEAD OF MULTI-CHOICE QA?

  • 66.4% of the time, text-only GPT-4o selects N1(C) among intentionally designed negative options, illustrating a centralized-caption shortcut in standard multiple-choice QA.The authors use this behavior to motivate multiple binary QA accuracy over the standard setting.Source: §5.1 · p. 11, Table 5 · p. 12

5.2 PERFORMANCE ON CATEGORIES

  • Models perform better on the “others” category than action-related categories and struggle most with action frequency.The evaluated categories include action order, frequency, type, motion magnitude, direction or orientation, and effector.Source: §5.2 · pp. 11–12

6 CONCLUSION AND FUTURE WORK

TemporalBench evaluates fine-grained temporal understanding, and the paper reports a substantial gap between humans and current state-of-the-art models. The authors also identify a multiple-choice QA pitfall and note limits on analyzing proprietary models.

  • TemporalBench is proposed as a benchmark for evaluating fine-grained temporal understanding in multimodal video models.Source: §6 · p. 12
  • GPT-4o failure cases include difficulty understanding motion direction, action frequency, and action type.Source: Figure 8 · p. 13
  • The empirical results reveal a substantial gap between human performance and current state-of-the-art models.Source: §6 · p. 13
  • The authors identify a critical pitfall in multiple-choice QA and propose multiple binary accuracy (MBA) to address it.Source: §6 · p. 13
  • The authors cannot fully analyze GPT-4o, Gemini-1.5-Pro, and Claude 3.5 Sonnet because they lack access to these proprietary models.Source: §6 · p. 13

ETHICS STATEMENT

The paper describes its data sources and annotation practices as ethically compliant and says it sought to avoid personally identifiable information. It also urges caution about real-world deployment given potential societal risks.

ETHICS STATEMENT

  • The research uses publicly available video datasets, reports ethical compliance, and says it made efforts to avoid personally identifiable information.Source: ETHICS STATEMENT · p. 14

A BROADER IMPACT

  • The authors urge caution in real-world deployment because advanced video understanding may contribute to surveillance, privacy violations, or harmful applications.Source: A BROADER IMPACT · p. 21

B MORE VISUALIZATIONS OF OUR BENCHMARK

The appendix presents video examples and their positive and negative descriptions for the benchmarks listed in Table 1.

C PER SUBSET RESULTS FOR SHORT AND LONG VIDEO QA UNDER BINARY ACCURACY (BA)

The appendix reports per-subset short- and long-video QA results under Binary Accuracy and states that humans outperform all multimodal video models.

D MORE RESULTS WITH EXTENDED FRAMES

The appendix extends the reported model results to additional frame counts in Table 8.

D MORE RESULTS WITH EXTENDED FRAMES

  • Table 8 extends results across different frame counts; the main paper reports each model’s performance only at its best-performing frame count.The passage does not provide the additional frame-count results.Source: D MORE RESULTS WITH EXTENDED FRAMES · p. 21

E DATA ANNOTATION PLATFORM

The annotation workflow combines AMT captioning and author refinement for positive captions with LLM-generated candidates selected by authors for negative captions. The appendix also includes visualizations of annotations and tables describing evaluations across video lengths, models, metrics, and frame counts.

Positive Captions

Negative Captions

  • Figures 9 and 10 visualize fine-grained video annotations with both positive and negative descriptions.Source: Figure 9 · p. 22, Figure 10 · p. 23
  • Table 8 covers short- and long-video QA across frame counts, reporting BA and MBA and defining Overall as the average of both video-length results.Source: Table 8 · p. 24
  • Table 9 reports short-video BA and MBA for generative and embedding models, including T- BA for the annotated subset and each model’s best-average-MBA frame count.Source: Table 9 · p. 27
  • Table 10 reports long-video BA and MBA for generative and embedding models, includes dataset BA, and identifies each model’s best-average-MBA frame count.Source: Table 10 · p. 28

Related papers

Loading 2410.10818v2…