PaperlayerView exact revision on arXiv

Source-linked AI summary

Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short Videos

Jianrui Zhang, Mu Cai, Yong Jae Lee

arXiv:2410.02763v1cs.CVcs.AIcs.CLcs.LG

TL;DR

Short-video comprehension is often treated as largely solved, but the paper questions whether current models can handle dense temporal information. Vinoground evaluates this with 1,000 short, natural video-caption pairs, and its results show persistent temporal-reasoning gaps; the authors conclude that short-video comprehension remains unsolved.

  • Problem

    Existing short-video benchmarks may not adequately test temporal reasoning because some can be solved from a single frame or use unnatural negative videos.

    Source: §1 · p. 1

  • Method

    Vinoground is a temporal counterfactual benchmark of 1,000 short, natural video-caption pairs testing temporal differences in actions and object transformations.

    Source: §1 · p. 2

  • Results

    Most models are near chance on video score and worse on group score, both below text score, while GPT-4o remains much weaker than average humans.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that short-video comprehension remains far from solved and propose Vinoground as a checkpoint for evaluating temporal understanding.

    Source: §5 · p. 10

    The behavior of proprietary models GPT-4o, Gemini-1.5-Pro, and Claude 3.5 Sonnet cannot be fully analyzed because the authors lack access to them.

    Source: LIMITATIONS · p. 11

Abstract

from arXiv · show

There has been growing sentiment recently that modern large multimodal models (LMMs) have addressed most of the key challenges related to short video comprehension. As a result, both academia and industry are gradually shifting their attention towards the more complex challenges posed by understanding long-form videos. However, is this really the case? Our studies indicate that LMMs still lack many fundamental reasoning capabilities even when dealing with short videos. We introduce Vinoground, a temporal counterfactual LMM evaluation benchmark encompassing 1000 short and natural video-caption pairs. We demonstrate that existing LMMs severely struggle to distinguish temporal differences between different actions and object transformations. For example, the best model GPT-4o only obtains ~50% on our text and video scores, showing a large gap compared to the human baseline of ~90%. All open-source multimodal models and CLIP-based models perform much worse, producing mostly random chance performance. Through this work, we shed light onto the fact that temporal reasoning in short videos is a problem yet to be fully solved. The dataset and evaluation code are available at https://vinoground.github.io.

1 INTRODUCTION

The introduction argues that strong short-video performance can obscure persistent gaps in dense temporal reasoning. Vinoground is designed to test those gaps with natural temporal counterfactuals, and the reported findings show that current models remain weak on temporal tasks.

1 INTRODUCTION

  • Modern LMMs remain far from fully understanding short videos, particularly when those videos contain dense temporal information.This challenges the view that short-video comprehension is mostly solved and that research can shift primarily to longer videos.Source: §1 · p. 1
  • Existing benchmarks may inflate temporal-understanding performance because single-frame bias and unnatural negative videos can make evaluation less demanding.Some tasks can be solved from one frame, while some counterfactual benchmarks lack corresponding negative videos or use unnatural video pairs.Source: §1 · pp. 1–2
  • Vinoground contains 1,000 short, natural video-caption pairs that test temporal differences in actions and object transformations.Positive and negative captions use the same words in different orders; text, video, and group scores assess caption matching, video matching, and matching both pairs.Source: §1 · p. 2
  • Most modern models are near chance on video score and perform even worse on group score, both below text score.The introduction also reports that models tend to miss fine-grained details despite analyzing video frames at a coarse level.Source: §1 · p. 2

2 RELATED WORK

Related work covers counterfactual evaluation and temporal reasoning benchmarks. The authors position Vinoground as a temporal counterfactual benchmark with natural video pairs and actions that cannot be inferred from text alone.

Counterfactual Reasoning

  • In computer vision, counterfactual reasoning typically manipulates original data to create negative examples and observes how outcomes change (Morgan & Winship, 2015; Hendricks et al., 2018; Yeh et al., 2019; Goyal et al., 2019; Verma et al., 2020; Guo et al., 2023; Zhang et al., 2021; Thrush et al., 2022; Le et al., 2023; Zhang et al., 2024a).The authors note that meaningful, hard negatives matter for evaluation; Winoground pairs two images with two captions, while COCO-Counterfactual edits images and generates negative captions.Source: §2 · p. 3

Single-Frame Bias and Temporal Reasoning

Temporal Counterfactuals

  • Existing temporal counterfactual benchmarks have limitations including no negative videos, uncertain positive-negative relations, or unnatural concatenated or reversed clips (Mangalam et al., 2023; Li et al., 2024b; Saravanan et al., 2024; Liu et al., 2024b).The authors report that LMMs perform better on some such videos than on Vinoground’s more natural negative videos.Source: §2 · p. 3

3 VINOGROUND

Vinoground is curated by generating temporal counterfactual captions, finding matching videos, and categorizing the resulting pairs. Its design targets temporal distinctions while retaining natural videos and a fine-grained category structure.

3.1 GENERATING COUNTERFACTUAL CAPTIONS

  • GPT-4 generates caption candidates with identical word composition in different orders, designed to require understanding the order of events across a video.The process avoids pairs solvable from a single frame, such as swapping who waves at whom, and focuses on temporal event-order differences.Source: §3.1 · pp. 3–4

3.2 VIDEO CURATION

  • The video-curation process retrieves candidate matches from VATEX and manually checks both caption fit and video fidelity, using YouTube searches when needed.The pool contains 9,000 videos and 45,000 captions; the final benchmark has 500 counterfactual pairs, or 1,000 video-caption pairs, reviewed in three human-evaluation rounds.Source: §3.2 · pp. 4–5

3.3 CATEGORIZATION

  • Vinoground categories distinguish object-state changes, action order, viewpoint changes, interaction, cyclical activity, spatial movement, and contextual background changes.Each pair belongs to exactly one major category—object, action, or viewpoint—while minor categories may overlap or be absent; contextual cases require attention to background changes when camera motion can make an object appear still.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

The experiments evaluate contrastive and generative vision-language models with separate text, video, and joint group scores. Results show that dense temporal reasoning remains difficult for models, with performance affected by frame sampling and category.

4.1 MODELS AND EVALUATION METRICS

  • Group score requires success across both text and video domains, while CLIP-based and generative models are evaluated with different scoring procedures.The experiments include CLIP-based models and proprietary and open-source generative models; group score is 1 only when both text and video scores are 1.Source: §4.1 · pp. 6–7

4.2 EXPERIMENTAL SETUP

  • Generative-model video questions pair positive and negative clips with a two-second black interval; 170 workers qualified through a 10-question, all-correct screening.Human evaluation also compares full videos with audio against uniformly sampled 32-frame videos without audio; each question receives answers from 10 workers.Source: §4.2 · p. 7

4.3 MAIN RESULTS

  • For models performing better than chance, text scores are significantly higher than video scores, while group scores are lowest; GPT-4o’s video score (38.20%) is significantly lower than its text score (54.0%).Source: §4.3 · p. 8
  • CoT prompting improves GPT-4o’s video score by 12.8% and group score by 10.4%; Qwen2-VL-72B reaches 50.4% text score.The authors report that CoT helps open-source models less, especially those near chance; other models perform at or below chance.Source: §4.3 · p. 8

4.4.1 FRAMES SAMPLED

  • More sampled frames generally improve model performance, but GPT-4o’s 64-frame variant scores 5% below its 32-frame variant on all three metrics.The authors suggest models may have difficulty discarding redundant information and isolating signal from noise when visual tokens are excessive.Source: Table 3 · p. 9, §4.4.1 · p. 9

4.4.2 CATEGORY

  • Models score higher on contextual and viewpoint categories than on other categories, consistent with stronger coarse-level than fine-grained temporal comprehension.The authors describe interaction and cyclical categories as requiring logical understanding and attention to small temporal changes, while spatial tests object-location changes.Source: Figure 3 · p. 10, §4.4.2 · p. 10

5 CONCLUSION

Vinoground evaluates temporal reasoning with 1,000 short, natural video-caption pairs. The authors report that models struggle on short videos, while humans perform much better, leaving short-video temporal reasoning unresolved.

5 CONCLUSION

  • Models remain quite incapable of temporal reasoning even on videos shorter than 10 seconds, with most barely exceeding random chance.The conclusion identifies actions, background transitions, and object transformations as aspects of temporal understanding the benchmark tests.Source: §5 · p. 10
  • Vinoground comprises 1,000 short, natural video-caption pairs designed as a temporal counterfactual benchmark.The authors propose it as a checkpoint for evaluating models’ temporal understanding.Source: §5 · p. 10
  • Average human performance is described as easy and accurate, whereas GPT-4o performs much worse than humans.The conclusion does not give exact scores for this comparison.Source: §5 · p. 10

LIMITATIONS

The authors cannot fully analyze the behavior of the proprietary models evaluated because they lack access to them.

LIMITATIONS

  • Without access to GPT-4o, Gemini-1.5-Pro, and Claude 3.5 Sonnet, the authors cannot fully analyze their behavior.This limitation applies to the proprietary models included in the paper.Source: LIMITATIONS · p. 11

A DATA CURATION PROCESS

Figure 4 provides an overall illustration of Vinoground’s data curation process.

A DATA CURATION PROCESS

B METRICS ILLUSTRATION

Figure 5 visualizes the text and video score metrics, showing four questions derived from one counterfactual data point. The appendix also gives random-chance baselines for text, video, and group scores.

B METRICS ILLUSTRATION

  • Figure 5 visualizes the text and video score metrics; one counterfactual data point yields four possible questions.The passage identifies the figure’s scope and the number of questions, but not their individual encodings.Source: B METRICS ILLUSTRATION · p. 15, Figure 5 · p. 15

C RANDOM CHANCE PERFORMANCE

  • 25% is the random-chance baseline for text score, 25% for video score, and 16.67% for group score.The group-score baseline reflects six possible video-caption matchings for a counterfactual pair.Source: C RANDOM CHANCE PERFORMANCE · p. 16

D SURVEY INTERFACE

The survey was delivered through Qualtrics with videos hosted on Google Drive, and its interface is shown in Figure 6.

D SURVEY INTERFACE

  • The videos were uploaded to Google Drive and embedded in Qualtrics surveys, with the survey interface shown in Figure 6.The 2,000 questions were split into 50 surveys of 40 random questions; Prolific participants received $2 per completed survey, or $0.05 per question.Source: D SURVEY INTERFACE · p. 16, Figure 6 · p. 16

E FULL CATEGORICAL RESULTS

The categorical results for the selected top six strongest models follow the previously reported trend: performance is better on contextual and viewpoint categories and worse on other categories. Tables 4 and 5 and Figures 7 and 8 present category-grouped results for proprietary and selected open-source models, including text and video scores.

  • Models perform much better on contextual and viewpoint categories, and worse on other categories, matching the trend reported in Section 4.4.2.Source: E FULL CATEGORICAL RESULTS · p. 17
  • Tables 4 and 5 present the best category-grouped performances for proprietary and selected open-source models, respectively.The tables mark significantly high performances in blue and significantly low performances in red.Source: Table 4 · p. 17, Table 5 · p. 17
  • Figures 7 and 8 show text and video scores by category, respectively, grouped by model.Source: Figure 7 · p. 18, Figure 8 · p. 18

F FULL RESULTS ON EVALUATED MODELS

This appendix reports evaluation results omitted from the main paper, covering additional model settings across model types, sampled-frame counts, and the stated metrics.

Loading 2410.02763v1…