PaperlayerView exact revision on arXiv

Source-linked AI summary

TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models

Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, Arman Cohan

arXiv:2410.23266v2cs.CVcs.AIcs.CL

TL;DR

Many video benchmarks may not establish whether multimodal models can genuinely reason over time, since tasks can be solved from single or out-of-order frames. TOMATO applies three temporal-reasoning principles to a new benchmark and finds that models struggle to interpret videos as continuous sequences.

  • Problem

    Existing video benchmarks often allow tasks to be solved from a single frame and rely little on correct frame order, limiting evidence of genuine temporal reasoning.

    Source: §1 · p. 2

  • Method

    The authors define three principles and corresponding metrics—Multi-Frame Gain, Frame Order Sensitivity, and Frame Information Disparity—and construct TOMATO, a benchmark of six temporal-reasoning tasks.

    Source: §1 · p. 2

  • Results

    Across three metrics, TOMATO tasks show greater multi-frame gain and frame-order sensitivity and more evenly distributed information than existing benchmarks; Multi-Frame Gain is 66.3% versus less than 5%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings identify continuous-sequence interpretation as a key challenge for video systems intended to comprehend changing scenes.

    Source: §7 · p. 13

    MFMs struggle to reason across multiple time steps and interpret frames as a continuous sequence.

    Source: §6.3 · p. 10

Abstract

from arXiv · show

Existing benchmarks often highlight the remarkable performance achieved by state-of-the-art Multimodal Foundation Models (MFMs) in leveraging temporal context for video understanding. However, how well do the models truly perform visual temporal reasoning? Our study of existing benchmarks shows that this capability of MFMs is likely overestimated as many questions can be solved by using a single, few, or out-of-order frames. To systematically examine current visual temporal reasoning tasks, we propose three principles with corresponding metrics: (1) Multi-Frame Gain, (2) Frame Order Sensitivity, and (3) Frame Information Disparity. Following these principles, we introduce TOMATO, Temporal Reasoning Multimodal Evaluation, a novel benchmark crafted to rigorously assess MFMs' temporal reasoning capabilities in video understanding. TOMATO comprises 1,484 carefully curated, human-annotated questions spanning six tasks (i.e., action count, direction, rotation, shape & trend, velocity & frequency, and visual cues), applied to 1,417 videos, including 805 self-recorded and -generated videos, that encompass human-centric, real-world, and simulated scenarios. Our comprehensive evaluation reveals a human-model performance gap of 57.3% with the best-performing model. Moreover, our in-depth analysis uncovers more fundamental limitations beyond this gap in current MFMs. While they can accurately recognize events in isolated frames, they fail to interpret these frames as a continuous sequence. We believe TOMATO will serve as a crucial testbed for evaluating the next-generation MFMs and as a call to the community to develop AI systems capable of comprehending human world dynamics through the video modality.

1 INTRODUCTION

The paper argues that MFMs’ visual temporal reasoning is overestimated because existing benchmarks permit frame-based shortcuts, and introduces TOMATO to require reasoning across video sequences. Its evaluation reports benchmark-design differences and substantial model–human performance gaps, alongside deeper model limitations.

1 INTRODUCTION

  • Analysis of four widely used benchmarks (Li et al., 2023; 2024b; Liu et al., 2024; Chen et al., 2024a) found questions answerable from single, few, or out-of-order frames.The authors propose three principles with corresponding metrics to assess whether benchmarks rigorously test visual temporal reasoning.Source: §1 · pp. 1–2
  • Existing benchmarks averaged less than 5% Multi-Frame Gain, less than 8% Frame Order Sensitivity, and over 27% Frame Information Disparity, versus TOMATO’s 66.3%, 34.1%, and 4.6%.Higher Multi-Frame Gain and Frame Order Sensitivity indicate greater reliance on multiple frames and correct frame order; lower Frame Information Disparity indicates more evenly distributed information.Source: §1 · p. 2
  • TOMATO contains 1,484 human-annotated multiple-choice questions across six tasks and 1,417 human-centric, real-world, and simulated videos, including 805 self-recorded and -generated originals.Tasks include action count, direction, rotation, shape and trend, velocity and frequency, and visual cues; question-answer pairs require reasoning across transitions in all frames.Source: §1 · p. 2
  • Qwen2-VL-72B achieved 37.9% overall accuracy, ahead of GPT-4o at 37.7%; humans scored 95.2% on full videos and 79.7% on 16 frames.The evaluation covered 21 open-source models and 10 proprietary models.Source: §1 · p. 2
  • Beyond the human–model gap, the authors find MFMs fail to interpret frames as continuous sequences, over-rely on common sense, and are susceptible to noisy information.The paper notes that models can recognize events in isolated frames but fail to interpret them as a continuous sequence.Source: §1 · pp. 1, 3

2 RELATED WORK

Video benchmarks have shifted from action recognition toward temporal and human-like understanding, but many questions in existing benchmarks still do not rigorously test visual temporal reasoning. TOMATO is motivated by this gap.

General video understanding benchmarks

  • Earlier benchmarks emphasized action recognition, while later work and MFM-era benchmarks expanded toward temporal dynamics, causal events, long-form and multidisciplinary video understanding, and broader task coverage.The paper positions TOMATO as building on this progress to evaluate visual temporal reasoning specifically.Source: §2 · p. 3
  • MMBench-Video (Fang et al., 2024) calls for more temporal questions, yet its Attribute Recognition, Object Recognition, and OCR dimensions remain static.This motivates TOMATO’s focus on visual temporal reasoning.Source: §2 · p. 3

Visual temporal reasoning benchmarks

  • VITATECS (Li et al., 2023), TempCompass, MVBench (Li et al., 2024b), and ReXTime broaden temporal tasks, but many questions remain answerable from single, few, or out-of-order frames.The paper argues this limits their effectiveness for evaluating true visual temporal reasoning and motivates TOMATO’s more rigorous evaluation.Source: §2 · p. 3

3 BENCHMARKING PRINCIPLES FOR VISUAL TEMPORAL REASONING TASKS

The section defines three principles and metrics to distinguish visual temporal reasoning from static-frame recognition: requiring multiple frames, sensitivity to frame order, and even information contribution across frames.

3.1 MULTI-FRAME GAIN

  • Multi-Frame Gain κ measures the relative performance gain from a single frame to m frames, testing whether a task requires reasoning across multiple frames.The single-frame input may be random or handpicked, while m frames are uniformly sampled at equal intervals; lower κ indicates stronger single-frame solvability, and higher κ indicates greater need for multiple frames.Source: §3.1 · p. 4

3.2 FRAME ORDER SENSITIVITY

  • Frame Order Sensitivity τ measures the relative performance gain from shuffled m frames to ordered m frames, indicating whether solving a task depends on temporal order.Lower τ indicates that out-of-order frames can answer the question more accurately; higher τ indicates stronger reliance on the original frame order.Source: §3.2 · p. 4

3.3 FRAME INFORMATION DISPARITY

  • Frame Information Disparity ρ measures the relative performance gain from a random single frame to a handpicked single frame, assessing whether informativeness is evenly distributed across frames.Higher ρ indicates an advantage for the handpicked frame; lower ρ indicates more even informativeness, with ρ = 0 ideal for a perfectly even distribution.Source: §3.3 · pp. 4–5

4 TOMATO: A VISUAL TEMPORAL REASONING BENCHMARK

TOMATO is a benchmark designed to test visual temporal reasoning across frames, addressing weaknesses in prior benchmarks. It contains 1,484 human-annotated questions across six tasks and 1,417 videos spanning human-centric, real-world, and simulated scenarios.

4 TOMATO: A VISUAL TEMPORAL REASONING BENCHMARK

  • TOMATO applies three temporal-reasoning principles to 1,484 human-annotated questions across six tasks and 1,417 videos, including 805 self-recorded and -generated videos.Its design addresses tasks solvable from a single or few frames, insensitive to frame order, or uneven in frame information.Source: §4 · p. 5

4.1 TEMPORAL TASKS IN TOMATO

  • The six tasks are rotation, direction, velocity and frequency, shape and trend, visual cues, and action count, covering three demonstration categories.The categories are human-centric interactions, real-world object actions, and simulated temporal actions.Source: §4.1 · p. 5

4.2 VIDEO COLLECTION

  • Videos come from YouTube, existing datasets, and benchmark-specific recordings and generated videos, spanning three scenarios and edited with counterfactual scenes, composite motions, and zoomed-in views.The collection includes interactive human-centric scenes, which existing sources lacked, alongside simulated object patterns and 3D human videos.Source: §4.2 · pp. 6–7

4.3 QUESTION-ANSWER ANNOTATION AND QUALITY CHECK.

  • Questions are designed to require reasoning across video transitions, with annotation adapted to video source and a three-stage review process.Annotators created or re-annotated temporal questions for some sources, while existing questions were retained and given close numerical distractors for others; disagreements were resolved through collective review.Source: §4.3 · p. 7

5 COMPARISONS AMONG VISUAL TEMPORAL REASONING BENCHMARKS

The section compares TOMATO with four recent visual temporal reasoning benchmarks using three metrics, with approximately 200 sampled questions and two state-of-the-art MFMs.

5.1 MULTI-FRAME GAIN

  • TOMATO achieves significantly higher κ than the compared benchmarks with both random and handpicked single-frame inputs relative to 16-frame inputs.The comparison uses random single frames in Table 3 and handpicked single frames in Table 4.Source: §5.1 · p. 7, Table 3 · p. 8, Table 4 · p. 8
  • Handpicked frames lower κ on existing benchmarks, while ReXTime has negative multi-frame gain, possibly because added frames introduce noise.This indicates that existing benchmark questions can be relatively easy to answer from a handpicked single frame.Source: §5.1 · p. 8, Table 4 · p. 8

5.2 FRAME ORDER SENSITIVITY

  • TOMATO achieves significantly higher τ under shuffled versus ordered 16-frame inputs, indicating stronger dependence on preserving frame order.The shuffled and ordered settings use the same set of 16 frames.Source: §5.2 · p. 8, Table 5 · p. 8

5.3 FRAME INFORMATION DISPARITY

  • TOMATO has the lowest performance gain from random to handpicked single frames, indicating more consistent informativeness across its frames than in other benchmarks.The comparison uses single-frame and handpicked single-frame inputs.Source: §5.3 · p. 8, Table 6 · p. 8

6 EVALUATING VISUAL TEMPORAL REASONING IN ADVANCED MFMS

TOMATO evaluation shows that MFMs achieve low accuracy relative to humans and struggle to reason over video as a continuous sequence. Analyses reveal limits in using added frames and visual evidence, as well as variation across video scenarios.

6.1 EXPERIMENTAL SETUP

  • The evaluation covers 31 MFMs—10 proprietary and 21 open-source—and includes text-only, guess, and human-performance baselines.Human performance is measured with video input and with 16 frames as input.Source: §6 · p. 9, §6.1 · p. 10

6.2 EXPERIMENTAL RESULTS

  • Qwen2-VL-72B achieves 37.9% accuracy versus GPT-4o’s 37.7%, but the best model trails humans’ 95.2% accuracy by 57.3%.Many models score below 30.0%.Source: §6.2 · p. 10

6.3 MAIN ANALYSIS

  • Models can describe changes in individual frames yet fail to infer the sequence, and they may favor common-sense assumptions or misleading cues over visual evidence.Reported failures include missing an overall clockwise rotation, misreading a reversed video’s motion, and being misled by text on an object or an unusual visual shape.Source: §6.3 · pp. 10–11
  • Qwen2-VL’s M-RoPE explicitly encodes temporal information, while four tested MFMs’ accuracy plateaus beyond 8 frames despite human improvement.The authors suggest temporal-aware positional encoding may help models retain temporal context; models’ gains from 1 to 8 frames do not continue with additional frames.Source: §6.3 · p. 11
  • On the same 101 QA pairs, GPT-4o improves 21.9% and Qwen2-VL-72B 7.8% from real-human to simulated-human scenarios.The comparison spans five reasoning types; the authors suggest semantic video abstraction may help temporal reasoning.Source: §6.3 · pp. 11–12

6.4 SCENARIO ANALYSIS

  • Counterfactual examples reduce GPT-4o performance by 37.7% and Qwen2-VL-72B performance by 29.4%, suggesting reliance on background knowledge over video evidence.The four models perform similarly on non-counterfactual questions; counterfactual videos are edited to depict events impossible in real life.Source: §6.4 · p. 12, Table 8 · p. 12
  • Zooming yields modest or negative accuracy changes, while models perform better on first-person than third-person questions, including a 61.3 score for Qwen2-VL-72B.Zoom changes are GPT-4o +6.0%, Qwen2-VL-7B +1.9%, Video-CCAM-v1.1 14B -1.0%, and Qwen2-VL-72B -4.2%. First-person gains are 101.6% for Qwen2-VL-72B and 72.1% for Qwen2-VL-7B; Qwen2-VL-72B outperforms GPT-4o by 28.9% under those conditions. The first-person set has 80 QAs versus 668 third-person QAs.Source: §6.4 · p. 12, Table 8 · p. 12

7 CONCLUSION · A EXPERIMENT SETUP · A.1 MODEL CONFIGURATION

TOMATO addresses likely overestimation of MFMs’ visual temporal reasoning by providing a benchmark grounded in three evaluation principles. Its evaluation finds that models recognize isolated frames but fail to interpret videos as continuous sequences.

7 CONCLUSION

  • Existing benchmarks likely overestimate MFMs’ visual temporal reasoning, motivating three principles and corresponding metrics for systematic evaluation.Source: §7 · p. 12
  • The authors introduce TOMATO, a video-understanding benchmark designed to rigorously assess MFMs’ visual temporal reasoning capabilities.Source: §7 · p. 12
  • MFMs fail to interpret videos as continuous sequences, instead relying on isolated-frame understanding that undermines visual temporal reasoning.The conclusion presents this limitation as a critical finding of the comprehensive evaluation.Source: §7 · p. 13
  • The conclusion identifies developing AI systems that comprehend changing real-life scenes through video as a direction for future work.Source: §7 · p. 13

A.1 MODEL CONFIGURATION

  • Table 9 presents evaluation model configurations; unset values use defaults, and its markers identify video-processing and frame-count constraints.It also notes a 0.61 token-kept ratio for dagger-marked models and that configurations use official repositories where available.Source: Table 9 · p. 21

A.2 IMPLEMENTATION DETAILS FOR MODEL INFERENCE · B PROMPTS

The paper evaluates models zero-shot and uses prompts that require answers from sampled video frames, with separate prompts for answer extraction and video-free guessing. Inference preserves unrestricted model outputs and uses GPT-4o-mini when standard answer parsing fails.

A.2 IMPLEMENTATION DETAILS FOR MODEL INFERENCE

  • MFMs were evaluated zero-shot across benchmarks, using official preprocessing when available and NVIDIA A100 GPUs for non-API evaluations.VideoLLaMA 2 7B and 72B process videos directly without frame splitting; Gemini 1.5 Flash processes video directly at 1 FPS.Source: §A.2 · p. 21
  • The authors imposed no output-format restrictions and used GPT-4o-mini to extract multiple-choice answers when standard parsing failed.The paper notes that unrestricted outputs complicate parsing with simple heuristics.Source: §A.2 · p. 21

B.1 EVALUATION PROMPT

  • The evaluation prompt provides uniformly sampled, separate video frames in chronological order and asks models to answer the video-content question.Source: §B.1 · p. 22
  • The prompt requires answers based on the frames rather than external knowledge or commonsense, and asks models to guess rather than abstain when frames are insufficient.Source: §B.1 · p. 22

B.2 ANSWER EXTRACTING PROMPT

  • The answer-extraction prompt asks GPT to return only the parsed option letter using the response, choices, and index-to-answer mapping.Source: §B.2 · p. 22

B.3 VIDEO-FREE RANDOM GUESS PROMPT

  • The video-free prompt asks for a reasonable random answer based only on the question and options, and forbids responding that the answer cannot be determined.Source: §B.3 · p. 23

C DATA ANALYSIS … E.3 FULL-SCALE ANNOTATION

The appendix documents TOMATO’s data distributions, benchmark comparisons, licensing, and annotation workflow, including task selection and full-scale verification.

C.1 VIDEO SOURCE DISTRIBUTION; C.4 VIDEO DURATION DISTRIBUTION

  • The appendix presents TOMATO video-source and video-duration distributions.The supplied figure captions identify these distributions but provide no numerical breakdowns.Source: Figure 3 · p. 24, Figure 6 · p. 26

C.2 PERFORMANCE COMPARISONS ACROSS DIFFERENT NUMBER OF FRAMES

  • Figure 4 presents performance comparisons of VITATECS, TempCompass, MVBench, RexTime, and TOMATO across different numbers of frames.The caption provides no performance values or winning benchmark.Source: Figure 4 · p. 25

C.3 MODEL PERFORMANCE WITH ≤8 FRAMES

  • Figure 5 presents Qwen2-VL-72B performance across five benchmarks with input restricted to fewer than 8 frames.The caption gives no benchmark scores.Source: Figure 5 · p. 25

C.5 RESPONSE LENGTH DISTRIBUTION

D LICENSE INFORMATION.

  • TOMATO’s videos are governed by source-specific licensing terms, and the benchmark is intended for academic research rather than commercial use.YouTube videos are Creative Commons-licensed; TOMATO-created videos use CC BY-NC-SA. Existing datasets include CLEVRER (Yi et al., 2020), TGIF-QA Jang et al. (2017), Perception Test (P˘atr˘aucean et al., 2023), and Music-AVQA (Li et al., 2022), each with stated licensing conditions.Source: D LICENSE INFORMATION · p. 29

E.1 ANNOTATOR BIOGRAPHIES

  • Table 10 provides biographies for TOMATO’s human annotators.Source: Table 10 · p. 30

E.3 FULL-SCALE ANNOTATION

  • Full-scale annotation drew on YouTube, TGIF-QA, CLEVRER, Music-AVQA, and benchmark-specific videos, followed by blind answer re-annotation and discrepancy resolution.For TGIF-QA, annotators selected pairs they could answer; for CLEVRER, they wrote questions and answers were extracted from video metadata. Of 1,484 QA pairs, 44 required collective resolution; four videos were replaced and the remaining answers corrected.Source: §E.3 · p. 30

F DATA ANNOTATION PLATFORM … G.2.4 EXAMPLE 4

The platform presents annotators with a question and video in a 4 × 4 frame grid, then asks them to select an answer. Rotation examples show model predictions that often differ from the provided answer keys.

F DATA ANNOTATION PLATFORM

  • The annotation platform displays a question beside the corresponding video as 16 frames in a 4 × 4 grid and asks the annotator to select an answer.Source: Figure 9 · p. 31

G.1.1 EXAMPLE 1

  • The hand’s answer key is clockwise throughout, while GPT-4o, Claude 3.5 Sonnet, and Qwen2 VL models predict other rotation patterns.Source: §G.1.1 · p. 32

G.2.1 EXAMPLE 1

  • The moon’s answer key is clockwise throughout, but GPT-4o, Claude 3.5 Sonnet, Qwen2 VL 72B, and Qwen2 VL 7B answer counter-clockwise throughout.Source: §G.2.1 · p. 33

G.2.2 EXAMPLE 2

  • The ping-pong ball’s answer key is counter-clockwise throughout, while GPT-4o and Qwen2 VL 72B answer clockwise and Claude 3.5 Sonnet and Qwen2 VL 7B predict a reversal.Source: §G.2.2 · p. 34

G.2.3 EXAMPLE 3

  • The horizontal-bar answer key is clockwise throughout, while GPT-4o and Video-CCAM-v1.1 answer counter-clockwise and Qwen2 VL 72B predicts a direction change.Source: §G.2.3 · p. 36

G.2.4 EXAMPLE 4

  • The gymnast’s answer key is clockwise throughout, but GPT-4o, Claude 3.5 Sonnet, and Qwen2 VL 72B answer counter-clockwise, while Qwen2 VL 7B predicts a reversal.Source: §G.2.4 · p. 37

G.2.5 EXAMPLE 5 … G.3 SIMULATED

These examples show that models often misidentify rotation direction, sometimes contradicting the stated answers. Some outputs also acknowledge that subtle motion or still frames make direction difficult to determine.

G.2.5 EXAMPLE 5

  • For the car, the stated answer is clockwise throughout, but GPT-4o and InternVL predict counter-clockwise throughout, and Claude also selects counter-clockwise while calling its judgment uncertain.Other listed models give different options, including clockwise then counter-clockwise and counter-clockwise then clockwise.Source: §G.2.5 · p. 38

G.2.6 EXAMPLE 6

  • For the person, the stated answer is clockwise throughout, while GPT-4o predicts a counter-clockwise-to-clockwise reversal and Claude and InternVL predict counter-clockwise throughout.Qwen2 VL 72B selects clockwise then counter-clockwise; Qwen2 VL 7B selects the reversal predicted by GPT-4o.Source: §G.2.6 · p. 39

G.2.7 EXAMPLE 7

  • For the wheel, the stated answer is counter-clockwise throughout, but GPT-4o and Claude predict clockwise throughout, while InternVL selects no rotation.Qwen2 VL 72B selects clockwise throughout and Qwen2 VL 7B selects clockwise then counter-clockwise.Source: §G.2.7 · p. 40

G.2.8 EXAMPLE 8

  • For the rope, the stated answer is counter-clockwise throughout, whereas GPT-4o and several listed models answer clockwise throughout.Claude also selects counter-clockwise, but bases its guess on typical jump-rope mechanics because the rope is not clearly visible.Source: §G.2.8 · p. 41
  • For the rope, Claude describes its counter-clockwise judgment as an inference from limited still images, and InternVL says the frames do not reveal the direction clearly.Claude notes that the rope itself is not clearly visible in most frames.Source: §G.2.8 · p. 41

G.2.9 EXAMPLE 9

  • For the object, the stated answer is clockwise throughout, but GPT-4o and Claude both predict counter-clockwise throughout.Claude says the provided sequence does not show a complete rotation or a reversal.Source: §G.2.9 · p. 43

G.3.1 EXAMPLE 1 … H.2 OBJECT

The examples show models often miss direction changes or infer motion incorrectly from frame sequences, despite sometimes recognizing motion in individual frames. Their answers also conflict with the provided answers in human-action and camera-instruction examples.

G.3.1 EXAMPLE 1

  • The sequence’s answer is counter-clockwise then clockwise; GPT-4o and InternVL 2 40B instead describe the minute hand moving clockwise throughout, contrary to the stated answer.Source: §G.3.1 · p. 46
  • For a minute hand whose answer is counter-clockwise throughout, GPT-4o reports no motion, while Claude 3.5 Sonnet and other models report clockwise motion.Qwen2 VL 7B instead answers D, counter-clockwise throughout.Source: §G.3.1 · p. 47
  • The wheel’s answer is clockwise throughout, but GPT-4o predicts a reversal, Claude 3.5 Sonnet and Qwen2 VL 72B predict counter-clockwise throughout, and InternVL 2 says rotation cannot be determined.Qwen2 VL 7B answers D, matching the provided answer.Source: §G.3.1 · pp. 48–49
  • For the blue circle, the answer is counter-clockwise then clockwise, while GPT-4o and Claude 3.5 Sonnet describe counter-clockwise motion throughout and Qwen2 VL 7B answers C.Qwen2 VL 72B instead answers D, counter-clockwise throughout; Video-CCAM-v1.1 and InternVL 2 give other directions.Source: §G.3.1 · p. 50

H.1.1 EXAMPLE 1

  • In a hand-movement example, the provided answer is right, while GPT-4o describes left then right and Claude 3.5 Sonnet describes right then left.Qwen2 VL answers E, first left then right, while Qwen2 VL 7B says the hand moved from the left side to the right side.Source: §H.1.1 · p. 53

H.2.1 EXAMPLE 1 … I.2 OBJECT

The examples examine models’ inconsistent readings of directional motion and rotation speed from frame sequences, including cases where static frames allow competing interpretations.

H.2.1 EXAMPLE 1

  • GPT-4o reads the drone as moving right then up, although the stated answer is right then left.Other responses also offer differing drone trajectories, including left then up and right then up.Source: §H.2.1 · pp. 57, 60
  • For the helicopter, GPT-4o describes descent and rightward motion, while Claude 3.5 Sonnet favors right then up.The stated answer is down then right; Qwen responses select different options as well.Source: §H.2.1 · p. 59

H.3.1 EXAMPLE 1

  • For the red diamond, GPT-4o predicts right then up, while Claude 3.5 Sonnet reports no visible movement and guesses right then up.The stated answer is up then down, and other model answers also vary.Source: §H.3.1 · p. 66
  • For the blue text block, the supplied answer is downwards, but GPT-4o and Claude choose upwards; Claude notes that the stationary block’s text says it is moving upwards.Source: §H.3.1 · p. 68
  • For the black-sweater direction question, the supplied answer is leftwards then downwards, while GPT-4o and InternVL describe downwards then leftwards and Claude chooses upwards then rightwards.Source: §H.3.1 · p. 69

I.1.1 EXAMPLE 1

  • For the person’s rotation speed, the supplied answer is decelerating, but GPT-4o and Qwen2 VL 72B choose constant speed, Claude makes a speculative constant-speed guess, and InternVL says not moving.Source: §I.1.1 · pp. 72–73

I.2.1 EXAMPLE 1 … J.2 SIMULATED

These examples show models struggling to infer speed, frequency, and shape from frame sequences or sparse frames, sometimes relying on speculative motion narratives or guessing when the queried object is absent. They may also disagree with supplied answers or acknowledge that static images are insufficient for reliable motion judgments.

I.2.1 EXAMPLE 1

  • On a plane-speed question keyed as decelerating, GPT-4o chose constant speed while Claude 3.5 Sonnet and InternVL 2 40B chose accelerating.Claude 3.5 Sonnet acknowledged that its acceleration judgment was an educated guess from still frames.Source: §I.2.1 · pp. 74–75
  • For a roller coaster keyed as decelerating, Claude 3.5 Sonnet inferred acceleration from a presumed chain lift, while GPT-4o said constant speed and InternVL 2 40B said stationary.Claude's reasoning treated a presumed lift as evidence that the car gained speed, although the frames did not show speed directly.Source: §I.2.1 · p. 76
  • In the answer choices, the ball was keyed as decelerating, but GPT-4o inferred acceleration, saying its distance covered in each subsequent frame seemed to increase.InternVL 2 40B said no balls were visible and guessed that the answer was “B”: not moving at all.Source: §I.2.1 · p. 80
  • For a laser-pointer frequency question keyed as decelerating, GPT-4o and Claude 3.5 Sonnet inferred acceleration from increasing apparent spacing or blur trails.InternVL 2 40B instead judged the movement to be constant-speed.Source: §I.2.1 · p. 81
  • For a pendulum keyed as accelerating, GPT-4o chose constant speed, while Claude 3.5 Sonnet reported no pendulum was visible and admitted its constant-speed answer was a guess.Claude explicitly said its guess was not based on visual evidence.Source: §I.2.1 · p. 82
  • Models often called rotation constant when the supplied answer was decelerating or accelerating, and another response said the pendulum’s motion could not be determined.For two rotational-speed questions, GPT-4o chose constant speed despite supplied answers of decelerating and accelerating; a separate response said the frames showed no pendulum.Source: §I.2.1 · pp. 82–84

I.2.10 EXAMPLE 10

  • For the disk, models disagreed: GPT-4o inferred deceleration, Claude inferred acceleration, and other models selected constant speed despite an accelerating supplied answer.Claude cited increasing motion blur across the frames, while GPT-4o cited smaller changes in the mark’s position between frames.Source: §I.2.10 · p. 85

I.2.11 EXAMPLE 11

  • For camera movement, the supplied answer was decelerating, while GPT-4o judged the camera stationary and other models inferred constant speed.Claude acknowledged uncertainty from static frames, while InternVL described the speed as relatively constant.Source: §I.2.11 · pp. 86–87

I.2.12 EXAMPLE 12

  • For the hula hoop, models characterized the rotation as constant speed although the supplied answer was accelerating.GPT-4o, Claude, and InternVL described a consistent rotational pace.Source: §I.2.12 · p. 88

I.3.1 EXAMPLE 1

  • For the car, most responses favored constant speed, while InternVL said the frames were insufficient to determine the pattern and then guessed acceleration; the supplied answer was deceleration.The responses illustrate the gap between recognizing movement and determining its speed pattern.Source: §I.3.1 · p. 89

J.1.1 EXAMPLE 1

  • For the air-drawn shape, GPT-4o inferred a square or rectangle, Claude saw no clear shape, and InternVL inferred a circle, while the supplied answer was trapezoid.Claude described its square-or-rectangle interpretation as tentative because the frames did not clearly show a specific shape.Source: §J.1.1 · pp. 90–91

J.2.1 EXAMPLE 1 … K.1.13 EXAMPLE 13

These examples examine how models infer object trajectories and instrument timing or order from visual frames. They highlight cases where frame evidence may not establish these details precisely and model answers can diverge from annotated answers.

J.2.1 EXAMPLE 1

  • For a black-circle trajectory annotated as an infinity shape, GPT-4o and Claude 3.5 Sonnet describe a circular path, while InternVL 2 40B calls it a diamond.Source: §J.2.1 · p. 95

K.1.1 EXAMPLE 1

  • For the question about which instrument sounds last, the annotated answer is flute, but GPT-4o and Claude 3.5 Sonnet infer that all instruments play simultaneously.InternVL 2 40B notes that the frames do not provide sound timing, yet also selects simultaneous playing.Source: §K.1.1 · p. 99
  • For which instrument plays first, the annotated answer is violin, while GPT-4o says the frames do not indicate which starts first and guesses simultaneous onset.Source: §K.1.1 · p. 100
  • For a flute–ukulele onset question answered “Flute,” GPT-4o guessed simultaneous playing while Claude inferred a ukulele solo from late-frame emphasis.Claude noted that the frames could not establish the exact order without audio.Source: §K.1.1 · p. 102
  • For a two-violin question answered “The right one,” GPT-4o instead inferred simultaneous playing because both violinists appeared active throughout the frames.The examples provide no clear visual indication of which violin started first.Source: §K.1.1 · p. 103
  • For a question answered “Cello on the middle” as the song’s final instrument, models guessed a left or right violin or simultaneous sound despite lacking clear ending cues.Claude explicitly described its answer as a guess from limited visual information; InternVL also said the frames did not indicate which instrument finished.Source: §K.1.1 · pp. 106–107
  • In Example 1, instrument-start questions elicit guesses about which player starts first as well as claims that both instruments begin simultaneously.One prompt’s answer key says both begin simultaneously, while model responses include guesses of individual instruments and agreement with simultaneity.Source: §K.1.1 · pp. 111, 113

K.1.13 EXAMPLE 13

  • In Example 13, the answer key marks the right guitar (D) as first; among cited model responses, GPT-4o also selects D, InternVL 2 40B selects simultaneous playing, and Qwen2 VL 72B instruct AWQ selects A (left).Source: §K.1.13 · p. 115

K.1.14 EXAMPLE 14 … L.2 OBJECT

The examples show models inferring event timing and repeated actions from sampled frames, but often disagreeing or acknowledging that the frames do not establish the answer. The disagreements span musical onset questions and counts of shapes or hand movements.

K.1.15 EXAMPLE 15

  • For the violin onset question, GPT-4o selected the right violin, while Claude and InternVL 2 40B answered that both sounded simultaneously, with InternVL noting the frames lack audio.Qwen2 VL 72B and 7B and Video-CCAM-v1.1 answered that the left violin was first.Source: §K.1.15 · p. 118

K.1.16 EXAMPLE 16

  • For the guitar–ukulele onset question, GPT-4o answered simultaneous onset, while Claude and Qwen2 VL answered ukulele; InternVL 2 40B said the frames were insufficient to determine which played first.Claude’s rationale relied on visual position and gesture despite describing both musicians as playing in all frames.Source: §K.1.16 · p. 119

L.1.1 EXAMPLE 1

  • For triangle counts, GPT-4o inferred three complete triangles from the frame sequence, whereas Claude saw no clear complete triangles and offered two as an uncertain guess.The two responses illustrate conflicting interpretations of the sampled hand gestures.Source: §L.1.1 · pp. 120–121
  • For diamond counts, GPT-4o inferred four completions, while Claude and InternVL 2 40B said the frames did not clearly establish diamond drawing and guessed one and two, respectively.Claude’s one-diamond answer was described as a conservative guess from limited still images.Source: §L.1.1 · pp. 122–123
  • For square counts, GPT-4o counted four, Claude guessed one because the sampled gestures were unclear, and Qwen2 VL 7B reported two squares.The responses give differing counts for the same hand-motion sequence.Source: §L.1.1 · p. 124
  • For rightward hand swings, GPT-4o counted six instances, while Claude 3.5 Sonnet counted four clear instances in the sampled frames.Both responses present counts based on frame inspection.Source: §L.1.1 · p. 126

L.2.1 EXAMPLE 1 · L.3 SIMULATED

Across five example counting questions, model answers often diverge, and some explicitly say the still frames do not establish exact event counts before guessing.

L.2.1 EXAMPLE 1

  • For the car’s complete circles, answers range from 0 to 2: InternVL says the count cannot be determined but guesses 0, while Claude guesses 2 and Qwen2 VL 7B answers 1.Claude also calls its answer an educated guess based on limited information.Source: §L.2.1 · pp. 128–129
  • For the rectangle question, GPT-4o describes a star rather than a rectangle but guesses 1; Claude likewise sees no rectangle and guesses 1.Claude says 1 is the lowest non-zero option despite seeing no rectangle.Source: §L.2.1 · p. 130
  • For the person’s steps, GPT-4o estimates 3, Claude estimates 1, and InternVL estimates 5 from the sampled frames.Claude says the exact step count is difficult to determine from static images; InternVL calls its estimate a guess.Source: §L.2.1 · pp. 131–132
  • For gyrations, GPT-4o guesses 4, Claude guesses 2 complete gyrations despite seeing 3–4 possible movements, and InternVL says the exact count is uncertain before guessing 5.Claude distinguishes complete gyrations from partial or preparatory motions.Source: §L.2.1 · pp. 133–134
  • For the triangle question, GPT-4o and Claude describe a butterfly rather than a triangle, yet both guess 1; InternVL also guesses 1 while noting no clear triangle is shown.Claude says its answer is based on limited information.Source: §L.2.1 · p. 135
  • Other triangle responses differ: Qwen2 VL 72B instruct AWQ answers C, Qwen2 VL 7B Instruct answers D, and Video-CCAM-v1.1 14B answers A.Source: §L.2.1 · p. 135

L.3.1 EXAMPLE 1

Across these collision-count examples, models give different counts for the videos and sometimes qualify their answers because still frames may not reveal every event. The examples illustrate uncertainty in interpreting collisions across a sequence.

L.3.1 EXAMPLE 1

  • For one clip, GPT-4o identifies 4 distinct collisions, while Claude 3.5 Sonnet guesses 2 and notes that exact collisions are difficult to determine from still images.Claude bases its estimate on objects appearing close together across frames.Source: §L.3.1 · p. 136
  • For another clip, GPT-4o counts 2 collisions, while Claude identifies at least 1 and cautions that 16 sampled frames may miss events.GPT-4o describes collisions involving a gray cylinder and purple cylinder, then a blue sphere and gray cylinder.Source: §L.3.1 · p. 138
  • In a further clip, GPT-4o reports 1 collision, whereas Claude identifies 2: the yellow sphere with the green cube and the blue sphere with the beige cylinder.The two models therefore differ in their count for this clip.Source: §L.3.1 · p. 140
Loading 2410.23266v2…