Let's Think Frame by Frame with VIP: A Video Infilling and Prediction Dataset for Evaluating Video Chain-of-Thought

1 Introduction

• Existing video question-answering and summarization datasets do not assess challenging reasoning about relationships between multiple frames.Source: §1 · p. 1
• Selecting a small sample of keyframes can capture much of a video's meaning while reducing the computational burden of processing video frames.Reasoning across the selected keyframes still requires stronger language-model capabilities.Source: §1 · p. 1
• Video Infilling predicts masked keyframe descriptions from preceding and following descriptions, whereas Video Prediction forecasts subsequent descriptions from preceding keyframes.Source: §1 · p. 1
• VIP pairs highly detailed unstructured captions with structured FAMOuS descriptions organized around Focus, Action, Mood, Objects, and Setting.The two formats support visually descriptive and more focused reasoning, respectively.Source: §1 · pp. 1–2, Figure 1 · p. 2
• Existing models show potential for multi-hop, multi-frame video reasoning, but substantial improvement remains possible.Source: §1 · p. 2

2 Related Work

• VIP offers extracted real-world video keyframes as a resource for evaluating reasoning abilities in the video domain.Source: §2 · p. 2
• Existing video datasets are often domain-specific or require supplementary representations, and some simplify tasks to textual summaries or a single frame.Source: §2 · p. 2
• Some prior datasets consider multiple frames, while VIP targets multi-hop reasoning over real-life videos without significant computation to train on videos.Source: §2 · p. 3
• Unlike prior video-text approaches, VIP introduces textual representations at the keyframe level and uses them to reason about specific video segments through VideoCOT.Source: §2 · p. 3

3 VIP Dataset Construction

• The dataset uses YouTube-8M videos, downweights visually static categories, and automates keyframe and textual-description generation to reduce manual construction costs.Videos may belong to multiple categories.Source: §3 · p. 3

3.1 Representative Keyframe Selection

• Keyframe selection starts with KATNA candidates, removes blurry or object-poor frames, then prunes frames with high neighboring similarity using image and object embeddings.A safeguard retains frames containing people unless a neighboring frame also includes people.Source: §3.1 · pp. 3–4

3.2 Textual Representations of Keyframes

• VIP pairs unstructured dense captions with FAMOuS descriptions that organize each frame’s focus, action, mood, objects, and setting.Dense captions combine frame captions, object lists, and object-level descriptions, with confidence scores and video-topic grounding used to mitigate hallucinations; crowdsourcing checks description quality.Source: §3.2 · pp. 4–5, §3.3 · p. 5

3.3 Dataset Contributions

• VIP introduces a dataset for evaluating multi-hop video reasoning through video chain-of-thought, with detailed frame descriptions that capture visual and semantic changes.Descriptions average 114 tokens, compared with traditional video caption datasets.Source: §3.3 · p. 5

4 Video Reasoning Tasks

• Video Infilling and Video Prediction adapt masked-language modeling and text completion to test models’ multi-frame video reasoning.Infilling reconstructs keyframes from preceding and following context; prediction generates the sequence that follows given frames.Source: §4 · p. 5

4.1 Video Infilling Task

• Infilling reconstructs masked keyframe descriptions from preceding and following context, and its difficulty rises with less context or a longer masked span.The task probes temporal changes and transitions, including visual elements, object positions, and contextual factors.Source: §4.1 · p. 6

4.2 Video Prediction Task

• Prediction generates future keyframes from past context, with reduced context or longer prediction spans increasing difficulty.Because prediction uses only past context, possibilities can increase exponentially as the generated sequence grows.Source: §4.2 · p. 6

5.1 Primary Results

• Models show weak overall performance, and generated outputs lack some semantic congruence with the ground truth.The authors conclude that unimodal language performance does not generalize to VIP’s video reasoning tasks.Source: Table 3 · p. 7, §5.5 · p. 9

5 Experiments

• The benchmark uses one in-context demonstration, greedy decoding, and three generated keyframes per task, averaging their metric scores against ground truth.It evaluates dense captions and FAMOuS descriptions with ROUGEL, BERTSCORE, and SENTENCEBERT.Source: §5 · p. 7

5.1 Primary Results

• Infilling generally outperforms prediction, while additional context yields higher but marginal gains; dense captions score better than FAMOuS descriptions on the selected metrics.The authors caution that word-similarity metrics may favor verbose dense captions over category-based FAMOuS descriptions.Source: §5.1 · pp. 7–8
• Model rankings show no clear overall trend: GPT-3 and VICUNA perform slightly better than GPT-4 on ROUGEL, while GPT-3 and GPT-4 have an edge on SENTENCEBERT.The authors suggest exact-word consistency may favor the earlier models on ROUGEL, whereas the SENTENCEBERT results suggest stronger semantic alignment for GPT-3 and GPT-4.Source: §5.1 · p. 8

5.2 FAMOuS Component Analysis

• For Prediction-3, models perform better on FAMOuS objects by ROUGEL and focus by BERTSCORE, while SENTENCEBERT favors mood, objects, and setting.The authors identify action as a weaker component and suggest that its more dynamic nature could benefit from video representations.Source: §5.2 · p. 8, Table 4 · p. 8

5.3 Causal Aspect Analysis

• Social causal-reasoning tasks score higher on BERTSCORE, while physical causal-reasoning tasks score higher on SENTENCEBERT.The authors connect these patterns to character and object consistency in social scenarios and environmental qualities in physical scenarios.Source: §5.3 · p. 8

5.4 Domain Analysis

• Games perform well and Jobs & Education fall behind, although several domain categories are noisy because of low sample sizes.The authors hypothesize that domain-specific training-data availability and the intrinsic dimensionality needed to model topic interactions contribute to these observations.Source: §5.4 · pp. 8–9

5.5 Qualitative Observations

• Qualitative examples show limited semantic congruence with ground truth and suggest possible overfitting, despite a baseline level of video understanding.The paper notes that quantitative and qualitative results indicate unimodal language models do not generalize their strong language-task performance to VIP’s video reasoning tasks.Source: §5.5 · p. 9

6 Future Work

• Targeted inference-time techniques, fine-tuning, a train-validate-test paradigm, and additional data are proposed as directions beyond the standard few-shot benchmark.The paper frames these as opportunities to develop skills beyond the general pretrained learnset.Source: §6 · p. 9
• A longer-term direction is bridging video reasoning end-to-end with video, with image synthesis and then video synthesis proposed as steps toward unifying the tasks.The authors suggest unification could reduce errors and improve usability.Source: §6 · p. 9
• Robust evaluation metrics should account for spatial and temporal invariance, including interchangeable actions across frame regions and sequences.The authors describe these properties as desirable but difficult to capture.Source: §6 · p. 9

7 Conclusion

• VIP evaluates video reasoning by having models perform video chain-of-thought over keyframes represented with dense captions or structured FAMOuS descriptions.The dataset introduces a pipeline for systematically extracting keyframes and generating the textual representations.Source: §7 · p. 9

• Multimodal models’ limited support for multiple image inputs per query means the benchmark currently evaluates only large language models.The authors identify this as the primary limitation and encourage future work on video scene-description generation.Source: Limitations · p. 10
• Potential errors in AI-generated examples are addressed through crowdworker classification and expert correction of descriptions flagged as poor quality.Source: Limitations · p. 10

• English-only captions bias the dataset toward culturally Western scenes, and multilingual settings may show differential behavior between language classes.Source: Ethics Statement · p. 10
• Human evaluation was restricted to workers in Australia, Canada, New Zealand, the United Kingdom, or the United States with at least a 98% HIT approval rating.Scene-description checking and correction were compensated at $15 per hour under an approved exempt IRB protocol.Source: Ethics Statement · p. 10

A.1 Human Evaluation Interface

• Prompt 1 asks for three intermediate keyframe descriptions from descriptions of frames 1 and 5, matching their structure and length.Its framing is “Let’s think frame by frame.”Source: §A.1 · p. 14, Figure 6 · p. 14
• Across five infilling and prediction tasks, Games, Sports, People & Society, Health, and Science perform better on average by domain, while Jobs & Education, Law & Government, and Food & Drink perform less well.The table caption identifies the scope and domain trend but does not provide specific cell values.Source: Table 6 · p. 15
• The prediction example asks for descriptions of the next three keyframes from the description of frame 1, matching its structure and length.Source: §A.1 · p. 16, Figure 8 · p. 16
• The human-evaluation interface asks crowdworkers to judge the accuracy of FAMOuS scene descriptions.Source: Figure 9 · p. 16