PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition

Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Meishan Zhang, Mong-Li Lee, Wynne Hsu

arXiv:2501.03230v1cs.AIcs.CV

TL;DR

Complex-video understanding remains limited by gaps in fine-grained spatial-temporal perception and cognitive semantic reasoning. The paper introduces MotionEpic with STSG grounding and builds VoT to solve video reasoning through sequential low-to-high-level sub-problems. Across complex video QA benchmarks, the framework improves existing state-of-the-art performance, while the authors note training, data, environmental, and misuse concerns.

  • Problem

    Complex video reasoning requires fine-grained pixel understanding and cognitive semantic interpretation, while existing approaches often remain shallow or lack video-specific CoT frameworks.

  • Method

    MotionEpic integrates STSG representations for spatial-temporal grounding, and VoT decomposes video questions into sequential sub-problems from pixel perception to cognitive interpretation.

  • Results

    Across 8 complex video QA benchmarks, the framework boosts current performance in fine-tuning and zero-shot settings and establishes new state-of-the-art results.

  • Takeaways & Limitations

    The framework supports more nuanced, human-level video analysis by combining fine-grained grounding with stepwise cognitive reasoning.

  • Takeaways & Limitations

    The framework may require substantial training energy and larger data corpora, and its capabilities create potential misuse risks requiring responsible licensing.

Abstract

from arXiv · show

Existing research of video understanding still struggles to achieve in-depth comprehension and reasoning in complex videos, primarily due to the under-exploration of two key bottlenecks: fine-grained spatial-temporal perceptive understanding and cognitive-level video scene comprehension. This paper bridges the gap by presenting a novel solution. We first introduce a novel video Multimodal Large Language Model (MLLM), MotionEpic, which achieves fine-grained pixel-level spatial-temporal video grounding by integrating video spatial-temporal scene graph (STSG) representation. Building upon MotionEpic, we then develop a Video-of-Thought (VoT) reasoning framework. VoT inherits the Chain-of-Thought (CoT) core, breaking down a complex task into simpler and manageable sub-problems, and addressing them step-by-step from a low-level pixel perception to high-level cognitive interpretation. Extensive experiments across various complex video QA benchmarks demonstrate that our overall framework strikingly boosts existing state-of-the-art. To our knowledge, this is the first attempt at successfully implementing the CoT technique for achieving human-level video reasoning, where we show great potential in extending it to a wider range of video understanding scenarios. Project is open at https://haofei.vip/VoT

1. Introduction

Complex video reasoning requires both fine-grained spatial-temporal perception and cognitive semantic understanding. The paper addresses these needs with MotionEpic and the stepwise Video-of-Thought framework.

  • Motivation: Existing video methods often handle straightforward perceptual understanding but leave in-depth reasoning about complex videos underexplored.Complex-video reasoning requires understanding spatiotemporal characteristics and implications beyond visible pixels.
  • Motivation: Complex video reasoning requires precise pixel-level perception together with cognitive capabilities for semantic explanation and causal imagination.The paper identifies perceptual pixel understanding and cognitive semantic understanding as complementary requirements.
  • Contributions: MotionEpic integrates video and spatial-temporal scene-graph representations to support fine-grained pixel-level grounding and autonomous STSG parsing.Grounding-aware tuning uses annotations initially, followed by STSG-free inference for downstream reasoning.
  • Contributions: VoT decomposes complex video questions into sequential sub-problems that progress from pixel grounding and tracking to semantic interpretation and answer verification.The process uses grounded tracklets, scene interactions, commonsense, answer ranking, and verification.
  • Results: Across 8 complex video QA benchmarks, the system improves performance over current methods in both fine-tuning and zero-shot settings and establishes new state-of-the-art results.The paper also analyzes MotionEpic’s grounding and VoT’s reasoning abilities.

2. Related Work

Prior work established strong language and multimodal reasoning, but video understanding remained largely shallow and video-specific Chain-of-Thought reasoning was underexplored. The paper introduces a genuine video thought chain that follows human cognitive progression.

  • MLLMs and CoT: Large language models and multimodal models brought strong semantic and commonsense reasoning capabilities to language and image understanding.CoT further decomposes problems into constituent parts and provides rationale at each step.
  • Video Understanding: Earlier video models trained on small in-domain datasets were mainly limited to shallow perception of contents and movements in simple videos.These methods lacked the depth associated with human-level cognition.
  • Video-of-Thought: Although CoT and multimodal CoT advanced reasoning, a framework specifically integrating CoT with video MLLMs remained underexplored.This gap concerns establishing a powerful reasoning framework tailored to video input.
  • Video-of-Thought: VoT implements a genuine thought chain by decomposing video problems before reasoning, moving from low-level pixel grounding to high-level semantic inference.Its design follows a human-cognitive procedure rather than relying only on a simple step-by-step prompt.

3. MotionEpic: Fine-grained Spatial-temporal Grounded Video MLLM

MotionEpic is a video MLLM that integrates STSGs for fine-grained spatial-temporal grounding. Its representation links frame-level objects, relationships, and persistent identities, enabling generated STSGs to serve as reasoning rationale.

  • Architecture: MotionEpic combines text prompts, video, and STSG inputs using a video encoder, Q-Former projector, graph transformer, and recurrent STSG propagation.The backbone LLM is Vicuna-7B (v1.5), while recurrent propagation encodes multi-frame graph information.
  • STSG Representation: An STSG represents each video frame as a scene graph of subject-predicate-object relations and preserves compositional spatiotemporal semantics.Its structural features filter less-informative background information and support perceptive understanding.
  • STSG Representation: The system reduces temporal redundancy by evenly sampling frames before constructing its adapted STSG representation.This modification is intended to reduce computation costs.
  • STSG Representation: Each frame graph stores object categories, neural features, bounding boxes, predicate edges, and temporal coreference edges linking the same objects across frames.The temporal links mimic tracking and enhance connectivity across single-frame graphs.
  • Grounding: MotionEpic can generate partial STSGs from prompted video, grounding specific content and recycling the output STSG as rationale for subsequent processing.The generated expression corresponds to a structural STSG representation.
  • Training: Grounding-aware tuning targets coarse video-STSG correspondence and fine-grained tracklet, action, object-label, and bounding-box correspondence.The objectives include whole-STSG generation, action-conditioned tracklets, and object tracking from a frame bounding box.

4. Video-of-Thought Reasoning Framework

VoT decomposes complex video reasoning into five sequential steps, progressing from target identification and pixel-level grounding to action interpretation, answer ranking, and verification.

  • Framework overview: VoT divides complex video reasoning into five chained steps that progress from low-level perceptive pixel grounding to high-level cognitive semantic comprehension.The framework is designed as a finer-grained decomposition than vanilla CoT’s single step-by-step prompt.
  • Step-1: Task Definition and Target Identification: The first step defines the task and identifies the video targets involved in the question.Targets may be explicitly visible or implicitly related to the question.
  • Step-2: Object Tracking: The second step tracks each target’s full spatial-temporal characteristics through STSG-based grounding, producing tracklets as evidence for behavior analysis.The resulting target tracklets provide low-level supporting rationale for the next reasoning stage.
  • Step-3: Action Analyzing: The third step analyzes target actions by combining STSG tracklets, neighboring scene nodes, and commonsense knowledge.The output is an observation and implication describing the action and its semantic meaning.
  • Step-4: Question Answering via Ranking: The fourth step scores each candidate answer from 1 to 10 using the inferred action observations and commonsense, then selects the highest-ranked option.The ranking mechanism supports multiple-choice questions and can unify open-ended QA through candidate generation.
  • Step-5: Answer Verification: The fifth step verifies the selected answer against both pixel-grounding facts and commonsense implications, re-running answer ranking when inconsistencies appear.This verification checks perceptual alignment with the video and cognitive consistency with inferred observations.

5. Experiments

Experiments evaluate MotionEpic and VoT on complex video QA benchmarks in fine-tuning and zero-shot settings, with analyses covering grounding, tuning objectives, reasoning errors, and qualitative behavior.

  • Task and Data: The evaluation uses six complex video QA benchmarks for fine-tuning and adds MSR-VTT and ActivityNet in the zero-shot setting.The complex benchmarks require abilities such as explanation, causality, foresight, and imagination.
  • Main Performance on Video QA Reasoning: MotionEpic with VoT consistently boosts state-of-the-art baselines by large margins across the main video QA results.The reported comparisons span Tables 1–3 and include multiple video QA datasets and reasoning settings.
  • Main Performance on Video QA Reasoning: STSG integration improves video understanding, while implicit scene-graph integration in MotionEpic is reported as superior to explicit integration under the same CoT setting.The comparison also reports that vanilla CoT with MotionEpic beats state-of-the-art methods on certain datasets.
  • Zero-shot Performance: In zero-shot evaluation, VoT improves over CoT more clearly than in fine-tuning, especially on complex cognitive video QA tasks rather than simpler perceptive tasks.The paper attributes this pattern to differences between cognitive understanding and perceptive understanding across datasets.
  • Zero-shot Performance: For simpler datasets, pixel-grounding verification is more crucial than commonsense verification, whereas both verification types are pivotal for complex videos.This conclusion comes from removing either verification mechanism and comparing the resulting performance.
  • Analyses on VoT Reasoning Framework: MotionEpic with VoT achieves results comparable to human performance on selected hard instances and reduces Video-LLaVA with CoT’s errors, especially for action semantics and commonsense understanding.The analysis uses 200 hard instances each from Causal-VidQA and Social-IQ and includes human evaluation.
  • Video Reasoning Visualization: A qualitative case study shows VoT combining STSG grounding, commonsense interpretation, candidate ranking, and verification to answer an abstract question that baselines miss.The example emphasizes an explainable reasoning process through decomposition across perception and cognition.

6. Conclusion

The paper introduces VoT for complex video reasoning alongside MotionEpic, which provides fine-grained pixel-level spatial-temporal grounding through video STSG representation. Experiments across complex video QA benchmarks report improved state-of-the-art performance.

  • VoT decomposes intricate video tasks into manageable sub-problems and solves them sequentially from low-level pixel perception to advanced cognitive interpretation.
  • MotionEpic achieves fine-grained pixel-level spatial-temporal video grounding by integrating video STSG representation.
  • Experiments across various complex video QA benchmarks demonstrate the efficacy of the approach and improvements over existing state-of-the-art standards.

Statement of Potential Broader Impact

The paper identifies energy consumption, data requirements, and potential malicious misuse as broader-impact concerns for its human-level video reasoning framework. It calls for licensing mechanisms to support responsible deployment and mitigate misuse.

  • Training the LLM-based system may require substantial energy consumption and more extensive data corpora, with potential environmental degradation.
  • The framework’s powerful video reasoning and comprehension capabilities could be exploited by malicious actors for nefarious intents, posing a societal threat.
  • The paper proposes specific licensing mechanisms to ensure responsible deployment and mitigate potential misuse.

A.1. Detailed Prompt Construction and System I/O

The VoT prompting process converts video QA into a staged reasoning pipeline: identify targets, ground and track them, analyze actions with commonsense, rank answers, and verify the result. Open-ended questions can first be transformed into multiple-choice questions.

  • Task Definition and Target Identification: VoT first identifies targets mentioned or implicitly involved in the video question using the raw video and task prompt.
  • Target Tracking: The second step requests target tracklets by outputting corresponding partial expressions in the STSG.
  • Action Analyzing: Action analysis integrates target tracking, neighboring STSG nodes, and commonsense to interpret motion behavior, interactions, and semantics.
  • Transforming Open-ended Question Answering into Multi-choice: For open-ended QA, VoT generates multiple distinct optional answers and converts the task into a unified multi-choice format.
  • Multi-choice Question Answering via Ranking: For each candidate answer, the model assigns a 1-10 rationality score with a rationale, then ranks candidates to determine the final answer.
  • Answer Verification: Answer verification checks whether the selected answer aligns with pixel-grounding facts and contradicts commonsense implications or observations from earlier reasoning.
  • The prompting process is presented as an example that can be adapted to other video reasoning tasks by transforming them into a QA format.

A.2. Implementations

MotionEpic combines a Vicuna-7B backbone, a ViT-L/14 video encoder, a Q-Former projector, and a six-layer Recurrent Graph Transformer. Videos are uniformly sampled at 8 fps, and fine-tuning and zero-shot settings use different training procedures.

  • MotionEpic uses Vicuna-7B (v1.5) as its backbone LLM, ViT-L/14 as video encoder, and Q-Former as projector.
  • The Recurrent Graph Transformer has six layers with 768-dimensional hidden sizes, while the LLaMA tokenizer contains approximately 32,000 classes.
  • Videos are uniformly sampled at 8 fps because preliminary studies found this setting offered the best trade-off.
  • Fine-tuning uses training-set data splits and evaluation methods aligned with prior baselines, whereas zero-shot video QA uses no in-domain training set.

B. More Qualitative Visualizations

The qualitative cases span perception-level and cognitive-level video QA, illustrating how MotionEpic handles structured object grounding and commonsense interpretation. The examples also reflect differing dataset biases toward perceptual recognition or cognitive comprehension.

  • Dataset perspectives: The qualitative analysis covers datasets biased toward either content recognition or cognitive comprehension involving physical, cultural, or humanities knowledge.
  • Perception-level reasoning: Perception-level cases show MotionEpic accurately grounding object counts when both baselines answer incorrectly.The example attributes this result to MotionEpic’s use of structured STSG representation.
  • Perception-level reasoning: A second perception-level case is solvable through straightforward temporal understanding, with both MotionEpic and Video-LLaVA answering correctly.
  • Cognitive-level reasoning: Cognitive-level cases require combining visual scene content with commonsense to infer a supermarket setting and the intention to release a crab into the sea.The crab example is difficult from visual content alone because the woman’s implicit intention is not directly observable.
Loading 2501.03230v1…