PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?

Junhao Cheng, Yuying Ge, Teng Wang, Yixiao Ge, Jing Liao, Ying Shan

arXiv:2505.21374v1cs.CV

TL;DR

Existing video benchmarks mainly test perception and grounding, leaving open whether MLLMs can perform complex, expert-like video reasoning. Video-Holmes evaluates that ability with questions requiring clues to be found and connected across video segments. Models generally struggle to integrate clues, and Gemini-2.5-Pro achieves 45% accuracy.

  • Problem

    Existing video benchmarks primarily assess visual perception and grounding, leaving the question of MLLMs’ complex, expert-like video reasoning open.

    Source: §1 · p. 2

  • Method

    Video-Holmes uses seven reasoning tasks based on manually annotated suspense films, with questions requiring MLLMs to locate and connect clues across video segments.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    45% accuracy for Gemini-2.5-Pro; evaluated MLLMs generally perform well at visual perception but struggle to integrate information and often miss critical clues.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The evaluation highlights ongoing challenges for MLLMs in integrating evidence for complex video reasoning.

    Source: §1 · p. 2

Abstract

from arXiv · show

Recent advances in CoT reasoning and RL post-training have been reported to enhance video reasoning capabilities of MLLMs. This progress naturally raises a question: can these models perform complex video reasoning in a manner comparable to human experts? However, existing video benchmarks primarily evaluate visual perception and grounding abilities, with questions that can be answered based on explicit prompts or isolated visual cues. Such benchmarks do not fully capture the intricacies of real-world reasoning, where humans must actively search for, integrate, and analyze multiple clues before reaching a conclusion. To address this issue, we present Video-Holmes, a benchmark inspired by the reasoning process of Sherlock Holmes, designed to evaluate the complex video reasoning capabilities of MLLMs. Video-Holmes consists of 1,837 questions derived from 270 manually annotated suspense short films, which spans seven carefully designed tasks. Each task is constructed by first identifying key events and causal relationships within films, and then designing questions that require models to actively locate and connect multiple relevant visual clues scattered across different video segments. Our comprehensive evaluation of state-of-the-art MLLMs reveals that, while these models generally excel at visual perception, they encounter substantial difficulties with integrating information and often miss critical clues. For example, the best-performing model, Gemini-2.5-Pro, achieves an accuracy of only 45%, with most models scoring below 40%. We aim that Video-Holmes can serve as a "Holmes-test" for multimodal reasoning, motivating models to reason more like humans and emphasizing the ongoing challenges in this field. The benchmark is released in https://github.com/TencentARC/Video-Holmes.

1 Introduction

Advances in CoT reasoning and RL post-training, adapted to video MLLMs [1, 2, 6, 7, 8, 9], motivate asking whether models can reason about video like human experts. Video-Holmes tests this with suspense films and questions requiring clue search and integration.

1 Introduction

  • Existing video reasoning benchmarks [10] [11] [12] [13] [14] [15] [16] [17] emphasize perception and grounding, often using explicit prompts or isolated visual cues.On these benchmarks, advanced thinking abilities yield only marginal gains, from 68.3% to 69.4% [18].Source: §1 · p. 2
  • Its suspense-film questions require active clue-seeking and multi-clue inference, and the benchmark analyzes models’ reasoning processes and factors behind their answers.The films feature suspense, reasoning, and supernatural themes; questions may require progressively integrating clues to infer a cause.Source: §1 · p. 2
  • 45% accuracy is the best result, achieved by Gemini-2.5-Pro, while most evaluated MLLMs score below 40%.Despite generally strong visual perception, models struggle to integrate information and often miss critical clues.Source: §1 · p. 2
  • Video-Holmes contains 1,837 questions from 270 manually annotated suspense short films, requiring models to locate and link clues across video segments.The benchmark is designed to assess complex video reasoning and analyze factors behind correct and incorrect answers.Source: §1 · p. 2

2 Related Works

Prior work extends image-based MLLMs and reasoning strategies to video, while existing benchmarks largely test perception and grounding rather than complex clue integration. Video-Holmes addresses this gap by requiring models to locate and connect clues across video segments.

MLLMs for Video Understanding and Reasoning

  • VideoChat [23], Video-ChatGPT [24], CogVLM2Video [25], InternVL [26] [27], LLaVA-Video [28], and Qwen-VL [29] [30] process videos as image sequences for understanding and reasoning.CoT and reinforcement post-training have been extended from LLM reasoning [3] to multimodal image and video tasks [31] [32] [33] [34] [35] [36] [7] [6] [37] [8] [38]. Video-R1 [6] and VideoChat-R1 [7] apply SFT followed by GRPO [2] on Qwen-VL-2.5 [39] and outperform vanilla Qwen-VL-2.5 across diverse reasoning tasks.Source: §2 · p. 3

Video Reasoning Benchmarks

  • Existing benchmarks span basic video QA [41] [42], short- and long-clip reasoning [43] [16] [15] [14], and domain-specific or CoT evaluation [12] [11] [10] [13], but generally test perception and grounding rather than complex clue integration.In contrast, Video-Holmes requires models to actively locate and connect multiple relevant visual clues, presenting a more complex reasoning scenario.Source: §2 · p. 3

3 Video-Holmes

Video-Holmes combines manually annotated suspense films with seven reasoning tasks that require active search for and connection of clues across video segments. The resulting dataset contains 270 videos and 1,837 question-answer pairs.

Video Collection and Annotation

  • The authors curated 270 high-quality, 1–5-minute suspense films from an initial pool of over 2,500 YouTube videos and manually annotated them.Annotations cover segmented plot descriptions, character relationships, reasoning shots and evidence, supernatural elements, and core themes; the authors describe these materials as supporting dataset reliability and quality.Source: §3 · pp. 3–4

Task Definition

  • The seven tasks require models to locate and connect clues across video segments, rather than answer questions with clues explicitly supplied.They cover Social Reasoning, Physical Anomaly Reasoning, Multimodal Hint Reasoning, Intention & Motive Chaining, Temporal Causal Inference, Timeline Analysis, and Core Theme Inference.Source: §3 · p. 4, Figure 3 · p. 5

Question-Answer Generation

  • 270 videos and 1,837 question-answer pairs make up the final dataset, with questions generated from formatted annotations using DeepSeek-R1 [3].Questions were manually sampled for quality and relevance, and the model also generated answer explanations for analyzing reasoning processes.Source: §3 · pp. 4–5

4 Experiments

Video-Holmes remains difficult for current MLLMs: the best model reaches only 45% accuracy, and analysis attributes many errors to missed clues and difficulty reasoning across them. Experiments also show that performance depends on model capability, available modalities, and how visual clues are represented.

4.1 Setup

  • The evaluation covered open- and closed-source MLLMs, used official checkpoints and guidelines, and prompted models to reason step by step before answering.Models with native video input received videos directly; others were evaluated using uniformly sampled, timestamped frames as multi-image input.Source: §4.1 · pp. 5–6

4.2 Main Results

  • 45% accuracy was highest (Gemini-2.5-Pro), most models scored below 40%, and Qwen2.5-VL (7B) reached 27.8% on Video-Holmes.Models also remained below 40% accuracy on most of the seven tasks.Source: §4.2 · pp. 6–7, Table 2 · p. 6
  • Thinking variants improved performance, including a 12.5% gain for Gemini-2.0-Flash-Thinking over Gemini-2.0-Flash, while other benchmarks did not show the same pattern.The authors interpret this as evidence that Video-Holmes distinguishes models’ reasoning abilities.Source: §4.2 · pp. 6–7, Table 3 · p. 6

4.3 Analytical Study

  • Approximately 60% of errors stemmed from logical comprehension of multiple visual clues (reasoning errors), while around 35% came from omitted critical visual information; TRAR exceeded 80% across most models’ correctly answered questions.Source: §4.3 · pp. 7–8
  • Increasing input frames generally improved performance without substantial gains, while adding audio helped especially with social reasoning tasks.Conversational cues can provide insights into interpersonal dynamics.Source: §4.3 · pp. 8–9, Table 5 · p. 9
  • CoT prompting improved accuracy for stronger closed-source models but had the opposite effect for weaker open-source models, which may amplify errors during reasoning.The results suggest that CoT’s effectiveness depends on the model’s overall capability.Source: §4.3 · p. 9, Table 5 · p. 9
  • With text-only inputs, advanced models reached around 90% accuracy using human annotations, while frame-level and video-level captions performed significantly worse.The annotations supplied logical connections between clues, whereas captions could miss details or provide incorrect interpretations.Source: §4.3 · p. 9, Table 5 · p. 9

5 Conclusion and Discussion · Appendix

Video-Holmes evaluates complex video reasoning by requiring MLLMs to locate and connect clues across suspense films. Its evaluation finds that models often struggle to integrate information despite generally strong visual perception.

5 Conclusion and Discussion

  • Video-Holmes is a benchmark for evaluating MLLMs’ complex video reasoning capabilities.It also analyzes model reasoning processes and factors associated with correct and incorrect answers.Source: §5 · p. 9
  • The benchmark comprises 1,837 questions derived from 270 manually annotated suspense short films.Source: §5 · p. 9
  • Its seven tasks require models to locate and connect relevant visual clues scattered across different video segments.Source: §5 · p. 9
  • The evaluation finds that MLLMs generally excel at visual perception but struggle to integrate information and often miss critical clues.Source: §5 · p. 9
  • The authors propose Video-Holmes as a “Holmes-test” for multimodal reasoning.They hope it will motivate models to reason more like humans and highlight ongoing challenges in the field.Source: §5 · p. 9

A Model Implementation Details … Question Generation Prompt (Continue)

The section specifies model configurations and a structured prompt for generating reasoning questions from detective-film plots. The prompt spans seven reasoning types and requires carefully constrained multiple-choice questions with explanations.

A Model Implementation Details

  • For QwenVL inference, top-p is 0.001, temperature is 0.01, and frame resolution is 256 × 28 × 28 pixels; InternVL images are resized to 448 × 448, and Gemini, GPT, and Claude use official APIs.Official checkpoints are used for QwenVL and InternVL models; GPT image URLs are retrieved with the detail parameter set to low.Source: A Model Implementation Details · p. 10

Question Generation Prompt

  • The prompt asks a detective-story enthusiast to generate reasoning questions from plot chronology, character relationships, reasoning shots, supernatural rules, and the film’s possible main idea.Questions are intended for viewers who have watched the short film.Source: Question Generation Prompt · p. 10

Question Generation Prompt (Continue)

  • Social Reasoning infers character relationships from appearance, interactions, and identity across time; Intention and Motive Chaining infers unspoken mental states or intentions from observable clues.Social questions are based on key relationships, including repeated characters across time spans; the intention question format names a character, time, brief action, and asks for the underlying intention or state.Source: Question Generation Prompt · p. 10, Question Generation Prompt (Continue) · p. 11
  • Temporal Causal Inference connects non-continuous events using audiovisual clues, while Timeline Analysis asks viewers to reorder five key events chronologically.The prompt specifies one to two questions for Temporal Causal Inference and one timeline question.Source: Question Generation Prompt (Continue) · p. 11
  • Multimodal Hint Reasoning probes camera, object, sound, and image clues; Physical Anomaly Reasoning asks about supernatural rules and their implied meaning when such elements occur.The prompt calls for one hint question per supplied reasoning shot and two questions per supernatural element; Physical Anomaly Reasoning is skipped if none are provided.Source: Question Generation Prompt (Continue) · p. 11
  • Core Theme Inference asks viewers to infer a film’s deeper meaning from its plot, dialogue, or symbols, while treating the supplied main idea as potentially subjective.Source: Question Generation Prompt (Continue) · p. 11
  • Each question must provide six options, a uniquely correct answer, plausible overlapping distractors, its type, and an explanation; stems must not reveal the clues or conclusions viewers should infer.The specified format lists the type and question first, then options A–F, the answer, and an explanation.Source: Question Generation Prompt (Continue) · p. 11

Model Evaluation Prompt (Thinking)

The thinking prompt asks the model to answer a single-choice question based on a video, separating its reasoning from its final answer with specified tags.

Model Evaluation Prompt (Thinking)

Model Evaluation Prompt (Without Thinking)

The prompt asks the model to answer a single-choice question based on a video and place its final answer between `<answer>` tags.

Model Evaluation Prompt (Without Thinking)

  • The template supplies placeholders for the question and options, then requests the answer between `<answer>` and `</answer>` tags.It begins, “Based on the given video,” and ends with “Your answer:”.Source: Model Evaluation Prompt (Without… · p. 12

Reasoning Process Analysis Prompt (Thinking Wrong) · Reasoning Process Analysis Prompt (Thinking Right)

These prompts classify video-reasoning responses by comparing the model’s thinking and answer with the film plot and correct explanation. The wrong-answer prompt distinguishes four error types, while the correct-answer prompt distinguishes flawed from explanation-consistent reasoning.

Reasoning Process Analysis Prompt (Thinking Wrong)

  • For incorrect answers, the evaluator compares the model’s thought process with the correct explanation and film plot to identify the critical error cause.The prompt supplies the plot description, question, options, correct answer, and explanation before the model’s thought process and incorrect answer.Source: Reasoning Process Analysis Prompt… · p. 12
  • VPE and VOE classify errors as, respectively, extracting incorrect visual information and failing to extract key visual information.Both categories link the visual error to the wrong answer.Source: Reasoning Process Analysis Prompt… · p. 12
  • RE and TRAW distinguish faulty interpretation of relationships among visual information from choosing a wrong option despite reasoning consistent with the explanation.The prompt requests the error abbreviation in <Type> tags and a brief reason in <Reason> tags.Source: Reasoning Process Analysis Prompt… · p. 12

Reasoning Process Analysis Prompt (Thinking Right)

  • For correct answers, the evaluator compares the model’s thought process with the correct explanation and film plot to classify how it reasoned.The prompt provides the plot, question, options, correct answer and explanation, followed by the model’s thought process and correct answer.Source: Reasoning Process Analysis Prompt… · p. 12
  • TWAR labels a correct answer reached through thought that significantly deviates from the explanation and uses incorrect information.The prompt asks for the type in <Type> tags and a brief judgment reason in <Reason> tags.Source: Reasoning Process Analysis Prompt… · p. 12
  • TRAR labels thought generally consistent with the explanation, allowing minor deviations, when the model’s answer is correct.The evaluator returns either TWAR or TRAR with a brief judgment reason.Source: Reasoning Process Analysis Prompt… · p. 12

C Key Statistics of Video-Holmes · D Examples of Video-Holmes

Section C summarizes Video-Holmes’ film-selection and task distributions, while Section D illustrates human annotations and generated questions, explanations, answers, and reasoning analyses.

C Key Statistics of Video-Holmes

D Examples of Video-Holmes

E Broader Impact

Video-Holmes may support progress in complex video reasoning by providing a rigorous, comprehensive evaluation benchmark. Its suspense-film content may be distressing or inappropriate for some audiences, so responsible use should include content warnings and audience considerations.

  • Video-Holmes may impact complex video reasoning by providing a rigorous and comprehensive evaluation benchmark.Source: E Broader Impact · p. 13
  • Because its suspense short films may contain horror or thriller elements, the benchmark could distress or be inappropriate for some audiences.Source: E Broader Impact · p. 13
  • Researchers and developers should use the benchmark responsibly, with appropriate content warnings and consideration for the intended audience.Source: E Broader Impact · p. 13

Related papers

Loading 2505.21374v1…