PaperlayerView exact revision on arXiv

Source-linked AI summary

CinematicVQA: Benchmarking Film-Grammar Reasoning in Large Vision-Language Models

Shuo Xing, Pooja Verlani, Balu Adsumilli, Zhengzhong Tu

arXiv:2609.28813v1cs.CV

TL;DR

Existing video benchmarks largely test cinematic technique identification rather than how techniques shape visual presentation and narrative. CinematicVQA uses Cinematic Scene Graphs to evaluate these links, finding a gap between describing appearances and identifying techniques; fine-tuning improves higher-level interpretation.

  • Problem

    Existing cinematic benchmarks primarily assess low-level technique identification rather than techniques’ perceptual effects and narrative functions.

    Source: §I · p. 1

  • Method

    CinematicVQA uses Cinematic Scene Graphs to connect filming techniques with visual presentation and narrative effects for film-grammar evaluation.

    Source: §I · p. 1, A. Cinematic Scene Graph · p. 2

  • Results

    Models score 10–20% higher on Visual Presentation than Technique Recognition; Chain-of-Thought prompting degrades accuracy for most models, while fine-tuning improves higher-level interpretation.

    Source: B. Benchmark with CINEMATICVQA-EVAL · p. 4, D. Finetuning with CINEMATICVQA-TRAIN · p. 5

  • Takeaways & Limitations

    CinematicVQA serves as both a benchmark for cinematic evaluation and a training resource for film-aware video models.

    Source: §I · p. 1

    The benchmark focuses on composition, lighting, and camera operation and currently relies on expert annotations from VADB.

    Source: §V · p. 6

Abstract

from arXiv · show

Cinematography, the craft of visual storytelling through framing, lighting, and camera operation, fundamentally shapes how audiences perceive and emotionally engage with video content. While Large Vision Language Models (LVLMs) have made remarkable progress in video question answering, existing benchmarks primarily focus on identifying low-level techniques rather than understanding their storytelling impact. To address this, we introduce CinematicVQA, the first-of-its-kind benchmark for cinematic video understanding that goes beyond technique recognition to evaluate film-grammar reasoning, utilizing our introduced Cinematic Scene Graph (CSG), a structured representation that links filming techniques to their perceptual effects and narrative functions. Through comprehensive evaluation of state-of-the-art LVLMs, we reveal a striking semantic gap: models consistently perform higher on describing visual presentations than on identifying the underlying techniques. Surprisingly, Chain-of-Thought prompting fails to provide consistent gains and degrades performance for most models, suggesting that current LVLMs lack sufficient cinematic domain knowledge to benefit from step-by-step reasoning. Fine-tuning on \textsc{CinematicVQA-train} yields consistent improvements, particularly for narrative function and multi-hop reasoning. Overall, \textsc{CinematicVQA} serves both as a rigorous benchmark for cinematic evaluation in LVLMs and as a practical dataset for training more film-aware video models.

I. INTRODUCTION

Existing cinematic benchmarks emphasize identifying techniques rather than understanding their perceptual and narrative effects. CinematicVQA uses expert annotations and Cinematic Scene Graphs to evaluate these connections, and fine-tuning on its training set improves cinematic reasoning.

I. INTRODUCTION

  • Existing benchmarks primarily test technique identification, leaving the storytelling effects and narrative functions of shot construction less evaluated.The authors frame these effects as central to the value of cinematographic techniques.Source: §I · p. 1
  • CinematicVQA evaluates how models connect composition, lighting, and camera-operation choices to perceptual effects and storytelling implications, beyond recognizing techniques.The benchmark probes technique-relevant visual evidence, its perceptual effects, and plausible narrative intent conditioned on those effects.Source: §I · p. 1
  • Cinematic Scene Graphs structure expert annotations by linking filming techniques with visual presentation and narrative effects through typed relations.The graph decomposes clips into technique, presentation, and narrative nodes, with relations such as technique → presentation → narrative.Source: §I · p. 1
  • Fine-tuning on CINEMATICVQA-TRAIN yields consistent gains in cinematic reasoning, positioning the benchmark as both an evaluation and training resource.The authors report these gains in their evaluation of representative LVLMs on CINEMATICVQA-EVAL.Source: §I · p. 1

II. RELATED WORK

Related work describes the LVLM paradigm and reviews cinematic benchmarks that largely evaluate low-level technique recognition. CinematicVQA is presented as extending evaluation to visual aesthetics and storytelling.

II. RELATED WORK

  • LVLMs typically combine a vision encoder and language model through a projector, feeding projected image tokens alongside text embeddings to the language model.The passage presents this as the foundational paradigm for LVLMs.Source: §II · p. 2
  • Existing cinematic benchmarks focus mainly on low-level technique recognition, while higher-level aesthetic and storytelling effects remain unevaluated.CameraBench covers camera motion, CineTechBench general cinematographic techniques, and ShotBench and RefineShot broaden or refine evaluation dimensions.Source: §II · p. 2
  • CinematicVQA introduces a multi-level evaluation protocol that extends beyond atomic recognition to higher-level cinematic understanding.The comparison is identified as Table I, whose caption describes a comparison with existing LVLM VQA benchmarks.Source: §II · p. 2, Table I · p. 2

III. CINEMATICVQA

CinematicVQA evaluates cinematic understanding from technique recognition through perceptual and narrative consequences. Its Cinematic Scene Graph represents these levels as typed nodes and directed relations, with a constrained technique-to-presentation-to-narrative path.

III. CINEMATICVQA

  • CinematicVQA is designed to evaluate LVLM understanding from filming-technique recognition through perceptual and narrative consequences.The benchmark is built on VADB human-written aesthetic comments to construct CSGs and generate training and evaluation sets.Source: §III · p. 2

A. Cinematic Scene Graph

  • A clip’s CSG separates technique nodes, presentation-effect nodes, and narrative-function nodes into three semantic types.Technique nodes are restricted to composition, lighting, or camera operation; presentation effects describe perception or appearance, while narrative nodes capture abstract narrative or emotional functions.Source: A. Cinematic Scene Graph · p. 2
  • CSG edges are directed and typed relations connecting nodes, with labels including enhances, reinforces, causes, limits, supports, and results_in.Each edge is represented as a triple linking two nodes with a relation type.Source: A. Cinematic Scene Graph · p. 2
  • Structural constraints require each technique node to link to a presentation effect and each presentation effect to a narrative function, forming technique → perception → narrative paths.These paths represent the stated chain from technique to perception to narrative.Source: A. Cinematic Scene Graph · p. 3

B. Data Curation

The curation pipeline builds Cinematic Scene Graphs from human-written VADB comments, then uses structured graph paths to create questions and answers across cinematic reasoning dimensions. The resulting training and evaluation sets are split at approximately 9:1.

B. Data Curation

  • The pipeline constructs each clip’s CSG from human-written comments in the Video Aesthetic Description Benchmark.The passage points to Figure 2 for the pipeline illustration.Source: B. Data Curation · p. 3
  • Input captions retain sentence indices, and GPT-4o mini extracts evidence-grounded technique, presentation-effect, narrative-function, and relation components.Extraction covers composition, lighting, and camera operation.Source: B. Data Curation · p. 3
  • The pipeline samples one technique-to-narrative path per category and uses the target reasoning hop to form each question, with the next path node as its answer.The questions cover technique recognition, visual presentation, narrative function, and multi-hop reasoning.Source: B. Data Curation · p. 3
  • The approximately 9:1 split yields 42,748 training VQA pairs and 4,596 evaluation pairs.LLMs also refine question-answer quality and generate plausible multiple-choice distractors.Source: B. Data Curation · p. 3

C. Human Annotations on Data Quality

Three experts assessed generated Cinematic Scene Graphs and evaluation distractors using rubrics where higher scores indicated better annotation quality. The CSGs scored 4.36 on average, and the distractors scored 4.07.

C. Human Annotations on Data Quality

  • Generated CSGs averaged 4.36, while CINEMATICVQA-EVAL multiple-choice distractors averaged 4.07 on expert annotation-quality ratings.Three experts independently scored 100 samples each using 1–5 rubrics, with higher scores indicating better quality.Source: C. Human Annotations on Data Quality · p. 3, Figure 3 · p. 3

IV. EXPERIMENT

The experiments assess current LVLM cinematic reasoning and whether training on CINEMATICVQA-TRAIN improves it. Evaluations cover zero-shot and Chain-of-Thought prompting, alongside supervised fine-tuning.

IV. EXPERIMENT

  • The evaluation compares zero-shot and Chain-of-Thought prompting on CINEMATICVQA-EVAL and uses supervised fine-tuning on CINEMATICVQA-TRAIN to assess training utility and transfer.The benchmark spans Technique Recognition, Visual Presentation, Narrative Function, and Multi-Hop Reasoning.Source: §IV · p. 3, Table II · p. 4
  • The experiments ask how well LVLMs reason about cinematic techniques and their visual–narrative functions, and how much CINEMATICVQA-TRAIN improves these abilities.The target abilities include technique recognition, visual-effect attribution, and narrative-function inference.Source: §IV · p. 4

A. Experimental Setup

The study evaluates state-of-the-art LVLMs on CINEMATICVQA-EVAL under zero-shot and Chain-of-Thought prompting, then reports that technique recognition trails visual description and CoT often fails to help.

A. Experimental Setup

  • The evaluation uses identical prompt templates, generation budgets of 100 tokens for zero-shot and 512 for CoT, and LoRA fine-tuning for one epoch.Fine-tuning uses rank 128 and alpha 256; models use BF16 mixed precision.Source: A. Experimental Setup · p. 4
  • Table II reports evaluated-model performance on CINEMATICVQA-EVAL across technique categories and cinematic-understanding dimensions, without specifying the individual dimensions in the cited passage.Evaluated models included InternVL-3.5 (4B and 8B) [13], Qwen2.5-VL-Instruct (3B and 7B) [24], and Qwen3-VL-Instruct (4B, 8B, and 32B) [12], under zero-shot and CoT prompting.Source: B. Benchmark with CINEMATICVQA-EVAL · p. 4, A. Experimental Setup · p. 4

B. Benchmark with CINEMATICVQA-EVAL

  • InternVL-3.5-8B has the strongest overall average among evaluated models under both prompting settings; GPT-4o mini scores 40.69 and Gemini-2.5-Flash scores 41.32.The two closed-source models perform at a similar level to the open-source models.Source: B. Benchmark with CINEMATICVQA-EVAL · p. 4
  • CoT prompting fails to deliver consistent gains and degrades accuracy for most models compared with zero-shot inference.The authors attribute this pattern to insufficient cinematic-domain grounding for step-by-step reasoning without drifting from visual evidence.Source: B. Benchmark with CINEMATICVQA-EVAL · p. 4
  • Models score 10–20% higher on Visual Presentation than Technique Recognition, with Qwen3-VL-8B-it scoring 60.45% versus 29.12% on composition under zero-shot prompting.The authors identify this perception–terminology gap as a central challenge in cinematic understanding.Source: B. Benchmark with CINEMATICVQA-EVAL · p. 4

C. Does Chain-of-Thought Help on CinematicVQA?

Chain-of-Thought prompting brings no overall mean gain across seven open-source models and can alter many answers. In right-to-wrong cases, models often favor their own generated descriptions over the video.

C. Does Chain-of-Thought Help on CinematicVQA?

  • CoT yields no overall mean gain across seven models, degrading Qwen2.5 most sharply and InternVL mildly while Qwen3 results stay within noise.On Qwen3-VL-8B, CoT changes 34.6% of predictions; 472 correct answers become incorrect and 460 incorrect answers become correct, with no significant difference (χ2 1 = 0.13, p = 0.72).Source: C. Does Chain-of-Thought Help on… · p. 5, Table III · p. 5
  • In right-to-wrong cases, models commit to a free-form description, then choose the option that best matches their own words rather than the video.The paper summarizes five recurring patterns of this failure mechanism.Source: C. Does Chain-of-Thought Help on… · p. 5, Table IV · p. 5

D. Finetuning with CINEMATICVQA-TRAIN

Fine-tuning on CINEMATICVQA-TRAIN consistently improves overall accuracy, with stronger gains on semantic reasoning than on technique recognition. Larger fine-tuned backbones tend to perform better overall and gain more on semantics-heavy questions.

D. Finetuning with CINEMATICVQA-TRAIN

  • The study fine-tunes InternVL-3.5 (4B), Qwen2.5-VL-Instruct (3B and 7B), and Qwen3-VL-Instruct (4B, 8B, and 32B), then evaluates zero-shot on CINEMATICVQA-EVAL.Figure 4 compares each model’s base and fine-tuned accuracy.Source: D. Finetuning with CINEMATICVQA-TRAIN · p. 5, Figure 4 · p. 5
  • SFT raises average accuracy from 42.87 to 47.20 for Qwen3-VL-4B-it and from 42.83 to 49.81 for Qwen3-VL-8B-it.Across question types, gains are strongest for Narrative Function and Multi-Hop; Technique Recognition improves less and can slightly regress.Source: D. Finetuning with CINEMATICVQA-TRAIN · p. 5, Figure 4 · p. 5
  • Larger fine-tuned backbones tend to achieve stronger overall performance and larger gains on semantics-heavy questions.The authors suggest they have greater capacity to absorb structured supervision and generalize to unseen videos.Source: D. Finetuning with CINEMATICVQA-TRAIN · p. 5

V. CONCLUSION AND FUTURE WORK

CinematicVQA evaluates whether LVLMs can connect cinematographic choices to visual presentation and narrative intent, using CSG to represent these relations. The authors report that models struggle with underlying techniques, while fine-tuning improves higher-level interpretation; they also outline extensions beyond the current scope.

V. CONCLUSION AND FUTURE WORK

  • CinematicVQA and its Cinematic Scene Graph represent relations among techniques, visual presentation, and narrative impact, beyond isolated technique recognition.The conclusion reports that models often describe visual patterns but struggle to identify underlying techniques; fine-tuning improves narrative-function and multi-hop reasoning, while technique recognition remains challenging.Source: §V · p. 5
  • The current benchmark focuses on Composition, Lighting, and Camera Operation, relies on expert annotations from VADB, and does not explicitly model soundtracks or sound effects.The authors propose broader data sources and subjective studies, along with extensions to events, characters, scene context, audio, dialogue, and scripts.Source: §V · p. 6
  • The authors identify CSG as a potential foundation for reinforcement learning and reward design, and as an interpretable evaluation framework or reward signal for cinematic video generation.Source: §V · p. 6
Loading 2609.28813v1…