PaperlayerView exact revision on arXiv

Source-linked AI summary

CinePile: A Long Video Question Answering Dataset and Benchmark

Ruchit Rawal, Khalid Saifullah, Miquel Farré, Ronen Basri, David Jacobs, Gowthami Somepalli, Tom Goldstein

arXiv:2405.08813v3cs.CVcs.LGcs.MM

TL;DR

Long-video understanding requires temporal and multimodal comprehension, while existing datasets may emphasize questions answerable from isolated frames. CinePile builds a large, diverse benchmark from human-written audio descriptions, and reports that models trail humans while fine-tuning improves performance.

  • Problem

    Existing video-understanding datasets may overemphasize visual-only or classification questions that do not require temporal understanding.

    Source: §1 · p. 2

  • Method

    CinePile uses transcribed, human-written audio descriptions aligned with movie clips to generate and filter diverse questions with language models.

    Source: §2.4 · p. 6

  • Results

    Humans exceed the best commercial vision/omni models by approximately 25% and open-source video understanding models by 37%, while fine-tuned Video-LLaVa improves from 25.72% to 44.16% accuracy.

    Source: §1 · p. 2, §4 · p. 10

  • Takeaways & Limitations

    CinePile’s scale supports both instruction tuning and evaluation, and its question-type breakdown helps developers identify model blind spots.

    Source: §1 · p. 2

    CinePile does not yet incorporate character grounding in time, and a motivated human given sufficient time can create more challenging questions than those generated by an LLM.

    Source: §6 · p. 11

Abstract

from arXiv · show

Current datasets for long-form video understanding often fall short of providing genuine long-form comprehension challenges, as many tasks derived from these datasets can be successfully tackled by analyzing just one or a few random frames from a video. To address this issue, we present a novel dataset and benchmark, CinePile, specifically designed for authentic long-form video understanding. This paper details our innovative approach for creating a question-answer dataset, utilizing advanced LLMs with human-in-the-loop and building upon human-generated raw data. Our comprehensive dataset comprises 305,000 multiple-choice questions (MCQs), covering various visual and multimodal aspects, including temporal comprehension, understanding human-object interactions, and reasoning about events or actions within a scene. Additionally, we fine-tuned open-source Video-LLMs on the training split and evaluated both open-source and proprietary video-centric LLMs on the test split of our dataset. The findings indicate that although current models underperform compared to humans, fine-tuning these models can lead to significant improvements in their performance.

1 INTRODUCTION

CinePile targets long-video understanding that requires temporal and multimodal comprehension, rather than questions answerable from isolated frames. Its large, diverse dataset is designed for both model training and evaluation, using questions generated from human-written scene descriptions.

1 INTRODUCTION

  • CinePile contains approximately 305,000 question-answer pairs from 9,396 videos, with question diversity and difficulty intended to support long-video understanding.The authors report that humans exceed the best commercial vision/omni models by approximately 25% and open-source video understanding models by 37%.Source: §1 · p. 2
  • Example questions require tracking a character’s emotional transition across a scene or locating objects and reasoning about their purpose.The latter combines spatial and temporal localization with world knowledge.Source: §1 · p. 2
  • Unlike datasets focused on visual-only or classification questions, CinePile covers vision, temporal understanding, and narrative reasoning, with question-type breakdowns to identify model blind spots.Its scale is intended to make it useful both as an instruction-tuning dataset and an evaluation benchmark.Source: §1 · p. 2
  • CinePile questions are generated from transcribed audio descriptions aligned with movie clips, while evaluated video models answer without access to those descriptions.The descriptions provide detailed human-written scene information for LLM-generated questions.Source: §1 · p. 2

2 CREATING A LONG VIDEO UNDERSTANDING BENCHMARK

CinePile’s four-stage curation process combines human-written source material with LLM-generated questions and refinement checks. It aligns film clips with human audio descriptions, derives reusable question templates, generates scene-specific MCQs, and filters or repairs weak items.

2.1 DATA COLLECTION AND CONSOLIDATION

  • MovieClips clips were aligned with AudioVault descriptions, then BERT classified transcript sentences as visual description or dialogue, reaching 96% evaluation accuracy.Clip localization used transcripts and sentence embeddings to find the clip within the full-movie description and extract the corresponding text.Source: §2.1 · pp. 2–3

2.2 AUTOMATED QUESTION TEMPLATES

  • About 30,000 human-curated questions were clustered and used to prompt GPT-4; manual review produced 86 templates grouped into five categories.The authors used 10 random questions from each cluster; they observed these yielded more general, higher-quality templates than the closest-to-center examples.Source: §2.2 · pp. 4–5

2.3 AUTOMATED QA GENERATION WITH LLMS

  • Scene-specific templates and annotated text yield approximately 32 questions per video; Gemini handles perceptual questions, while GPT-4 generates reasoning questions.Prototypical questions help prevent hallucination and improve distractors; rationales improve question quality, and timestamps improve temporal questions.Source: §2.3 · p. 5

2.4 DATASET QUALITY EVALUATION AND ADVERSARIAL REFINEMENT

  • 90.94% of weak training Q&As and 90.24% of weak test Q&As were corrected; unfixable items were removed from evaluation but retained in training.The process used up to five refinement attempts; about 80 unfixable evaluation Q&As were excluded, while about 4,500 remained in training.Source: §2.4 · p. 6
  • Vision reliance tests whether Gemini can answer from dialogue alone; hardness tests whether it can answer with the subtitles and visual descriptions used to create the questions.Authors reviewed questions flagged as hard and checked the evaluation split across iterations for systemic pipeline errors.Source: §2.4 · pp. 6–7

3 A LOOK AT THE DATASET

CinePile contains 9396 movie clips averaging about 160 seconds, with 298,887 training questions and 4,941 test questions. Its question categories vary in visual reliance and difficulty.

3 A LOOK AT THE DATASET

  • 298,887 training points and 4,941 test points cover 9396 clips averaging ∼160 seconds, with around 32 questions per scene.Each MCQ has one correct answer and four distractors; correct-answer positions are randomized, and degenerate questions are filtered from the test split but retained in training.Source: §3 · p. 7
  • Question diversity comes from varied movie clips and prompting strategies, with “Character Relationship Dynamics” the largest category.“Setting and Technical Analysis” is the next most common category; the authors attribute the prevalence of relationship questions to the many templates derived from human-written questions.Source: §3 · p. 7
  • “Temporal” questions are the hardest category, while “Setting and Technical Analysis” depends most on visual elements.“Thematic Exploration” follows “Temporal” in hardness; “Character Relationship Dynamics” and “Temporal” follow “Setting and Technical Analysis” in visual reliance.Source: §3 · pp. 7–8

4 MODEL EVALUATION

Across evaluated models, Gemini 1.5 Pro leads overall and LLaVA-OV leads open-source models, while performance falls on CinePile’s hard split. Fine-tuning Video-LLaVa on CinePile’s training data improves its accuracy substantially.

4 MODEL EVALUATION

  • The evaluation covers 24 commercial and open-source LLMs, with human performance included for comparison.The authors distinguish careful human viewing, including rewatching when needed, from the model evaluation.Source: §4 · pp. 8–9
  • Gemini 1.5 Pro leads commercial models overall, while LLaVA-OV leads open-source models at 49.34% accuracy.Gemini 1.5 Flash reaches 58.75% overall accuracy and ranks second; GPT-4 models are competitive on Narrative and Plot Analysis, while Gemini performs particularly well on visually reliant Setting and Technical Analysis.Source: Table 2 · p. 8, §4 · p. 9
  • Most top models lose 15%–20% on hard questions, but their relative rankings remain unchanged.Gemini 1.5 Flash declines by ≈21%, compared with 13% for Gemini 1.5 Pro.Source: §4 · p. 9, Figure 8 · p. 10
  • Alternative response-matching strategies improve some open-source models’ scores, but they still trail the best open-source models, indicating video-understanding limits beyond instruction following.Embedding similarity and GPT-4 judging largely align with each other and with the original response-extraction rankings.Source: §4 · pp. 9–10, Figure 7 · p. 9
  • Fine-tuning Video-LLaVa on CinePile improves accuracy by approximately 71%, from 25.72% to 44.16%, with gains across all question subcategories.The experiment uses 4-bit quantization, a frozen base model, LoRA, five epochs, and AdamW.Source: §4 · p. 10

5 RELATED WORK

Prior video datasets vary in video length, modality, and the kinds of understanding they test. CinePile is presented as a scalable, multimodal dataset with longer videos and diverse perceptual, temporal, and reasoning questions.

5 RELATED WORK

  • Existing benchmarks cover different gaps: MovieQA focuses on whole-film plots but has few vision-focused questions, while EgoSchema emphasizes egocentric perceptual knowledge.LVU centers on year, genre, or like-ratio prediction; Perception Test targets core perception in short videos.Source: §5 · p. 10
  • MAD supplies dialogue and visual descriptions for scene captioning, while TVQA (Lei et al., 2018) and MoVQA (Zhang et al., 2023b) provide video question-answering at different scales.MAD does not provide raw visual data in the cited discussion, and CMD is a text-to-video retrieval benchmark rather than a QA dataset.Source: §5 · pp. 10–11
  • Many recent long-video benchmarks contain only a few thousand Q&A pairs because human question generation becomes costly as video length grows; CinePile has 70–75× more.The passage names Video-MME (Fu et al., 2024), MVBench, and LVBench (Wang et al., 2024) among these benchmarks.Source: §5 · p. 11
  • CinePile combines longer videos and many questions covering perceptual, temporal, and reasoning aspects, with questions often requiring both video and dialogue.The authors contrast its automated generation from human-generated questions with earlier fixed-template approaches and describe future extensions to videos and question categories.Source: §5 · p. 11

6 DISCUSSION AND CONCLUSION

The conclusion presents CinePile as a long-video dataset and benchmark, and notes remaining gaps in character grounding and question difficulty. Its appendices describe related work, additional question-generation details, and template categories.

6 DISCUSSION AND CONCLUSION

  • CinePile contains ∼300k training questions and ∼5000 test questions; the authors report scalable, cost-effective curation, model benchmarking, and a human study.They describe it as the only large-scale dataset focused on multimodal understanding, and say fine-tuning experiments demonstrate training-split quality.Source: §6 · p. 11
  • The dataset still lacks character grounding in time, and motivated humans given sufficient time can create more challenging questions than current LLM-generated ones.The authors say their goal is to narrow the human-question difficulty gap in future iterations.Source: §6 · p. 11

B ADDITIONAL RELATED WORK

  • The related-work discussion connects human-in-the-loop synthetic data to instruction-dataset generation and presents automated template generation and question synthesis for vision and video as an early application.It draws on Self-Instruct, UltraChat, and Evol-Instruct as examples of synthetic language-data approaches.Source: B ADDITIONAL RELATED WORK · p. 16

C ADDITIONAL QA GENERATION DETAILS

  • For additional long-form questions, the pipeline prompts a model using a scene’s visual summary; summary-generated MCQs are restricted to training because some summaries hallucinate.The visual summary is built from shot-selected frames described by Gemini, then condensed by Gemini’s text model.Source: C ADDITIONAL QA GENERATION DETAILS · p. 16

D QUESTION TEMPLATE CATEGORY DETAILS

  • The template categories cover character and relationship dynamics, plot and narrative, themes, setting and technical analysis, and temporal understanding.The categories span character actions and relationships, event sequences, themes, spatial/object details, and action counts or ordering.Source: D QUESTION TEMPLATE CATEGORY DETAILS · pp. 16–17

E QA GENERATION BY DIFFERENT MODELS

GPT-4 is reported to produce high-quality questions across categories, while Gemini performs well only in selected categories. The model pairings also differ in degenerate-question rate and cost efficiency.

E QA GENERATION BY DIFFERENT MODELS

  • GPT-4 consistently produces high-quality questions across categories, whereas Gemini works well only for Character Relationships and Interpersonal Dynamics and Setting and Technical Analysis.The authors attribute the quality gap to differences in captured concepts and Gemini’s hallucination tendencies.Source: E QA GENERATION BY DIFFERENT MODELS · p. 17
  • Across the evaluated question-generation and template-selection model choices, GPT-4 paired with GPT-4 produced the fewest degenerate questions; Gemini paired with GPT-4 also performed well and was cost-efficient at large scale.Source: E QA GENERATION BY DIFFERENT MODELS · p. 17
  • Examples show Gemini misreading dialogue as conflict resolution and choosing an incorrect answer after misremembering temporal order.The examples compare questions generated for two movie clips.Source: E QA GENERATION BY DIFFERENT MODELS · p. 17

F TRAIN DATA STATISTICS

The supplied text identifies a figure showing the train split’s question-category composition.

F TRAIN DATA STATISTICS

G ADDITIONAL EVALUATION DETAILS

The additional evaluation details describe model versions, evaluation coverage, and hardware, alongside a comparison of question-generation pairings.

G ADDITIONAL EVALUATION DETAILS

  • Gemini 1.5 Pro and Flash (001 versions), evaluated from May 20 to June 1 and on June 28, 2024, were assessed on approximately 4.2k of 4,941 samples because safety-blocking exceptions affected a few videos.Source: G ADDITIONAL EVALUATION DETAILS · p. 18
  • Open-source model experiments used two NVIDIA A40 GPUs with 48GB each and two NVIDIA A100 GPUs with 82GB each.The passage also distinguishes proprietary closed-source model licenses from the various open-source licenses used by other models.Source: G ADDITIONAL EVALUATION DETAILS · p. 18
  • The GPT-4/GPT-4 pairing performs best for generating better questions, while template selection has minimal effect.The table caption frames the comparison in terms of lower numbers of degenerate questions.Source: Table 6 · p. 19

H ADDITIONAL EVALUATION STRATEGIES

The appendix evaluates model responses with semantic-similarity metrics and examines cases where raw responses complicate accuracy scoring. Results from four open-source models align with the accuracy trend reported in the main paper.

H ADDITIONAL EVALUATION STRATEGIES

  • Accuracy is computed by extracting a model’s selected option from its raw response and mapping it to a predefined answer choice.Prompt tuning and post-processing reduce, but do not eliminate, errors from verbose or instruction-ignoring responses.Source: H ADDITIONAL EVALUATION STRATEGIES · p. 19
  • Table 7 analyzes raw open-source-model responses, evaluation-pipeline scores, and corresponding failure modes for one Area 51 scene.The supplied table caption identifies the scope but does not report the table’s specific findings.Source: Table 7 · p. 20
  • BERTScore measures contextual embedding similarity, CIDEr compares response n-grams with the single reference answer, and ROUGE-L uses longest-common-subsequence similarity.The appendix uses these captioning metrics to assess semantic similarity rather than exact-string matching.Source: H ADDITIONAL EVALUATION STRATEGIES · p. 20
  • Four open-source models’ BERTScore, CIDEr, and ROUGE-L results align with the main paper’s accuracy trend.The authors say these findings support the reliability of their response normalization and post-processing steps.Source: H ADDITIONAL EVALUATION STRATEGIES · p. 20, Table 8 · p. 20

I HUMAN STUDY DETAILS

A small human study assessed question quality, and the authors audited errors and removed ambiguous test questions. A second study found the final test questions answerable but difficult; the appendix also defines and illustrates questions answerable without video context.

I HUMAN STUDY DETAILS

  • The initial study involved 25 graduate-student volunteers, each answering ten randomly sampled questions about two clips after giving informed consent.The survey collected no personally identifiable information; the dataset uses English movies produced in the United States.Source: I HUMAN STUDY DETAILS · p. 21, Figure 10 · p. 21
  • Some questions missed by humans or GPT-4V were described as difficult and answerable only through careful attention to the video.Figure 12 presents examples of such questions.Source: Figure 12 · p. 22
  • Authors audited questions humans missed; they removed test questions with ambiguous distractors after finding that two choices could seem plausible.Other reported errors involved difficulty attending to an entire clip, understanding dialogue or cultural references, and carelessness.Source: Figure 11 · p. 21, I HUMAN STUDY DETAILS · p. 22
  • Human accuracy on the final test set was 73%, while the authors’ independently measured accuracy was 86%.The authors conclude that many questions are answerable but difficult, and attribute most observed wrong answers to human error or confusion over scene events.Source: I HUMAN STUDY DETAILS · p. 22

J EXAMPLE DEGENERATE QUESTIONS

  • A question is labeled degenerate when models can select its correct option from the question and choices alone, without video context.Examples include common-sense answers and questions potentially answered through memorized movie scripts.Source: J EXAMPLE DEGENERATE QUESTIONS · p. 22, Table 9 · p. 23

K.1 FRAME RATE ABLATION

A subtitles-only ablation tests the contribution of visual frames, while the appendix describes the question-generation prompt’s categories, inputs, and constraints. Example figures pair movie frames and subtitles with questions and templates.

K.1 FRAME RATE ABLATION

  • With subtitles only, performance significantly declined for every model except Video-ChatGPT; the effect was stronger for commercial models.The authors note that stronger overall models also appear to use visual information more effectively.Source: §K.1 · p. 23, Table 10 · p. 24

L QA GENERATION PROMPT

  • The prompt organizes film-analysis questions into five subcategories: character analysis, narrative understanding, scene setting, temporal, and theme.It asks for evenly distributed questions across subcategories and concise wording that does not reveal the answer.Source: L QA GENERATION PROMPT · pp. 23, 25
  • The prompt input distinguishes subtitle dialogue from visual descriptions, which cover character presence, attributes, and thematic elements.The subtitle field is to identify character presence, not actions or dialogue content.Source: L QA GENERATION PROMPT · p. 25
  • The prompt requires five concise answer options, a separate answer key, and a rationale supported by verbatim subtitle or visual-description evidence.Questions should avoid answer hints, and incorrect options should plausibly match the correct answer’s length and form.Source: L QA GENERATION PROMPT · p. 25
  • The appendix illustrates movie frames, accompanying subtitles, questions, and template labels across examples from multiple films.The examples identify TEMP as Temporal and direct readers to Table 3 for other category acronyms.Source: Figure 14 · p. 26, Figure 15 · p. 27, Figure 16 · p. 28

Related papers

Loading 2405.08813v3…