PaperlayerView exact revision on arXiv

Source-linked AI summary

HourVideo: 1-Hour Video-Language Understanding

Keshigeyan Chandrasegaran, Agrim Gupta, Lea M. Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, Li Fei-Fei

arXiv:2411.04998v1cs.CVcs.AIcs.LG

TL;DR

Short-video benchmarks offer limited evaluation of long-form video-language understanding, where questions must require cross-segment comprehension. HourVideo pairs a varied task suite with curated egocentric videos and human- and LLM-assisted question creation; multimodal models remain far below human experts on its benchmark.

  • Problem

    Common multimodal benchmarks focus on images or short clips, while long-form evaluation needs questions requiring long-range dependencies rather than brief-segment answers or language-model prior knowledge.

    Source: §1 · p. 1

  • Method

    HourVideo combines multiple long-form comprehension tasks with curated Ego4D videos and a question-generation pipeline using human annotators and LLMs.

    Source: §1 · pp. 1–2, §2.2 · pp. 4–5

  • Results

    Human experts reach 85.0% accuracy versus Gemini 1.5 Pro’s 37.3%, while GPT-4V and LLaVA-NeXT are only marginally above the 20% random predictor.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark exposes a substantial gap between current multimodal models and human experts in long-form video understanding.

    Source: §1 · p. 2, §5 · p. 10

    The benchmark may contain inconsistent questions, uses a limited range of video sources, and does not yet support audio.

    Source: §5 · p. 10

Abstract

from arXiv · show

We present HourVideo, a benchmark dataset for hour-long video-language understanding. Our dataset consists of a novel task suite comprising summarization, perception (recall, tracking), visual reasoning (spatial, temporal, predictive, causal, counterfactual), and navigation (room-to-room, object retrieval) tasks. HourVideo includes 500 manually curated egocentric videos from the Ego4D dataset, spanning durations of 20 to 120 minutes, and features 12,976 high-quality, five-way multiple-choice questions. Benchmarking results reveal that multimodal models, including GPT-4 and LLaVA-NeXT, achieve marginal improvements over random chance. In stark contrast, human experts significantly outperform the state-of-the-art long-context multimodal model, Gemini Pro 1.5 (85.0% vs. 37.3%), highlighting a substantial gap in multimodal capabilities. Our benchmark, evaluation toolkit, prompts, and documentation are available at https://hourvideo.stanford.edu

1 Introduction

HourVideo addresses the limits of short-clip benchmarks by evaluating long-form video-language understanding with a diverse task suite and curated egocentric videos. Models remain well below human experts on the benchmark.

1 Introduction

  • Existing multimodal benchmarks commonly use single images or short clips, usually lasting seconds to no more than three minutes.The paper argues that longer-form video-language understanding merits deeper evaluation as a basis for future autonomous agents and assistants.Source: §1 · p. 1
  • Long-form questions must require long-range dependencies, rather than being answerable from a brief segment or from language-model prior knowledge.The paper identifies both short-segment answerability and questions trivially resolved by prior knowledge as benchmark-design challenges.Source: §1 · p. 1
  • HourVideo combines summarization, perception, visual reasoning, and navigation tasks with 500 curated Ego4D videos and 12,976 five-way questions.The videos span 20 to 120 minutes and 77 everyday activities; question creation used more than 800 hours of human effort alongside LLMs.Source: §1 · p. 2
  • 85.0% human-expert accuracy substantially exceeds Gemini 1.5 Pro’s 37.3%, while GPT-4V and LLaVA-NeXT achieve only marginal gains over random prediction.GPT-4V and LLaVA-NeXT score 25.7% and 22.3%, respectively, against a 20% random predictor.Source: §1 · p. 2

2 Benchmark Design and Construction

HourVideo uses a broad task suite and a human-and-LLM question-generation pipeline to build a benchmark from long egocentric videos. Its 500 videos yield 12,976 five-way questions across everyday scenarios.

2.1 Task Suite

  • The 18-subtask suite spans summarization, perception, visual reasoning, and navigation, using handcrafted prototypes to test long-form understanding.The benchmark design combines focused question prototypes with a diverse task suite; open-ended evaluation is avoided because automated scoring of free-form answers is challenging.Source: §2.1 · p. 2, §2 · p. 2, Table 1 · p. 4

2.2 Dataset Generation Pipeline

  • Experts selected 500 of 1,470 reviewed Ego4D videos, then task-specific templates and structured 20-minute video summaries support LLM candidate-question generation.Ego4D was chosen for its egocentric perspective, visual narrations, and accessibility under its license.Source: §2.2 · pp. 4–5
  • Annotators check question validity, answer accuracy, and distractors before LLM refinement; blind LLMs then filter questions answerable without video.Human-feedback refinement used over 400 hours of annotator effort; blind filtering removes an item if either of two LLMs answers it correctly without video input.Source: §2.2 · p. 5

2.3 HourVideo Statistics

  • 12,976 questions accompany 500 Ego4D videos spanning 77 daily-life scenarios; videos average 45.7 minutes, and 113 exceed one hour.The dataset totals 381 hours of footage, averaging 26 questions per video; questions are distributed across tasks except for manually generated causal, counterfactual, and navigation items.Source: Figure 3 · p. 6, §2.3 · p. 6

2.2 Dataset Generation Pipeline

  • Expert review refines remaining questions, while causal, counterfactual, spatial-layout, and navigation questions are manually generated when the pipeline is unsuitable.Expert refinement involved over 300 hours; manual generation produced 658 questions, or 5.1% of the benchmark.Source: §2.2 · p. 6

3 Experiments

The experiments compare task-batched and independent evaluation, several multimodal baselines, and human performance on HourVideo. Gemini 1.5 Pro leads the model baselines, but remains well below human experts.

3.1 Evaluation Protocol

  • Questions are evaluated in task or sub-task batches to reduce cross-question information leakage; predictive tasks use timestamps to trim videos.The benchmark reports accuracy by task and in aggregate.Source: §3.1 · p. 7

3.2 Baselines

  • Baselines span video-blind GPT-4, caption-aggregating Socratic models, native Gemini processing, and sampled expert evaluation.Socratic models aggregate captions from one-minute video segments; native models process the long video directly.Source: §3.2 · p. 7

3.3 Results

  • Gemini 1.5 Pro reaches 37.3% versus human experts’ 85.0%, and outperforms blind and Socratic models across all tasks (14 of 18 sub-tasks).Human performance was evaluated on a sample of 14 videos and 213 questions.Source: §3.2 · p. 7, Table 2 · p. 8, §3.3 · p. 9
  • Blind LLMs score 19.6% overall; GPT-4 Socratic reaches 41.1% on summarization versus blind GPT-4 at 24.4% and LLaVA-NeXT-captioned Socratic at 34.6%.Random guessing corresponds to 20% accuracy.Source: §3.3 · pp. 7–8
  • Independent MCQ scoring caused a 2.1% performance drop while increasing evaluation costs by more than threefold, supporting task/sub-task batching.The comparison used Gemini 1.5 Pro on 570 questions across 25 videos.Source: §3.3 · p. 9, Table 3 · p. 9

4 Related Work

Related video benchmarks often focus on specific domains or short clips, while HourVideo targets long-form understanding. Prior work has advanced video-language capabilities and long-context methods, but long-form understanding remains challenging.

Dataset Comparison

  • Many existing benchmarks use short videos or narrow domains; HourVideo instead averages 45.7 minutes per video, compared with EgoSchema’s 3 minutes.The paper describes HourVideo tasks as requiring long-term comprehension.Source: §4 · p. 9

Video Understanding Tasks

  • Video-language research spans visual perception and reasoning benchmarks, while long-form methods use memory banks, token compression, caption aggregation, or temporal localization.The paper notes that long-form video understanding remains a significant challenge for current multimodal models.Source: §4 · pp. 9–10
  • Many questions in prior datasets can be answered from a single frame; HourVideo adds challenging long-duration reasoning and navigation questions.The benchmark is designed to assess limits in current models’ comprehension of long-form videos.Source: §4 · p. 10

5 Conclusion

HourVideo introduces a benchmark for long-form video-language understanding, where model performance remains far below human experts. The authors note limits in question consistency, video-source diversity, and sensory coverage.

5 Conclusion

  • Human experts reach 85.0% accuracy, compared with 37.3% for Gemini 1.5 Pro; GPT-4V and LLaVA-NeXT perform only marginally above random guessing.The benchmark contains 12,976 five-way questions across 500 egocentric videos lasting 20 to 120 minutes.Source: §5 · p. 10
  • HourVideo evaluates one-hour video understanding through summarization, perception, visual reasoning, and navigation tasks.Its 500 Ego4D videos span 20 to 120 minutes and support 12,976 high-quality five-way multiple-choice questions.Source: §5 · p. 10
  • The authors identify possible inconsistencies in multiple-choice questions and call for more diverse video sources, audio, and other sensory modalities.They specifically name sports and YouTube videos as examples of sources for more holistic benchmarks.Source: §5 · p. 10

HourVideo Supplementary Material

The supplementary material is organized into release information, data-generation details, evaluation guidance, additional experiments, and broader impact.

HourVideo Supplementary Material

A HourVideo Release v1.0

HourVideo v1.0 is released as a benchmark dataset with supporting materials for loading, inspecting, and evaluating its questions. Users obtain the corresponding Ego4D videos subject to that dataset’s license agreement.

A HourVideo Release v1.0

  • The release provides a JSON benchmark file, metadata, five-choice questions, predictive-task timestamps, and a PyTorch dataloader.Users can download the 500 corresponding videos from Ego4D after reviewing and accepting its license agreement.Source: A HourVideo Release v1.0 · p. 17
  • The release organizes 12,976 questions in HourVideo_v1_0.json and provides navigation and spatial-layout task images.Ground-truth annotations are not public; CSV annotations are supplied for two sample videos, and per-video CSVs improve accessibility.Source: A HourVideo Release v1.0 · p. 17
  • The release includes video-processing and dataloader scripts, baseline prompts and code, documentation, and a stated MIT license for HourVideo.Ego4D videos remain subject to the Ego4D License, and the authors say they will announce updates and new versions publicly.Source: A HourVideo Release v1.0 · p. 17

B Data Generation Pipeline: Additional details

The supplementary pipeline details describe prompt design, narration compilation, and human refinement. The process combines structured video-segment descriptions with substantial annotation and expert effort.

B.1 Prompt Design

  • The team designed 25 prompts and varied question, answer, and distractor generation by task, using manual generation for causal, counterfactual, spatial-layout, and navigation tasks.Prompts also cover narration compilation and answer paraphrasing for selected tasks.Source: §B.1 · p. 18

B.2 Narration Compilation Details

  • Videos are segmented at 20-minute intervals into structured descriptions, then compiled by an LLM; the narration set with more tokens is selected.The authors report that their empirical observation linked more narration tokens with more comprehensive visual-element coverage; Figure B.1 plots visual coverage against narration-token count using object collections and Tiktoken token counts.Source: §B.2 · p. 18, Figure B.1 · p. 18

B.3 Human Feedback and Expert Refinement Details

  • Seven trained annotators spent more than 400 hours on feedback-based MCQ refinement, and four experts spent over 250 hours refining questions, answers, and wrong answers.The team conducted continuous quality assessments during the feedback stage.Source: §B.3 · p. 18

C Evaluation details

HourVideo evaluates models independently at the task and sub-task level. Its structured assessment is intended to reduce information leakage across questions and the costs of evaluating each MCQ.

C Evaluation details

  • Tasks and sub-tasks are assessed independently under the proposed evaluation protocol.The protocol identifies which tasks and sub-tasks require independent evaluation.Source: C Evaluation details · p. 18
  • The structured assessment minimizes information leakage across questions and the costs of individual MCQ evaluation.Individual evaluation costs scale with the number of questions.Source: C Evaluation details · p. 18
  • Expert refinement can filter MCQs, ground questions and answers, and refine correct and incorrect answers to increase challenge.Source: Figure B.2 · p. 19

D Additional Experiments

Additional experiments evaluate Tarsier-7B on HourVideo using 16 frames sampled uniformly across each video. Its performance is comparable to Socratic LLMs, and the paper notes that Ego4D appeared in its video-captioning pretraining.

D.1 Additional Baselines

  • The evaluation uses publicly available Tarsier-7B with 16 frames uniformly sampled from the entire video, following Tarsier's long-video setup.Source: §D.1 · p. 19
  • Baseline experiments use a generic task-agnostic prompt, while advanced prompting strategies are left for future work.Source: §D.1 · p. 19
  • The evaluation protocol independently assesses the tasks and sub-tasks identified in the table.Source: Table C.1 · p. 20
  • Tarsier-7B with 16 frames performs comparably to Socratic LLMs on HourVideo.Ego4D was used in Tarsier pretraining for video captioning.Source: Table D.1 · p. 20

D.2 Model Refusal Rates

The authors report refusal rates for proprietary models and note that Gemini 1.5 Pro refuses more often than GPT-4. For Socratic models, both named models caption more than 96% of one-minute segments.

D.2 Model Refusal Rates

  • GPT-4 and LLaVA-34B-DPO caption more than 96% of one-minute segments as Socratic models.Source: §D.2 · p. 20
  • Gemini 1.5 Pro has a significantly higher refusal rate than GPT-4.Source: Table D.2 · p. 20, §D.2 · p. 20
  • Refusal rates are reported for 500 videos and 12,976 MCQs; for Socratic LLMs, the rates cover question answering.Source: Table D.2 · p. 20

E Broader Impact

HourVideo may advance AI video understanding, with potential applications in embodied AI, world models, and AR/VR. The authors also emphasize privacy and ethical risks, calling for responsible development and safeguards.

E Broader Impact

  • The authors describe long-form video understanding as potentially supporting applications including embodied AI, robotics, world models, and AR/VR.Source: E Broader Impact · p. 21
  • Long-form video understanding could support embodied agents' navigation and interaction, including multi-stage tasks and planning based on extended observations.The paper gives home and industrial robots as examples.Source: E Broader Impact · p. 21
  • Processing extended video sequences could help world models represent event flow, object changes, and interactions over time.The paper connects this temporally informed knowledge with reasoning, prediction, planning, and generalization.Source: E Broader Impact · p. 21
  • Long-form video understanding could enable AR systems to adapt to users' visual context and make VR experiences more responsive to user actions.Source: E Broader Impact · p. 21
  • Long-running egocentric video can reveal personal details, raising privacy concerns about data handling, transparency, and unauthorized surveillance.Source: E Broader Impact · p. 21
  • HourVideo uses research-licensed videos and focuses on high-level semantic understanding rather than invasive personal-information extraction.Source: E Broader Impact · p. 21
  • The authors frame HourVideo as a step forward for AI video understanding, while stressing responsible development and societal values.Source: E Broader Impact · p. 21
Loading 2411.04998v1…