PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding

Karttikeya Mangalam, Raiymbek Akshulakov, Jitendra Malik

arXiv:2308.09126v1cs.CVcs.AIcs.CL

TL;DR

Long-form video benchmarks may use long clips without requiring genuinely long-term understanding, motivating a metric that measures intrinsic temporal evidence. EgoSchema introduces temporal certificates and a curated multiple-choice benchmark, finding a large temporal-length gap from prior datasets and low zero-shot model accuracy relative to humans.

  • Problem

    Existing long-form video tasks may disguise short-term tasks as long-term ones, while narrow labels and difficult free-form evaluation limit broad assessment of natural video understanding.

  • Method

    EgoSchema defines temporal certificates as the minimum necessary and sufficient subclips for verifying an annotation and builds a curated benchmark from filtered three-minute Ego4D clips.

  • Results

    EgoSchema’s certificate length is approximately 5.7× longer than the next-longest dataset and 25× to 100× longer than other datasets, while models score below 33% and humans 76%.

  • Takeaways & Limitations

    EgoSchema provides a diagnostic probe for very long-form video-language understanding and exposes a substantial gap between current systems and human performance.

  • Takeaways & Limitations

    EgoSchema inherits egocentric video biases from Ego4D and may contain text-distribution biases and some mislabelled or ill-formed question-answer sets.

Abstract

from arXiv · show

We introduce EgoSchema, a very long-form video question-answering dataset, and benchmark to evaluate long video understanding capabilities of modern vision and language systems. Derived from Ego4D, EgoSchema consists of over 5000 human curated multiple choice question answer pairs, spanning over 250 hours of real video data, covering a very broad range of natural human activity and behavior. For each question, EgoSchema requires the correct answer to be selected between five given options based on a three-minute-long video clip. While some prior works have proposed video datasets with long clip lengths, we posit that merely the length of the video clip does not truly capture the temporal difficulty of the video task that is being considered. To remedy this, we introduce temporal certificate sets, a general notion for capturing the intrinsic temporal understanding length associated with a broad range of video understanding tasks & datasets. Based on this metric, we find EgoSchema to have intrinsic temporal lengths over 5.7x longer than the second closest dataset and 10x to 100x longer than any other video understanding dataset. Further, our evaluation of several current state-of-the-art video and language models shows them to be severely lacking in long-term video understanding capabilities. Even models with several billions of parameters achieve QA accuracy less than 33% (random is 20%) on the EgoSchema multi-choice question answering task, while humans achieve about 76% accuracy. We posit that \name{}{}, with its long intrinsic temporal structures and diverse complexity, would serve as a valuable evaluation probe for developing effective long-term video understanding systems in the future. Data and Zero-shot model evaluation code are open-sourced for both public and commercial use under the Ego4D license at http://egoschema.github.io

1 Introduction

EgoSchema targets very long-form video-language understanding with diverse, manually curated questions and introduces temporal certificate length to measure intrinsic temporal difficulty rather than relying on clip duration alone.

  • Motivation: EgoSchema combines long-form video, natural-language questions, and multiple-choice evaluation to capture diverse human behavior while avoiding difficult-to-score free-form outputs.The benchmark is designed around interconnected spatial, temporal, social, and long-term reasoning abilities.
  • Temporal certificates: Temporal certificate length measures the minimum video evidence needed to convince a human verifier that an annotation is correct.The notion applies beyond question answering to tasks including action classification, detection, and temporal localization.
  • Temporal certificates: Certificate length distinguishes short, long-form, and very long-form tasks at approximately 1, 10, and 100 seconds, respectively.The metric is intentionally decoupled from raw clip duration, separating retrieval of relevant subclips from understanding them.
  • Results: EgoSchema’s median temporal certificate is about 100 seconds, substantially exceeding other video understanding datasets.The introduction reports a 5× gap over the second-longest dataset and 25× to 100× gaps over all other existing datasets.
  • Results: Billion-parameter video-language systems achieve below 33% zero-shot accuracy on EgoSchema, compared with about 76% for humans.Random selection yields 20% accuracy on the five-choice task.
  • Dataset construction: EgoSchema uses a staged pipeline that filters Ego4D videos, generates question-answer triplets with chained LLM prompts, applies automated filtering, and performs two rounds of human curation.Stage width ratios indicate the relative filtering ratios across the pipeline.

2 Related Works

Prior video-language datasets evaluate many specialized capabilities, but long-form video understanding remains sparsely explored and often uses narrow tasks or short clips.

  • Video Question-Answering Datasets: Existing video question-answering datasets target capabilities including compositional, causal, instructional, social, spatiotemporal, and external-knowledge reasoning.Large-scale video-language resources also support pretraining and question generation from narrated or ASR text.
  • Long-form Video Understanding Datasets: Earlier long-form benchmarks used narrow tasks such as like-ratio and view-count prediction, limiting the breadth of behavior assessed.The cited benchmark’s average certificate length is about 5.7× smaller than EgoSchema’s.
  • Long-form Video Understanding Datasets: Other prior datasets address efficient inference or event-boundary detection without long-term annotations or with clips only 10 seconds long.These settings therefore provide shorter temporal certificates than EgoSchema.
  • Dataset construction: The paper positions its prompt-generation and filtering examples as part of a pipeline intended to construct richer long-form video question-answering data.The full prompt versions are provided in the supplementary material.

3 Collecting EgoSchema

EgoSchema is curated from filtered Ego4D clips through staged LLM generation, automated filtering, and human quality control. The pipeline targets diverse, visually grounded questions requiring substantial temporal evidence.

  • Stage I: Raw Data Filtering: Three-minute Ego4D clips with at least 30 timestamped human narrations form the raw material for EgoSchema.The filtering standardizes clip length while retaining narration-rich videos for question-answer construction.
  • Stage II: Question Answer Generation: LLMs generate multiple question-answer triplets per clip, each containing one correct answer and four wrong answers.The selected configuration uses N = 3 questions and M = 4 wrong answers per question.
  • Stage II: Question Answer Generation: Q(AW)-shot is chosen because it preserves answer diversity while using two chained calls, making generation 30% cheaper than QAW-shot.The procedure jointly generates questions, then generates correct and incorrect answers together.
  • Stage II: Question Answer Generation: Generation quality depends strongly on prompting and LLM choice, with good-quality QAW found only often enough using GPT-4, Bard, and Claude.Weaker LLMs tend to produce narrow question diversity and obviously wrong or overly similar negatives.
  • Stages III–IV: Filtering and Curation: Rule-based and LLM-based filters remove malformed, ungrounded, or logically flawed questions before manual curation.The blind baseline tests whether the correct answer can be guessed without video narrations, while rule-based checks discard prompt leakage and parsing failures.
  • Temporal Certificates: A temporal certificate is the minimum set of subclips sufficient to convince a human that an annotation is correct without watching the rest of the video.Certificate length is the summed duration of those subclips.

4 Benchmarking EgoSchema

EgoSchema benchmarks zero-shot multiple-choice video question answering under conventional and maximum-frame inference settings, alongside human evaluations. The results show strong dependence on model and frame sampling, while humans retain high accuracy even under constrained viewing.

  • Certificate-Length Evaluation: EgoSchema certificate lengths are compared across 15 datasets spanning action classification, detection, relationship classification, concept classification, video classification, and question answering.EgoSchema estimates use five hours of randomly selected video, while other datasets receive coverage of labels and at least two hours of human effort.
  • Certificate-Length Evaluation: 5.7× longer than the second-longest dataset and 10× to 100× longer than all other video understanding datasets is EgoSchema’s temporal certificate length.The comparison is made by plotting certificate length against actual clip length.
  • Zero-shot Model Evaluation: State-of-the-art video-language models are evaluated zero-shot using both their training-frame setting and the maximum frame count executable on an 80G A100.Frames are sampled uniformly from each input clip in both settings.
  • Human Evaluation: 67.0% accuracy is achieved by humans in under one minute, increasing only 1.0% when the time limit reaches three minutes.With 1 fps video, humans achieve 67.2% accuracy.
  • Human Evaluation: 76.2% accuracy is achieved when humans watch the video before reading the question, versus 75.0% with no viewing-order constraint.The no-constraint estimate covers nine hours of video, while the other human estimates cover five hours.

5 Conclusion

EgoSchema measures very long-form video-language understanding with temporal certificates and exposes a substantial gap between current models and humans. The benchmark is intended to support future model development and evaluation, while inheriting data and curation limitations.

  • EgoSchema has temporal certificate lengths approximately 5.7× longer than the next longest dataset and 25× to 100× longer than all others.
  • Zero-shot models achieve less than 33% question-answering accuracy on EgoSchema, whereas humans achieve 76%.
  • EgoSchema is designed as a diagnostic benchmark for developing and evaluating future very long-form video-language models.
  • EgoSchema inherits egocentric video biases, text-distribution biases from LLM-generated outputs, and some possible mislabelled or ill-formed question-answer sets.

EgoSchema Datasheet

EgoSchema is a zero-shot benchmark of 3-minute videos paired with questions and five answer options, built from selectively filtered Ego4D clips. It is publicly released under the Ego4D license and retains inherited privacy, representation, and curation considerations.

  • EgoSchema introduces temporal certificate sets to measure intrinsic temporal understanding length rather than relying on video clip length alone.
  • Each instance represents a 3-minute video with one question and five answer options, with one option marked correct.
  • The dataset contains 5063 instances, each with one video, one question, and five answer options.
  • The video subset uses non-overlapping three-minute Ego4D segments containing at least 30 human-annotated timestamped narrations.
  • EgoSchema is designed specifically for zero-shot testing of out-of-the-box long-term video-language understanding.
  • EgoSchema is publicly released under the Ego4D license for research and commercial use, with videos and text available through project resources.
  • The dataset includes inherited Ego4D privacy protections, including informed consent and de-identification procedures such as obscuring personally identifiable information.
  • EgoSchema may contain egocentric video biases, LLM-related text distribution biases, and residual errors despite manual curation.

Full Prompts

The prompts construct difficult multiple-choice questions from timestamped action narrations while testing abstraction over long video segments. They emphasize concise conclusions, high-level reasoning, and closely related distractors rather than timestamp recall.

  • 5.1.1 Question prompt: The question prompt asks for three difficult, diverse questions testing long-range summarization, information compression, and identification of important video content.
  • 5.1.1 Question prompt: Questions should target high-level video details without mentioning particular timestamps or narrations, and answers should compress rather than list actions.
  • 5.1.1 Question prompt: Example questions ask about overarching relationships, primary activities, and how changing actions, interactions, and spatial movement shape the narrative.
  • 5.1.2 Answer prompt: The answer prompt generates one correct answer and four closely related wrong answers, with correct answers presented as concise conclusions.
  • 5.1.2 Answer prompt: Illustrative answers distinguish casual, friendly interactions and a relaxed guitar-playing environment from conflict, urgency, or criminal scenarios.
  • 5.1.2 Answer prompt: The answer examples also classify changing actions and interactions as reflecting a comfortable environment rather than conflict or professional problem-solving.
  • 5.2.1 Question and answer prompt: The alternate question-and-answer prompt retains the same goals: high-level questions, long-part comparison, information compression, and focus on important content.
  • 5.2.1 Question and answer prompt: The prompts explicitly avoid when, how many, and how much questions in favor of difficult questions about main actions and reasons.

6 Our clip length and narration density choice

The dataset balances clip duration against narration density when selecting standardized evaluation clips. It chooses three-minute clips with ten narrations per minute as a compromise between temporal extent and sufficient narration.

  • The heatmap compares viable clips across clip length and narration density, revealing only a few balanced choices.
  • Three-minute clips with five narrations per minute risk insufficient narration for high-quality question-answer generation, while one-minute clips with 20 narrations per minute are too brief.
  • The dataset selects three-minute clips with a narration density of 10 narrations per minute as a satisfactory balance.

7 Human curation

EgoSchema was human-curated through trained annotation procedures that screened question–answer sets for answerability, correctness, temporal length, and question quality.

  • Curation process: Annotation used trained third-party personnel and separate procedures for data curation and human accuracy testing.Curators were responsible for assessing the quality of released question–answer sets.
  • Quality criteria: A set was good only if the question was answerable from the video and required more than a brief glimpse.Questions answerable from absent content or fewer than roughly one second of video failed this condition.
  • Quality criteria: The marked correct answer had to be correct, while all four marked wrong answers had to be less correct than it.If even one marked answer was also correct, the set was labeled bad.
  • Quality criteria: Questions needed an intrinsic temporal certificate exceeding 30 seconds, including the summed duration of non-contiguous sub-certificates.Questions answerable from only a few frames failed this long-term condition.
  • Quality criteria: Frequency questions were excluded, and any failed condition caused the entire question–answer set to be marked bad.Annotators could label near-good cases as MAYBE for later revision; the expected share was no more than 3%.
  • Edge cases: Curators also handled ambiguous or duplicate cases by accepting qualifying multiple-answer questions, retaining only one of similar good questions, and documenting edge cases.The expected proportion of good questions was roughly 20–25%, varying by three-minute clip.

8 Benchmarking details

Benchmarking evaluated several video-language models and human annotators under different frame, timing, and video-to-text access conditions.

  • Model benchmarking: Violet was evaluated using one pretrained model and three models fine-tuned on LSMDC-MC, MSRVTT-QA, and MSRVTT-retrieval.Each model was tested with both 5 and 75 frames, and the model with the best overall accuracy was selected.
  • Model benchmarking: mPLUG-Owl was adapted to multiple-choice video QA by selecting the answer option with the highest Yes-token confidence.Ties among top-scoring options were broken randomly, and the model’s multi-frame ability was described as emergent.
  • Task format: EgoSchema was formulated as zero-shot multiple choice because open-ended video QA has restricted answer vocabulary.InternVideo supports both open-ended video QA and zero-shot multiple-choice formats.
  • Human benchmarking: Human benchmarking used ten employees from the annotation company, with randomized answer choices presented as a test.The protocol varied time limits, frame rate, and whether participants could revisit the video after reading the text.
Loading 2308.09126v1…