ALLVB: All-in-One Long Video Understanding Benchmark

Introduction

• Prior long-video benchmarks use manual QA annotation, which limits scalability and results in shorter videos and fewer Q&As per video.The authors also identify a lack of a unified standard for designing video-comprehension questions.Source: Introduction · p. 1
• Because current LLMs do not natively accept video input, ALLVB uses related text descriptions, especially movie scripts, to represent hour-long video content.Scripts are segmented into plots and sub-plots before GPT-4o generates overarching and detailed Q&As.Source: Introduction · p. 1
• ALLVB integrates nine video-understanding tasks through 91 question templates, bringing multiple capabilities into a single video-QA framework.The templates assess capabilities including summarization, information extraction, and temporal reasoning.Source: Introduction · p. 2
• Its GPT-4o pipeline uses external text information and rigorous quality control, and is designed for scalability and easier maintenance.The pipeline checks plot continuity and question-answer format, followed by a three-stage manual review.Source: Introduction · p. 2
• ALLVB contains 1,376 videos averaging nearly two hours, with 252k Q&As—nearly 2× the second-longest benchmark’s average video length and 11.5× its QA count.The paper describes it as the largest long-video understanding benchmark on these dimensions.Source: Introduction · p. 2

Multi-model LLMs

• Long-video models such as MovieChat and LLaMA-VID had not demonstrated proven performance on hour-long videos, motivating reliable evaluation.The passage describes MLLMs as needing expanded inferential capabilities for long-video comprehension.Source: Multi-model LLMs · p. 2

Video Understanding Benchmarks

• Video-QA benchmarks span short and long videos, but manual annotation and the lack of a unified QA-design standard motivate task-based template generation.The authors aim to make generated Q&As more objective and aligned by basing templates on established video-understanding tasks.Source: Video Understanding Benchmarks · p. 3

Benchmark Construction Pipeline

• The dataset begins with scripts; TV scripts are temporarily excluded, and filtering yields 1,376 text-formatted movie scripts and corresponding movies.The authors initially scrape 2,520 scripts, then filter out unsuitable files and scripts with incomplete or problematic text.Source: Benchmark Construction Pipeline · p. 3
• GPT-4o segments scripts in two stages, dividing plots into finer-grained sub-plots to cover details and support question construction.Single-pass segmentation produced plots spanning considerable durations.Source: Benchmark Construction Pipeline · p. 3
• Nine established video-understanding tasks expand into 91 sub-tasks and templates, covering recognition, reasoning, summarization, and related capabilities.Needle-in-a-Haystack questions target details in specific frames to assess long-context retrieval without inserting unrelated content.Source: Benchmark Construction Pipeline · pp. 3–4

Benchmark Statistics

• Most videos run 90–150 minutes; most have 5–20 sub-plots and 100–250 Q&As, and the collection spans 16 genres.The paper also notes that multiple-choice format supports objective evaluation and subtitles enable questions drawing on more than video content alone.Source: Figure 2 · p. 4, Benchmark Statistics · p. 5

Benchmark Construction Pipeline

• Questions target the whole video, each sub-plot, or evenly divided needle segments; for a movie with n sub-plots, the specified count is 21 + 14n.Needle segments are independent of the script-derived sub-plots.Source: Benchmark Construction Pipeline · p. 4
• Each QA is multiple choice, with five options, one correct answer, and four distractors, enabling accuracy scoring without open-ended subjective judgment.Options are constrained to similar lengths, and distractors must relate to the video content.Source: Benchmark Construction Pipeline · p. 4
• Quality control checks plot boundaries and coverage, flags invalid or incomplete questions for regeneration, and includes three-stage manual review.Authors and 12 volunteers also verify discrepancies between scripts and movies, especially for object and action recognition questions.Source: Benchmark Construction Pipeline · pp. 4–5

Benchmark Statistics

• ALLVB contains 1,376 movies and 252,420 Q&As, and leads long-video benchmarks in video count, average duration, and QA quantity.Its 1,376 movies are divided into 15,966 sub-plots, averaging 11.6 per movie; the benchmark averages 183 questions per movie.Source: Benchmark Statistics · p. 5, Table 1 · p. 5

Implementation Details

• Models receive 16 uniformly sampled frames and corresponding subtitles in zero-shot testing; open-source models answer each video’s Q&As individually, while closed-source models answer them together.The test set contains 140 videos, and the train/test split is 1,236/140.Source: Implementation Details · pp. 5–6

Results and Analysis

• Claude 3.5 Sonnet leads all nine tasks, GPT-4o ranks second throughout, and Claude reaches 75.2% accuracy.Table 2 reports accuracy across nine task types and average accuracy.Source: Table 2 · p. 6, Results and Analysis · pp. 6–7

Implementation Details

• GPT-4o required two answer-matching regex rules, compared with 34 for mPlug-Owl-V and 26 for VideoChat.The authors report that closed-source models needed significantly fewer regex rules than open-source models.Source: Implementation Details · p. 6

Results and Analysis

• VC, VCap, and VER score higher, while ODT, TAL, and NH are harder; NH scores are lower across models because it requires very fine-grained frame information.The authors contrast tasks needing less interaction with detailed visuals against those requiring fine visual detail or logical interactions.Source: Results and Analysis · pp. 6–7
• Open-source video MLLMs show no significant advantage over image MLLMs under the same 7B parameter size and input conditions.The authors suggest the limited number of input frames could partly explain this result.Source: Results and Analysis · p. 7

Further Discussion

• All models tend to lose accuracy as video length increases, while GPT-4o remains stable from 105–115 to 130–140 minutes.With a fixed 16-frame input, longer videos mean more information is missed; the authors say the overall drop is not particularly significant because ALLVB videos are generally long.Source: Further Discussion · p. 7, Table 3 · p. 7
• Average accuracy improves for all models with more input frames, but 64 frames still miss substantial information in hour-long videos.The authors also note that models have not fully utilized longer-context input.Source: Further Discussion · p. 7, Table 4 · p. 7

Conclusion

• ALLVB combines nine major video-understanding tasks with a GPT-4o automated long-video annotation pipeline.The authors describe it as having the largest number of videos, longest average duration, and most Q&As among long-video understanding benchmarks.Source: Conclusion · p. 7
• The authors report that existing MLLMs still have significant room to improve in long-video understanding.They hope ALLVB can serve as an objective evaluation metric for future MLLMs.Source: Conclusion · p. 7
• ALLVB currently uses movie-sourced videos and relatively simple Q&As; the authors plan to add other video types and specialized content.Examples proposed include sports, documentaries, instructional videos, and course videos for assessing professional knowledge.Source: Conclusion · p. 7

A.1 Motivation

• The dataset targets the gap left by video QA benchmarks that mainly cover videos under five minutes, offering a more comprehensive and challenging long-video resource.The paper’s authors and recruited volunteers built the dataset; their organization’s names are withheld for anonymity.Source: §A.1 · p. 10

A.2 Composition

• 1,376 movies span 2,628 hours and yield 252,420 Q&As, with films selected for quality and variety across 16 common genres.Each movie averages 114 minutes and 183 five-option Q&As, with one correct answer; the data is split 9:1 into training and test sets. Similar questions may recur across different subplots.Source: §A.2 · p. 10

A.3 Collection Process

• Movies and subtitles were collected from open-source websites using web crawlers, and GPT-4 automatically generated Q&As that collaborators and volunteers manually verified.The collection took place from June to August 2024; the team did not hire crowd workers.Source: §A.3 · p. 11

A.4 Preprocessing/cleaning/labeling

• Movies and scripts were screened for quality and to exclude sensitive or harmful material; raw data was not retained, and cleaning scripts were not publicly released.The Python scripts follow rules described in the paper’s construction-pipeline section and can be provided on request.Source: §A.4 · pp. 11–12

A.5 Uses

• The dataset is intended primarily for multimodal video QA and can assess MLLMs across long-video understanding tasks, including classification, scene recognition, action recognition, and needle-in-a-haystack questions.As of August 2024, only the original paper had used the dataset.Source: §A.5 · p. 12

A.6 Distribution

• The authors planned public distribution as JSON on HuggingFace under CC-BY-NC-SA-4.0, with no DOI.The datasheet stated that release was planned for 2024.Source: §A.6 · p. 12

Maintenance Who will be supporting/hosting/maintaining the dataset?

• The authors plan to maintain the dataset by adding instances and correcting errors, with significant updates communicated on HuggingFace.They also stated that they would host it there; updates to the current version would be communicated through the platform.Source: Maintenance · pp. 12–13
• The authors do not currently plan to let third parties modify the dataset, citing consistency and fairness.Source: Maintenance · p. 13

B Further discussion on GPT-4o

• The authors report that models were evaluated zero-shot on movie videos, not scripts, after Q&A quality-control revisions.The annotations were initially based on movie scripts, but models had no access to the scripts during testing; multiple verification rounds and revisions changed Q&As that did not meet standards. The authors say zero-shot testing prevented data leakage and use of training data.Source: B Further discussion on GPT-4o · p. 13

C More Q&A Examples

• The appendix presents examples of sub-tasks from all nine main tasks, with correct options marked in bold green.Source: C More Q&A Examples · p. 14

C.1 Sub-tasks for Video Classification (VC)

• Figure 3 presents examples of the Video Classification (VC) sub-tasks.Source: Figure 3 · p. 14
• The figure covers 11 sub-tasks.Source: Figure 3 · p. 14
• The examples are for Video Classification (VC).Source: Figure 3 · p. 14

C.2 Sub-tasks for Scene Recognition (SR)

• Figure 4 presents examples of the Scene Recognition (SR) sub-tasks.Source: Figure 4 · p. 15
• The figure covers 12 sub-tasks.Source: Figure 4 · p. 15
• The examples are for Scene Recognition (SR).Source: Figure 4 · p. 15

C.3 Sub-tasks for Object Detection and Tracking (ODT)

• Figure 5 presents examples of the Object Detection and Tracking (ODT) sub-tasks.Source: Figure 5 · p. 16
• The figure covers 10 sub-tasks.Source: Figure 5 · p. 16
• The examples are for Object Detection and Tracking (ODT).Source: Figure 5 · p. 16

• Figure 6 illustrates examples of the 11 Action Recognition sub-tasks.Source: Figure 6 · p. 17

C.5 Sub-tasks for Temporal Action Localization (TAL)

• The TAL task is illustrated with examples of 7 sub-tasks.Source: Figure 7 · p. 18

C.6 Sub-tasks for Event Detection (ED)

• The ED task is illustrated with examples of 12 sub-tasks.Source: Figure 8 · p. 19

C.7 Sub-tasks for Video Captioning (VCap)

• The VCap task is illustrated with examples of 8 sub-tasks.Source: Figure 9 · p. 20

C.8 Sub-tasks for Video Emotion Recognition (VER)

• The VER task is illustrated with examples of 10 sub-tasks.Source: Figure 10 · p. 21

C.9 Sub-tasks for Needle-in-a-Haystack (NH)

• The NH task is illustrated with examples of 10 sub-tasks.Source: Figure 11 · p. 22

D.1 Plot Segmentation Prompt

• GPT-4o is prompted to segment a script into plots and subplots, then generate 11 questions about the full script’s content and classification tasks.The prompt asks for plot and subplot page ranges and summaries, and specifies 11 question types spanning video classification and narrative understanding.Source: §D.1 · p. 23

D.2 Q&A Construction Prompt (For sub-plots)

• For sub-plots, GPT-4o is prompted to create 14 questions from randomly selected sub-tasks using distinct, evenly distributed scenes.Each question uses five options with one correct answer and includes scene context; the correct option must require information from the video.Source: §D.2 · p. 25

C.5 Sub-tasks for Temporal Action Localization (TAL)

• Action frequency questions require counting occurrences and identifying the observed segment’s start and end points.The prompt also frames frequency as counting a specified action, such as jumping.Source: §D.2 · p. 26
• Action sequence and simultaneity questions test event ordering and detection of actions that occur at the same time.Examples include ordering walking, sitting, and standing up, or recognizing talking and gesturing together.Source: §D.2 · p. 27
• Other prompts ask where actions occur and how a specific action varies in form or style.Examples include identifying locations for dancing or eating and distinguishing different dance moves.Source: §D.2 · p. 27
• The prompts also cover character interaction through actions and the context or setting in which actions occur.Examples include handshakes, hugs, or arguments, and cooking in a kitchen or playing in a park.Source: §D.2 · p. 27

Question Type 34: Action Start and End Recognition (TAL)

• Questions identify the landmark scenes marking an action’s start and end, such as a character entering a room or a car driving away.Question Type 34 asks for specific scenes or moments that indicate both boundaries.Source: §D.2 · p. 27

Question Types 35–36: Initial Trigger Recognition and Action Termination Indicator (TAL)

• Questions distinguish the event that triggers an action from the scene or event that signals its termination.Examples of triggers include a signal, dialogue, or environmental change; termination indicators include leaving the frame, a closing door, or changed lighting.Source: §D.2 · p. 27

Question Types 37–38: Pre-Action Context Analysis and Post-Action Consequence Analysis (TAL)

• Questions examine the events and context preceding an action and its immediate consequences after it ends.They ask what conditions lead to initiation and what immediate impact or outcomes follow.Source: §D.2 · p. 27

Question Types 39–40: Character Motivation and Immediate Reactions (TAL)

• Questions ask why a character performs an action and how other characters respond immediately afterward.Motivations may be explained using dialogue, interactions, or the situation leading up to the action.Source: §D.2 · p. 27

D.2 Q&A Construction Prompt (For sub-plots)

• The prompts ask about a character’s primary emotion, changes through a scene and its resolution, including events that cause emotional shifts.Source: §D.2 · p. 29
• They also probe predominant group emotions and interactions, emotional responses during conflicts, and contrasting reactions between two characters.Source: §D.2 · p. 29
• Other prompts infer underlying feelings from subtle cues and analyze how word choice, tone, and pacing in dialogue reflect emotion.Source: §D.2 · p. 29
• The prompts also examine how setting influences or reflects emotion and how facial expressions, body language, and other non-verbal behaviors convey it.Source: §D.2 · p. 29

D.3 Q&A Construction Prompt (For the Needle-in-a-Haystack task)

• Needle-in-a-haystack questions use five options, with similar-length answers, plausible but distinct distractors, video-dependent correct answers, randomized answer positions, and varied question types.Source: §D.3 · pp. 29–30
• Answers must give the correct option letter and reasoning; the example asks for a pencil’s color amid detailed scene clues and identifies yellow as correct.Source: §D.3 · p. 30

E.1 Accuracy for Video Classification (VC)

• Figure 12 reports accuracy across Video Classification sub-tasks.Source: Figure 12 · p. 31

E.2 Accuracy for Scene Recognition (SR)

• Figure 13 reports accuracy across Scene Recognition sub-tasks.Source: Figure 13 · p. 32

E.3 Accuracy for Object Detection and Tracking (ODT)

• Figure 14 reports accuracy across Object Detection and Tracking sub-tasks.Source: Figure 14 · p. 33

E.4 Accuracy for Action Recognition (AR)

• Figure 15 reports accuracy across Action Recognition sub-tasks.Source: Figure 15 · p. 34

E.5 Accuracy for Temporal Action Localization (TAL)

• Figure 16 reports accuracy across Temporal Action Localization sub-tasks.Source: Figure 16 · p. 35

E.6 Accuracy for Event Detection (ED)

• Figure 17 reports accuracy across Event Detection sub-tasks.Source: Figure 17 · p. 36

E.7 Accuracy for Video Captioning (VCap)

• Figure 18 reports accuracy across Video Captioning sub-tasks.Source: Figure 18 · p. 37

E.8 Accuracy for Video Emotion Recognition (VER)

• Figure 19 reports accuracy across Video Emotion Recognition sub-tasks.Source: Figure 19 · p. 38