PaperlayerView exact revision on arXiv

Source-linked AI summary

LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding

Haoning Wu, Dongxu Li, Bei Chen, Junnan Li

arXiv:2407.15754v1cs.CVcs.CLcs.LG

TL;DR

LongVideoBench addresses the lack of public benchmarks for long-context multimodal video understanding by testing retrieval and reasoning over hour-long subtitled videos. Its referring-reasoning task challenges proprietary and open-source models, with performance improving as models process more frames while both remain challenged.

  • Problem

    Public benchmarks for long-context multimodal inputs remain limited, while existing video benchmarks show single-frame bias and under-test contextual relational reasoning.

  • Method

    LongVideoBench uses referring-reasoning questions over diverse hour-long subtitled videos, requiring models to identify referred contexts and reason about their multimodal details.

  • Results

    Evaluations show significant challenges for proprietary and open-source models, while GPT-4o and Gemini-1.5-Pro improve more than 10% on videos longer than 180 seconds when increasing input from 16 to 256 frames.

  • Takeaways & Limitations

    LongVideoBench provides a challenging benchmark for assessing long-context multimodal video understanding and informing future exploration of multimodal models.

Abstract

from arXiv · show

Large multimodal models (LMMs) are processing increasingly longer and richer inputs. Albeit the progress, few public benchmark is available to measure such development. To mitigate this gap, we introduce LongVideoBench, a question-answering benchmark that features video-language interleaved inputs up to an hour long. Our benchmark includes 3,763 varying-length web-collected videos with their subtitles across diverse themes, designed to comprehensively evaluate LMMs on long-term multimodal understanding. To achieve this, we interpret the primary challenge as to accurately retrieve and reason over detailed multimodal information from long inputs. As such, we formulate a novel video question-answering task termed referring reasoning. Specifically, as part of the question, it contains a referring query that references related video contexts, called referred context. The model is then required to reason over relevant video details from the referred context. Following the paradigm of referring reasoning, we curate 6,678 human-annotated multiple-choice questions in 17 fine-grained categories, establishing one of the most comprehensive benchmarks for long-form video understanding. Evaluations suggest that the LongVideoBench presents significant challenges even for the most advanced proprietary models (e.g. GPT-4o, Gemini-1.5-Pro, GPT-4-Turbo), while their open-source counterparts show an even larger performance gap. In addition, our results indicate that model performance on the benchmark improves only when they are capable of processing more frames, positioning LongVideoBench as a valuable benchmark for evaluating future-generation long-context LMMs.

1 Introduction

LongVideoBench addresses the lack of long-context multimodal benchmarks by evaluating LMMs on hour-long subtitled videos. It centers on referring reasoning, which requires retrieving referred video contexts and answering multimodal questions about them.

  • Motivation: Long-context multimodal benchmarks remain scarce despite foundation models expanding from 2K-token contexts to millions of tokens.Existing benchmarks predominantly focus on text-only inputs.
  • Benchmark motivation: LongVideoBench evaluates LMMs on hour-long subtitled videos and shows consistent performance improvements when models process more frames.This contrasts with findings from previous benchmarks.
  • Referring reasoning: Referring reasoning first presents a query identifying referred video contexts, then asks a question requiring multimodal understanding of those contexts.Questions test capabilities including visual perception and relational reasoning.
  • Dataset: 6,678 multiple-choice questions span 3,763 diverse-theme videos across durations from 8–15 seconds to 15–60 minutes, with original or transcribed subtitles.The benchmark covers movies, news, life, and knowledge videos.
  • Question categories: The benchmark includes perception questions about individual scenes and relation questions requiring associations across scenes, organized into 17 fine-grained categories.Relation questions address temporal ordering, attribute change, and referred-object tracking.

2 The Referring Reasoning Task

LongVideoBench defines referring reasoning to test two central challenges in long-video understanding: retrieving granular details and reasoning about contextual relations. Questions reference one or more video moments containing frames and subtitles, then require perception or relational reasoning across 17 fine-grained categories.

  • Challenges for the LONGVIDEOBENCH: LongVideoBench requires models to retrieve granular details such as objects, events, or attributes rather than provide summaries or topic overviews.
  • Challenges for the LONGVIDEOBENCH: Questions also require models to analyze interconnections among diverse content within a long video to derive the correct answer.
  • General Scheme for Referring Reasoning: In referring reasoning, each question identifies one or more video moments as referred context, combining frames and subtitles, followed by a question body requiring reasoning over that context.
  • Two Levels: Perception and Relation: L1 Perception references a single moment and tests locating it and understanding visual concepts such as objects, actions, or events.
  • Two Levels: Perception and Relation: L2 Relation references multiple moments and requires reasoning about temporal order or shared concepts, making it generally more challenging than L1.
  • 17 Finer-grained Question Categories: The benchmark subdivides the two levels into 17 finer-grained categories based on referring-query type and target-answer type, using video-frame and subtitle descriptions.

3 Dataset Construction

LONGVIDEOBENCH is constructed to evaluate long-video understanding across progressive durations and diverse content categories using temporally interleaved video-subtitle inputs. Its referring-reasoning questions are produced through trained, multi-stage annotation with explicit referred-context requirements.

  • 3.1 Video Categories and Duration Groups: LONGVIDEOBENCH uses four progressive duration groups spanning videos from 8 seconds to 3600 seconds, including a unique hour-long subset.The groups are (8s, 15s], (15s, 60s], (180s, 600s], and (900s, 3600s].
  • 3.1 Video Categories and Duration Groups: Videos are collected from 99 landscape-video channels and 20 portrait-video channels across diverse content categories.The passage contrasts this coverage with benchmarks focused on narrower categories such as egocentric videos or movies.
  • 3.2 Interleaved Video-Subtitle Data: The collection pipeline downloads videos at least 720P, retains or generates English subtitles, and uses Whisper-V3-Large when subtitles are unavailable.Videos without transcribed subtitles or with non-English subtitles are removed when source platforms provide subtitles.
  • 3.2 Interleaved Video-Subtitle Data: The benchmark feeds temporally aligned subtitles alongside concurrent video frames as interleaved multimodal sequences.This format reflects subtitle-assisted viewing and uses speech text to reduce ambiguity from purely visual scenes.
  • 3.3 Human Annotation: Annotation uses trained annotators who watch each full video, explicitly highlight referring queries, and label frame indices for all referred moments.Each video passes through a primary annotator, examiner, and reviser to filter unusable videos, check requirements, and correct incorrect annotations.

4 Evaluation of LONGVIDEOBENCH

Evaluation shows that long-context LMMs struggle with longer videos, complex temporal relations, and interleaved modalities, while proprietary models benefit more from increased input length and subtitles than open-source models. Test results further indicate advantages from stronger language backbones, but persistent gaps in detailed long-video understanding.

  • Validation findings: More than 10% improvement occurs for GPT-4o and Gemini-1.5-Pro on videos longer than 180 seconds when input length increases from 16 to 256 frames.On EgoSchema, Gemini-1.5-Pro improves only 2.5% when increasing from 16 to 150 frames.
  • Validation findings: Open-source LMMs cannot improve results beyond 16 frames, and Idefics2 and Mantis-Idefics2 degrade severely at 64 frames.The degradation occurs before these models reach their context-length limits.
  • Validation findings: All six models perform worst on the longest (900,3600] videos, followed by (180,600] videos and then shorter groups.This duration pattern establishes LONGVIDEOBENCH as a challenging test of video understanding.
  • Validation findings: Subtitles improve every model, but open-source LMMs integrate them less effectively than GPT-4o, while removing video frames causes much worse results for all models.These findings show both a gap in interleaved multimodal understanding and the fundamental importance of visual input.
  • Test-set findings: 5.9% and 14.3% improvements over PLLaVA-7B are achieved by PLLaVA-13B and PLLaVA-34B, respectively, with PLLaVA-34B ranking highest among open-source models.The results indicate that scaling the language-model backbone supports more comprehensive video understanding.
  • Test-set findings: All models perform worst on SSS, an L2 category requiring scene-order reasoning, showing that temporal relations are harder than perception.Distracting options in SSS are permutations of the correct scene sequence, and models also perform worse when referred moments are nearer the video beginning, especially in longer videos.

5 Related Works

Prior video LMMs primarily target short-video understanding, whereas newer long-context models address hour-long inputs through token compression, memory banks, and extended context windows. Existing benchmarks range from global-summary short-video QA to more focused temporal-comprehension evaluations.

  • Video LMMs and Long-context LMMs: Early video LMMs focus on videos shorter than one minute, using pretrained video backbones, temporal pooling, and video-specific supervised tuning.Image LMMs also achieve competitive performance on traditional short-video understanding tasks.
  • Video LMMs and Long-context LMMs: Long-video research compresses frames into fewer tokens or adds memory banks to support hour-long content within LMMs.These approaches target the scaling challenges of longer video inputs.
  • Video LMMs and Long-context LMMs: Leading open-source and proprietary models now support context lengths over 128K tokens.Examples include LWM and Phi-3-128K among open-source models, and GPT-4o and Gemini-1.5-Pro among proprietary models.
  • Benchmarks for Video LMMs: Classical benchmarks such as MSVD-QA, MSRVTT-QA, and ActivityNet-QA mainly assess global-summary questions that can often be answered from a few key frames.NeXT-QA and MVBench instead target temporal dynamics in short videos.

6 Conclusion

LONGVIDEOBENCH is introduced as a comprehensive benchmark for evaluating LMMs on hour-long subtitled videos across diverse themes. It uses referring-reasoning questions to address single-frame bias and challenges both proprietary and open-source LMMs.

  • 6 Conclusion: LONGVIDEOBENCH evaluates Large Multimodal Models on hour-long subtitled videos spanning diverse themes.The benchmark is designed as a comprehensive evaluation resource for long-form video understanding.
  • 6 Conclusion: Referring reasoning is introduced as a novel video question-answering paradigm addressing single-frame bias in existing video understanding benchmarks.The questions require models to understand referred video contexts rather than relying on isolated frames.
  • 6 Conclusion: Evaluation results show that LONGVIDEOBENCH presents significant challenges for both proprietary and open-source LMMs.The passage identifies difficulty across both model groups, without reporting specific scores.

Checklist

The checklist confirms coverage of claims, limitations, societal impacts, ethics, reproducibility, asset documentation, data safeguards, and human-subjects procedures. It also records that theoretical-result items were not applicable and models were evaluated zero-shot.

  • Research integrity: The authors affirmed that the abstract and introduction accurately reflect the paper’s contributions and scope, described limitations, discussed potential negative societal impacts, and followed ethics guidelines.Limitations are referenced in Sec. C, and societal impacts in Sec. D.
  • Reproducibility: Theoretical-result assumptions and complete proofs were marked N/A, while code, data, and reproduction instructions were provided at the LongVideoBench website.The models were tested in the zero-shot setting, making training details and error bars N/A.
  • Assets and resources: The checklist confirms citation and licensing of existing assets, release of new assets online, and documentation of compute resources in Sec. A.All assets can be assessed at https://longvideobench.github.io.
  • Data governance: The authors documented consent procedures and stated that the data were checked to eliminate personally identifiable information and offensive content.These safeguards are reported alongside the asset documentation.
  • Human subjects: For crowdsourcing and human-subjects work, participant instructions, IRB approval, estimated wage, and total compensation were documented separately.The instructions and wage appear in Sec. E.1, while approval appears in the dataset sheet.

A Additional Experimental Settings … B More Visualizations w.r.t. Referring Query Depth

The appendix details computational resources, participating models, prompt-processing procedures, and evaluation settings, then extends visualization of performance trends by referring query depth. Most remaining models perform worse when queried moments are near the video beginning or middle, although some show no clear trend.

  • A Additional Experimental Settings: Open-source LMM experiments used 2 NVIDIA A800 80G GPUs and 64-core Intel Xeon Platinum 8336C CPUs, with batch size=1 for all 18 models.
  • A.1 A Brief Introduction on Participating Models: The six long-context LMMs comprised GPT-4o, Gemini-1.5-Pro, Idefics2, Phi-3, Mantis-Idefics2, and Mantis-BakLLaVA.
  • A.1 A Brief Introduction on Participating Models: Idefics2 has 8B parameters and 32K context length, while Phi-3-Vision-Instruct has 4B parameters and 128K context length.
  • A.2 Prompts and Settings: Long-context LMMs received interleaved images or image placeholders and text, ending with the question and answer options, while image LMMs processed 8 concatenated image embeddings.
  • A.2 Prompts and Settings: Video LMM evaluations followed official settings: PLLaVA, VideoChat2, VideoLLaVA, and ShareGPT4Video used 16 frames, whereas LLaVA-Next-Video used 32 frames.
  • B More Visualizations w.r.t. Referring Query Depth: Most of the 14 remaining LMMs performed worse when queried moments were near the video beginning or middle, while some showed no clear trend because overall performance was limited.

C Limitations … H Hosting, Licensing and Maintenance Plan

The paper documents LongVideoBench’s scope limitations, detailed annotation procedures across perception and relation question types, human-study logistics, public resources, and plans for responsible licensing and maintenance.

  • C Limitations: The benchmark currently covers only vision and language, excludes audio and videos longer than 1 hour, and is intended for future expansion.This defines the stated limitations and planned broader scope.
  • E.1 Instructions and Annotation Interface; I. SCENE-REFERRED EVENT (S2E); II. SCENE-REFERRED OBJECT (S2O); III. SCENE-REFERRED OBJECT ATTRIBUTE (S2A); IV. EVENT-REFERRED OBJECT (E2O); V. OBJECT-REFERRED EVENT (O2E); VI. TEXT-REFERRED EVENT (T2E); VII. TEXT-REFERRED OBJECT (T2O); VIII. TEXT-REFERRED OBJECT ATTRIBUTE (S2A): Perception annotations construct questions from scenes, events, subtitles, people, objects, and object attributes, using observed content as answers and absent or confusing content as distractors.The procedures cover scene-referred event, object, and attribute questions; event-referred object; object-referred event; and text-referred event, object, and attribute categories.
  • VIII. TEXT-REFERRED OBJECT ATTRIBUTE (S2A); IX. EVENT BEFORE/AFTER EVENT (E3E); X. OBJECT BEFORE/AFTER OBJECT (O3O); XI. SEQUENCE OF SCENES (SSS); XII. SCENE-REFERRED OBJECT TRACKING (SOS); XIII. SCENE-REFERRED OBJECT ATTRIBUTE CHANGE (SAA): Relation annotations require reasoning across timestamps by testing event order, object order, scene sequences, tracking, and attribute changes.These procedures include event-before/after-event, object-before/after-object, sequence-of-scenes, scene-referred tracking, and scene-referred attribute-change questions.
  • XIV. EVENT BEFORE/AFTER TEXT (T3E); XV. OBJECT BEFORE/AFTER TEXT (T3O); XVI. TEXT-REFERRED OBJECT TRACKING (TOS); XVII. TEXT-REFERRED OBJECT ATTRIBUTE CHANGE (TAA): Subtitle-linked relation questions connect text with temporally preceding or following events, objects, tracking contexts, and attribute changes.The procedures cover event-before/after-text, object-before/after-text, text-referred object tracking, and text-referred object attribute change.
  • E.1 Instructions and Annotation Interface; XVII. TEXT-REFERRED OBJECT ATTRIBUTE CHANGE (TAA): The annotation interface records a question, 4-5 candidate options, one correct choice, the highlighted referring query, and its referred moment, with timestamp and frame-number support.The progress bar can display the current frame number, illustrated as 1 of 12113.
  • E.2 Hourly Wage and Total Compensation: Experienced annotators receive 90 USD daily for 3 working hours, while 1500 total hours of annotation, examination, and revision cost 45,000 USD.The stated hourly wage is 30 USD per hour.
  • E.3 IRB Approval for Human Study; F URLs to Websites: The paper points readers to a dataset sheet for further explanation and provides the project homepage, Hugging Face dataset, and GitHub repository URLs.These resources are listed as the benchmark’s key URLs.
  • G Author Statement on Responsibility; H Hosting, Licensing and Maintenance Plan: The authors accept responsibility for potential copyright and privacy violations, while hosting the dataset on Hugging Face under CC-BY-NC-SA-4.0 and maintaining updates at least twice yearly.The license prohibits commercial use and permits sharing under the stated terms.

I Dataset Sheet for LONGVIDEOBENCH … Any other comments?

LONGVIDEOBENCH was created to evaluate long-context multimodal understanding in long videos, addressing video-question-answering benchmarks’ limited need for long frame inputs. It contains raw web-sourced videos and human-annotated multiple-choice questions, with public validation and hidden test subsets.

  • I Dataset Sheet for LONGVIDEOBENCH: LONGVIDEOBENCH evaluates large multimodal models on long-context understanding in long videos, requiring hundreds of input frames for most advanced models to reach optimal accuracy.The dataset addresses existing video question-answering benchmarks’ limited requirement for long frame inputs.
  • Any other comments?: 3,763 videos and 6,678 question-answering pairs comprise the dataset, with each pair containing one question, one correct option, and 3-4 distractors.Instances are web-sourced videos and question-answering pairs about those videos.
  • How many instances are there in total (of each type, if appropriate)?: The videos are uniformly randomly sampled from videos downloaded from 119 web channels spanning 10 content categories.This sampling process is described in the main paper’s Table 3.
  • How many instances are there in total (of each type, if appropriate): Each video is raw data obtained directly from the web without feature extraction, while each question-answering pair comes from human annotation.The benchmark annotates up to three question-answering pairs per video, each with a question, correct answer, and distracting options.
  • How many instances are there in total (of each type, if appropriate): The benchmark provides no training set: validation contains 20% with public labels, while the hidden-label test set contains the remaining 80% to reduce overfitting.The validation and test subsets are the recommended benchmark splits.
  • Any other comments?: Annotation contractors were paid 90 USD per day and limited to 3 working hours daily to avoid fatigue.Videos were downloaded using official software APIs, and annotations were checked by examiners and revised when marked incorrect.
  • Any other comments?: The videos were collected during April 2024, although associated video data may date from 2010 through April 2024.The videos were uploaded to web platforms no later than April 2024 and may have been created as early as 2010.
  • Any other comments?: The dataset excludes private or proprietary information from collected videos and annotations, and videos becoming non-public will be removed with their question-answering pairs.Some videos may contain subtitles identifying individuals, such as movie characters.

What (other) tasks could the dataset be used for? · Any other comments?

The dataset is restricted from commercial use and prohibited from post-processing or sub-sampling intended to demonstrate bias. It is distributed through Hugging Face datasets and has no DOI.

  • What (other) tasks could the dataset be used for?: Future users should consider dataset composition and collection, preprocessing, cleaning, and labeling factors that could affect reuse.
  • What (other) tasks could the dataset be used for?: Potential future-use concerns include unfair treatment, stereotyping, quality-of-service issues, financial harms, and legal risks.
  • What (other) tasks could the dataset be used for?: The dataset should not be used for commercial applications under any circumstances.
  • What (other) tasks could the dataset be used for?: Post-processing or sub-sampling to demonstrate racial or gender bias is strictly prohibited.
  • Any other comments?: The dataset has been distributed to third parties through Hugging Face datasets.
  • Any other comments?: The dataset does not have a digital object identifier (DOI).

When will the dataset be distributed? · Any other comments? · Who will be supporting/hosting/maintaining the dataset?

The dataset has already been released under the CC-BY-NC-SA 4.0 license, which prohibits commercial use. The authors will maintain the dataset.

  • When will the dataset be distributed?: The dataset has already been released.
  • When will the dataset be distributed?: The distribution documentation asks whether copyright or other IP licenses, terms of use, access points, and associated fees apply.
  • When will the dataset be distributed?: It is distributed under the Creative Commons Attribution Non Commercial Share Alike 4.0 (CC-BY-NC-SA 4.0) license.
  • When will the dataset be distributed?: No commercial use is allowed.
  • When will the dataset be distributed?: The documentation also addresses possible third-party IP restrictions and export-control or other regulatory restrictions.
  • Who will be supporting/hosting/maintaining the dataset?: The authors will be maintaining the dataset.

How can the owner/curator/manager of the dataset be contacted (e.g., email address)? · Any other comments?

The dataset is reviewed twice yearly, with updates announced on GitHub and Hugging Face. Older versions remain available through independent download links, while contributors may contact the owner and have their contributions reviewed.

  • How can the owner/curator/manager of the dataset be contacted (e.g., email address)?: The provided materials include a question about errata but do not provide an erratum link or access point.This point reflects the absence of an answer in the supplied passages rather than evidence that no erratum exists.
  • How can the owner/curator/manager of the dataset be contacted (e.g., email address)?: Twice-yearly reviews will maintain the dataset, with updates announced on GitHub and Hugging Face.The provided passage does not specify who performs the reviews beyond stating that the dataset will be reviewed.
  • How can the owner/curator/manager of the dataset be contacted (e.g., email address)?: Annotators understand that their annotations will remain public for an unlimited time period.The passage marks retention limits as not applicable while stating the expected public availability period.
  • How can the owner/curator/manager of the dataset be contacted (e.g., email address)?: Older dataset versions will remain available through independent download links.The passage confirms continued access but does not describe additional maintenance procedures.
  • How can the owner/curator/manager of the dataset be contacted (e.g., email address)?: People seeking to extend or contribute to the dataset can contact the owner.The passage identifies owner contact as the contribution mechanism but gives no email address or other contact detail.
  • How can the owner/curator/manager of the dataset be contacted (e.g., email address)?: Contributions from others will be reviewed by the owner.No further validation, distribution, or communication process is specified in the provided answer.
Loading 2407.15754v1…