Source-linked AI summary
LVBench: An Extreme Long Video Understanding Benchmark
Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, Jie Tang
TL;DR
Short-video progress has not resolved the need to evaluate multimodal models on videos spanning extended temporal contexts. LVBench addresses this gap with a diverse, visually grounded benchmark covering six long-video understanding capabilities. Evaluations show that current models still underperform on these demanding tasks.
Problem
Existing multimodal models and benchmarks mainly target short videos, leaving long-range video understanding insufficiently evaluated despite applications requiring extended temporal comprehension.
Method
LVBench constructs a benchmark from filtered long videos and evaluates six composable temporal and semantic capabilities through annotated question-answer tasks.
Results
Current multimodal models still underperform on LVBench’s demanding long-video understanding tasks.
Takeaways & Limitations
LVBench provides a challenging evaluation framework intended to stimulate development of models for extended video comprehension.
Takeaways & Limitations
The benchmark excludes audio because most current models lack effective audio processing capabilities.
Abstract
from arXiv · showhide
Recent progress in multimodal large language models has markedly enhanced the understanding of short videos (typically under one minute), and several evaluation datasets have emerged accordingly. However, these advancements fall short of meeting the demands of real-world applications such as embodied intelligence for long-term decision-making, in-depth movie reviews and discussions, and live sports commentary, all of which require comprehension of long videos spanning several hours. To address this gap, we introduce LVBench, a benchmark specifically designed for long video understanding. Our dataset comprises publicly sourced videos and encompasses a diverse set of tasks aimed at long video comprehension and information extraction. LVBench is designed to challenge multimodal models to demonstrate long-term memory and extended comprehension capabilities. Our extensive evaluations reveal that current multimodal models still underperform on these demanding long video understanding tasks. Through LVBench, we aim to spur the development of more advanced models capable of tackling the complexities of long video comprehension. Our data and code are publicly available at: https://lvbench.github.io.
1. Introduction
Existing multimodal language models perform well on short-video understanding but face substantial challenges with longer temporal sequences. LVBench addresses the resulting benchmark gap with a framework for evaluating long video understanding.
- Existing MLLMs have advanced across video tasks but struggle with longer temporal sequences.
- Most existing video benchmarks emphasize static images or short clips, overlooking temporal understanding in long-form videos.
- LVBench introduces a systematic framework of six temporal understanding capabilities that can be combined into complex evaluation tasks.
2. Related Works
Prior multimodal video models and benchmarks have introduced architectural and evaluation advances, but long-range video comprehension remains insufficiently addressed. LVBench is positioned against this limitation in existing models and datasets.
- Architectural advances include temporal modeling, multi-frame processing, temporal extrapolation, memory modules, long contexts, and dense-video feature pooling.
- LVBench compares datasets using whether video sources are diversified and whether questions span more than two categories.
- Despite these advances, experiments indicate that current video understanding models still fall short on tasks requiring long-range comprehension.
- Existing vision-language benchmarks largely focus on images and short videos, while several long-video datasets are domain-specific or provide limited temporal context.
3. LVBench
LVBench constructs a long-video benchmark from systematically filtered videos and evaluates six composable comprehension capabilities through structured question-answer annotation. Its dataset and questions emphasize broad temporal coverage, specificity, and visual grounding.
- 3.1. Dataset Collection: Long videos are defined as videos lasting at least 30 minutes with rich visual information, multiple events, and scene transitions.
- 3.1. Dataset Collection: 500 YouTube videos were initially collected across six domains, then screened using protagonist presence, coherence, event density, visual clarity, and modality independence.
- 3.1. Dataset Collection: 103 high-quality videos totaling 117 hours remained after filtering.
- 3.2. Task Definition and Question Types: The benchmark defines six core skills: temporal grounding, summarization, reasoning, entity recognition, event understanding, and key information retrieval.
- 3.2. Task Definition and Question Types: Entity Recognition and Event Understanding comprise larger question proportions because many questions inherently require identifying entities and understanding events.
- 3.3. Question-Answer Pair Generation: Annotators watch videos fully, mark events and entities, generate questions averaging 24 per hour across the video, and construct four-choice answers with annotated clue durations.
4. Experiments
LVBench experiments compare models adapted for long-video inputs with models supporting long videos natively, revealing substantial gaps among models, humans, and instruction-following reliability. Performance generally improves with denser visual sampling, while zero-frame controls support the benchmark’s visual-grounding requirement.
- Experimental Setup: 13 multi-image or short-video models were evaluated with fixed frame samples, alongside 13 models supporting long videos natively.The evaluation included models such as GPT-4o, GPT-4.1, Gemini 2.5 Pro, Qwen2.5-VL-72B, and Seed1.5-VL.
- Performance across Core Capabilities: Gemini-2.5-Pro achieved the top score of 67.4 and outperformed other models in 5 of 6 tasks, while Seed1.5-VL-Thinking ranked second at 64.6.The winning tasks were EU, KIR, TG, Rea, and Sum.
- Performance across Core Capabilities: Open-source leaders VideoLLaMA3-7B and Qwen2.5-VL-72B scored 45.3 and 44.0, respectively, trailing Gemini-2.5-Pro by over 20%.The gap was reported for both individual tasks and the overall score.
- Analysis of Failure Modes: Some native long-video models performed unexpectedly poorly because models struggled to follow constrained answer instructions.Gemini-1.5-Pro produced answers outside the four options 20.9% of the time, while MovieChat and LWM strongly favored option A.
- Performance across Video Categories: Humans averaged 94.4% accuracy, compared with 67.4% for Gemini-2.5-Pro and 64.0% for Seed1.5-VL across video categories.Gemini-2.5-Pro ranged from 72.1% on Event Records to 54.3% on Documentaries, whereas Seed1.5-VL ranged from 60.3% to 67.5%.
- Ablation on Frame Density: All evaluated models improved as input frame counts increased, with the largest performance leap occurring from 50 frames to dense 1 FPS sampling.The 0 Frames control produced near-random performance, supporting the need for visual understanding rather than linguistic shortcuts.
5. Discussion
LVBench introduces a challenging benchmark for long video understanding using lengthy videos and annotated question-answer pairs. Evaluations show that state-of-the-art multimodal models remain below human-level accuracy on long videos, while audio is excluded from the benchmark.
- LVBench combines diverse lengthy videos with a meticulously annotated question-answer dataset for evaluating complex long video understanding.It is designed as a robust evaluation framework for multimodal models.
- State-of-the-art multimodal models still perform below human-level accuracy on long videos despite progress on short video understanding.
- Audio data is excluded because most current models lack effective audio processing capabilities.Future work aims to incorporate audio information into the evaluation framework.
- The dataset relies on YouTube content and provides download links rather than source videos, although stored copies may be available with creator permission.The original videos are not guaranteed to remain constant over time.
A.1. Motivation
The dataset was created to evaluate and optimize long video understanding models. It addresses the scarcity of benchmarks capable of assessing performance on extra-long videos.
- The dataset was created to evaluate and optimize long video understanding models.
- The benchmark addresses a gap by assessing model performance on extra-long videos.The authors describe it as the first benchmark capable of this assessment to their knowledge.
A.2. Composition
LVBench contains YouTube videos across six categories, paired with question-answer labels and supporting metadata. The videos were manually annotated and inspected for quality, while the dataset can support several long video understanding tasks.
- The dataset contains 103 YouTube videos and 1,549 question-answer pairs across Sports, Documentary, Event Record, Lifestyle, TV Show, and Cartoon categories.
- Video metadata stores question-answer pairs under “qa,” category under “type,” and duration, resolution, and related information under “video info.”
- Each video instance is associated with question-answer pairs that serve as its labels or targets.
- Hired annotators manually annotated the videos after they were downloaded from YouTube using dedicated servers and an annotation platform.
- The videos were manually inspected to ensure high quality, content richness, and freedom from harmful information.
- The dataset could support long video summarization, captioning, question answering, and multimodal understanding tasks.
A.6. Distribution
LVBench is freely and publicly available for distribution to third parties. Its distribution details and access point remain anonymous for review.
- The dataset is freely and publicly available and accessible to third parties.
A.7. Maintenance
The dataset’s maintenance plan is anonymous for review, with updates and long-term support details not specified. Content may be removed when original creators request it.
- Original creators can request removal of their videos from the dataset.
- Dataset maintenance and update procedures are listed as anonymous for review.
B. Distribution of Core Capability Combinations
The six core competencies combine into 26 fine-grained question types, with their proportions quantified to support diverse, multidimensional evaluation.
- 26 fine-grained question types are formed by combining six core competencies.
- The combinatorial design supports dataset richness and diversity across multiple evaluation dimensions.
- Figure 5 presents the proportions of the core capability combinations.