Source-linked AI summary
Towards Event-oriented Long Video Understanding
Yifan Du, Kun Zhou, Yuqi Huo, Yifan Li, Wayne Xin Zhao, Haoyu Lu, Zijia Zhao, Bingning Wang, Weipeng Chen, Ji-Rong Wen
TL;DR
Existing video benchmarks may allow answers from a few frames because their videos lack rich events, limiting assessment of event understanding. The paper introduces Event-Bench and VIM, which merges video instructions into event-intensive training data; VIM surpasses GPT-4V on average on Event-Bench.
Problem
Existing video benchmarks may suffer shortcut bias because answers can sometimes be deduced without watching the full video, and their test instances may not accurately estimate human-like understanding.
Source: §1 · p. 1Method
Event-Bench evaluates six event-related tasks using existing data and human annotations, while VIM merges similar video instructions and uses an adaptive architecture that supports image and video inputs.
Source: §1 · p. 2, §4.2 · p. 6Results
41.64 v.s. 32.65: VIM surpasses GPT-4V on average on Event-Bench and outperforms open-source models of comparable parameter scales.
Source: §1 · p. 2Takeaways & Limitations
Event-Bench provides a systematic comparison of video MLLMs across event-understanding capabilities, and the proposed approach exceeds open-source models of comparable parameter scales and GPT-4V on average.
Source: §6 · p. 9Event-Bench considers only visual modality, and training used 500K instructions because of limited computational resources.
Source: §7 · p. 9
Abstract
from arXiv · showhide
With the rapid development of video Multimodal Large Language Models (MLLMs), numerous benchmarks have been proposed to assess their video understanding capability. However, due to the lack of rich events in the videos, these datasets may suffer from the short-cut bias that the answers can be deduced from a few frames, without the need to watch the entire video. To address this issue, we introduce Event-Bench, an event-oriented long video understanding benchmark built on existing datasets and human annotations. Event-Bench includes six event-related tasks and 2,190 test instances to comprehensively evaluate video event understanding ability. Additionally, we propose Video Instruction Merging~(VIM), a cost-effective method that enhances video MLLMs using merged, event-intensive video instructions, addressing the scarcity of human-annotated, event-intensive data. Extensive experiments show that the best-performing model, GPT-4o, achieves an overall accuracy of 53.33, significantly outperforming the best open-source model by 41.42%. Leveraging an effective instruction synthesis method and an adaptive model architecture, VIM surpasses both state-of-the-art open-source models and GPT-4V on the Event-Bench. All code, data, and models are publicly available at https://github.com/RUCAIBox/Event-Bench.
1 Introduction
Existing video benchmarks may allow answers from a few frames because their videos lack rich events, limiting evaluation of human-like video understanding. The paper introduces Event-Bench and VIM, an event-intensive instruction synthesis method for improving video MLLMs.
1 Introduction
- Existing benchmarks may have shortcut bias: answers can sometimes be deduced without watching the full video, undermining reliable assessment of event understanding.The paper links this bias to a lack of rich events in videos and notes that test instances may not accurately estimate human-like understanding.Source: §1 · pp. 1–2
- Event-Bench contains 2,190 samples for six event-related tasks, with longer time scopes and an event-oriented focus than existing benchmarks.Its evaluation covers atomic, composite, and overall event understanding; Table 1 contrasts its event-oriented design and longer time scope with earlier benchmarks.Source: §1 · p. 2, Table 1 · p. 2
- 41.64 v.s. 32.65: VIM surpasses GPT-4V on average and outperforms open-source models of comparable parameter scales on Event-Bench.The paper attributes VIM to merging similar videos into event-intensive instructions and using an adaptive architecture that also supports high-quality image instructions.Source: §1 · p. 2
- VIM addresses the cost of annotating event-intensive instructions by combining existing video and image instructions, including merged videos containing events from similar source videos.An adaptive architecture handles both image and video inputs; experiments report improved video MLLM performance on Event-Bench.Source: §1 · p. 2
2 Related Work
The related work reviews video MLLM architectures and video understanding benchmarks, then identifies gaps that motivate event-oriented long-video evaluation.
2.1 Video Multimodal Large Language Model
- Prior video MLLM work explores stronger encoders, long-context video, and large-scale instruction tuning; this paper instead targets complex-event instruction synthesis and scalable architecture.Typical systems combine a video or image encoder, an LLM, and a connector.Source: §2.1 · p. 2
2.2 Video Understanding Benchmark
- Earlier video benchmarks often use short or limited-domain videos, while long-video benchmarks may lack complex event reasoning or open-domain coverage.These gaps motivate a diverse, event-oriented long-video benchmark for evaluating complex event narratives.Source: §2.2 · pp. 2–3
3 Event-oriented Benchmark
Event-Bench organizes long-video evaluation around event understanding and combines existing VideoQA data with human-annotated internet videos. Its construction includes filtering, targeted annotation, and task-level statistics.
3 Event-oriented Benchmark
- Event-Bench pairs videos with multiple-choice questions from event-related subtasks, using a hierarchical task taxonomy to guide data collection.Its data come from existing datasets and newly human-annotated internet videos.Source: §3 · p. 3, §3.2 · p. 3
3.1 Hierarchical Task Taxonomy
- Atomic event understanding tests recognition and description; composite understanding tests temporal order and causal relations between events.These tasks include Event Description, Temporal Reasoning, and Causal Reasoning.Source: §3.1 · p. 3
- Overall understanding covers relationships across all video events through contextual, episodic, and counter-intuitive reasoning tasks.These tasks respectively address event sequences and environment, story-level semantics involving characters or objects, and abnormal details.Source: §3.1 · p. 3
3.2 Data Construction
- The benchmark combines selected instances from four VideoQA datasets with newly annotated internet videos to broaden its event and storyline coverage.The existing sources are STAR, NExT-QA, EgoSchema, and FunQA; YouTube videos are collected for episodic reasoning.Source: §3.2 · p. 3, §3.2.1 · p. 3, §3.2.2 · p. 5
3.2.1 Construction Based on Existing Datasets
- STAR, NExT-QA, EgoSchema, and FunQA instances undergo format unification, single-frame shortcut filtering, and answer-video inconsistency filtering.Questions are converted to multiple choice and human-reviewed; shortcut items are removed when three image MLLMs all answer from a single frame, while disagreements between two video MLLMs and the human answer identify inconsistent samples.Source: §3.2.1 · pp. 3–4
3.2.2 Annotation Based on Internet Videos
- For episodic reasoning, annotators caption videos at 30-second intervals, GPT-4 generates questions, and humans verify answers and identify relevant event intervals.Human reviewers resolve mismatches through additional review and voting.Source: §3.2.2 · p. 5
3.3 Data Statistics
- 2,190 video question-answer pairs span six tasks, with 172 to 400 test samples per task and videos of varied lengths and real-world domains.The benchmark's hierarchical taxonomy supports evaluation at different capability levels.Source: §3.3 · p. 5, Table 2 · p. 5
4 Methodology
VIM combines semantically similar video instructions into event-rich training examples and uses an adaptive architecture that supports image and video inputs. This design also lets training combine merged video instructions with high-quality image instructions.
4.1 Video Instruction Merging
- VIM merges semantically similar video instructions into a new event-rich sample, using ChatGPT to generate a question and answer for the temporally concatenated videos.The instructions are first represented as text embeddings, and the most similar ones are selected to support coherent merging.Source: §4 · p. 5, §4.1 · pp. 5–6
4.2 Adaptive Model Architecture
- Merged video instructions and high-quality image instructions are combined for training because the model architecture handles both input types.Source: §4 · p. 5, §4.2 · p. 6
- The adaptive architecture samples video frames, processes them into visual tokens, and concatenates those tokens into a video representation; image input follows the regular image-MLLM path.Its variable sequence-length design supports a single image, short videos, and long videos.Source: §4 · p. 5, §4.2 · p. 6
5 Experiment
Experiments evaluate image, video, and proprietary MLLMs on Event-Bench using a circular evaluation protocol. Overall understanding remains especially challenging, and open-source models trail proprietary models on that task; frame-count and training-strategy analyses examine performance differences.
5.2 Main Results
- 38.67 for Gemini-1.5-Pro and 37.33 for GPT-4o on episodic reasoning; overall understanding is harder than composite and atomic event understanding.Source: Table 3 · p. 7, §5.2 · pp. 7–8
5.1 Experimental Setup
- Circular evaluation tests each multiple-choice question under every ordering of its options and counts a sample as successful only if every round is correct.Source: §5.1 · p. 7
5.3 Analysis
- 47.50 to 56.75 in GPT-4o temporal reasoning when input frames increase from 8 to 32; open-source models do not consistently improve with more frames.Most open-source models perform best with 16 or 24 frames, and often degrade at 32.Source: Figure 5 · p. 8, §5.3 · p. 8
5.2 Main Results
- Open-source models are comparable to proprietary models on atomic and composite tasks, but all lag behind on overall understanding; the paper's model leads open-source models on average accuracy.MovieChat leads on episodic reasoning, while PLLaVA (13B) is slightly better on counter-intuitive reasoning.Source: §5.2 · p. 8
5.3 Analysis
- Removing instruction merging or image instructions lowers performance across all tasks, and selecting semantically similar videos performs better than random selection.Source: Table 4 · p. 8, §5.3 · p. 8
6 Conclusion
Event-Bench evaluates long-video event understanding, while the authors use merged event-intensive video instructions and revised architecture to improve video MLLMs. Their experiments report systematic comparisons and performance above comparable-scale open-source models and GPT-4V on average.
6 Conclusion
- Event-Bench comprises six event-related tasks and 2,190 test instances for evaluating event understanding in long videos.The benchmark is based on existing datasets and human annotations.Source: §6 · p. 8
- The training strategy merges simple video instructions into event-intensive instructions and revises the architecture to support high-quality image instructions.The authors describe this as addressing the lack of human-annotated, event-intensive video instructions.Source: §6 · pp. 8–9
- The approach outperforms open-source video MLLMs of comparable parameter scales and surpasses GPT-4V on average.Experiments also show Event-Bench provides systematic comparisons across capabilities and identifies major shortcomings of open-source MLLMs.Source: §6 · p. 9
7 Limitation
The authors identify limits in Event-Bench’s modalities and in VIM’s training data scale and synthesis quality, and plan to address them in future work.
7 Limitation
- Event-Bench currently considers only visual events, although real-world events also involve textual, audio, and speech modalities.The authors plan to add other modalities to the benchmark.Source: §7 · p. 9
- Training used 500K instructions because computational resources were limited.The authors report that including more high-quality video and image instructions positively impacts performance and aim to scale training data and model size.Source: §7 · p. 9
- Synthesized video instructions still have lower diversity and complexity than human-annotated instructions.The authors plan to construct more event-intensive training data through human annotation.Source: §7 · p. 9
A.1 Internet Videos Source
The appendix identifies YouTube channels as sources for internet videos.
A.1 Internet Videos Source
- The internet videos are collected from YouTube channels.The passage introduces a list of channels but does not specify their names here.Source: §A.1 · p. 12
A.2 Data Statistics
The benchmark contains 2,190 video question-answer pairs across six event-understanding tasks, with 172–400 test samples per task.
A.2 Data Statistics
- 2,190 video question-answer pairs span six tasks corresponding to different event-understanding abilities.Figure 6 presents the dataset distribution for the benchmark.Source: §A.2 · p. 12, Figure 6 · p. 12
- Each task has 172–400 test samples for evaluation.Source: §A.2 · p. 12
A.3 Ablation Study
The ablation finds that merging more videos can reduce performance, while appendix prompts describe question generation and conversion, and an example illustrates merging similar instructions.
A.4 Prompt
- For HumorQA and MagicQA, the conversion prompt asks why a video is humorous or magical, with plausible options grounded in details requiring complete-video viewing.The prompt also requires the question to use one of two specified formulations and options of roughly equal length.Source: §A.4 · p. 12
A.3 Ablation Study
- From 2 to 3 or 4 merged videos, performance falls on every task but remains better than training on a single video.The comparison varies the number of selected videos, with k = 1 indicating no merge operation.Source: Figure 7 · p. 12, §A.3 · p. 12
A.4 Prompt
- The episodic-question prompt requests ten difficult four-way questions from chronological scene descriptions, answerable only by viewers who watched the full video.It also asks for a rationale for each correct answer and excludes questions about scenes where the director makes the video.Source: §A.4 · p. 13
A.5 Video Instruction Merging
- The merging example pairs two similar video instructions before merging with the resulting merged instruction.The text identifies this process example as Figure 8.Source: §A.5 · p. 13, Figure 8 · p. 15