PaperlayerView exact revision on arXiv

Source-linked AI summary

Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs

Zijia Zhao, Haoyu Lu, Yuqi Huo, Yifan Du, Tongtian Yue, Longteng Guo, Bingning Wang, Weipeng Chen, Jing Liu

arXiv:2406.09367v3cs.CV

TL;DR

Existing video benchmarks are costly to construct and often combine skills, making targeted evaluation difficult. VideoNIAH separates video content from queries through synthetic needles and predefined rules, and VNBench evaluates retrieval, ordering, and counting. The evaluation finds proprietary advantages on temporal tasks, while most models struggle with counting.

  • Problem

    Existing video benchmarks are costly to construct and often combine multiple comprehension skills, making specific model weaknesses difficult to identify.

    Source: §1 · pp. 1–2

  • Method

    VideoNIAH inserts needles into original video haystacks and uses predefined rules to generate skill-specific query-response pairs for VNBench tasks.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Proprietary models lead on temporal retrieval and ordering, while most models perform poorly on counting; the highest overall accuracy gap between the top proprietary and open-source models is 8.0%.

    Source: §1 · p. 3, §4.2 · p. 6

  • Takeaways & Limitations

    The authors offer practical advice for video MLLM training based on analyses of model performance and training settings.

    Source: §1 · p. 3

    Synthetic data cannot fully replicate authentic real-world scenarios, so real-world benchmarks should complement synthetic ones.

    Source: §7 · p. 10

Abstract

from arXiv · show

Video understanding is a crucial next step for multimodal large language models (MLLMs). Various benchmarks are introduced for better evaluating the MLLMs. Nevertheless, current video benchmarks are still inefficient for evaluating video models during iterative development due to the high cost of constructing datasets and the difficulty in isolating specific skills. In this paper, we propose VideoNIAH (Video Needle In A Haystack), a benchmark construction framework through synthetic video generation. VideoNIAH decouples video content from their query-responses by inserting unrelated visual 'needles' into original videos. The framework automates the generation of query-response pairs using predefined rules, minimizing manual labor. The queries focus on specific aspects of video understanding, enabling more skill-specific evaluations. The separation between video content and the queries also allow for increased video variety and evaluations across different lengths. Utilizing VideoNIAH, we compile a video benchmark VNBench, which includes tasks such as retrieval, ordering, and counting to evaluate three key aspects of video understanding: temporal perception, chronological ordering, and spatio-temporal coherence. We conduct a comprehensive evaluation of both proprietary and open-source models, uncovering significant differences in their video understanding capabilities across various tasks. Additionally, we perform an in-depth analysis of the test results and model configurations. Based on these findings, we provide some advice for improving video MLLM training, offering valuable insights to guide future research and model development. The code and data are available at https://github.com/joez17/VideoNIAH.

1 INTRODUCTION

Existing video benchmarks are costly to construct and often combine multiple comprehension skills, making specific weaknesses difficult to pinpoint. VideoNIAH uses synthetic video construction to enable skill-targeted evaluation, and VNBench applies it to retrieval, ordering, and counting.

1 INTRODUCTION

  • Existing benchmarks are costly to build and often bundle multiple video-comprehension skills, complicating the identification of specific model weaknesses.Real-world benchmark construction can involve labor-intensive prompt engineering, annotation, and data filtering; query-response pairs may also risk data leakage.Source: §1 · pp. 1–2
  • VideoNIAH inserts unrelated image or text needles into original video haystacks and generates skill-specific query-response pairs using predefined rules.Decoupling queries from video content supports diverse video sources and flexible lengths while reducing human labor.Source: §1 · pp. 2–3
  • Proprietary models show clear advantages on temporal retrieval and ordering, while most models perform poorly on counting.The benchmark evaluates temporal perception, chronological order, and spatio-temporal coherence.Source: §1 · p. 3
  • VNBench covers retrieval, ordering, and counting, and the authors evaluate proprietary and open-source models and analyze benchmark and training-setting factors.The analysis examines context length, needle number, position, and characteristics, and offers advice for video MLLM training.Source: §1 · p. 3

2 RELATED WORKS

Prior work develops video MLLMs and benchmarks for broad video understanding, including long-context evaluation. Synthetic benchmarks offer more control over task factors, and VideoNIAH applies that approach to video understanding.

2.1 VIDEO MLLMS AND VIDEO BENCHMARKS

  • Video MLLM research includes open-source models trained on video data and proprietary models such as GPT-4V and Gemini 1.5 Pro.The related-work discussion also describes models using memory mechanisms, additional video SFT data, or external modality information.Source: §2.1 · p. 3
  • A VNBench sample combines inserted needles, query-response pairs generated by predefined rules, and a randomly selected video haystack.These components are shown as the construction overview within the VideoNIAH framework.Source: Figure 2 · p. 4
  • Realistic video benchmarks cover broad or long-context video QA, but may require laborious annotation and face data-leakage risks.The cited work distinguishes comprehensive benchmarks from benchmarks focused on long-context videos.Source: §2.1 · p. 4

2.2 SYNTHETIC BENCHMARKS

  • Synthetic benchmarks control factors such as sequence length and task complexity; VideoNIAH is presented as a holistic synthetic video benchmark with varied needles and query formats.The related work traces needle-in-a-haystack evaluation from language models to more complex synthetic benchmarks.Source: §2.2 · p. 4

3 EVALUATION DATA RECIPE

VideoNIAH constructs synthetic video tests by separating inserted needles from the original video and generating queries from predefined rules. VNBench applies this framework to retrieval, ordering, and counting, with needle filtering and repeated shuffled-option evaluation.

3.1 VIDEONIAH: A FLEXIBLE SYNTHETIC FRAMEWORK

  • VideoNIAH uses independent inserted needles and query-response rules, combining intra-frame edits, inter-frame insertions, multiple needles, and long-dependency queries.Needles may be subtitles, static images, or dynamic clips.Source: §3.1 · pp. 4–5

3.2 VNBENCH: A BASIC VIDEO UNDERSTANDING BENCHMARK

  • VNBench tests single-needle retrieval, chronological ordering, and repeated-object counting to probe temporal perception, sequence understanding, and spatio-temporal coherence.It samples 150 videos from three sources to create 1,350 samples; VNBench-Long* adds four tasks with videos lasting 10–30 minutes.Source: §3.2 · pp. 5–6

3.3 AUTOMATIC FILTER IN NEEDLE SELECTION

  • Candidate needle images with maximum CLIP similarity above 0.7 to any haystack frame are discarded.The threshold is the one used in VNBench.Source: §3.3 · p. 6

3.4 EVALUATION STRATEGY

  • Each four-choice sample is evaluated four times with shuffled options and counts as correct only if all four answers are correct.Reported task performance is accuracy.Source: §3.4 · p. 6

4 EVALUATION RESULTS

Across VNBench, proprietary models outperform open-source models on most tasks, while results vary across retrieval, ordering, and counting. The evaluation covers 12 models, and the reported findings show particular difficulty with long-dependency tasks, ordering, and fine-grained counting.

4.1 MODELS

  • The evaluation includes 12 video-understanding models: nine open-source models and three proprietary models.The proprietary set includes Gemini 1.5 Pro and two GPT-4 versions.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • 58.7% for LLaVA-OneVision-72B and 66.7% for Gemini 1.5 Pro differ by 8.0% in overall accuracy, with proprietary models ahead on most tasks.The table reports results across VNBench's three tasks and their splits.Source: §4.2 · p. 6, Table 2 · p. 7
  • Models generally perform better on retrieval than on ordering and counting, and multiple-needle long-dependency tasks trail single-needle short-dependency tasks.Some proprietary models reach 100% accuracy on Retrieval-E.Source: §4.2 · pp. 6–7
  • Gemini 1.5 Pro scores 72.9% and GPT-4o 73.4% on ordering, while most open-source models are nearly incapable of the task except the LLaVA-OneVision series.The authors suggest that open-source training may overlook temporal-sequence modeling.Source: §4.2 · p. 7
  • Counting remains difficult even for advanced proprietary models, and every evaluated model performs poorly on Counting-E-2.Counting-E-2 tests information embedded in specific spatial areas of video segments.Source: §4.2 · p. 7

5 RESULT ANALYSIS

VNBench analyses how video duration, needle count and type, and needle position affect model performance. Proprietary models remain stable across duration and recall inserted information within tested context windows, while open-source models show duration- and sampling-related weaknesses.

Effect of Haystack Length

  • Open-source models such as VideoChat2, LLaVA-NeXT-Video, and ST-LLM decline on longer videos, while proprietary models fluctuate little across 10–180-second duration splits.GPT-4 and Gemini 1.5 Pro are reported to have 128k- and 1M-token context windows, respectively.Source: §5 · p. 7, Figure 5 · p. 8

Effect of Needle Number

  • Counting-task performance significantly declines as the number of inserted needles increases, exposing weaknesses in object tracking and memory.The authors identify spatio-temporal relationships, attention, and long-term memory as areas needing improvement.Source: §5 · p. 8, Figure 6 · p. 8

Effect of Recognizing Ability

  • Gemini 1.5 Pro and GPT-4o excel at recognizing subtitles and common images, but accuracy drops on landmark images in Retrieval-I-2 versus common images in Retrieval-I-1.The retrieval subtasks probe local patches, common objects, and landmark images, respectively.Source: §5 · p. 8, Figure 7 · p. 8

Effect of Needle Position

  • Proprietary models precisely recall inserted information within the tested intervals, which are shorter than their context windows.With sequential sampling, LLaMA-VID and LLaVA-NeXT-Video recall more from the beginning and end than the middle; uniform-sampling models show recall linked to whether a needle was sampled.Source: §5 · pp. 8–9, Figure 8 · p. 9

6 MODEL ANALYSIS

Model analyses vary model size, tokens per frame, sampled frames, and temporal prompts. Larger models improve across tasks, while more sampled frames and temporal prompts improve selected temporal tasks.

Model Size

  • Increasing language-model parameters in the LLaVA-OneVision family improves all VNBench subtasks, with the largest gains in ordering and only marginal gains in counting.The authors relate the limited counting improvement to language models’ known difficulty with counting.Source: §6 · p. 9

Token Number Per Frame

  • Varying tokens per frame produces minimal performance variation across VNBench subtasks.The authors designed VNBench primarily to evaluate temporal relationships in long contexts rather than spatial comprehension.Source: §6 · p. 10

Frame Sampling Number

  • Increasing sampled frames improves retrieval and ordering, but yields only marginal improvement in counting.The authors say further optimization is needed for higher-level temporal relationships such as spatio-temporal coherence.Source: §6 · p. 10

Temporal Prompt

  • Temporal prompts significantly improve performance in both ordering and counting tasks.These prompts provide textual time cues, such as frame order or timestamps.Source: §6 · p. 10, Table 3 · p. 10

7 LIMITATIONS

The authors present VideoNIAH as scalable and flexible, but note that synthetic benchmarks cannot fully replicate authentic scenarios represented by real-world videos and human-validated questions.

7 LIMITATIONS

  • Synthetic data generated by VideoNIAH cannot fully replicate authentic scenarios represented by real-world videos with human-validated questions.The authors say real-world benchmarks should complement synthetic ones.Source: §7 · p. 10
  • VideoNIAH allows researchers to tailor increasingly complex construction rules for broader video-model evaluations.The authors describe the framework as scalable and flexible.Source: §7 · p. 10
  • The authors position real-world and VideoNIAH-created synthetic benchmarks as complementary approaches to video-model research.Real-world benchmarks retain value through authentic videos and human-validated questions.Source: §7 · p. 10

8 CONCLUSION

The conclusion presents VideoNIAH as a scalable synthetic framework that separates video content from query-response pairs and supports evaluation across video contexts.

8 CONCLUSION

  • VideoNIAH separates video content from query-response pairs and supports probing video MLLMs across diverse sources, lengths, and comprehension dimensions.VNBench assesses temporal perception, chronological order, and spatio-temporal coherence; the evaluation finds models still struggle with long-distance dependencies.Source: §8 · p. 10

Overview of Appendix

  • The appendix covers construction details, task correlation, long and action benchmarks, robustness, training settings, complete results, task samples, and consistency evaluation.These topics are listed in the appendix overview.Source: Overview of Appendix · p. 15

A.1 SUBTITLE NEEDLE

  • Subtitle needles use the format “The secret word is NAME,” with the keyword drawn from a list of candidate names.The listed names run from Alice to Zach.Source: §A.1 · p. 15

A.2 IMAGE NEEDLE

  • Image needles include fruit images, MSCOCO objects, and landmarks from Seed-Bench2.The fruit candidates are shown in Figure 12.Source: §A.2 · p. 15, Figure 12 · p. 15

A.3 QUERY-RESPONSE GENERATION

  • Retrieval negatives are sampled from needle candidates, ordering negatives shuffle the correct answers, and counting negatives are sampled around the true count using a normal distribution.Figure 13 depicts samples as needles, rule-generated query-response pairs, and a randomly selected video.Source: §A.3 · p. 15, Figure 13 · p. 16

A.4 CONSTRUCTION DETAILS

  • Retrieval and ordering use subtitle or image needles, with image subtasks divided by recognizability; ordering asks for the temporal order of four inserted needles.The tasks include subtitle-based Retrieval-Edit and Ordering-Edit, and image-based Retrieval-Insert and Ordering-Insert.Source: §A.4 · p. 16
  • Counting-Edit tests repeated appearances of inserted objects, while Counting-Insert asks how often an object category appears in video.Counting-Edit level 2 places an image across four random video clips and requires counting appearances across spatial and temporal dimensions.Source: §A.4 · pp. 16–17

B.1 SUB-TASK CORRELATION ANALYSIS

  • Retrieval and ordering subtasks cluster cohesively, whereas counting subtasks are more autonomous and reflect differences in their needle types and demands.Counting-Edit-Level2 requires capturing spatial and temporal details across multiple image needles and frames.Source: §B.1 · p. 18, Figure 14 · p. 18

B.2 CROSS-TASK CORRELATION ANALYSIS

VNBench performance correlates with real-world video benchmark performance, especially on MLVU and VideoMME. The analysis evaluates models with different frame counts and uses distinct answer-matching procedures for models that do not follow the output format.

B.2 CROSS-TASK CORRELATION ANALYSIS

  • The correlation analysis selected six models with 16, 32, 48, 64, 96, and 128 sampled frames and evaluated them on real-world video benchmarks VideoMME, MLVU, and LongVideoBench.The evaluation included 10 video MLLMs: 7 open-source models and 3 proprietary models.Source: §B.2 · p. 19, §C.1 · p. 20
  • VNBench correlations exceed 0.9 with MLVU and VideoMME, while all benchmark correlations exceed 0.75.The authors interpret this as evidence that VNBench can, to a certain extent, reflect real-world video-processing ability.Source: Table 4 · p. 19, Figure 15 · p. 19, §B.2 · p. 19

C.2 INFERENCE SETTING

  • VideoChatGPT and Video-LLaMA2 responses that do not follow the option-letter instruction are evaluated with GPT-3.5 as a judge.For most other models, the authors extract the option letter using rule-based matching.Source: §C.2 · p. 21

D EVALUATION RESULTS ON VNBENCH-LONG

VNBench-Long tests retrieval and ordering over 10–30-minute videos, and Gemini 1.5 Pro maintains stable performance through 30 minutes. VNBench-Act extends needles to short video clips, making ordering more challenging.

D EVALUATION RESULTS ON VNBENCH-LONG

  • Gemini 1.5 Pro maintains stable performance as VNBench-Long videos extend to 30 minutes.The benchmark has four tasks—Retrieval-Edit, Retrieval-Insert, Ordering-Edit, and Ordering-Insert—and its 10–30-minute videos exceed most evaluated models’ context windows.Source: Figure 16 · p. 22, D EVALUATION RESULTS ON VNBENCH-LONG · p. 22

E EVALUATION RESULTS ON VNBENCH-ACT

F.1 CIRCULAR EVALUATION

Circular evaluation across shuffled options reduces inference randomness, while the needle-content and sample-size analyses report robustness to both changes. The appendix also specifies the model-training setup and provides complete results and generated task examples.

F.1 CIRCULAR EVALUATION

  • Increasing circular-evaluation iterations from one to four reduces inference randomness and supports more accurate and fair evaluation.The analysis focuses on Gemini 1.5 Pro and GPT-4o; Table 7 reports task accuracy across one to four iterations.Source: §F.1 · p. 23, Table 7 · p. 23

F.2 NEEDLE CONTENT

  • Replacing fruit-image needles with animal images has minimal impact on results across five tasks, supporting robustness to needle content.Table 8 compares task-result vectors before and after replacement using correlation coefficients.Source: §F.2 · pp. 23–24, Table 8 · p. 24

F.3 SAMPLE NUMBER

  • Results on the original 1,350-sample benchmark correlate relatively highly with results on a 2,700-sample version, indicating robustness to the current sample size.The expanded benchmark follows the same methodology and was tested on five models.Source: §F.3 · p. 24, Table 9 · p. 24

G TRAINING SETTING IN MODEL ANALYSIS

H COMPLETE RESULTS ON VNBENCH

  • Table 11 presents complete VNBench results for three tasks with three splits each, covering three proprietary and nine open-source models.Source: Table 11 · p. 26

I TASK SAMPLES OF VNBENCH

J CONSISTENCY EVALUATION

Circular evaluation distinguishes a model’s ability to answer from its consistency under shuffled answer options. The authors caution that low accuracy on unsolvable tasks is not a meaningful basis for model comparison.

J CONSISTENCY EVALUATION

  • Circular evaluation shuffles options across four iterations; Consist. measures unchanged answer content, and Acc. measures accuracy under this evaluation.Acc.|Consist. is the accuracy conditional on consistent predictions.Source: J CONSISTENCY EVALUATION · p. 32
  • For Counting-E1 and Counting-I, Acc.|Consist. substantially above 25% makes accuracy useful for reflecting task ability.Source: J CONSISTENCY EVALUATION · p. 32
  • When Acc.|Consist. is around or below 25% on Counting-E2, accuracy comparisons have limited significance because models are unable to solve the task.The authors describe this as a common issue across models.Source: J CONSISTENCY EVALUATION · p. 32
  • On tasks models can solve, GPT-series Consistency Rate is high; on unsolvable tasks it is low, reaching 5.3% for GPT-4o and 3.6% for GPT-4-turbo.The authors associate the latter pattern with sensitivity to option order and report 0.0% circular-evaluation accuracy for these models.Source: J CONSISTENCY EVALUATION · p. 32
  • The authors conclude that high accuracy can reflect model ability, whereas low accuracy indicates a shared difficulty; low-accuracy comparisons are therefore not significant.Source: J CONSISTENCY EVALUATION · pp. 32–33

Related papers

Loading 2406.09367v3…