PaperlayerView exact revision on arXiv

Source-linked AI summary

StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding

Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, Maosong Sun

arXiv:2411.03628v1cs.CVcs.AI

TL;DR

Most MLLMs and video benchmarks focus on offline comprehension, leaving streaming understanding—where queries can arise during a video and audio, visual, and interaction context matter—less comprehensively evaluated. StreamingBench assesses these capabilities across 18 tasks; Gemini 1.5 Pro achieves 67.07% average accuracy, 24.59% below human performance. The analysis identifies real-time processing, context, proactive output, and continuous querying as persistent challenges.

  • Problem

    Most video benchmarks preload the entire video before queries, unlike streaming scenarios with midstream queries, synchronized audio and visual inputs, and interaction history.

    Source: §1 · pp. 1–2

  • Method

    StreamingBench evaluates streaming video understanding across 18 tasks grouped into real-time visual, omni-source, and contextual categories.

    Source: §1 · pp. 2–3

  • Results

    Gemini 1.5 Pro achieves 67.07% average accuracy on StreamingBench, 24.59% lower than human performance.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Models perform poorly in real-time processing and struggle with redundant context, proactive output, and continuous queries.

    Source: §1 · p. 3

    The GPT-4o API does not support video inputs.

    Source: §2 · p. 3

Abstract

from arXiv · show

The rapid development of Multimodal Large Language Models (MLLMs) has expanded their capabilities from image comprehension to video understanding. However, most of these MLLMs focus primarily on offline video comprehension, necessitating extensive processing of all video frames before any queries can be made. This presents a significant gap compared to the human ability to watch, listen, think, and respond to streaming inputs in real time, highlighting the limitations of current MLLMs. In this paper, we introduce StreamingBench, the first comprehensive benchmark designed to evaluate the streaming video understanding capabilities of MLLMs. StreamingBench assesses three core aspects of streaming video understanding: (1) real-time visual understanding, (2) omni-source understanding, and (3) contextual understanding. The benchmark consists of 18 tasks, featuring 900 videos and 4,500 human-curated QA pairs. Each video features five questions presented at different time points to simulate a continuous streaming scenario. We conduct experiments on StreamingBench with 13 open-source and proprietary MLLMs and find that even the most advanced proprietary MLLMs like Gemini 1.5 Pro and GPT-4o perform significantly below human-level streaming video understanding capabilities. We hope our work can facilitate further advancements for MLLMs, empowering them to approach human-level video comprehension and interaction in more realistic scenarios.

1 INTRODUCTION

Most video benchmarks evaluate offline comprehension, while streaming understanding requires answering midstream queries, integrating synchronized audio and video, and using interaction history. StreamingBench evaluates these capabilities across 18 tasks, and experiments reveal substantial gaps from human performance and weaknesses in real-time processing.

1 INTRODUCTION

  • Offline benchmarks preload the full video, whereas streaming questions can arise midstream and require synchronized modalities and interaction history.The only related benchmark identified, VStream-QA, covers 32 videos, five task types, visual content only, and independent questions.Source: §1 · p. 2
  • StreamingBench comprises 900 videos and 4,500 manually curated questions across 18 tasks and eight video categories.Each video has five questions designed for streaming scenarios.Source: §1 · p. 2
  • Its three task categories assess real-time visual understanding, synchronized audio-visual understanding, and contextual understanding across streaming interactions.Contextual tasks include detecting anomalies, filtering misleading information, maintaining continuity, and responding proactively.Source: §1 · pp. 2–3
  • Gemini 1.5 Pro reaches 67.07% average accuracy, 24.59% below human performance on StreamingBench.The evaluation converts each task into an offline input containing the video segment up to the query point.Source: §1 · p. 3
  • Current models perform poorly in real-time processing and struggle with redundant context, proactive output, and continuous queries.The authors suggest offline-video training may contribute to these weaknesses.Source: §1 · p. 3

2 RELATED WORK

Video MLLMs generally process sampled frames from entire videos, and thus focus on offline comprehension; streaming video models and GPT-4o have explored streaming inputs. The related benchmark VStream-QA has limited video and task coverage, motivating a more comprehensive streaming benchmark.

Video MLLMs

  • Video MLLMs commonly sample and encode frames from entire videos for language-model processing, a setup limited to offline understanding.Streaming video MLLMs include VideoLLM-online, Flash-VStream, and VideoLLM-MoD.Source: §2 · pp. 3–4

Video Understanding Benchmarks

  • VStream-QA assigns timestamps to questions but has limited video types and task designs for thorough streaming evaluation.The GPT-4o API does not support video inputs.Source: §2 · pp. 3–4

3 STREAMINGBENCH

StreamingBench organizes streaming video understanding into real-time visual, omni-source, and contextual tasks, each targeting a distinct aspect of continuous video streams. It combines timestamped questions with a manually curated video set and a hybrid QA-generation process.

3.1 TAXONOMY

  • The taxonomy separates streaming understanding into real-time visual, omni-source, and contextual categories, addressing queries during a stream, synchronized audio-visual content, and streaming context.Each category focuses on one of these distinctions and evaluates related capabilities.Source: §3.1 · p. 4

3.1.1 REAL-TIME VISUAL UNDERSTANDING

  • Real-time visual questions use timestamps and refer only to preceding video content; ten tasks assess abilities ranging from object perception and causal reasoning to counting.Questions are multiple choice with four options for deterministic evaluation.Source: §3.1.1 · p. 4

3.1.2 OMNI-SOURCE UNDERSTANDING

  • Omni-source tasks test real-time integration or temporal alignment of visual and audio content across emotion recognition, scene understanding, source discrimination, and multimodal alignment.Questions have timestamps and four options, and are designed to require both visual and audio understanding.Source: §3.1.2 · pp. 4–5

3.1.3 CONTEXTUAL UNDERSTANDING

  • Contextual tasks address misleading or anomalous information, sequential questions that require episodic memory, and proactive outputs triggered by predefined conditions.Proactive output questions specify when an output should occur; the current benchmark evaluates 50 of the 250 questions created for this task.Source: §3.1.3 · p. 5, Figure 2 · p. 6

3.2 DATA CONSTRUCTION

  • Data construction combines 900 manually curated videos across eight scenario categories with hybrid QA generation and human verification of every pair.The benchmark contains 4,500 questions; videos range from 3 seconds to 24 minutes, and low-quality or answerable-without-video questions are revised or discarded.Source: §3.2 · p. 6

4 EXPERIMENTS

The experiments assess 13 open-source and proprietary MLLMs on StreamingBench, then analyze performance across task types, clip lengths, clue timing, and contextual challenges.

4.1 SETTINGS

  • Most tasks use offline clips ending at query time; SQA adds prior QA pairs, while PO models are polled each second and judged within a two-second window.The evaluation uses accuracy for multiple-choice questions.Source: §4.1 · p. 7

4.2 RESULTS ON STREAMINGBENCH

  • Models generally perform well on real-time visual understanding but poorly on omni-source and contextual understanding tasks.The authors link the weaker areas to audio-dependent questions and to context such as streaming interactions or high-redundancy visual inputs.Source: §4.2 · p. 7
  • Gemini 1.5 Pro scored 67.07%, versus 91.66% for humans; all three proprietary models exceeded the best open-source model, LLaVA-OneVision (56.36%).Human evaluation used a randomly sampled 10% of questions.Source: §4.2 · p. 7, Table 2 · p. 8

4.3 MODEL PERFORMANCE ON DIFFERENT VIDEO LENGTHS

  • All three top open-source models perform worse overall on videos longer than 60 seconds, though Qwen2-VL does better on long videos for Causal Reasoning and Clip Summarization.The comparison divides videos into ≤60-second and >60-second groups.Source: §4.3 · p. 8, Table 3 · p. 9

4.4 MODEL PERFORMANCE ON TASKS WITH DIFFERENT TEMPORAL CLUES

  • Models perform better on prior clues than on concurrent or subsequent clues, while performance is not related to clue count.The authors suggest this pattern may reflect training datasets that underrepresent concurrent- and subsequent-type tasks.Source: §4.4 · p. 9, Figure 4 · p. 9

4.5 ANALYSES ON CONTEXUAL UNDERSTANDING TASKS

  • Removing redundant visual information improves four open-source models' MCU and ACU performance, indicating weak robustness to high-redundancy inputs.The analysis sampled 125 questions and removed misleading frames for MCU or retained only anomaly segments for ACU.Source: §4.5 · pp. 9–10, Table 5 · p. 10
  • Resolving references between sequential questions improves most models' performance, with MiniCPM-V 2.6 as the exception.The analysis rewrites references in the questions explicitly, such as replacing a reference to a team with its name.Source: §4.5 · p. 10, Figure 5 · p. 10
  • Relaxing the PO timing threshold rapidly improves accuracy, and directly querying at the target time also improves performance, indicating difficulty with timing precision and proactive output format.Figure 3 compares timing tolerances, while Figure 5 compares original PO prompts with transformed concurrent questions.Source: Figure 3 · p. 8, §4.5 · p. 10, Figure 5 · p. 10

5 CONCLUSION

StreamingBench assesses streaming video understanding with a broad benchmark, and experiments show that even Gemini 1.5 Pro falls significantly short of human-level performance.

5 CONCLUSION

  • Even the best-performing model, Gemini 1.5 Pro, falls significantly short of human-level streaming video understanding.The experiments evaluate 13 open-source and proprietary MLLMs.Source: §5 · p. 11
  • StreamingBench comprises 900 videos and 4,500 QA pairs across 18 tasks and three categories of streaming video comprehension.The categories assess real-time visual, omni-source, and contextual understanding.Source: §5 · p. 11
  • The authors analyze the performance gap and propose potential directions for improving MLLMs.They hope the work contributes to future AI systems for real-world scenarios.Source: §5 · p. 11

A MORE DETAILS OF EVALUATION

The evaluation settings impose model-specific constraints on frame sampling and video-frame capture.

A.1 MODEL INFERENCE SETTINGS

  • GPT-4o evaluation uses only 64 frames per video because extracting more frames causes many API access failures.The authors plan to try other methods for using more frames.Source: §A.1 · p. 13
  • Qwen2-VL frame rates vary by video length: 1 fps under 5 minutes, 0.5 fps for 5–10 minutes, and 0.2 fps above 10 minutes.For ACU image inputs, a video-like extraction strategy is used to reduce evaluation costs, based on the assumption that images embedded in video require fewer resources.Source: §A.1 · p. 13
  • MiniCPM-V 2.6 and InternVL-V2 may fail to capture the last few frames under their official inference strategies, which the authors assume may affect evaluation results.They plan to address the issue in future work.Source: §A.1 · p. 13

A.2 EVALUATION PROTOCALS

Evaluation generally supplies each model with video only up to the question timestamp, while separate prompting procedures handle ordinary, sequential, and proactive tasks.

A.2 EVALUATION PROTOCALS

  • For real-time visual, omni-source, ACU, and MCU questions, evaluation crops the video from its start through the question timestamp before model input.The ordinary multiple-choice prompt asks the model to choose the best of four options.Source: §A.2 · p. 13
  • The standard prompt presents a video question with four answer options and requests only the letter of the best choice.This prompt applies to tasks except SQA and Proactive Output.Source: §A.2 · p. 13
  • For SQA, each prompt includes earlier timestamps, questions, options, and ground-truth answers so the model can answer a sequence of related questions.Ground-truth answers stand in for prior model responses because the protocol assumes humans can correct mistakes during streaming conversations.Source: §A.2 · pp. 13–14
  • In PO tasks, evaluation polls at up to 10 timestamps: the query timestamp and every second in [-4,4] seconds, using the ground-truth timestamp as origin.Source: §A.2 · p. 14

B MORE DETAILS OF DATA CONSTRUCTION

StreamingBench samples diverse video scenarios and constructs timestamped questions using different procedures for visual, audio-dependent, and contextual tasks.

B.1 VIDEO SELECTION

  • The benchmark divides streaming scenarios into eight categories, including everyday life, competition, video games, education, TV shows, unusual events, documentaries, and animation and movies.Examples include travel vlogs, sports, lectures, news, accidents, and science documentaries.Source: §B.1 · pp. 14–15

B.2 QA GENERATION

  • Five timestamps per video serve as query points, placing questions throughout the stream rather than only at its end.Frames sampled at 1 fps are timestamped, and questions are tagged with the minimal video segment containing the relevant clues.Source: §B.2 · p. 15
  • For visual tasks, single-frame questions are generated directly from sampled images, while multi-frame questions use captions before question generation.Questions for audio-dependent omni-source and contextual tasks, except Proactive Output, receive manual annotation; Figure 6 depicts the automatic QA-construction pipeline.Source: §B.2 · p. 15, Figure 6 · p. 15

B.3 PROMPT FOR QA GENERATION

  • The question-generation prompt requests relevant, challenging multiple-choice QA pairs with four similarly formatted options and precise timestamps.It instructs the generator to avoid unsupported topics and contradictory or unreasonable caption content.Source: §B.3 · pp. 16–17
  • The caption prompt asks for objective, detailed descriptions of visible actions, objects, and relationships, then adds timestamps matching the sampled frames.Descriptions should not be frame-by-frame, should avoid uncertain details, and use timestamped ranges or single timestamps.Source: §B.3 · p. 16

C.1 RESULTS OF STREAMING VIDEO MLLMS

Flash-VStream and VideoLLM-online show substantial difficulties on StreamingBench, including poor Proactive Output results and problems following instructions. Examples also record when models responded relative to the expected output time.

C.1 RESULTS OF STREAMING VIDEO MLLMS

  • Flash-VStream has extremely low accuracy on most tasks, usually answering “A”; its polling-based Proactive Output results are similarly poor.The authors used polling for Proactive Output after official scripts took excessive processing time.Source: §C.1 · p. 18
  • VideoLLM-online cannot follow instructions accurately and generates substantial redundant information; streaming evaluation performs similarly to or worse than polling.The model accepts streaming video input, but the streaming evaluation strategy does not outperform polling in the reported results.Source: §C.1 · p. 18
  • 00:05:36: Flash-VStream responded “Yes” eight seconds before the 00:05:44 ground-truth timestamp.Source: §C.1 · p. 18

C.2 IMPACT OF VIDEO LENGTH ON STREAMING VIDEO UNDERSTANDING

Video length affects performance, though the reported effects vary by task; limiting context to 60 seconds generally improves results, while adding ASR generally helps. The verbosity evaluation measures response length at the first judged-correct answer.

C.2 IMPACT OF VIDEO LENGTH ON STREAMING VIDEO UNDERSTANDING

  • Video length affects performance: counting is substantially affected, while causal reasoning and clips summarization show no particularly significant differences.Source: §C.2 · pp. 19–20, Table 7 · p. 19
  • In an experiment limited to the 60 seconds preceding query time, overall performance generally improved versus the main experiment, though counting declined while MA and SD scores generally increased.Source: §C.2 · p. 20

C.3 IMPACT OF ADDITIONAL ASR INFORMATION ON STREAMING VIDEO UNDERSTANDING

C.4 DEGREE OF VERBOSITY OF THE MODELS

  • Verbosity is measured as the number of response tokens when an answer is first judged correct, across four open-source models.Source: §C.4 · p. 21, Table 10 · p. 21

C.5 DETAILS OF HUMAN EVALUATION

The human evaluation sampled questions from StreamingBench and constrained how participants could answer them.

C.5 DETAILS OF HUMAN EVALUATION

  • Five participants evaluated questions, with 10% of each task’s questions randomly selected.Source: §C.5 · p. 21
  • Each participant had only one chance to answer each question.Source: §C.5 · p. 21
  • Participants could not rewind or replay a video after watching it.Source: §C.5 · p. 21

D DATA EXAMPLES

The appendix provides examples spanning the benchmark’s task types and illustrates transformations used for proactive output and sequential question answering.

D DATA EXAMPLES

  • Figures 8 and 9 show examples for object, attribute, text-rich, causal, clip-summarization, and prospective reasoning tasks.Source: Figure 8 · p. 22, Figure 9 · p. 23
  • Figure 10 shows examples for event understanding, emotion recognition, and scene understanding.Source: Figure 10 · p. 24
  • Figure 11 shows examples for source discrimination, misleading-context understanding, and anomaly-context understanding.Source: Figure 11 · p. 25
  • Figures 12 and 13 show examples for multimodal alignment, sequential question answering, proactive output, spatial understanding, counting, and action perception.Source: Figure 12 · p. 26, Figure 13 · p. 27
  • Figures 14 and 15 illustrate transformations for proactive output and reference resolution in sequential question answering.Source: Figure 14 · p. 28, Figure 15 · p. 28
Loading 2411.03628v1…