PaperlayerView exact revision on arXiv

Source-linked AI summary

SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding

Zhenyu Yang, Yuhang Hu, Zemin Du, Dizhan Xue, Shengsheng Qian, Jiahong Wu, Fan Yang, Weiming Dong, Changsheng Xu

arXiv:2502.10810v2cs.CV

TL;DR

Existing video benchmarks do not adequately test sustained temporal reasoning in long-context streaming video. SVBench addresses this gap with temporal multi-turn QA chains, temporal linkages, and a semi-automated annotation pipeline, while evaluations show GPT-4o leads and StreamingChat substantially improves over open-source LVLMs.

  • Problem

    Existing benchmarks focus on isolated single-instance inputs and do not measure temporally sequential reasoning throughout streaming video.

  • Method

    SVBench constructs temporal multi-turn QA chains and linkages across 1,353 streaming videos using a semi-automated annotation pipeline.

  • Results

    GPT-4o outperforms other evaluated models, while StreamingChat improves over top-performing open-source LVLMs by 9.41% in dialogue evaluation and 3.30% in streaming evaluation.

  • Takeaways & Limitations

    SVBench provides a challenging evaluation of streaming video understanding, and StreamingChat achieves comparable performance on diverse conventional vision-language benchmarks.

Abstract

from arXiv · show

Despite the significant advancements of Large Vision-Language Models (LVLMs) on established benchmarks, there remains a notable gap in suitable evaluation regarding their applicability in the emerging domain of long-context streaming video understanding. Current benchmarks for video understanding typically emphasize isolated single-instance text inputs and fail to evaluate the capacity to sustain temporal reasoning throughout the entire duration of video streams. To address these limitations, we introduce SVBench, a pioneering benchmark with temporal multi-turn question-answering chains specifically designed to thoroughly assess the capabilities of streaming video understanding of current LVLMs. We design a semi-automated annotation pipeline to obtain 49,979 Question-Answer (QA) pairs of 1,353 streaming videos, which includes generating QA chains that represent a series of consecutive multi-turn dialogues over video segments and constructing temporal linkages between successive QA chains. Our experimental results, obtained from 14 models in dialogue and streaming evaluations, reveal that while the closed-source GPT-4o outperforms others, most open-source LVLMs struggle with long-context streaming video understanding. We also construct a StreamingChat model, which significantly outperforms open-source LVLMs on our SVBench and achieves comparable performance on diverse vision-language benchmarks. We expect SVBench to advance the research of streaming video understanding by providing a comprehensive and in-depth analysis of current LVLMs. Our benchmark and model can be accessed at https://github.com/sotayang/SVBench.

1 INTRODUCTION

SVBench targets a gap in evaluating LVLMs on long-context streaming video, where models must sustain temporally linked, multi-turn reasoning as video content unfolds. It introduces a large annotated benchmark and reports that existing models remain weak on this setting, motivating StreamingChat.

  • Streaming video understanding requires dialogues to update with temporal flow without access to future content.
  • Existing benchmarks emphasize isolated single-turn questions and omit temporally sequential reasoning across related video clips and questions.
  • SVBench defines temporal multi-turn dialogue through QA chains and temporal linkages that require models to use current and historical video-dialogue context.
  • 1,353 videos from 6 streaming platforms contain 49,979 QA pairs and temporal dialogue paths designed to assess reasoning through time.The dataset averages 36.94 QA pairs per video, reported as the highest number among known video datasets.
  • Existing LVLMs are far from satisfactory on streaming video understanding, while StreamingChat improves dialogue evaluation by 9.41% and streaming evaluation by 3.30% over top-performing open-source LVLMs.StreamingChat also achieves comparable performance on conventional image and video benchmarks.

2 RELATED WORK

Prior video benchmarks assess LVLMs across established video-understanding settings, but many focus on isolated events or clips. SVBench instead frames evaluation around temporal dialogue paths and streaming video reasoning.

  • Recent video benchmarks provide standardized criteria for comparing LVLMs and encouraging progress in video understanding.
  • SVBench evaluates long-context streaming understanding through temporal dialogue paths that connect questions and video segments over time.
  • The framework combines video filtering, scene splitting, QA-chain construction, manual quality assessment, temporal-linkage identification, and path construction.
  • Many existing datasets use videos capturing single events, overlooking temporal dependencies in longer videos.

3 DATASET

SVBench is constructed as a streaming-video dataset centered on temporal multi-turn QA chains and linkages across successive video clips. Its semi-automated, manually refined pipeline targets coherent cross-clip reasoning and high-quality annotations.

  • 42,605 QA pairs from 1,153 videos form the training split, while 7,374 QA pairs from 200 videos form the evaluation split.
  • The dataset is built from diverse public video sources, filtered for quality and suitable length before scene-based segmentation.
  • Constructing QA Chains for Video Dialogues: Each QA chain represents consecutive multi-turn interactions over one video clip, generated with LVLM assistance and manually aligned to the video.
  • Quality Control: GPT-4 evaluates QA chains on seven dimensions plus an overall score, and chains below the 90-point threshold undergo further manual revision.
  • Identifying Temporal Linkages: Temporal linkages connect QA pairs in adjacent chains through shared entities, scenes, events, and categorized relations.
  • Linking QA Chains for Temporal Reasoning: Manual modifications preserve within-chain coherence, establish cross-chain logical links, and replace repetitive or simple questions with deeper ones.

4 STATISTICAL ANALYSIS

SVBench contains diverse long-form streaming videos and organizes questions into skill-focused categories. Its annotations emphasize multi-turn dialogue density and temporal structure.

  • Dataset Statistics: 1,353 videos from 6 distinct sources contain an average of 36.94 QA pairs per video, the highest number among existing datasets.Videos average more than 2 minutes, with 4.29 QA pairs per multi-turn dialogue and 8.61 multi-turn dialogues per video.
  • Video Categories: The videos span 12 primary categories and 36 subcategories, illustrating broad content diversity.
  • Question Categories: Questions are classified into 9 categories, each targeting a distinct LVLM capability such as intention inference, potentiality assessment, counterfactual reasoning, comparison, and event-centric analysis.

5 STREAMINGCHAT

StreamingChat is a streaming LVLM built on InternVL2 with a vision encoder, projector, and language model. It processes video frames at 1 FPS and is fine-tuned on sequential multi-turn, multi-image dialogue data.

  • Model Architecture: StreamingChat combines InternViT, an MLP projector, and InternLM2, extracting video-frame embeddings at 1 FPS.
  • Supervised Fine-Tuning Data: Its fine-tuning data represents temporal dialogue paths as sequential video segments followed by multiple dialogue rounds in a multi-turn, multi-image format.

6 EXPERIMENTS

The experiments evaluate LVLMs under dialogue and streaming settings designed to test long-context continuity and temporal reasoning. GPT-4o leads overall, while StreamingChat is the strongest open-source model and improves substantially over InternVL2.

  • Experimental Setup: The evaluation compares state-of-the-art open- and closed-source LVLMs in dialogue and streaming settings on SVBench.Dialogue evaluation preserves all preceding QA context, whereas streaming evaluation introduces probabilistic transitions between temporally linked QA chains.
  • Experimental Setup: Dialogue evaluation preserves preceding QA history through the video, while streaming evaluation uses an 80% jump probability for temporally linked subsequent QA chains.The streaming setup tests reasoning across different but related video segments and evolving temporal dependencies.
  • Overall Performance: GPT-4o achieves Overall Scores of 66.29 in dialogue evaluation and 58.17 in streaming evaluation, outperforming the other evaluated models.StreamingChat and MiniCPM-V 2.6 are the leading open-source models, with dialogue scores of 59.41 and 54.30 and streaming scores of 53.90 and 52.19, respectively.
  • Overall Performance: StreamingChat improves over InternVL2 by 28.79% in dialogue evaluation and 26.20% in streaming evaluation.The reported gains support the effectiveness of the SVBench training data for streaming video understanding.
  • Overall Performance: StreamingChat improves SVBench performance after fine-tuning, with slight decreases on MMBench and MMBench-Video but modest gains on VideoMME and MVBench.The results suggest improved streaming video understanding without compromising fundamental image and video comprehension skills.
  • Overall Performance: Streaming evaluation scores are consistently lower than dialogue scores because it requires integrating extended temporal contexts and dynamically evolving scenes.The comparison contrasts relatively stable dialogue inputs with streaming inputs that demand seamless processing over time.
  • Performance Analysis of Video Understanding Skills: Intention Inference and Potentiality Assessment are relatively strong skills, whereas Counterfactual Reasoning and Spatio-Temporal Speculation are comparatively difficult.StreamingChat outperforms other open-source LVLMs across all nine skills, while nearly all models score below 60 overall across these skills.
  • Ablation Study: Multi-turn QA generally improves METEOR and GPT4-Score compared with single-instance QA when previous answers provide additional context.The improvement is reported for both open-source and closed-source LVLMs.

7 CONCLUSION

SVBench benchmarks long-context streaming video understanding with diverse videos and temporal multi-turn QA, while StreamingChat provides a strong open-source model for this setting. The authors report that current LVLMs remain below human-level accuracy on streaming videos, motivating further model development.

  • Benchmark and Model: SVBench contains 1,353 streaming videos from 6 platforms and 49,979 annotated QA pairs for temporal multi-turn dialogues.The benchmark targets long-context streaming video understanding.
  • Benchmark and Model: StreamingChat significantly outperforms open-source LVLMs on SVBench and achieves comparable performance on diverse vision-language benchmarks.The model is developed alongside the benchmark to address streaming video understanding.
  • Conclusion: State-of-the-art LVLMs perform below human-level accuracy on streaming videos despite progress on single-instance video QA.The conclusion frames SVBench as a challenging evaluation for future streaming video understanding models.

A.1 DATA FILTERING

The dataset is filtered for sufficient duration, visual quality, motion characteristics, and scene diversity. Most retained videos are longer than one minute and are concentrated in the 60–240-second range.

  • Data Filtering: The filtering pipeline starts from diverse public video datasets and excludes videos shorter than one minute.The source collections include YT-Temporal-1B, YouCook2, ActivityNet, MovieChat, Panda-70M, and Ego4D.
  • Data Filtering: Videos with aesthetic scores below 4 are removed to retain content with higher visual quality.Open-Sora assigns the aesthetic scores used for filtering.
  • Data Filtering: Videos are filtered to optical-flow scores between 0.5 and 100 to control visual complexity and motion coherence.The optical-flow criterion is applied using Open-Sora.
  • Data Filtering: Scene detection retains videos containing 5 to 15 scenes, excluding content that is excessively monotonous or overly intricate.The pipeline also calculates average scene duration during filtering.
  • Video Length Distribution: 95.05% of the videos are longer than one minute, primarily ranging from 60 to 240 seconds.Figure 6 reports the distribution of video lengths in the dataset.

B.7 COMPARISON AND TREND ANALYSIS

The modification guidelines construct comparison and trend questions by relating adjacent QA chains through actions, quantities, people, objects, and events. These transformations target temporal, relational, counterfactual, and contextual reasoning skills.

  • Guideline Structure: The guideline organizes QA-pair modification into Action, Person, Object, Event, Environment, and Quantity modules.Each module provides methods for revising questions to evaluate diverse LVLM skills.
  • Action: For actions, adjacent-chain questions can probe sequence, purpose, feasibility, spatio-temporal context, counterfactual outcomes, or comparisons.The modifications include asking what happens before or after an action and how one action affects another.
  • Quantity: Quantity questions compare categories and can examine trends, periodicity, or regularity over time.One example compares the number of balls with the number of people on a field.
  • Person: Person-related questions distinguish whether adjacent chains involve different or identical people.Different people prompt relationship, position, and interaction questions, while identical people prompt emotional-state and identity questions.
  • Object: Object-related questions examine effects, counterfactual attribute changes, state evolution, and action trajectories.These modifications connect object properties to other events or objects and track changes across context or time.
  • Event: Event-related questions ask about what happens next, causal or chain-reaction relations, event categories, and impacts.These transformations deepen temporal and comparative reasoning across adjacent chains.

D PRIMARY EVALUATION METRICS

SVBench uses multiple evaluation metrics and compares them with GPT-4-based assessment to distinguish LVLM capabilities. It also examines how inference speed relates to video length.

  • The four primary metrics broadly agree with GPT-4 evaluation, supporting SVBench’s ability to distinguish different LVLM capability levels.The comparison is reported in Table 5.
  • GPT-4-based assessment is used as a comparison point for the benchmark’s metric-based evaluations.
  • Inference speed is evaluated in relation to video length as part of the benchmark’s primary metrics analysis.

E.1 BENCHMARK CASE

The benchmark case studies show that contextual information and cross-turn coherence remain challenging for current LVLMs. Even the strongest open-source model can overlook intent and relationships across questions.

  • MiniCPM-V 2.6 cannot infer the questioner’s intent from contextual information in the benchmark case study.It instead relies mainly on the current question and video segments.
  • MiniCPM-V 2.6 overlooks coherence and contextual links between questions when information from the current input is insufficient.

F.1 COMPARISON OF LLM-BASED EVALUATIONS WITH HUMAN EVALUATION

The additional evaluations compare model, human, and metric-based judgments across SVBench, while also examining efficiency, scaling, and model-design factors. Results show substantial gaps between models and humans, and performance varies with model size and language-model choice.

  • F.2 CORRELATION BETWEEN INFERENCE SPEED AND CONSUMPTION WITH VIDEO LENGTH: Inference time generally increases with video length, while some models’ resource consumption plateaus as input frames and memory reach preset limits.
  • F.3 ADDITIONAL EVALUATION RESULTS: Human performance significantly exceeds open-source and closed-source LVLMs across all metrics in both dialogue and streaming evaluations.Humans perform relatively better on Temporal Understanding than Informational Completeness.
  • F.4 IMPACT OF MODEL SIZE ON PERFORMANCE: Model size significantly affects SVBench performance across InternVL2 variants ranging from 2B to 76B parameters.
  • F.4 IMPACT OF MODEL SIZE ON PERFORMANCE: Changing the language model from Yi-based to Llama3-based while retaining InternViT-6B is associated with lower performance on SVBench and MVBench.
Loading 2502.10810v2…