PaperlayerView exact revision on arXiv

Source-linked AI summary

RIVER: A Real-Time Interaction Benchmark for Video LLMs

Yansong Shi, Qingsong Zhao, Tianxiang Jiang, Xiangyu Zeng, Yi Wang, Limin Wang

arXiv:2603.03985v1cs.CV

TL;DR

Online video interaction requires models to process streaming inputs while maintaining temporal-aware dialogue, yet existing benchmarks and offline paradigms do not adequately evaluate this setting. The paper introduces RIVER Bench, a precisely annotated benchmark spanning retrospective memory, live perception, and proactive response, together with an interactive training dataset. Across model categories, offline models perform well with full-video context but struggle in strict real-time processing, while fine-tuning on the proposed data improves online video comprehension; the benchmark currently excludes audio.

  • Problem

    Existing benchmarks inadequately evaluate multimodal large language models’ temporal-aware interaction with streaming video, despite the importance of online applications.

  • Method

    RIVER Bench defines precisely timed online interactions and evaluates retrospective memory, live perception, and proactive response using rigorously selected and annotated video data.

  • Results

    Offline models excel at single-question answering with full-video context but struggle in strict real-time processing, while the proposed training dataset substantially improves existing models’ online comprehension.

  • Takeaways & Limitations

    RIVER Bench identifies limitations in real-time interaction and supports improved online video understanding through specialized fine-tuning data.

  • Takeaways & Limitations

    The dataset currently excludes audio, although the paper identifies audio integration as important for real-time interaction.

Abstract

from arXiv · show

The rapid advancement of multimodal large language models has demonstrated impressive capabilities, yet nearly all operate in an offline paradigm, hindering real-time interactivity. Addressing this gap, we introduce the Real-tIme Video intERaction Bench (RIVER Bench), designed for evaluating online video comprehension. RIVER Bench introduces a novel framework comprising Retrospective Memory, Live-Perception, and Proactive Anticipation tasks, closely mimicking interactive dialogues rather than responding to entire videos at once. We conducted detailed annotations using videos from diverse sources and varying lengths, and precisely defined the real-time interactive format. Evaluations across various model categories reveal that while offline models perform well in single question-answering tasks, they struggle with real-time processing. Addressing the limitations of existing models in online video interaction, especially their deficiencies in long-term memory and future perception, we proposed a general improvement method that enables models to interact with users more flexibly in real time. We believe this work will significantly advance the development of real-time interactive video understanding models and inspire future research in this emerging field. Datasets and code are publicly available at https://github.com/OpenGVLab/RIVER.

1 INTRODUCTION

RIVER Bench addresses the lack of benchmarks for dynamic online multimodal interaction by evaluating retrospective memory, live perception, and proactive response with precisely timed interactions. Evaluations show that offline models perform well on single-question answering but remain limited in strict real-time processing, while the proposed training data improves online comprehension.

  • Motivation: Online multimodal interaction requires temporal-aware dialogue over streaming visual inputs, but existing benchmarks inadequately evaluate these dynamic requirements.The gap affects applications such as augmented-reality navigation and robotic task supervision.
  • RIVER Bench: RIVER Bench evaluates retrospective memory, live perception, and proactive response through temporal feedback, latency-accuracy tradeoffs, and future-event prediction.Retro-Memory analyzes performance decay across temporal intervals; Pro-Response jointly models event-forecasting confidence and temporal localization precision.
  • RIVER Bench: The benchmark reconstructs data from diverse video datasets into online interactions with explicitly defined timing, content, and reference points.Both manual and automated methods were used to select and validate the data.
  • Findings: Offline models excel at single-question answering with full-video context but remain severely limited in strict real-time video processing.The evaluation compares vanilla offline, sliding-window adapted, existing online, and fine-tuned models.
  • Findings: The proposed interactive training dataset substantially improves existing state-of-the-art models’ real-time comprehension in complex streaming environments.The paper also incorporates a long-short term memory module to dynamically preserve visual information.

2 RELATED WORKS

Prior work extends offline video models or introduces online interaction mechanisms, but online-video benchmarks have not consistently captured interactivity, temporal grounding, and response accuracy. RIVER Bench organizes these requirements around retrospective memory, live perception, and proactive anticipation.

  • Online Interaction: Offline video models extend frame capacity but lack effective memory mechanisms for sustained online interaction.The related work contrasts large-context offline comprehension with the need to retain information over time.
  • Online Interaction: Online systems such as VideoLLM-Online and MMDuet use specialized training data to support proactive interaction and response timing.VideoLLM-Online predicts special tokens that determine when the model should respond.
  • Online Interaction: Some multimodal systems add voice input or memory modules, but their designs can involve less streamlined multi-component integrations.IXC2.5-OL selectively retains relevant information while integrating multiple voice components.
  • Online Video Benchmarks: Existing online-video benchmarks often retain traditional evaluation formats and do not precisely define interactive task types or temporal requirements.This limits their ability to capture the real-time nature of online video understanding.
  • Online Video Benchmarks: MovieChat-1K supports timestamped immediate-response questions, while StreamingBench adds timestamp-linked multimodal and contextual evaluation tasks.MovieChat-1K nevertheless lacks a comprehensive framework for recalling or awaiting responses.

3 RIVER BENCH

RIVER Bench formulates online video comprehension as a window-based interactive task spanning retrospective memory, live perception, and proactive response. It defines temporally grounded data and evaluates these capabilities with task-specific procedures and metrics.

  • Formulation: RIVER Bench models online interaction as a window-based video-text-to-text task involving streaming video, user queries, responses, and historical model state.The formulation permits multiple end-of-sequence tokens to simulate silence or pauses in live conversation.
  • Task Types: The benchmark distinguishes Retro-Memory, Live-Perception, and Pro-Response according to whether the queried event is past, current, or future.Retro-Memory uses history, Live-Perception concerns current or short-term inputs, and Pro-Response monitors the stream until a specified future condition occurs.
  • Data Construction: RIVER constructs temporally grounded questions by specifying query, cue, and response times and curating data from diverse video datasets.Filtering combines language-model screening, human evaluation, and semantic-similarity procedures to remove visually answerable failures and ambiguous event descriptions.
  • Retro-Memory: Retro-Memory evaluates recall across short, medium, long, and very long intervals, ranging from 15–30 seconds to 1800–3600 seconds.Each question is designed so its answer depends on one specific video moment.
  • Live-Perception: Live-Perception tests current-window understanding of environments, actions, object attributes, dynamic changes, and static visual semantics.The questions target visual information available within a window typically lasting a few seconds.
  • Pro-Response: Pro-Response includes instant and stream questions, requiring either one timely answer or continuous real-time descriptions and guidance.Unlike post-video dense captioning, stream questions require outputs at different timestamps during conversational interaction.
  • Metrics: The benchmark evaluates retrospective and live answers with accuracy procedures, while Pro-Response uses temporal alignment to ground-truth timestamps within a tolerance window.The proactive metric gives full credit inside the acceptable window and penalizes early responses with zero.

4 EXPERIMENTS

The experiments evaluate online video understanding across model categories and test a sliding-window, long-short-term-memory adaptation for offline models. Results show substantial real-time limitations in offline and existing streaming models, while memory mechanisms and benchmark-specific fine-tuning improve targeted capabilities.

  • Evaluation Setup: The evaluation covers commercial, native-online, open-source video, and online-adapted MLLMs, using recommended sampling strategies for videos up to 120 minutes.GPT-4o and Gemini-1.5-pro use 50 frames, several open-source models use 16 frames, and native-online models process video at 4 fps.
  • Making Offline Models Work Online: Typical offline inference processes the entire video before answering, limiting online interaction; the proposed adaptation combines sliding-window sampling with long-short-term memory.The implementation uses 1 fps windows, current-window tokens as short-term memory, and compressed earlier-frame tokens in fixed-length long-term memory slots.
  • Evaluation Results: GPT-4o achieves the best performance across live-perception, retro-memory, and pro-response tasks, while offline models remain strong mainly on single-question answering.The table reports frame counts and pro-response localization alongside core online capability results.
  • Evaluation Results: Fine-tuning VideoLLM-Online on RIVER Bench pro-response data improves accuracy by 11.28% over baseline, demonstrating gains especially for proactive response.Existing streaming models otherwise underperform substantially, partly because of offline training and optimization for long-video comprehension rather than interactive QA.
  • Model Memory Capability: As recall duration increases, most models show declining memory retrieval and reasoning, whereas online-inference modifications improve medium-to-long-term memory performance.Flash-VStream maintains consistent accuracy across durations despite modest overall performance.
  • Model Memory Curve: Adding memory modules reduces the recall-performance decay slope by 12% compared with models without memory, with greater retention stability within one hour.The result differs from the classic Ebbinghaus forgetting curve and suggests distinct retention behavior in equipped MLLMs.

5 CONCLUSION

RIVER Bench evaluates online multimodal interaction through retrospective memory, live perception, and proactive response, and identifies real-time limitations while supporting improvement through specialized fine-tuning. The benchmark currently excludes audio data, which the authors plan to incorporate.

  • RIVER Bench quantifies retrospective memory, live perception, and proactive response to identify limitations in real-time multimodal interaction.
  • The dataset currently excludes audio, although the authors identify audio integration as crucial for evaluating real-time video interaction.

ETHICS STATEMENT

The benchmark uses publicly available or open-dataset videos selected under applicable licenses and excludes private, sensitive, or personally identifiable information. It focuses on general video understanding rather than sensitive human activities.

  • Videos come from publicly available platforms and open datasets and are selected according to their licenses and terms of use.
  • The benchmark excludes private, sensitive, or personally identifiable information and omits clinical, offensive, and invasive content.

REPRODUCIBILITY STATEMENT

The authors describe the benchmark’s sources, selection criteria, task construction, evaluation metrics, and experimental setups to support reproducibility. Code and related implementation details are documented across the paper and appendix.

  • The paper and Appendix A document video sources, selection criteria, dataset composition, and duration and source distributions.
  • Section 3 details construction of the Retrospective Memory, Live-Perception, and Proactive Anticipation tasks.
  • Section 4 specifies evaluation metrics and experimental setups for all compared models.

A.1 BENCHMARK DETAILS

RIVER Bench combines QA-based and dense-action video data, filters and reconstructs annotations for online interaction, and generates proactive questions from timestamped events using templates and distractors.

  • RIVER Bench contains QA pairs with visual information and videos annotated with dense actions.
  • QA data are filtered by removing personal names, excessively long segments, overly general questions, and items answerable without visual input.
  • Dense event annotations are deduplicated, key events are selected using sentence-embedding similarity, and an LLM generates QA pairs from templates.
  • Retro-Memory and Live-Perception questions are reconstructed from Vript-RR, LVBench, and LongVideoBench after filtering and quality checks.
  • Query timestamps are sampled after an event for Retro-Memory or synchronized with it for Live-Perception.
  • For Proactive Anticipation Instant questions, timestamped dense events are supplied to an LLM that generates questions, correct answers, and plausible distractors.
  • The question-generation prompt instructs the assistant to provide concise real-time language assistance and generate three to five confusing wrong choices.
  • A template library is randomly sampled to synthesize Proactive Anticipation Instant questions.

A.2 PROACTIVE ANTICIPATION ABILITY

The section describes an online interaction model that combines long- and short-term visual memory, processes video in batches, and produces responses from combined embeddings. It also classifies visual questions into three cue categories for performance analysis.

  • Training configuration: The training parameters include flash_attention_2, a 0.0002 learning rate, adamw_torch optimization, and a cosine learning-rate scheduler.These settings are summarized in Table 6.
  • Memory mechanism: The long-short-term memory mechanism reduces redundant visual tokens through average pooling to limit memory use and preserve clip similarity calculations.The memory module stores long-term and short-term representations before model inference.
  • Question categorization: Visual questions are categorized into three types: video content and details, characters and events, and scenes and environments.The classification prompt requires selecting exactly one category for each question.
  • Memory mechanism: The memory pipeline initializes separate long- and short-term embedding banks, updates the long-term bank when it exceeds capacity, and merges the most similar pairs.Final embeddings combine the updated long-term bank with short-term embeddings before response generation.
  • Inference: The model formats each question with its options and generates a response using the final combined embeddings.

B DATA EXAMPLE

The examples illustrate RIVER-style data across multiple video sources, durations, question types, and interaction formats. They include recalling past events, responding to immediate or ongoing events, and anticipating what happens next, alongside rejected question designs.

  • Dataset instances: The examples cover LVBench and Ego4D videos with durations ranging from 74.5 to 3665.5 seconds and frame rates from 4 to 24 fps.
  • Interaction formats: Retrospective examples ask about details such as an opening caption, with answer choices and a time reference tied to the relevant video moment.
  • Interaction formats: Questions are labeled with interaction types such as Awaiting@short, Ongoing, Immediate, and Recalling@short.The examples associate these labels with event triggers, ongoing descriptions, immediate alerts, and retrospective questions.
  • Question curation: Rejected examples remove questions that are easy to guess, use overly long time spans, rely on answer-choice preferences, or ask overly broad whole-video content.
  • Interaction formats: Awaiting questions request alerts when specified actions occur and may ask what happened before or what will happen next.Examples include scrolling through a phone, interacting with a woman, and cleaning a bicycle.

C DATASET OPEN SOURCE PROTOCOL

The dataset uses an index-only release strategy that provides metadata, annotations, and download links without redistributing the original videos. This approach is intended to support reproducibility while complying with licensing restrictions.

  • Release strategy: The release provides indexes, metadata, annotations, and download links rather than redistributing video files.
  • Release strategy: The stated purpose is to respect original licenses, avoid video redistribution, and enable reproducible research with legal and ethical compliance.
Loading 2603.03985v1…