PaperlayerView exact revision on arXiv

Source-linked AI summary

Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge

Haomiao Xiong, Zongxin Yang, Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Jiawen Zhu, Huchuan Lu

arXiv:2501.13468v1cs.CVcs.AI

TL;DR

Long-video understanding systems struggle with memory, multi-turn interaction, and real-time processing. StreamChat is a training-free framework that combines hierarchical memory and parallel scheduling, evaluated with the diverse, interactive StreamBench benchmark. It reports higher accuracy and faster processing across online and offline settings, while basic retrieval and VRAM scalability remain limitations.

  • Problem

    Existing video models struggle to process long sequences, support multi-turn dialogue, and maintain rapid, consistent performance in dynamic scenarios.

  • Method

    StreamChat uses training-free adaptability, hierarchical short-, long-, and dialogue-memory storage, and parallel scheduling for streaming video reasoning; StreamBench evaluates diverse interactive scenarios.

  • Results

    64.7% accuracy on StreamBench improves 8.3% over the previous best, while processing reaches 32 FPS and text-generation latency stays under 0.9 seconds.

  • Takeaways & Limitations

    StreamChat provides a real-time streaming video understanding framework with multi-round interaction and robust performance across online and offline settings.

  • Takeaways & Limitations

    Basic retrieval can occasionally produce incorrect responses, and tree-structured storage may become difficult to scale as video duration and complexity grow because of VRAM constraints.

Abstract

from arXiv · show

Recent advances in Large Language Models (LLMs) have enabled the development of Video-LLMs, advancing multimodal learning by bridging video data with language tasks. However, current video understanding models struggle with processing long video sequences, supporting multi-turn dialogues, and adapting to real-world dynamic scenarios. To address these issues, we propose StreamChat, a training-free framework for streaming video reasoning and conversational interaction. $\StreamChat$ leverages a novel hierarchical memory system to efficiently process and compress video features over extended sequences, enabling real-time, multi-turn dialogue. Our framework incorporates a parallel system scheduling strategy that enhances processing speed and reduces latency, ensuring robust performance in real-world applications. Furthermore, we introduce StreamBench, a versatile benchmark that evaluates streaming video understanding across diverse media types and interactive scenarios, including multi-turn interactions and complex reasoning tasks. Extensive evaluations on StreamBench and other public benchmarks demonstrate that StreamChat significantly outperforms existing state-of-the-art models in terms of accuracy and response times, confirming its effectiveness for streaming video understanding. Code is available at StreamChat: https://github.com/hmxiong/StreamChat.

1 INTRODUCTION

Existing video-understanding models struggle with long sequences, multi-turn dialogue, and real-time interaction. STREAMCHAT addresses these gaps with hierarchical memory, parallel scheduling, and STREAMBENCH, reporting improved accuracy and efficiency.

  • Motivation: Offline models lose information in extended videos and generally support only single-turn dialogue, limiting interactive understanding.These models also face computational bottlenecks when compressing long video features within limited memory.
  • Motivation: Previous online models enable multi-round interaction but still struggle with rapid processing and consistent performance in unfamiliar scenarios.These limitations matter for real-time applications such as robotic navigation and human-robot collaboration.
  • STREAMCHAT: STREAMCHAT combines training-free adaptability, hierarchical memory, and parallel system scheduling for streaming video understanding.Its short-term, long-term, and dialogue memories manage ongoing events, compressed past events, and conversational history, while three parallel threads process frames, memory, and responses.
  • STREAMBENCH: STREAMBENCH evaluates streaming understanding with multi-turn dialogues, diverse question formats, and varied video content designed to expose shortcomings of existing benchmarks.The benchmark targets both accuracy and latency in realistic interactive scenarios.
  • Results: 64.7% accuracy on STREAMBENCH improves over the previous best by 8.3%, while offline performance exceeds the state-of-the-art by 2.5% on average across four public benchmarks.These results are reported for online and offline settings respectively.
  • Results: 32 FPS processing speed is six times higher than existing methods, with text-generation latency remaining below 0.9 seconds.These measurements characterize STREAMCHAT’s interactive processing efficiency.

2 COLLECTION AND COMPOSITION OF STREAMBENCH

STREAMBENCH is constructed to evaluate online video understanding across diverse video sources, lengths, and interaction types. Its semi-automated curation and annotation pipeline produces a 306-video benchmark with six task formats.

  • Benchmark scope: STREAMBENCH targets online scenarios with four major domains, sixteen video subclasses, and broader video-length coverage than primarily offline benchmarks.The benchmark is designed to simulate online video understanding rather than simultaneous access to all frames and questions.
  • Video collection: The collection draws from EgoSchema and YouTube-8M to provide egocentric, web, work-related, and cinematic video content.Source selection prioritizes diversity in video type and length.
  • Video collection: Machine classification and human review remove redundant, static, noisy, or otherwise unsuitable videos from the benchmark.Human judgment specifically checks scene changes and filters content such as advertisements and video-game footage.
  • Benchmark composition: 306 videos totaling 24.8 hours, averaging 4.5 minutes each, form the final StreamBench collection.The collection spans different categories and lengths.
  • Tasks: Six annotated tasks simulate agent-human conversation across object search, long-term memory, short-term memory, conversational interaction, knowledge-based QA, and simple factual questions.Their timing constraints probe retrieval of recent or delayed events and use of conversational history.
  • Annotation: Manual annotation feedback checks question diversity, clarity, video consistency, and removal of sensitive topics.Together with multimodal-model-assisted collection, this forms a semi-automated benchmark construction pipeline.

3 STREAMCHAT

StreamChat combines selective frame stacking, hierarchical memory formation, contextual summarization, and concurrent system scheduling for streaming video understanding and multi-round interaction. Its memory structures organize, compress, and retrieve visual and dialogue information while supporting real-time responses.

  • Hierarchical memory storage: Hierarchical memory combines long-term, short-term, and dialogue memories to manage visual and conversational data over extended video sequences.Long-term and short-term memories are constructed along the video timeline, while dialogue memory stores conversation history.
  • System overview: StreamChat uses selective frame stacking, memory formation, and contextual summarization as its three main processing components.The components prepare vision features, organize them into structured memory, and retrieve relevant context for user queries.
  • Selective frame stacking: Selective frame stacking uses optical-flow motion intensity to encode and buffer frames whose motion exceeds a predefined threshold.The module computes motion between consecutive frames, then pushes qualifying vision embeddings into the vision buffer.
  • Memory formation: Short-term memory randomly selects vision embeddings using normalized forgetting probabilities, while long-term memory stores clustered visual features and text clues.Text clues index long-term memory retrieval, and long-term nodes are organized into a tree structure.
  • Contextual summarization: Dialogue memory pre-encodes each question-answer pair, and retrieval searches long-term clues and dialogue history for context relevant to a new query.Long-term retrieval uses cosine similarity, while dialogue retrieval uses a FAISS index.
  • System overview: Independent threads execute the three components concurrently to optimize inference speed and minimize latency.Scheduling allows selective frame stacking, memory formation, and contextual summarization to operate without interference.
  • Efficiency: StreamChat processes video at 32 FPS and maintains text-generation latency below 0.9 seconds.The reported design supports real-time video processing and sub-second response latency.

4 EXPERIMENTS

Experiments evaluate StreamChat’s memory configurations, efficiency, streaming benchmark performance, offline performance, and ablations. Results show gains from hierarchical memory and tradeoffs among accuracy, latency, processing speed, and memory usage.

  • Experimental Setup: StreamChat uses Base, Fast, and Slow configurations that vary threshold, chunk length, group size, and clustering goals to balance speed and response accuracy.The Fast variant prioritizes rapid processing, while Slow prioritizes response accuracy.
  • Evaluation: StreamChat is evaluated on StreamBench and offline benchmarks using streaming RPD, score, and accuracy metrics.The experiments compare online and offline video-understanding performance across multiple tasks and datasets.
  • Hierarchical Memory Ablation: Adding dialogue memory improves CI performance by 4.1%, long-term memory improves LM by 6.2%, and short-term memory improves SM by 3.2%.Combining long-term and short-term memory increases average accuracy by 0.9%.
  • Parameter Tradeoffs: Increasing threshold to 0.55 reaches 32 FPS, but higher speed reduces performance from 64.0% to 60.7%.Larger chunk lengths, group sizes, and clustering dimensions improve performance but increase latency or VRAM use.

5 RELATED WORK

Related work spans multimodal language models, streaming video understanding, and retrieval-augmented generation. These approaches address video compression, online processing, memory-based interaction, and real-time information access, but streaming systems still face speed and generalization challenges.

  • Multimodal Language Models: Multimodal language models extend image-language methods to video, while video processing research focuses on compressing visual information within limited context windows.Examples include mapping layers, visual instruction tuning, K-NN clustering, and token reduction.
  • Streaming Video Understanding: Streaming video understanding requires responses from potentially indefinite video streams, motivating online models that process current and past frames rather than uniformly sampled clips.Existing online models still struggle with slow processing speeds and inadequate generalization.
  • Retrieval-Augmented Generation: Retrieval-augmented generation combines retrieval with text generation, while MemoryBank stores conversations and retrieves relevant context to support coherent dialogue.These methods target knowledge retention and real-time information access.

6 CONCLUSION

The paper introduces StreamBench and StreamChat to evaluate and perform streaming video understanding across diverse, interactive scenarios. It reports real-time processing and robust online and offline performance, while identifying retrieval accuracy and VRAM scalability as limitations.

  • Contributions: StreamBench broadens evaluation to diverse video lengths and types with six question formats simulating real-world human-robot interactions.The benchmark is designed for complex and dynamic streaming scenarios.
  • Contributions: StreamChat treats video frames as compressible, storable units managed through hierarchical memory and uses system scheduling for real-time processing.The method is training-free and targets efficient streaming video understanding.
  • Limitations: The retrieval algorithm can produce incorrect responses, while tree-structured storage may limit scalability as video duration and complexity increase.The authors propose finer-grained retrieval and more efficient or adaptive compression as future directions.

A DATA PIPELINE

The StreamBench data pipeline uses automated classification, human judgment, and annotation checking to prepare videos. Classification prompts map videos to dataset-specific category options before outputs are saved and used for category assignment.

  • Pipeline Stages: The StreamBench collection pipeline has three stages: classification, human judgment, and annotation checking.The first filtering step uses a multimodal large language model for video classification.
  • Classification: The classification prompt requires a category prediction and score in a Python-dictionary-style output containing the keys pred and score.The prompt prohibits additional explanatory text.
  • Dataset-Specific Categories: All_Class denotes the available category options, which change across datasets such as YouTube-8M and EgoSchema.The pipeline saves classification outputs to JSON and later retrieves categories as needed.

B MORE VISUALIZATIONS

STREAMCHAT is visualized across egocentric, web, working, and movie videos, with annotations spanning interactive questions and memory-based retrieval. Its retrieval algorithm selects high-similarity knowledge from a tree and combines it with the user request for response generation.

  • STREAMCHAT visualizations cover egocentric, web, working, and movie video analysis.
  • Egocentric examples use six questions, beginning with Simple Start and varying the remaining five across object identification, memory recall, and knowledge-based interactions.
  • The retrieval process computes similarity between the user request and memory captions, then follows the best-matching path through the tree.
  • The selected long-term knowledge and contextual information are reconstructed with the user request as a mixed prompt for the multimodal language model.

D DETAILS OF METRICS

The evaluation measures semantic correctness, cross-turn coherence, and request processing delay. These metrics respectively assess answer quality, consistency across dialogue turns, and the time from completed input to response generation.

  • Score and Accuracy: Semantic score and accuracy evaluate single-turn answer correctness using LLaMA-3 8B Instruct as the scoring model.The scoring model assigns a score from 0-5 and evaluates semantic correctness against the reference answer.
  • Score and Accuracy: The evaluation prompt combines the question, reference answer, and model response before scoring semantic correctness.
  • Coherence: Coherence averages the absolute differences between semantic scores of consecutive dialogue turns within one scenario.A smaller coherence score indicates a better coherence experience.
  • Request Processing Delay: Request processing delay measures the time from user input completion until the model starts generating its response.Generation delay is treated separately and depends on context length, model parameters, and deployment methods.

E FAILURE CASE AND ANALYZE

The failure analysis groups observed problems around object search, long-term memory, short-term memory, and conversational interaction tasks. The reported cases are used to explain why these failures occurred.

  • Testing revealed failures mainly in object search, long-term memory, short-term memory, and conversational interaction tasks.
  • The analysis presents failure cases and explains their causes rather than reporting only aggregate performance.
  • Temporal fine-grained failures occur when queried objects or events appear too briefly or sporadically for reliable identification.The reported example involves a small candle that appears infrequently.
  • Spatial fine-grained failures occur when target objects are small or blend into the background, hindering detection and localization.The examples include a porcelain bowl and a red cup that are small relative to the foreground.

F MODEL SELECTION AND DEPLOYMENT

The deployment discussion identifies long-form video handling and robustness to prompt variations as important model characteristics. STREAMCHAT is integrated with LongVA and deployed using two GPUs for tensor-parallel scheduling, while benchmark scale remains under expansion.

  • Model Selection: Long-form video understanding is required because memory-compressed information can still produce relatively long retrieved sequences.
  • Model Selection: Robustness to prompt variations is needed to maintain accurate and coherent multi-turn conversations when memory information changes the prompt wording.The stated concern is inconsistency or hallucination in model outputs.
  • Deployment: STREAMCHAT extends LongVA to streaming video processing and multi-turn conversations while preserving the cited model characteristics.
  • Deployment: The deployment uses 2 GPUs so tensor parallelism can distribute computational load during system scheduling.
  • Future Expansion: The authors plan to expand StreamBench to thousands of diverse videos and annotations at the order of ten thousand.
Loading 2501.13468v1…