Source-linked AI summary
Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction
Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, Jiaqi Wang
TL;DR
Offline video LLMs cannot provide continuous interactive feedback, and VideoLLM-online blocks perception during response generation. Dispider disentangles perception, decision, and reaction into asynchronous modules, and experiments report stronger performance across streaming and conventional video QA settings.
Problem
Offline video LLMs process entire videos before answering, while VideoLLM-online cannot perceive and generate responses simultaneously during streaming.
Method
Dispider uses asynchronous perception, decision, and reaction modules with proactive streaming processing and specialized training for response and silence decisions.
Results
Dispider outperforms existing methods across streaming and conventional video QA benchmarks in proactive response capabilities, temporal awareness, reasoning, and computational efficiency.
Takeaways & Limitations
The disentangled architecture supports timely, contextually accurate, computationally efficient interaction over long-duration video streams.
Takeaways & Limitations
VideoLLM-online’s single-LLM design alternates perception and reaction, delaying video input during answer generation.
Abstract
from arXiv · showhide
Active Real-time interaction with video LLMs introduces a new paradigm for human-computer interaction, where the model not only understands user intent but also responds while continuously processing streaming video on the fly. Unlike offline video LLMs, which analyze the entire video before answering questions, active real-time interaction requires three capabilities: 1) Perception: real-time video monitoring and interaction capturing. 2) Decision: raising proactive interaction in proper situations, 3) Reaction: continuous interaction with users. However, inherent conflicts exist among the desired capabilities. The Decision and Reaction require a contrary Perception scale and grain, and the autoregressive decoding blocks the real-time Perception and Decision during the Reaction. To unify the conflicted capabilities within a harmonious system, we present Dispider, a system that disentangles Perception, Decision, and Reaction. Dispider features a lightweight proactive streaming video processing module that tracks the video stream and identifies optimal moments for interaction. Once the interaction is triggered, an asynchronous interaction module provides detailed responses, while the processing module continues to monitor the video in the meantime. Our disentangled and asynchronous design ensures timely, contextually accurate, and computationally efficient responses, making Dispider ideal for active real-time interaction for long-duration video streams. Experiments show that Dispider not only maintains strong performance in conventional video QA tasks, but also significantly surpasses previous online models in streaming scenario responses, thereby validating the effectiveness of our architecture. The code and model are released at \url{https://github.com/Mark12Ding/Dispider}.
1 The Chinese University of Hong Kong 2 Shanghai AI Laboratory
Dispider is presented as a system for active real-time interaction with streaming video, contrasting with VideoLLM-online’s single-model design.
- Dispider separates streaming perception, response decisions, and asynchronous response generation into distinct modules.The perception module monitors continuously, the decision module determines when to trigger interaction, and the reaction module generates detailed responses.
1. Introduction
Offline video LLMs wait for the entire video before answering, limiting their suitability for continuous interactive scenarios. Dispider addresses this through parallel perception, decision, and asynchronous reaction modules.
- Offline video LLMs process the entire video before generating a response, which is impractical when users expect continuous feedback.
- VideoLLM-online alternates perception and reaction because autoregressive decoding prevents its single LLM from handling both simultaneously.This blocking behavior delays video inputs during response generation, especially in long-duration streams.
- Dispider runs perception, decision, and reaction asynchronously, allowing continuous video processing while delivering timely responses.Its decision process remains uninterrupted by asynchronous reaction generation.
- Scene-based perception segments streams at scene boundaries, while interleaved scene features, historical responses, and decision tokens support response timing decisions.
- Dispider’s specialized streaming QA training covers both response-required and silence-appropriate situations across diverse streaming contexts.
- Dispider outperforms existing methods across streaming and conventional long-video benchmarks in proactive response, temporal grounding, reasoning, and efficiency.The reported evaluations include StreamingBench, EgoSchema, VideoMME, MLVU, and ETBench.
2. Related Work
Prior video LLM research largely targets offline processing, while streaming systems remain limited in long-context handling, latency, or interactive response generation. Dispider targets these combined requirements.
- Video LLMs face challenges from processing frame sequences alongside the language model’s context-length constraints.
- Dispider introduces a pipeline for streaming video inputs and real-time outputs, emphasizing long-context handling together with interactive responses.
- Streaming video understanding requires continuous processing, long-term temporal consistency, and interactive responses, but only a few works explore it.
- Existing systems variously lack efficient extended-stream handling, real-time interaction, or focus on offline rather than streaming processing.
3. Method
Dispider formulates real-time video dialogue as deciding when to respond from partial streams, then implements this with proactive processing and asynchronous interaction modules.
- 3.1. Problem Formulation: The task is to generate timely, accurate responses from video observed up to the current time rather than processing the entire stream.
- 3.1. Problem Formulation: At each timestamp, a decision function determines whether sufficient information exists for the reaction function to generate a response or remain silent.
- 3.1. Problem Formulation: The framework disentangles continuous Perception, response-timing Decision, and Reaction so the system can respond without waiting for the complete video.
- 3.2. Proactive Streaming Video Processing: Scene-based perception adaptively segments long streams into non-uniform clips using scene boundaries and visual-change detection.
- 3.2. Proactive Streaming Video Processing: The decision module combines clip features, question information, historical memory, and special decision tokens to determine whether to respond.
- 3.2. Proactive Streaming Video Processing: The ⟨ANS⟩ token marks response actions, while decision inputs exclude reaction-generated tokens so monitoring remains unblocked.
- 3.3. Asynchronous Interaction: The asynchronous interaction module uses the current query, prior answers, grounded clip features, and cosine-similarity retrieval for temporal multi-hop reasoning.
- 3.3. Asynchronous Interaction: Training aligns predicted and ground-truth clip relevance with KL divergence and includes positive and negative samples for response or silence.
4. Experiments
Dispider is evaluated on streaming, conventional long-video, and temporal-awareness benchmarks, where it generally outperforms prior online models and remains competitive offline. Its scene-based segmentation and disentangled architecture support temporal grounding, proactive responses, and non-blocking interaction.
- Streaming Video Understanding: Dispider significantly outperforms Flash-VStream and VideoLLM-online on StreamingBench, particularly in Proactive Output.Proactive Output requires determining the precise timing of responses while maintaining an internal state over relevant frames.
- Streaming Video Understanding: 25.3 is Dispider’s score on Proactive Output, while other streaming models fail this task.The comparison concerns questions posed at the beginning of the video, unlike offline evaluation after the video has played.
- Streaming Video Understanding: Dispider consistently outperforms VideoLLM-Online across all ETBench streaming metrics, with particularly notable improvements in temporal grounding.On dense video captioning and step localization, streaming achieves more precise temporal grounding and more comprehensive descriptions than the conventional setting.
- Streaming Video Understanding: Dispider gradually identifies clues and generates informative answers step by step on multi-step reasoning questions, whereas VideoLLM-online gives simpler scene or action descriptions.The example links “thirsty” with drinks appearing in the video and infers contextually appropriate actions.
- Streaming Video Understanding: Dispider’s disentangled architecture processes streaming input and generates responses simultaneously without blocking ongoing video perception.VideoLLM-online instead experiences interruptions in the video stream during answer generation.
- Conventional Video Understanding: On conventional long-video benchmarks, Dispider achieves a leading EgoSchema accuracy of 55.6 and competitive performance across varying video lengths.The reported benchmarks include EgoSchema, MLVU, and VideoMME; the authors associate performance across minutes-to-hours videos with efficient contextual processing.
- Conventional Video Understanding: Dispider captures timestamps on ETBench and achieves the highest F1 score in temporal video grounding and episodic memory subtasks without specialized time tokens.Dense video captioning and step localization further demonstrate accurate temporal grounding and precise visual perception.
- Ablation Study: Scene-based non-uniform clip segmentation yields superior results in both conventional and streaming settings compared with uniform segmentation.The authors attribute this to preserving more structural video information, facilitating model learning and more timely responses.
5. Conclusion
Dispider disentangles perception, decision, and reaction through asynchronous processing for active real-time video interaction. Its experiments evaluate streaming comparisons and special-token designs while reporting strong overall effectiveness.
- Dispider disentangles perception, decision, and reaction and uses asynchronous processing to support active real-time interaction.
- Figure 3 compares Dispider with VideoLLM-online in streaming video understanding using answer texts or silence states.
- Table 5 ablates the ⟨ANS⟩, ⟨TODO⟩, and ⟨SILENT⟩ special tokens used in the streaming processor and final LLM.
- The system continuously processes video while generating timely, contextually accurate, and precise responses through separate processing and interaction modules.
- Experiments cover conventional and streaming video QA benchmarks, evaluating proactive response capabilities, temporal awareness, and computational efficiency.