PaperlayerView exact revision on arXiv

Source-linked AI summary

Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction

Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, Jiaqi Wang

arXiv:2501.03218v1cs.CV

TL;DR

Offline video LLMs process the full video before responding, limiting their fit for interactive streaming. Dispider separates perception, decision, and reaction into asynchronous modules, and experiments report stronger streaming interaction performance alongside strong conventional video QA results.

  • Problem

    Most video LLMs process the entire video before responding, while real-time streaming interaction calls for timely dialogue without waiting for the video to finish.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    Dispider disentangles perception, decision, and reaction into asynchronous modules so video processing continues while detailed responses are generated.

    Source: §1 · p. 2

  • Results

    Experiments report that Dispider outperforms previous online models in streaming interaction and existing methods across conventional and streaming video QA benchmarks.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Dispider’s disentangled architecture is presented as a solution for real-time interaction over long-duration video streams.

    Source: §5 · p. 9

    VideoLLM-online’s single LLM alternates between perception and response generation, blocking video processing during responses.

    Source: §1 · p. 2

Abstract

from arXiv · show

Active Real-time interaction with video LLMs introduces a new paradigm for human-computer interaction, where the model not only understands user intent but also responds while continuously processing streaming video on the fly. Unlike offline video LLMs, which analyze the entire video before answering questions, active real-time interaction requires three capabilities: 1) Perception: real-time video monitoring and interaction capturing. 2) Decision: raising proactive interaction in proper situations, 3) Reaction: continuous interaction with users. However, inherent conflicts exist among the desired capabilities. The Decision and Reaction require a contrary Perception scale and grain, and the autoregressive decoding blocks the real-time Perception and Decision during the Reaction. To unify the conflicted capabilities within a harmonious system, we present Dispider, a system that disentangles Perception, Decision, and Reaction. Dispider features a lightweight proactive streaming video processing module that tracks the video stream and identifies optimal moments for interaction. Once the interaction is triggered, an asynchronous interaction module provides detailed responses, while the processing module continues to monitor the video in the meantime. Our disentangled and asynchronous design ensures timely, contextually accurate, and computationally efficient responses, making Dispider ideal for active real-time interaction for long-duration video streams. Experiments show that Dispider not only maintains strong performance in conventional video QA tasks, but also significantly surpasses previous online models in streaming scenario responses, thereby validating the effectiveness of our architecture. The code and model are released at \url{https://github.com/Mark12Ding/Dispider}.

1. Introduction

Dispider addresses the limits of offline video LLMs for streaming interaction by separating perception, decision, and reaction into asynchronous modules. It is reported to outperform online and offline models on streaming and video benchmarks.

1. Introduction

  • Offline video LLMs wait until the full video is viewed before responding, making them impractical for interactive streaming that requires continuous feedback.VideoLLM-online [5] alternates between processing frames and generating responses, so reaction blocks perception and delays incoming video, particularly in long streams.Source: §1 · pp. 1–2
  • Dispider separates perception, decision, and reaction into parallel asynchronous modules, allowing decisions to continue while responses are generated.The decision module uses interaction history, previous decision tokens, and visual information to determine whether to trigger a response.Source: §1 · p. 2
  • Dispider segments streams into non-uniform, scene-based clips and interleaves their features with historical responses and decision tokens for response-or-wait decisions.When triggered, its asynchronous interaction module generates context-aware responses without interrupting ongoing video processing.Source: §1 · p. 2
  • Dispider is trained on a specialized streaming QA dataset to handle both cases requiring responses and situations where silence is more appropriate.The paper says this supports timely, relevant responses across diverse dynamic streaming contexts.Source: §1 · p. 2
  • Dispider significantly outperforms VideoLLM-online [5] in temporal grounding, proactive response generation, and multi-step reasoning, and offline models on long-video benchmarks and time-sensitive tasks.The paper also reports strong temporal reasoning across diverse video lengths.Source: §1 · p. 2

2. Related Work

Related work progresses from general LLMs to video-capable models, while prior streaming systems leave gaps in long-context handling and real-time interaction that Dispider targets.

2.1. Large Language Models

2.2. Video Large Language Models

  • Video LLMs address long-form reasoning through temporal modeling in TimeChat [53], sparse memory in MovieChat [55], and chat-centric spatiotemporal reasoning in VideoChat [31], while Dispider targets streaming inputs and real-time outputs.Source: §2.2 · pp. 2–3

2.3. Streaming Video Understanding

  • Prior streaming work leaves a gap: VideoLLM-online [5] lacks efficient long-term handling and real-time interactivity, VideoStream [48] is offline-focused, and Flash-VStream [69] overlooks interaction; Dispider targets long-context interactive responses.Source: §2.3 · p. 3

3. Method

Dispider formulates real-time video dialogue as deciding when observed clips and prior context suffice for a response, then separates continuous perception and decision from asynchronous response generation.

3.1. Problem Formulation

  • At each timestamp, the model uses prior dialogue context and observed clips to decide whether enough information exists to answer without processing the entire stream.If it is not ready to respond, it remains silent and gathers more information.Source: §3.1 · p. 3

3.2. Proactive Streaming Video Processing

  • Scene-based perception samples frames, uses SigLip [66] embeddings and cosine similarity to identify scene changes, then segments the stream into non-uniform clips with an exclusion window to avoid very short clips.This segmentation is intended to preserve video structure while reducing redundancy.Source: §3.2 · p. 3
  • The decision module combines question text, clip features, and pooled historical memory to predict whether to respond or wait, without using reaction-generated tokens that would block monitoring.A compact LLM uses a binary classification head on the ⟨TODO⟩ token to predict response timing.Source: §3.2 · pp. 3–4

3.3. Asynchronous Interaction

  • For each answer, the asynchronous interaction module receives the query, prior answers, and grounded clip features, retrieving relevant historical clips through ⟨TODO⟩-embedding similarity to support multi-hop reasoning.The retrieved clips can provide evidence distributed across multiple temporal segments.Source: §3.3 · pp. 4–5
  • The interaction module trains on response-required and no-response samples, producing incremental answers or ⟨SILENT⟩ as appropriate while video monitoring continues in parallel.Its adaptive reasoning focuses on unanswered content and past interactions to respond to new information.Source: §3.3 · p. 5

4. Experiments

Experiments evaluate Dispider’s streaming and conventional video understanding, finding strong streaming performance alongside competitive long-video QA results. Ablations support scene-based clip segmentation and the architecture’s response-control tokens.

4.1. Implementation Details; 4.2. Benchmarks

  • The evaluation covers streaming benchmarks, timestamped ETBench tasks, and long-video QA, while implementation uses compact and larger LLMs for streaming decisions and detailed responses.The implementation resizes frames to 224×224, uses CLIP-L/14 features and Qwen2-1.5B for streaming processing, and Qwen2-7B for responses; streaming questions are posed at video start, unlike conventional evaluation at video end.Source: §4.2 · pp. 5–6, §4.1 · p. 5

4.3. Streaming Video Understanding

  • Dispider significantly outperforms Flash-VStream and VideoLLM-online on StreamingBench, scoring 25.3 on Proactive Output despite receiving questions at video start.Unlike offline models given the question after the video, Dispider decides when to respond; its disentangled design also processes incoming video while generating answers without blocking.Source: §4.3 · p. 6, Table 1 · p. 7
  • In streaming ETBench, Dispider outperforms VideoLLM-Online across metrics, with particularly notable temporal-grounding gains and stronger streaming results than conventional results on dense captioning and step localization.The benchmark evaluates six timestamp-focused subtasks; it reports F1 for temporal grounding and sentence similarity for captioning.Source: §4.3 · p. 6, Table 3 · p. 8

4.4. Conventional Video Understanding

  • On conventional long-video QA, Dispider achieves leading EgoSchema accuracy of 55.6 and competitive performance on MLVU and VideoMME.On conventional ETBench, it achieves the highest F1 on temporal video grounding and episodic memory without specialized time tokens.Source: §4.4 · pp. 6–7, Table 2 · p. 8, Table 3 · p. 8

4.5. Ablation Study

  • Scene-based non-uniform clip segmentation yields superior conventional and streaming results to uniform 16-frame segmentation, consistent with preserving video structure for learning and timely responses.The ablation compares MLVU and VideoMME QA accuracy and ETBench temporal video grounding and dense captioning metrics.Source: §4.5 · p. 7, Table 4 · p. 8

5. Conclusion

Dispider enables active real-time interaction with video LLMs by disentangling perception, decision, and reaction and processing interactions asynchronously. Experiments show advantages over existing methods across conventional and streaming video QA benchmarks.

Conclusion

  • Dispider disentangles perception, decision, and reaction and uses asynchronous processing to address conflicts that hinder real-time video interaction.It continues processing the video stream while providing timely, contextually accurate responses.Source: §5 · pp. 8–9
  • Its Proactive Streaming Video Processing module focuses analysis on relevant content, while the Precise Interaction module generates detailed responses asynchronously without interrupting video processing.Source: §5 · p. 9
  • Experiments on conventional and streaming video QA benchmarks show that Dispider outperformed existing methods in proactive response capabilities, temporal awareness, and computational efficiency.The conclusion identifies long-duration video streams and real-time interactive applications as a suitable scope for Dispider.Source: §5 · p. 9

Related papers

Loading 2501.03218v1…