PaperlayerView exact revision on arXiv

Source-linked AI summary

StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant

Haibo Wang, Bo Feng, Zhengfeng Lai, Mingze Xu, Shiyu Li, Weifeng Ge, Afshin Dehghan, Meng Cao, Ping Huang

arXiv:2505.05467v2cs.CVcs.AIcs.CL

TL;DR

Offline Video-LLMs are built for prerecorded videos, while streaming requires multi-turn real-time understanding and proactive responses. StreamBridge adapts them with compressed memory, a separate activation model, and Stream-IT training data. The adapted models report strong streaming-benchmark performance, including results above proprietary models, while retaining or exceeding offline performance.

  • Problem

    Offline Video-LLMs process entire prerecorded videos, leaving limited support for multi-turn real-time understanding and proactive responses as frames arrive sequentially.

    Source: §1 · p. 1

  • Method

    StreamBridge pairs a memory buffer and round-decayed compression with a decoupled activation model, and supports fine-tuning with interleaved streaming data from Stream-IT.

    Source: §1 · p. 2

  • Results

    StreamBridge models achieve state-of-the-art results on OVO-Bench and Streaming-Bench, outperform GPT-4o and Gemini 1.5 Pro, and retain or exceed performance on conventional offline video tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments show that adapted offline Video-LLMs can provide timely, proactive responses across multi-turn, long-context streaming scenarios while preserving base-model strengths.

    Source: §6 · p. 10

    Stream-IT relies partly on synthetic QA generation and clip concatenation, which may introduce domain shift from truly continuous real-world video streams.

    Source: G Limitations · p. 19

Abstract

from arXiv · show

We present StreamBridge, a simple yet effective framework that seamlessly transforms offline Video-LLMs into streaming-capable models. It addresses two fundamental challenges in adapting existing models into online scenarios: (1) limited capability for multi-turn real-time understanding, and (2) lack of proactive response mechanisms. Specifically, StreamBridge incorporates (1) a memory buffer combined with a round-decayed compression strategy, supporting long-context multi-turn interactions, and (2) a decoupled, lightweight activation model that can be effortlessly integrated into existing Video-LLMs, enabling continuous proactive responses. To further support StreamBridge, we construct Stream-IT, a large-scale dataset tailored for streaming video understanding, featuring interleaved video-text sequences and diverse instruction formats. Extensive experiments show that StreamBridge significantly improves the streaming understanding capabilities of offline Video-LLMs across various tasks, outperforming even proprietary models such as GPT-4o and Gemini 1.5 Pro. Simultaneously, it achieves competitive or superior performance on standard video understanding benchmarks.

1 Introduction

Offline Video-LLMs process prerecorded videos, while streaming calls for multi-turn real-time understanding and proactive responses. StreamBridge combines memory and decayed compression with a decoupled activation model, supported by Stream-IT, and reports strong streaming-benchmark performance while retaining offline capability.

1 Introduction

  • Offline Video-LLMs process entire prerecorded videos, whereas robotics and autonomous driving require online causal perception and interpretation.This mismatch limits their readiness for streaming settings where timely understanding and responsiveness matter.Source: §1 · p. 1
  • Streaming interaction requires retaining accumulated visual and conversational history while focusing on each new segment, and can also require responses without explicit prompts.The introduction illustrates proactive step-by-step guidance as video content unfolds.Source: §1 · p. 1
  • StreamBridge adapts pretrained offline Video-LLMs using a memory buffer with round-decayed compression and a compact, decoupled activation model.The activation component enables proactive behavior without modifying the main model architecture.Source: §1 · p. 2
  • Stream-IT provides interleaved video-text sequences, multi-turn questions, proactive responses, and diverse task formats for streaming scenarios.It is built by concatenating semantically related short clips and generating multi-turn QA sequences.Source: §1 · p. 2
  • The framework and Stream-IT fine-tuning yield state-of-the-art results on OVO-Bench and Streaming-Bench, outperforming GPT-4o and Gemini 1.5 Pro while retaining or exceeding offline performance.The paper reports these results for adapted LLaVA-OV, Oryx-1.5, and Qwen2-VL models.Source: §1 · p. 2

2 Related Work

Prior work spans offline Video-LLMs, streaming tasks and benchmarks, and methods for online conversation or proactive response. StreamBridge instead adapts pretrained offline models with a dedicated dataset and a modular activation component.

2 Related Work

  • Video-LLMs combine a visual encoder, modality projector, and LLM, but are designed for fully accessible prerecorded videos rather than sequential streaming inputs.Prior systems achieve strong standard video-benchmark results, yet streaming requires real-time, temporally coherent, or proactive responses.Source: §2 · p. 3
  • Streaming research includes causal action recognition and anticipation, real-time conversation [50] [12] [13] [51], and benchmarks such as StreamingBench [21] and OVO-Bench [20].VideoLLM-Online and Flash-VStream [11] use online objectives and memory architectures; MMDuet [14] and ViSpeak [52] add proactive-response heads.Source: §2 · p. 3
  • Unlike approaches that retrain models or tightly couple proactive mechanisms to the backbone, StreamBridge adapts pretrained Video-LLMs [1] [3] [2] with a dedicated dataset and plug-and-play activation model.The paper notes that embedding activation in the main model can cause optimization conflicts and performance degradation [14].Source: §2 · p. 3

3 Methodology

The methodology frames streaming as interleaved multi-turn input and proactive output, then equips offline Video-LLMs with persistent context, bounded history, and a separate response trigger. Its components store and compress the interaction history while an activation model monitors incoming frames.

3.1 Preliminary Analysis

  • Streaming includes interleaved multi-turn queries and answers with immediate responses, plus proactive output in which the model chooses when to respond as video arrives.Existing benchmarks simplify multi-turn streaming into isolated single-turn offline tasks by discarding prior dialogue history.Source: §3.1 · p. 3

3.2 StreamBridge

  • StreamBridge combines a memory buffer, round-decayed compression, and a compact activation model; incoming frames are stored while the activation model signals when the LLM should answer.Figure 2 depicts frame storage, a query, and the activation model's binary decision.Source: §3.2 · p. 4, Figure 2 · p. 4

3.2.1 Memory Buffer

  • The memory buffer stores visual and textual embeddings, appends encoded frames and queries, and supplies accumulated history to the LLM for responses.Generated responses are also appended, preserving temporal continuity across multi-turn video-text interactions.Source: §3.2.1 · p. 4

3.2.2 Round-Decayed Compression

  • When input embeddings exceed MaxLen, round-decayed compression merges visual tokens from the earliest rounds first, preserving recent visual context while reducing memory use and inference overhead.The method averages adjacent frame tokens and continues until the input is below MaxLen.Source: §3.2.2 · pp. 4–5

3.2.3 A Plug-and-play Activation Model

  • A compact external MLLM independently evaluates each incoming frame, query, and optionally prior frames, triggering the main LLM when its predicted score exceeds threshold α.Training uses a binary score head and activation token; the last P% of each video segment's frames are labeled positive, with P sampled from 0% to 50%.Source: §3.2.3 · pp. 5–6, Figure 3 · p. 5

4 Stream-IT Dataset

Stream-IT is a video-text dataset for streaming instruction tuning, designed around interleaved multi-turn dialogue. Its formats support both multi-turn real-time understanding and proactive responses.

4 Stream-IT Dataset

  • Stream-IT is a video-text dataset for streaming instruction tuning with interleaved multi-turn dialogue, addressing gaps in support for real-time understanding and proactive responses.The dataset is designed for streaming scenarios where existing datasets and video sources fall short.Source: §4 · p. 6
  • Timestamp-annotated public datasets for captioning, step recognition, and grounded video QA are reformatted so video segments can delay answers after a query.The proactive-style sequence is <Q> <V1> <A1>, <V2> <A2>, and so on.Source: §4 · p. 6
  • StreamingQA-120K filters approximately 1.28 million caption-aligned clips, concatenates roughly 10 similar clips into videos averaging over 150 seconds, then generates eight-task QA pairs with GPT-4o.The source clips are around 12 seconds long, and their captions are preserved with timestamps.Source: §4 · p. 6
  • Sequence augmentation randomly drops QA pairs (Pdrop=0.55) and shifts QA before its clip (Pshift=0.1), supporting varied streaming interactions across tasks and timescales.The shift places video between the question and response as a temporal delay.Source: §4 · p. 6

5 Experiments

Experiments evaluate three offline Video-LLMs on multi-turn streaming, proactive response, and seven general video benchmarks. StreamBridge models show strong streaming results while retaining or improving offline performance.

5.2 Main Results

  • 65.5 on VideoMME, up 6.7, for Oryx-1.5-7B‡; adapted models generally maintain or improve base performance, with LLaVA-OV-7B‡ excepting LongVideoBench.Qwen2-VL-7B‡ surpasses its original counterpart on benchmarks other than MVBench.Source: Table 2 · p. 8, §5.2 · p. 8
  • Oryx-1.5† gains +11.92 on OVO-Bench and +4.2 on Streaming-Bench; Qwen2-VL† scores 71.30 and 77.04, beating GPT-4o and Gemini 1.5 Pro.The multi-turn evaluation streams video at 1 FPS while maintaining historical context within a predefined maximum token budget.Source: §5.2 · pp. 7–8
  • On ET-Bench DVC and SLC generation tasks, StreamBridge scores higher DVCSim and SLCSim than VideoLLM-Online and Dispider.All tasks present the question at the start of the video, leaving the model to decide when to respond.Source: Table 3 · p. 8, §5.2 · p. 8

5.3 In-Depth Analysis

  • Removing LLaVA-178K alone hurts streaming scores, using only Stream-IT hurts MVBench and VideoMME, and removing StreamingQA-120K degrades both benchmark types.Table 5 identifies StreamingQA-120K as a generated subset of Stream-IT.Source: Table 5 · p. 9, §5.3 · p. 9
  • Truncation performs worst by removing visual and textual history; Round-Uniform is better but still trails round-decayed compression, partly because it compresses recent visual tokens.The comparison is reported in Table 4.Source: Table 4 · p. 9, §5.3 · p. 9
  • With compression, latency stays near-constant beyond MaxLen; without it, delays rise sharply and inference reaches OOM at 2048 frames.Latency is evaluated on one A100-80G GPU at MaxLen settings of 8k, 16k, and 32k.Source: Figure 4 · p. 9, §5.3 · p. 9
  • Both very low and high α reduce ET-Bench DVCF1 and SLCF1: low thresholds trigger overly frequent responses, while high thresholds suppress them.The default activation threshold is α = 0.35; users can adjust it to control response frequency.Source: Figure 5 · p. 9, §5.3 · p. 10
  • Streaming OVO-Bench Real-Time accuracy stays between 70.49% and 71.30% across MaxLen settings, while VideoMME rises from 61.7% at 4k to 64.7% at 32k.Accuracy peaks at 16k for streaming tasks; the paper uses MaxLen = 16k as a balance across tasks.Source: §5.3 · pp. 9–10, Table 6 · p. 9

6 Conclusion

StreamBridge adapts offline Video-LLMs for streaming through memory, compression, and activation components, supported by Stream-IT. The authors report timely proactive responses across multi-turn, long-context scenarios while preserving base-model strengths.

6 Conclusion

  • StreamBridge combines a memory buffer and round-decayed compression with a compact activation model to support long-context, multi-turn streaming and proactive responses.The framework also uses Stream-IT, a dataset with interleaved video-text sequences.Source: §6 · p. 10
  • Stream-IT supplies interleaved video-text sequences to support StreamBridge.Source: §6 · p. 10
  • Experiments across diverse benchmarks show timely proactive responses in multi-turn, long-context streaming scenarios while preserving the base models’ strengths.Source: §6 · p. 10

A Datasets Used to Train the Activation Model

The activation model is trained on video datasets spanning five task categories, using approximately 180,000 samples and randomly selected task-specific prompts. Stream-IT’s dense-captioning data includes both proactive and multi-turn formats.

A Datasets Used to Train the Activation Model

  • Activation-model training spans dense video captioning, sequential step recognition, temporal action detection, grounded VideoQA, and temporal video grounding datasets.The cited datasets are ActivityNet Captions and Shot2Story; YouCook2 and COIN; FineAction and HACS; Multihop-EgoQA and EgoTimeQA; and Charades and ET-Instruct’s TVG subset.Source: A Datasets Used to Train the… · p. 15
  • The training set contains approximately 180k video samples, with a randomly selected prompt from a predefined, task-specific pool for each sample.The prompt is inserted at the beginning of the input sequence.Source: A Datasets Used to Train the… · p. 15

B.1 Statistics of Stream-IT

  • For dense-captioning tasks, 20% of sequences use a proactive format and 80% use a multi-turn format.The multi-turn format pairs each video segment with a question and answer about the current situation.Source: §B.1 · p. 16

B.2 Concatenation Strategy for Constructing StreamingQA-120K

StreamingQA-120K construction orders short clips through repeated semantic-similarity sampling, then uses task-specific prompts to generate varied question-answer pairs. The prompts cover temporal, semantic, and other video-understanding tasks.

B.2 Concatenation Strategy for Constructing StreamingQA-120K

  • Starting from 1.28 million filtered short videos, the construction process repeatedly samples a clip by semantic similarity to the current anchor, then makes it the next anchor.Similarity is based on middle frames, and sampling is without replacement.Source: §B.2 · p. 16
  • The similarity-ordered clips can be concatenated as continuous spans of any length, and the authors also add hallucination questions irrelevant to the video inputs at a 0.01% ratio.Source: §B.2 · pp. 16–17

B.3 Prompt Templates for Generating QA Pairs

  • QA-pair generation uses diverse prompt templates for eight reasoning tasks, intended to broaden coverage and generalization across temporal and semantic challenges.Source: §B.3 · p. 17
  • The prompt categories include object perception, action recognition, spatial awareness, sequential relationships, causal reasoning, and optical character recognition.Source: §B.3 · p. 17
  • The remaining prompt categories address unexpected-event handling and event understanding, including temporally linked event sequences.Source: §B.3 · p. 17

C More Implementation Details

The implementation specifies separate frame-token configurations for three main Video-LLMs and a compact activation model. It also describes fine-tuning settings and frame-sampling rates for training and inference.

C More Implementation Details

  • LLaVA-OV-7B uses 384×384 center crops and bilinear ×4 downsampling, yielding 49 tokens per frame.Source: C More Implementation Details · p. 18
  • Oryx-1.5-7B uses dynamic 288–480-pixel resolution and average-pooling ×4 downsampling, yielding 33–59 tokens per frame.Source: C More Implementation Details · p. 18
  • Qwen2-VL-7B uses dynamic 224–448 resolution and average-pooling ×4 downsampling, yielding 36–64 tokens per frame.Source: C More Implementation Details · p. 18
  • Main Video-LLMs are fine-tuned for one epoch at learning rate 2e-5; the image encoder is frozen while the projector and LLM are trainable.The input-embedding maximum length is 16384.Source: C More Implementation Details · p. 18
  • The activation model uses LLaVA-OV-0.5B with 16 pooled frame tokens, training only LoRA adapters, the projector, score head, and activation token.It is trained for five epochs with distinct learning rates for the projector and other trainable components.Source: C More Implementation Details · p. 18
  • Training samples frames at 1 FPS; inference uses 2 FPS for short-video benchmarks and 1 FPS for streaming and long-video benchmarks.Videos longer than 256 seconds are uniformly sampled to 256 frames.Source: C More Implementation Details · p. 18

D Benchmarks and Metrics

The section defines streaming and general video benchmarks, discusses broader impacts, and distinguishes StreamBridge from related memory and retrieval approaches.

D Benchmarks and Metrics

  • OVO-Bench and Streaming-Bench evaluate multi-turn real-time understanding, while ET-Bench tasks assess when models should proactively respond.The section reports accuracy for the streaming benchmarks and uses localization F1 or sentence-level similarity for ET-Bench tasks.Source: D Benchmarks and Metrics · pp. 18–19

E Broader Impacts

  • The authors identify health monitoring, autonomous driving, and collaborative robots as possible applications, and advocate responsible use under applicable laws and regulations.Source: E Broader Impacts · p. 19

F More Related Works

  • Unlike ReKV [56], VideoStreaming, and StreamChat [53], StreamBridge combines round-decayed compression, a lightweight activation model, and Stream-IT for proactive streaming interactions.The section characterizes the earlier approaches as advancing long-context retention for reactive question-answering.Source: F More Related Works · p. 19

G Limitations

The authors identify limitations in Stream-IT’s construction and in the video-streaming settings StreamBridge currently supports.

G Limitations

  • Stream-IT relies partly on synthetic QA generation and clip concatenation, which may create domain shift from continuous real-world streams.The authors suggest organically collected long-form streams with naturally evolving events and dialogue as future work.Source: G Limitations · p. 19
  • StreamBridge currently targets frame-by-frame streaming at relatively low sampling rates, such as 1 FPS.Source: G Limitations · p. 19
  • Real-time handling of denser frame rates and multimodal streams such as audio-visual-text remains future work.Source: G Limitations · p. 19

H Full Results on OVO-Bench and Streaming-Bench

This section presents full results tables for two streaming benchmarks and pseudocode that splits interleaved inputs by round and compresses visual tokens.

H Full Results on OVO-Bench and Streaming-Bench

I Pseudo Code of the Round-Decayed Compression in a PyTorch-like Style

Loading 2505.05467v2…