PaperlayerView exact revision on arXiv

Source-linked AI summary

StreamingVLM: Real-Time Understanding for Infinite Video Streams

Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, Yao Lu, Song Han

arXiv:2510.09608v2cs.CVcs.AIcs.CL

TL;DR

VLMs struggle to understand near-infinite video streams while maintaining stable real-time responses, because full attention is costly and sliding-window methods can break coherence or recompute redundantly. StreamingVLM aligns overlapped-chunk training with streaming inference, using compact KV-state reuse and stable context management. It delivers strong streaming and VQA results, including a 66.18% win rate against GPT-4O mini on Inf-Streams-Eval and stable performance at up to 8 FPS on one NVIDIA H100.

  • Problem

    VLMs need to understand near-infinite video streams with low, stable latency, but existing full-attention and sliding-window approaches impose memory, coherence, or recomputation challenges.

  • Method

    StreamingVLM aligns training with streaming inference by fine-tuning on short, overlapped video chunks and reusing attention-sink and past KV states with contiguous position IDs during inference.

  • Results

    StreamingVLM improves streaming captioning and general VQA without VQA-specific fine-tuning, achieving a 66.18% win rate against GPT-4O mini on Inf-Streams-Eval.

  • Takeaways & Limitations

    The framework supports coherent, long-duration streaming video understanding with low, stable latency and practical real-time operation.

Abstract

from arXiv · show

Vision-language models (VLMs) could power real-time assistants and autonomous agents, but they face a critical challenge: understanding near-infinite video streams without escalating latency and memory usage. Processing entire videos with full attention leads to quadratic computational costs and poor performance on long videos. Meanwhile, simple sliding window methods are also flawed, as they either break coherence or suffer from high latency due to redundant recomputation. In this paper, we introduce StreamingVLM, a model designed for real-time, stable understanding of infinite visual input. Our approach is a unified framework that aligns training with streaming inference. During inference, we maintain a compact KV cache by reusing states of attention sinks, a short window of recent vision tokens, and a long window of recent text tokens. This streaming ability is instilled via a simple supervised fine-tuning (SFT) strategy that applies full attention on short, overlapped video chunks, which effectively mimics the inference-time attention pattern without training on prohibitively long contexts. For evaluation, we build Inf-Streams-Eval, a new benchmark with videos averaging over two hours that requires dense, per-second alignment between frames and text. On Inf-Streams-Eval, StreamingVLM achieves a 66.18% win rate against GPT-4O mini and maintains stable, real-time performance at up to 8 FPS on a single NVIDIA H100. Notably, our SFT strategy also enhances general VQA abilities without any VQA-specific fine-tuning, improving performance on LongVideoBench by +4.30 and OVOBench Realtime by +5.96. Code is available at https://github.com/mit-han-lab/streaming-vlm.

1 INTRODUCTION

StreamingVLM addresses the difficulty of understanding near-infinite video streams by aligning training with streaming inference and reusing compact attention states. It supports coherent, low-latency commentary while improving long-video and VQA performance.

  • Motivation: Full attention incurs O(T^2) cost and unbounded memory, while sliding-window variants either break coherence or repeatedly recompute context.These limitations make long-video understanding inefficient or unstable.
  • Approach: StreamingVLM trains on short, overlapped video chunks with full attention to align supervision with its streaming inference pattern.At inference, it combines attention sinks, sliding windows, reused KV states, and contiguous position IDs.
  • Evaluation: Inf-Streams-Eval contains videos averaging over two hours and requires dense, per-second alignment between frames and text.The paper also introduces Inf-Streams-Train with over 4000 hours of sports commentary data.
  • Results: +4.30 on LongVideoBench is obtained without VQA-specific fine-tuning, and the model improves general visual ability across VQA tasks.The reported gains accompany stable streaming captioning and long-term history retention.

2 METHOD

The method combines a compact streaming KV cache, contiguous positional encoding, overlapped full-attention training, and curated long-horizon sports data. These components align training with inference while providing data for streaming commentary and evaluation.

  • Method overview: StreamingVLM uses an inference scheme, a streaming-aware training strategy, and data-curation pipelines for long-horizon real-time understanding.The method is organized around low-latency inference, streaming capability, and long-horizon data.
  • Inference scheme: The KV cache retains attention-sink text, recent text, and recent vision tokens while evicting older vision tokens first.This asymmetric retention reuses previous states instead of recomputing them.
  • Inference scheme: Contiguous RoPE shifts positions after eviction so effective indices remain bounded and in-distribution during long-horizon streaming.For Qwen-VL, the approach extends to contiguous 3D indices for time, height, and width.
  • Training strategy: Training splits long streams into overlapped chunks, interleaves vision and text at one-second intervals, and applies full attention within each chunk.Text loss is aligned to per-second narration, with placeholder tokens when narration is absent.
  • Training strategy: Overlapped full-attention supervision approximates the inference-time cache pattern without requiring prohibitively long quadratic-cost contexts.The strategy teaches the intended recency bias for stable streaming behavior.
  • Data curation: The data pipeline collects five sports, cleans commentary with GPT-based keep, edit, or delete decisions, and forms aligned video-commentary pairs.The initial corpus exceeds 6,000 hours, with 46.32% kept, 37.89% edited, and 15.79% deleted.
  • Evaluation data: Inf-Streams-Eval contains 20 full games averaging 2.12 hours, scored by GPT-5 pairwise voting against ground-truth references.Evaluation segments are 100 seconds long and require at least 200 commentary words.

3 EXPERIMENTS

The experiments evaluate StreamingVLM on captioning, VQA, and efficiency, comparing it with strong baselines across chunked and infinite settings. Results show improved captioning, general VQA accuracy, stable latency, and bounded positional behavior for infinite streams.

  • Experimental setup: StreamingVLM is evaluated on captioning and four VQA tasks against strong baselines, including GPT-4o mini, LiveCC-7B-Instruct, ReKV, and its Qwen-2.5-VL-7B-Instruct base model.Inf-Streams-Eval averages 2.12 hours and tests long-horizon commentary; the VQA comparison measures gains over the pre-SFT base model.
  • Captioning: Short chunks break coherence, whereas long chunks exceed training-length limits and degrade, leaving existing designs unable to support infinite inference.Figure 6 highlights the difficulty of balancing cross-chunk coherence with training-length constraints.
  • Efficiency and ablations: Contiguous RoPE sustains infinite-stream performance by keeping positional indices bounded, while native RoPE degrades and 100 s chunking harms long-term memory.The ablation compares captioning win rates under infinite and 100 s chunked settings.
  • Efficiency and ablations: StreamingVLM latency stays low and stable, while full attention reaches OOM and sliding-window variants remain inefficient or exceed real-time limits.The dashed threshold corresponds to 10 tokens/s, or ≤0.1 s per token.

3.3 EFFICIENCY TESTS

The efficiency and ablation studies identify the design choices that support stable infinite-video inference. Fixed context, KV reuse, contiguous positions, recent vision retention, overlapped SFT, and annealing data each contribute to performance or efficiency.

  • Efficiency: StreamingVLM supports real-time commentary at 8 FPS on a single NVIDIA H100 while maintaining lower and stable latency through fixed context and KV reuse.Full attention reaches OOM, non-overlapping windows show periodic latency and coherence problems, and overlapping windows incur redundant computation.
  • Contiguous RoPE: Contiguous RoPE keeps position indices bounded and supports infinite inference without loss, whereas native RoPE degrades sharply on infinite streams.Splitting video into 100 s chunks partly recovers accuracy but harms long-term coherence.
  • Sliding window and sink: Evicting previous text tokens is important for infinite inference and improves performance under the matching eviction policy.The ablation compares models trained and evaluated with text eviction against models without eviction.
  • Sliding window and sink: A 16 s visual window balances recent-action coverage and efficiency, while retaining 0 s of vision context causes a clear performance drop.The ablation demonstrates that recent vision tokens are essential for continuous actions.
  • Training strategy and dataset: Overlapped SFT improves Inf-Streams-Eval win rate by +31.29 against GPT-4o mini and LiveCC-Sports-3K CC win rate by +3.68 against LLaVA-Video-72B-Qwen2.The strategy adapts the model to the interleaved vision–text pattern and long-video understanding.
  • Training strategy and dataset: The non-overlapping-chunk ablation performs significantly worse, supporting the importance of overlapped-chunk SFT for training–inference alignment.The ablation is designed to test whether the data alone is sufficient without the proposed SFT strategy.
  • Training strategy and dataset: Adding high-quality annealing data brings further gains on both captioning and VQA benchmarks, especially for Inf-Streams-Eval.The data focuses on real-time content and further improves commentary quality.

4 RELATED WORK

Related work spans long-context text inference, streaming video-language models, and video benchmarks. StreamingVLM targets a gap in existing evaluation by requiring dense frame-level understanding during near-infinite commentary.

  • Long-context and streaming inference: Long-context text methods include attention sinks with sliding windows, RoPE extension, and KV-cache compression for near-infinite inputs under fixed memory and delay.StreamingLLM exemplifies sink-token retention and recent-window decoding.
  • Positioning: StreamingVLM combines a compact streaming cache with training aligned to the inference pattern, extending ideas from long-context text systems to continuous video understanding.The supplied related-work passages establish the neighboring method families and the benchmark gap, while the paper’s method passages describe the unified extension.
  • Streaming and online video LLMs: Concurrent streaming video-language methods use streaming dialogue, fixed video-token budgets, or aligned ASR and video frames for long-context, low-latency applications.Examples include VideoLLM-online, VideoStreaming, and LiveCC.
  • Benchmarks and evaluation: Existing video benchmarks cover multimodal comprehension, real-time commentary, and ultra-long-video memory, including VideoMME, LiveSports-3K-CC, and LVBench.Their evaluation targets differ in task coverage and temporal scope.
  • Benchmarks and evaluation: Inf-Streams-Eval addresses a benchmark gap by requiring second-level frame–response alignment, high-FPS processing, and near-infinite commentary over videos longer than two hours.This design is intended to approximate real-world needs for VLM assistants, robots, and autonomous driving.

5 CONCLUSION

StreamingVLM combines streaming-aware training, inference, and data curation to support real-time perception over long video streams. It delivers stable commentary and introduces resources for second-level streaming evaluation.

  • StreamingVLM unifies training and inference to bring real-time streaming perception to existing VLMs.
  • The inference design supports stable commentary for over 3 hours at up to 8 FPS on a single NVIDIA H100.
  • Inf-Streams provides an SFT dataset and benchmark for second-level, real-time understanding on videos averaging over 2 hours.
  • The framework and data curation pipeline boost performance on both streaming tasks and VQA.

A.1 LLM USAGE STATEMENT

The authors used large language models only as writing assistants for manuscript preparation. They state that the research and scientific contributions were produced by the authors.

  • Claude and GPT-5 were used exclusively to improve manuscript language, grammar, phrasing, and organization.
  • The authors state that LLMs did not contribute to research ideation, hypothesis formulation, or core scientific aspects.
  • The authors reviewed and verified all content and take responsibility for the manuscript.

A.2 STABILITY OVER TIME

On a 2-hour test set divided into five temporal segments, StreamingVLM maintains performance across later segments. It remains close to Sliding Window w/ Overlap while other baselines degrade or stay lower.

  • StreamingVLM does not degrade across later segments of the 2-hour test set.Each video is split into five segments at 20% intervals.
  • StreamingVLM reaches performance close to Sliding-Window w/ Overlap over time.
  • Full Attention and Sliding Window w/o Overlap degrade or remain far lower across segments.
  • Table 8 presents sensitivity analysis over Tsink on Inf-Stream-Eval.

A.3 SENSITIVITY ANALYSIS OF SINK TOKEN WINDOW SIZE

The sensitivity analysis shows that attention-sink window size affects performance. Larger windows generally improve win rates, but gains plateau, creating a trade-off with computational efficiency.

  • Tsink is a key hyperparameter whose size affects performance across scenarios.
  • Larger Tsink capacities generally yield better win rates against GPT-4o and Livecc metrics.The evaluated sizes are 64, 128, 256, 512, and 1024.
  • Performance gains plateau at larger window sizes, indicating a trade-off between context retention and computational efficiency.
  • Tsink should be tuned according to the scenario’s context-length requirements.

A.4 DEMO

The supplementary demo shows StreamingVLM providing commentary after 100 minutes of continuous inference, while noting that occasional hallucinations may occur because the base model is modest in size.

  • 100 minutes of continuous inference are demonstrated in the supplementary commentary video.The demo video is randomly selected and edited to remove long pauses and mid-length ads.
  • The demo video is randomly selected and edited to remove long pauses and mid-length ads.
  • Occasional hallucinations may occur because the base model is modest in size.
Loading 2510.09608v2…