PaperlayerView exact revision on arXiv

Source-linked AI summary

ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding

Jitai Hao, Ke Yang, Qiang Huang, Jun Yu

arXiv:2609.02780v1cs.CVcs.CL

TL;DR

Streaming video understanding requires costly continuous processing, especially when full-depth MLLM prefill is repeatedly applied to incoming frames. ShallowStream uses shallow layers to encode and index the stream, then selectively applies full-depth computation to retrieved evidence. It matches leading streaming methods while reducing per-frame prefill and 10-second end-to-end latency by up to 52.1× and 11.9×.

  • Problem

    Streaming video understanding is computationally expensive because repeated full-depth prefill over continuously arriving frames creates substantial overhead and depth-proportional KV-cache growth.

  • Method

    ShallowStream uses shallow MLLM layers for continuous encoding and indexing, then applies selective full-depth answering to retrieved evidence through a query-time gate.

  • Results

    ShallowStream performs on par with current strongest streaming methods across backbones and benchmarks while reducing per-frame prefill latency by up to 52.1× and 10-second end-to-end latency by up to 11.9×.

  • Takeaways & Limitations

    ShallowStream establishes an efficient operating point for continuous video understanding by reducing computation, latency, and GPU-memory growth while preserving access to query-relevant evidence.

Abstract

from arXiv · show

Streaming video understanding is a critical capability for real-world applications, including embodied intelligence, autonomous driving, industrial monitoring, surveillance and early warning, and wearable assistants. However, processing continuous video streams with multimodal large language models (MLLMs) is computationally expensive. Existing efforts have explored reducing streaming overhead through visual token pruning, token merging, quantization, on-demand frame retrieval, and context offloading. However, most existing methods overlook the dimension of model depth. Repeatedly executing full-depth MLLM prefill over incoming frames is prohibitively expensive, incurring substantial computational overhead and causing the KV cache to grow at a rate directly proportional to the prefill depth. To address these challenges, we propose ShallowStream, a novel framework that leverages the shallow layers of an MLLM to simultaneously perform frame encoding and retrieval index building. During stream processing, ShallowStream maintains an always-on lightweight index using the KV cache of shallow layers. During query-time answering, we leverage the attention scores generated by the shallow layers to score context frames and employ a diversity-aware selection strategy to retrieve precise and comprehensive evidence. ShallowStream achieves performance on par with the strongest existing streaming methods, while reducing per-frame prefill latency and 10-second end-to-end latency by up to 52.1x and 11.9x, respectively. Our code is available at https://github.com/CURRENTF/ShallowStream.

1 Introduction

Streaming video understanding is costly because frames arrive continuously while queries are sparse, making repeated full-depth processing a dominant system burden. ShallowStream addresses this asymmetry by indexing frames with shallow layers and deferring deep computation until query time.

  • Streaming workloads are asymmetric: video frames arrive continuously, whereas queries occur intermittently.
  • Per-frame prefill is a first-order cost because incoming frames dominate steady-state processing and existing overhead translates into severe end-to-end latency.
  • Existing methods face expensive full-stack stream processing, lossy evidence reduction, and inefficient history use.
  • Shallow layers already provide strong retrieval capability, including at layer 4 of 28 for Qwen3-VL-8B and layer 3 of 32 for LLaVA-OneVision-7B.
  • ShallowStream builds a lightweight full-history visual index from shallow-layer KVs and applies full-depth processing only to retrieved evidence and recent context.
  • The framework decouples query-agnostic stream processing from query-time answering, preserving historical evidence while deferring expensive deep computation.

2 Related Work

Prior streaming-video systems construct and manage historical representations during continuous processing, then reuse or retrieve them when questions arrive. These approaches include recurrent memory, KV-cache management, compressed states, quantized token groups, and indexed scene recall.

  • Streaming systems operate causally on continuously growing video while questions may arrive at arbitrary times.
  • Query-agnostic methods build historical representations through recurrent memory, KV-cache management, or compact visual representations.
  • At query time, methods either reuse stored memory or retrieve question-relevant evidence from compressed, stored, quantized, or indexed history.

3 Observations

The paper defines causal streaming over an observed video prefix and partitions the MLLM into shallow and deep layers. Its observation is that shallow representations can retrieve relevant history while deeper stream-time processing costs more.

  • At query time, the system accesses only the observed video prefix and cannot see future video or know the future question.
  • The layer-wise diagnostic varies the layer used for video-question matching and measures retrieval capability alongside per-video-unit stream-time prefill cost.
  • The base MLLM combines a vision encoder with a language Transformer whose layers process visual-token hidden states.
  • The pruning boundary P separates shallow layers [0, P) from deep layers [P, L), while their union forms the full MLLM.
  • Shallow representations provide sufficient capability to select question-relevant video units from long histories.
  • Retrieval capability emerges in shallow layers, whereas streaming computational costs increase continuously with depth.

4 ShallowStream

ShallowStream separates lightweight query-agnostic stream processing from selective query-time answering by using shallow MLLM layers to build and search a full-history visual index. A gate routes queries, diversity-aware retrieval selects evidence, and full-depth processing generates answers from selected context.

  • Shallow index construction: Incoming video units are encoded only through shallow MLLM layers, producing reusable visual states and retrieval KVs without storing full-depth KV caches.The shallow cache retains visual encodings and shallow-layer KVs so selected units can later be re-prefilled from layer 0.
  • Shallow index construction: Skipping deep layers reduces per-unit model computation from L layers to P layers during streaming prefill.The causal KV cache accessible to each shallow layer supports this reduced-depth processing.
  • Shallow index construction: ShallowStream retains shallow-layer KVs across the observed video history, while recent units remain available for current-scene awareness.Historical units may optionally be compressed into fixed-size long-term cluster representatives when memory exceeds the retained-history budget.
  • Query-time answering: A query-logit gate distinguishes retrospective questions requiring historical retrieval from current-scene questions answerable using recent context.The gate uses a pretrained MLLM logit difference and a benchmark-independent frozen threshold, avoiding a separately trained router.
  • Query-time answering: For routed retrieval, shallow attention scores produce token votes, top candidates are ranked, and max-min diversity selection retains complementary evidence before temporal expansion.Cluster centroids can also be scored when long-term compression is enabled.
  • Query-time answering: Selected visual states, recent context, and the query are re-prefilled through all language layers so generation uses consistent full-depth representations.Shallow KVs support contextualization and selection but are not continued as partial-depth generation states.

5 Experiments

Experiments evaluate ShallowStream on OVO-Bench and StreamingBench, showing competitive accuracy with lower computational, latency, and memory demands. Ablations examine routing, retrieval ranking, diversity, and qualitative evidence selection.

  • Overall Performance: ShallowStream reaches 69.5 on OVO-Bench and 78.2 on StreamingBench with Qwen3-VL-8B, while its LLaVA-OneVision-7B variant reaches 62.2 and 75.5, respectively.The main experiments use two MLLM backbones and compare against strong streaming methods without task-specific training.
  • Efficiency: 52.1× lower per-frame prefill and 11.9× lower 10-s end-to-end latency are reported while maintaining performance on par with current state-of-the-art streaming methods.Efficiency measurements use one RTX 5090, Qwen3-VL-8B, five long OVO-Bench Backward videos, and fixed 16-token generation.
  • Efficiency: 18 GiB peak GPU memory remains nearly flat with long-cluster compression across 64–1,024 frames, whereas the uncompressed variant grows to 21.76 GiB.The comparison uses the final ShallowStream method with compression enabled or disabled; HERMES and OASIS require more memory at long prefixes.
  • Efficiency: At an 80-s query interval, both ShallowStream variants require about 2.6 s of compute, compared with 6.2 s for HERMES and 50.4 s for OASIS.The total combines query computation with continuous prefill on five long videos containing 588–1,198 sampled frames.
  • Analysis: The calibrated Gate activates historical retrieval mainly for retrospective questions while preserving recent context for current-scene questions.Its routing differs between Backward and Real-Time queries, indicating retrieval follows historical need rather than being applied indiscriminately.
  • Analysis: Token voting outperforms pooled shallow Q-K and SigLIP ranking, while max–min diversity further improves selection by discouraging redundant moments.A qualitative case shows the final retriever covering complementary views and correctly answering the rug-location question, unlike the alternatives.

6 Conclusions

The paper concludes that ShallowStream separates lightweight streaming processing from selective query-time answering. Across two MLLM backbones, it retains competitive performance while reducing computation, latency, and GPU-memory growth.

  • Conclusion: ShallowStream processes incoming video through shallow MLLM layers and activates full-depth computation only for query-relevant evidence selected by gating and diversity-aware token voting.This design decouples query-agnostic stream processing from selective query-time answering.
  • Conclusion: Across two MLLM backbones, ShallowStream achieves performance on par with current state-of-the-art streaming methods while reducing computation, latency, and GPU-memory growth.The conclusion describes the resulting efficiency-performance operating point for continuous video understanding.

A.1 Other Related Work

Related work spans streaming representation learning, proactive interaction, memory construction and retrieval, efficient execution, query-driven agents, and continuous evaluation. These lines address different components of efficient or interactive video understanding.

  • Representation Learning, Model Adaptation, and Architecture Design: StreamFormer adapts an image-pretrained vision transformer with causal temporal attention and jointly learns video-level, frame-level, and fine-grained spatial information.
  • Always-On and Proactive Interaction: StreamMind uses event-gated LLM invocation, while Dispider separates perception, decision, and reaction so lightweight monitoring can continue asynchronously.
  • Memory Construction, Compression, and Query-Time Retrieval: Streaming memory methods construct compact historical representations and retrieve query-dependent evidence through segmentation, computation reuse, compression, or adaptive retrieval scope.
  • Efficient Model Execution and Adaptation: Other approaches reduce redundant visual computation, compress token flow, or distill model knowledge through feature reuse, pruning, and compact adaptation.
  • Query-Driven Long-Video Agents: Query-driven agents acquire evidence progressively across temporal hierarchies or complementary memory types and scales.
  • Continuous and Interactive Evaluation: RTV-Bench and ProReady-QA evaluate continuous perception and reasoning through multi-timestamp, proactive, and response-timing-sensitive questions.

A.2 Implementation Details

The implementation archives shallow KVs and input-level visual states in host memory, with long-cluster compression replacing older detailed units by fixed-size clusters.

  • Implementation Details: Shallow KVs and input-level visual states are archived in host memory for both compression settings.LC-off preserves every detailed unit, whereas LC-on replaces units outside the detailed window with fixed-size clusters.

A.3 Attention Context and Retrieval Depth

ShallowStream targets bounded-context streaming by using shallow-layer representations for retrieval while retaining a full-history shallow index. Deeper processing can refine retrieval under dense attention, but its cost grows with accumulated history.

  • Attention Context: ShallowStream’s stream-time prefill uses an attention sink and bounded local window, while retrieval still covers the full observed stream.The bounded window applies to stream-time prefill rather than retrieval scope.
  • Retrieval Depth: The first shallow layers already provide useful retrieval representations under the deployment-oriented stream-time configuration.The diagnostic follows the stream-time prefill configuration used by ReKV.
  • Retrieval Depth: Dense full-history attention can refine retrieval representations, but its prefill cost grows with accumulated history and lacks bounded per-unit cost for open-ended streams.ShallowStream instead targets practical bounded-context streaming.
  • Implementation Settings: The evaluated backbones use different pruning boundaries and stream-processing settings, including P=5 and P=4 with distinct sampling and video-unit configurations.The implementation table lists the backbone-specific settings.

A.4 Pruning-Boundary Accuracy and Cost

A pruning boundary of P=5 provides the strongest reported accuracy without unnecessary continuous-stream depth, while reducing stream-time cost. Query-time latency remains stable as retained history grows under the evaluated settings.

  • Pruning-Boundary Accuracy: 58.27 Backward average is achieved at P=5, which outperforms deeper boundaries in the reported pruning sweep.The sweep holds routing, evidence selection, and recent-only outputs fixed; increasing the boundary through P=19 provides no further gain.
  • Pruning-Boundary Cost: 31.0% less per-frame prefill is required by P=5 than by P=19 while preserving the strongest reported accuracy.Gate routing and evidence selection occupy only a small fraction of total query computation.
  • Query-Time Scaling: Total query latency remains nearly unchanged across a 16× range of history lengths under both full-history LC-off and matched final LC-on settings.The measurements include host-archive transfer and selected-context assembly.
Loading 2609.02780v1…