PaperlayerView exact revision on arXiv

Source-linked AI summary

ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding

Jitai Hao, Ke Yang, Qiang Huang, Jun Yu

arXiv:2609.02780v1cs.CVcs.CL

TL;DR

Continuous video understanding with MLLMs is costly because incoming frames repeatedly undergo full-depth prefill. ShallowStream builds a lightweight index with shallow layers and defers full-depth processing to retrieved evidence at query time, matching the strongest existing streaming methods while reducing per-frame prefill and 10-s end-to-end latency by up to 52.1× and 11.9×.

  • Problem

    Existing streaming methods often overlook model depth, while repeatedly prefilling incoming frames through the full Transformer stack makes per-frame stream processing costly.

    Source: §1 · p. 2

  • Method

    ShallowStream uses shallow MLLM layers to encode incoming frames and build a lightweight index, then applies full-depth processing to retrieved evidence at query time.

    Source: §1 · p. 3, §3 · p. 4

  • Results

    Up to 52.1× and 11.9× reductions in per-frame prefill and 10-s end-to-end latency, respectively, while matching the strongest existing streaming methods.

    Source: §1 · p. 2, §5.3 · p. 9

  • Takeaways & Limitations

    These results establish an efficient operating point for continuous video understanding across two MLLM backbones.

    Source: §6 · p. 12

Abstract

from arXiv · show

Streaming video understanding is a critical capability for real-world applications, including embodied intelligence, autonomous driving, industrial monitoring, surveillance and early warning, and wearable assistants. However, processing continuous video streams with multimodal large language models (MLLMs) is computationally expensive. Existing efforts have explored reducing streaming overhead through visual token pruning, token merging, quantization, on-demand frame retrieval, and context offloading. However, most existing methods overlook the dimension of model depth. Repeatedly executing full-depth MLLM prefill over incoming frames is prohibitively expensive, incurring substantial computational overhead and causing the KV cache to grow at a rate directly proportional to the prefill depth. To address these challenges, we propose ShallowStream, a novel framework that leverages the shallow layers of an MLLM to simultaneously perform frame encoding and retrieval index building. During stream processing, ShallowStream maintains an always-on lightweight index using the KV cache of shallow layers. During query-time answering, we leverage the attention scores generated by the shallow layers to score context frames and employ a diversity-aware selection strategy to retrieve precise and comprehensive evidence. ShallowStream achieves performance on par with the strongest existing streaming methods, while reducing per-frame prefill latency and 10-second end-to-end latency by up to 52.1x and 11.9x, respectively. Our code is available at https://github.com/CURRENTF/ShallowStream.

1 Introduction

ShallowStream separates continuous, query-agnostic stream processing from selective query-time answering. It uses shallow-layer processing to maintain access to historical visual evidence while deferring deep computation until a query identifies relevant context.

Introduction

  • Streaming workloads process frames continuously but receive queries intermittently, making per-frame prefill a dominant cost and existing methods a source of substantial end-to-end latency.Existing methods often incur full-stack frame prefill before future questions are known, even when later cache reduction discards deep KV states [10] [19].Source: §1 · pp. 1–2
  • ShallowStream matches the strongest existing streaming methods [10] [23] [27] on OVO-Bench while reducing per-frame prefill latency by up to 52.1× and 10-s end-to-end latency by up to 11.9× on long videos.The latency comparison is shown in Figure 1, whose end-to-end setting uses one query every 10 seconds.Source: Figure 1 · p. 2, §1 · p. 2
  • Prior streaming methods face costly full-depth processing, lossy query-agnostic evidence reduction, and inefficient history use during answering [8] [10] [19] [22] [23].Retaining all states can consume prohibitive memory or require bandwidth-limited offloading [10], while appending history can slow answers and interfere with current-scene perception.Source: §1 · p. 2
  • Shallow layers already retrieve question-relevant historical evidence in Qwen3-VL-8B [2] at layer 4 of 28 and LLaVA-OneVision-7B [20] at layer 3 of 32.This observation motivates using shallow layers for retrieval rather than repeatedly pre-filling the full model.Source: §1 · p. 2
  • ShallowStream encodes incoming units through shallow layers, builds a visual index from shallow-layer KVs, optionally compresses long clusters, and uses a query-logit gate to trigger full-depth processing only for retrieved history and recent context.This decouples lightweight query-agnostic processing from selective query-time answering while preserving access to historical evidence.Source: §1 · pp. 2–3

2 Related Work

Streaming video understanding handles a continuously growing causal stream in which future frames are unavailable and questions may arrive at arbitrary times. Related systems encode and preserve history before a query, then decide whether and how to use it; ShallowStream maintains a shallow index and gates historical retrieval.

Streaming Video Understanding

  • Streaming video systems process sequential frames without future observations and answer questions posed at arbitrary times, unlike offline systems that receive the complete video and question before inference.They generally separate query-agnostic stream processing, which encodes incoming content and maintains reusable history, from query-time evidence localization and answering.Source: §2 · p. 3

Query-Agnostic Stream Processing

  • Prior query-agnostic systems use recurrent memory [28], KV-cache management [10] [8] [19], token reduction [7], or scene indexing [25]; ShallowStream retains a lightweight shallow index.ShallowStream can optionally compress that index for long streams.Source: §2 · p. 3

Query-Time Answering

  • At query time, methods reuse compressed KVs [19] or retrieve historical states, token groups, or indexed scenes [10] [8] [27] [7] [25], while history can impair current-scene perception.The passage distinguishes direct KV reuse from retrieval of historical evidence, including quantized-memory token groups and indexed scenes with high-resolution content.Source: §2 · p. 3
  • OASIS [23] and WeaveTime trigger retrieval conditionally but require answer-level routing and a second inference when retrieval activates; ShallowStream gates retrieval before full-depth prefill.ShallowStream applies full-depth prefill only to selected historical evidence or recent context, avoiding an MLLM reasoning process for routing.Source: §2 · p. 3

3 Observations

Shallow MLLM layers can retrieve question-relevant video units, while stream-time computation rises with layer depth. ShallowStream therefore indexes continuously with shallow layers and reserves full-depth processing for retrieved evidence after a query arrives.

Background and Notation

  • At query time T, the system accesses only the observed video prefix and does not know the future video or question in advance; each unit has 2 sampled frames for Qwen3-VL and 1 for LLaVA-OneVision.Source: §3 · p. 3

Observation: Shallow Layers Are Sufficient for Retrieval

  • Shallow-layer retrieval capability emerges before deeper layers, while stream-time prefill costs increase continuously with depth.Figure 2 compares layer-wise video-question matching and per-unit latency for Qwen3-VL-8B and LLaVA-OneVision-7B on LVBench.Source: Figure 2 · p. 4, §3 · p. 4

Background and Notation

  • The base MLLM has a vision encoder and L language layers; the pruning boundary P divides shallow layers [0, P) from deep layers [P, L).Source: §3 · p. 4

Observation: Shallow Layers Are Sufficient for Retrieval

  • On LVBench, representations from varying layers ranked historical units, and the full-depth model answered using only the highest-scoring units under attention-sink prefill [10].The experiment isolated retrieval from answer generation; prior findings on semantic information in shallow and middle layers are cited as [26].Source: §3 · p. 4

4 ShallowStream

ShallowStream separates frequent, query-agnostic stream processing from query-time answering: shallow layers maintain a visual index, and full-depth prefill is reserved for selected video units.

4.1 Query-Agnostic Shallow Index Construction

  • Each arriving video unit is processed only through shallow layers, retaining its visual input and shallow states for retrieval and later full-depth processing.This reduces per-unit computation from L layers to P layers by skipping the deeper layers.Source: §4.1 · p. 5
  • ShallowStream retains shallow-layer KVs across the observed history, always includes recent units, and can compress older history into fixed-size cluster representatives when memory exceeds its budget.The optional compression preserves recent and near-term history in detail; when the uncompressed history fits in memory, the approximation is disabled.Source: §4.1 · pp. 5–6

4.2 Query-Time Answering

  • A text-only query-logit gate distinguishes retrospective questions from current-scene questions, activating historical retrieval only when needed and using a benchmark-independent, frozen threshold.Without retrieval, the system uses recent context; when retrieval is activated, it searches retained detailed history and, if enabled, long-term clusters.Source: §4.2 · p. 7
  • For gated queries, shallow-layer attention votes rank historical units, max-min selection diversifies the evidence, and temporal neighbors are added before only selected inputs receive full-depth prefill.The attention-based retrieval preserves token-level evidence, while selected-unit full-depth processing avoids pre-filling the entire stream.Source: §4.2 · pp. 7–8

5 Experiments

ShallowStream matches the strongest streaming methods across two benchmarks while reducing stream-time costs. Its analyses attribute evidence retrieval to calibrated historical routing, token-level relevance, and diversity-aware selection.

5.1 Experimental Setup

  • Experiments evaluate ShallowStream on OVO-Bench and StreamingBench with Qwen3-VL-8B-Instruct and LLaVA-OneVision-7B, sampling both backbones at 1 FPS with frozen Gate and retrieval settings.Baselines include specialized online MLLMs and training-free methods spanning recent-window context, KV retention and retrieval, memory compression, fixed-budget memory, and on-demand retrieval; the original backbones are also reported.Source: §5.1 · pp. 8–9

5.2 Overall Performance

  • ShallowStream scores 69.5 on OVO-Bench and 78.2 on StreamingBench with Qwen3-VL-8B, and 62.2 and 75.5, respectively, with LLaVA-OneVision-7B, matching strong streaming methods without task-specific training.The results cover retrospective evidence tracing and broad real-time visual understanding.Source: §5.2 · p. 9, Table 1 · p. 9, Table 2 · p. 10

5.3 Efficiency

  • ShallowStream reduces per-frame prefill latency by up to 52.1× and 10-s end-to-end latency by up to 11.9× versus current state-of-the-art streaming methods.Efficiency measurements use Qwen3-VL-8B on five long OVO-Bench Backward videos sampled at 1 FPS, with generation fixed to 16 visible answer tokens.Source: §5.3 · pp. 9–10
  • Across 64–1,024 frames, compression keeps peak GPU memory nearly flat at about 18 GiB, while the uncompressed variant grows to 21.76 GiB.HERMES and OASIS require more GPU memory at long prefixes.Source: §5.3 · p. 10, Figure 4 · p. 11
  • At an 80-s query interval, LC-on and LC-off each require about 2.6 s of compute, compared with 6.2 s for HERMES and 50.4 s for OASIS.The comparison combines query computation and continuous prefill on five long videos containing 588–1,198 sampled frames without a frame cap.Source: §5.3 · p. 10, Figure 4 · p. 11

5.4 Analysis

  • The calibrated Gate retrieves historical evidence primarily for retrospective questions while preserving recent context for current-scene questions, avoiding both missed history and distracting indiscriminate retrieval.Its routing differs markedly between Backward and Real-Time queries.Source: §5.4 · pp. 11–12, Figure 5 · p. 11
  • Last-prompt-token voting outperforms pooled shallow Q-K and SigLIP, while max–min diversity further improves selection by reducing redundant moments.The ranker comparison disables diversity and temporal expansion; the diversity comparison fixes token voting and preceding-unit expansion.Source: Figure 6 · p. 11, §5.4 · p. 12
  • In the rug case study, only the final retriever combines query-specific token voting with diversity to select complementary evidence and answer E correctly.Pooled shallow Q-K and its max–min-diversity variant both predict B.Source: §5.4 · p. 12, Figure 7 · p. 12

6 Conclusions

ShallowStream decouples lightweight, query-agnostic stream processing from selective query-time answering: incoming video is processed through shallow MLLM layers, and full-depth computation is activated only for selected evidence.

6 Conclusions

  • Across two MLLM backbones, ShallowStream matches current state-of-the-art streaming methods while substantially reducing computation, latency, and GPU-memory growth.Query-time evidence is selected using a query-logit gate and diversity-aware token voting.Source: §6 · p. 12

Reproducibility Statement

  • The source code and experiment configurations are available, alongside documentation of implementation, calibration, and evaluation procedures needed to reproduce the results.The code is hosted at https://github.com/CURRENTF/ShallowStream.Source: Reproducibility Statement · p. 13

A Appendix · A.1 Other Related Work

Related work improves model representations, manages streaming memory and retrieval, or adapts interaction and evaluation. ShallowStream instead builds a lightweight shallow index and delays full-depth processing until a query identifies needed historical evidence.

A.1 Other Related Work

  • StreamFormer, LiveCC, and Streamo adapt model architectures or training for streaming, while Uni-X addresses modality conflict; ShallowStream instead selectively reuses historical representations.Source: §A.1 · p. 17
  • StreamMind [11], Dispider, StreamBridge, ViSpeak [12], and StreamReady [1] address when to invoke or respond, while ShallowStream's gate activates after a question arrives.Its gate determines whether answering requires historical evidence.Source: §A.1 · p. 17
  • ShallowStream preserves history in a lightweight shallow index and postpones full-depth prefill until a query identifies needed evidence, unlike caption-based memories or fully computed KV stores.Related approaches include StreamRAG, MuKV, FlexMem [6], DeltaKV, and EventMemAgent, which use event segmentation, KV representations, compression, or structured event memory.Source: §A.1 · p. 17
  • SelectStream exposes history through a query-conditioned latent memory graph, while SAVEMem uses semantic priors and adapts retrieval across memory timescales.Both distinguish current-scene perception from historical recall, using latent memory structures or structured semantic summaries.Source: §A.1 · p. 17
  • Streaming Token Compression reuses features from similar frames and prunes low-salience tokens, while Low-Rank Clone improves distillation efficiency [14].These approaches reduce redundant visual computation or transfer knowledge compactly, whereas ShallowStream controls when full model depth runs and which historical evidence is processed.Source: §A.1 · pp. 17–18
  • VideoTree, WorldMM, and LongVideo-R1 use adaptive multi-step retrieval, generally assuming complete random-access video and a query available before evidence search.These assumptions differ from query-agnostic streaming, where future questions and future observations are unavailable.Source: §A.1 · p. 18
  • RTV-Bench, ProReady-QA [1], and ViSpeak-Bench [12] evaluate evolving questions, answer-evidence timing, and visually triggered interaction, respectively.Together with OVO-Bench, StreamingBench, and SVBench [24], they underscore that comparisons depend on query timing, future-frame visibility, evidence availability, interaction history, and concurrent processing.Source: §A.1 · p. 18

A.2 Implementation Details

The main experiments use shared frame-rate and budget settings across two backbones, while architecture-specific choices govern pruning, gating, and video units.

A.2 Implementation Details

  • Both backbones use 1 FPS on both benchmarks and share evidence-selection and decoding budgets, while pruning boundary, Gate threshold, and video-unit definition are architecture-specific.Shallow KVs and input-level visual states are archived in host memory. LC-off preserves every detailed unit; LC-on replaces units outside the detailed window with fixed-size cluster KVs and one input representative per cluster.Source: §A.2 · p. 18
  • Query latency spans query entry through generation completion, including host-to-device transfer and selected-context assembly.Source: §A.2 · p. 18

A.3 Attention Context and Retrieval Depth

ShallowStream’s preferred retrieval depth depends on the stream-time attention regime: bounded-context streaming supports effective shallow retrieval, while dense full-history attention can benefit from deeper processing at growing cost.

A.3 Attention Context and Retrieval Depth

  • Stream-time prefill uses an attention sink and bounded local window, but ShallowStream retains a full-history shallow index for retrieval across the observed stream.The layer-wise diagnostic follows the stream-time prefill configuration used by ReKV [10].Source: §A.3 · p. 18
  • Under bounded-context streaming, shallow representations already provide effective video–question matching signals, while deeper stream-time prefill adds computation.Source: §A.3 · p. 18
  • Dense complete-causal-prefix attention can refine retrieval representations, but its prefill cost grows with accumulated history and is not bounded per unit for open-ended streams.ShallowStream reserves full-depth computation for query-relevant evidence in practical bounded-context streaming, where retrieval signals emerge early.Source: §A.3 · p. 18

A.4 Pruning-Boundary Accuracy and Cost … A.7 Ten-Shot Router Prompt

The appendices show that shallow-layer indexing preserves accuracy at lower cost, describe query-logit router calibration, and report query-time scaling and the ten-shot prompt used for routing.

A.4 Pruning-Boundary Accuracy and Cost

  • 31.0% less per-frame prefill at P=5 than P=19 preserves the strongest accuracy, while routing and evidence selection use only a small fraction of query computation.The cost summary covers stream-time prefill averaged over five long videos and query components under matched full-history LC-off settings.Source: Table 4 · p. 19, §A.4 · p. 19
  • P=5 achieves the strongest EPM, HLD, and Backward average (58.27) in the OVO-Bench Backward pruning-boundary sweep; increasing the boundary through P=19 adds no gain.Routing, evidence selection, and recent-only outputs are held fixed; P=5 was selected beforehand using LVBench.Source: §A.4 · p. 19, Figure 8 · p. 20

A.5 Query-Logit Gate Calibration

  • The gate calibration set uses synthetic labeled questions under the ten-shot prompt’s retrospective-versus-recent routing definition, excluding OVO-Bench and StreamingBench queries.For each backbone, the method computes s(q) = ℓret(q) − ℓrecent(q) using the ten-shot prompt.Source: §A.5 · p. 20
  • A backbone-specific threshold is frozen before benchmark evaluation using a precision-first criterion requiring a one-sided 95% Wilson lower bound on retrieval precision of at least 90%.Calibration uses the benchmark-independent query-only set.Source: §A.5 · p. 20, Table 5 · p. 20

A.6 Query-Time Scaling with Retained History

  • Across a 16× range of retained history, total query latency remains nearly unchanged, and long-cluster compression keeps evidence-selection cost essentially constant.The wall-clock measurements include host-archive transfer and selected-context assembly; Gate and decoding latency also remain nearly invariant.Source: §A.6 · p. 20, Figure 9 · p. 21

A.7 Ten-Shot Router Prompt

  • The ten-shot router prompt classifies whether a question requires earlier memory (A) or whether the latest segment suffices (B), using labeled examples of both query types.The prompt tells the router not to answer the video question itself and ends by requesting output only A or B.Source: §A.7 · pp. 21–22

Related papers

Loading 2609.02780v1…