PaperlayerView exact revision on arXiv

Source-linked AI summary

Cache Locally, Recompute Selectively for Streaming Video Understanding

Junhyeok Lee, Sangyoon Lee, Jaeho Lee

arXiv:2610.29229v1cs.CVcs.LG

TL;DR

Streaming video methods restrict idle-time prefill context to control costs, creating a mismatch with the context needed for later queries. FrameBlend selectively recomputes query-relevant KVs and enables shorter prefill windows; across benchmarks, it matches or improves accuracy while reducing computation.

  • Problem

    To control prefill cost over unbounded streams, SVU methods restrict context even though future queries are unknown, creating a mismatch between prefill and query-time context.

    Source: §1 · p. 1

  • Method

    FrameBlend re-prefills selected query-relevant tokens under query-time context and replaces their cached KVs, without additional training.

    Source: §1 · p. 2

  • Results

    Across three SVU frameworks and five benchmarks, FrameBlend matches or improves aggregate accuracy for every framework–benchmark pair, with gains up to 3.2 pp on streaming and 3.9 pp on offline benchmarks.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Query-time recomputation enables shorter idle-time attention windows, reducing prefill cost while matching or exceeding the accuracy of longer windows.

    Source: §1 · p. 2

    Experiments cover only 7B-scale backbones, use fixed prefill windows selected per framework and benchmark, and incur additional query-time latency from recomputation.

    Source: §6 · p. 9

Abstract

from arXiv · show

Streaming video understanding (SVU) requires video-LLMs to process incoming video continuously while answering user queries with low latency. Recent methods reduce query-time cost by prefilling frames into KV caches during idle time and reusing the cached KVs at query time. To bound per-frame computation over unbounded video streams, these methods restrict the context available during prefill. This creates a fundamental context mismatch: the restricted context may omit information needed to properly contextualize video tokens, yet, still contain information irrelevant to the eventual query, which is unknown during prefill. We propose FrameBlend, a simple yet effective plug-and-play method to address this mismatch through selective KV recomputation, without requiring any additional training. At query time, FrameBlend re-prefills selected video tokens using query-relevant context and replaces their cached KVs with recomputed ones. Crucially, this recomputation enables shorter attention windows during idle-time prefill, substantially reducing prefill cost while improving accuracy. Experiments across three SVU frameworks and five streaming and offline benchmarks demonstrate accuracy gains of up to 3.9 pp. On RVS-Ego, FrameBlend reduces per-frame prefill FLOPs by over 20% and total processing time by 14-59% for retrieval-based frameworks, including the cost of query-time recomputation. These results demonstrate that a simple change in how contextualization is distributed between idle time and query time can improve the accuracy-efficiency trade-off for SVU.

1 INTRODUCTION

FrameBlend addresses the context mismatch created when streaming video methods restrict idle-time prefill context to control computation. It selectively recomputes query-relevant KVs at query time, allowing shorter prefill windows while improving accuracy and efficiency.

1 INTRODUCTION

  • SVU requires continuous processing of potentially unbounded video streams under idle-time compute and memory limits, despite unknown future queries and tight query-time latency.Source: §1 · p. 1
  • Restricting prefill context creates a mismatch: sliding windows can omit context later needed for a query, while compression can preserve query-irrelevant information.Direct KV reuse leaves these representations uncorrected, potentially degrading answer quality.Source: §1 · pp. 1–2
  • FrameBlend selectively re-prefills query-relevant tokens under query-time context and overwrites their cached KVs, without training.Its QTS selection is lightweight and frequency-based, guided by query relevance.Source: §1 · p. 2
  • Across three frameworks and five benchmarks, FrameBlend matches or improves aggregate accuracy for every framework–benchmark pair, with gains up to 3.2 pp on streaming and 3.9 pp on offline benchmarks.For ReKV and LiveVLM on RVS, total FLOPs decrease by 19.0–23.2% and total latency by 14.1–58.7%.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • Query-time recomputation permits shorter idle-time attention windows, reducing prefill cost while matching or exceeding the accuracy of longer windows.Shorter windows limit query-irrelevant information baked into cached KVs when future queries are unknown.Source: §1 · p. 2

2 RELATED WORK

Prior SVU approaches shift video processing into idle time by caching incoming frames, but restrict context to control growing prefill costs. FrameBlend addresses the resulting mismatch through query-relevant KV recomputation.

2 RELATED WORK

  • Explicit visual-representation memories can still require costly backbone-LLM processing after a query arrives, contributing to long query-time latency.Source: §2 · p. 3
  • SVU methods prefill incoming frames into KV caches during idle time and reuse them at query time, restricting context with sliding windows or KV compression as streams grow.Some retrieval-based methods select query-relevant KVs during decoding; non-retrieval methods use the bounded memory.Source: §2 · p. 3
  • Directly reusing stored KVs can mismatch prefill and decoding context; FrameBlend mitigates this by recomputing selected KVs with query-relevant context.Source: §2 · p. 3
  • Prior cache-recomputation methods target long-context or vision-language serving, using query-agnostic heuristics or additional query-time operations rather than SVU-specific selection.FrameBlend selects query-relevant targets and can use existing backbone retrieval results to minimize additional selection operations.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries model SVU as an unbounded, temporally ordered token stream and describe KV-cache reuse as a way to divide inference between idle time and query time.

3 PRELIMINARIES

  • A streaming video question-answering input is an unlimited sequence of temporally ordered tokens, with a query arriving after an observed prefix.Source: §3 · pp. 3–4
  • KV-cache reuse separates inference into idle time, when incoming tokens are processed and caches stored, and query time, when the current memory is used to answer.This avoids storing and processing the entire observed token sequence only after the query arrives.Source: §3 · p. 4
  • Sliding-window prefill stores layerwise KV caches from the most recent tokens, while newer methods may update and compress memory to keep storage sublinear in stream length.Source: §3 · p. 4
  • At query time, a method can select per-layer KV subsets by similarity between the query and stored keys; compressed-memory approaches may instead use the whole memory.The selected subsets need not contain identical tokens across layers.Source: §3 · p. 4

4 METHOD

FrameBlend selectively recomputes cached KVs at query time to reduce prefill cost and improve SVU accuracy. This recomputation allows shorter attention windows during idle-time prefill.

4.1 SELECTIVE RECOMPUTATION OF KV CACHE

  • QTS selects token-level recomputation targets by cross-layer retrieval frequency, prioritizing query-relevant KVs while enabling constant computation across layers without inter-layer operations.The selected tokens are those most frequently retrieved across layers, capturing cross-layer agreement on query relevance.Source: §4.1 · p. 5
  • FrameBlend jointly re-prefills selected tokens in temporal order in one forward pass, replaces their cached KVs, and decodes using the updated cache.It retains observed video tokens in CPU memory during idle-time prefill to avoid repeated vision-encoder computation during recomputation.Source: §4.1 · p. 5

4.2 PREFILLING WITH A SHORT ATTENTION WINDOW

  • 196 tokens (one frame) can improve FrameBlend accuracy; prefill windows range from 1–68% of the backbone’s original size while improving accuracy over longer windows.The OVO-Bench comparison is for ReKV with FrameBlend.Source: Figure 3 · p. 5, §4.2 · p. 6

5 EXPERIMENTS

Experiments evaluate FrameBlend across three SVU frameworks and streaming and offline benchmarks, examining accuracy, efficiency, target selection, and component interactions. The reported efficiency gains accumulate with video length, while recomputation improves the trade-off between accuracy and latency.

5.1 EXPERIMENTAL SETUP

  • The setup evaluates ReKV, LiveVLM, and HERMES on five streaming and offline benchmarks, using two 7B video-LLM backbones and tuned prefill windows for each framework–benchmark pair.Inputs are sampled at 0.5 FPS; the recomputation budget is 7,840 video tokens, corresponding to 40 frames for LLaVA-OV-7B.Source: §5.1 · p. 6

5.2 MAIN RESULTS

  • Tables 1 and 2 compare the accuracy of SVU frameworks with and without FrameBlend on streaming and offline VQA benchmarks, respectively.The setup includes streaming benchmarks RVS and OVO-Bench and offline benchmarks MLVU, Video-MME, and EgoSchema.Source: §5.1 · p. 6, Table 1 · p. 7, Table 2 · p. 7

5.3 EFFICIENCY ANALYSIS

  • FrameBlend reduces per-frame prefill FLOPs by 12–26% and prefill latency by 13–61%; total FLOPs fall 10–23% and total latency 14–59% across three frameworks.Although recomputation raises TTFT to approximately one second, savings in idle-time computation exceed its additional query-time cost; the shorter prefill window reduces attention computation per incoming frame.Source: §5.3 · p. 7, Table 3 · p. 8
  • As video length grows, ReKV with FrameBlend increasingly reduces total latency and peak VRAM; for short inputs, peak VRAM remains similar because vision encoder activations dominate.The shorter prefill window keeps fewer KVs on the GPU and offloads older KVs to host memory.Source: §5.3 · p. 7, Figure 4 · p. 8

5.4 RECOMPUTATION TARGET SELECTION ANALYSIS

  • With the same prefill window and KV replacement policy, QTS achieves higher accuracy with lower added TTFT than CacheBlend across all streaming benchmarks.The controlled comparison isolates target selection: CacheBlend fully recomputes early layers to identify targets query-agnostically, while QTS directly selects query-relevant targets.Source: §5.4 · p. 8, Table 4 · p. 9

5.5 ABLATION STUDY

  • Combining a shorter prefill window with query-time recomputation achieves the highest overall accuracy across benchmarks, whereas either component alone has mixed effects.Recomputation complements the shorter window by selectively restoring query-relevant context among selected tokens.Source: §5.5 · p. 8, Table 5 · p. 9

6 CONCLUSION

FrameBlend selectively recomputes cached KVs at query time to address context mismatch, using QTS to choose query-relevant targets with low additional overhead. Across three SVU frameworks and five benchmarks, it matches or improves performance while reducing overall computational cost; evaluation scope and query-time latency remain limitations.

6 CONCLUSION

  • FrameBlend is a plug-and-play, training-free method that selectively recomputes cached KVs at query time to address context mismatch in prior SVU frameworks.QTS selects query-relevant targets with minimal additional overhead, and recomputation permits shorter idle-time attention windows that can improve performance while reducing computation.Source: §6 · p. 9
  • Across three SVU frameworks and five streaming and offline benchmarks, FrameBlend consistently improves or matches performance while reducing overall computational cost.The conclusion identifies additional query-time latency as a cost of selective recomputation.Source: §6 · p. 9
  • Experiments cover only 7B-scale backbones, use a fixed prefill window per framework and benchmark, and incur additional query-time latency from selective recomputation.The authors identify smaller and larger models, adaptive windows, and dynamically adjusted recomputation budgets as future directions.Source: §6 · p. 9

A FRAMEWORK-SPECIFIC IMPLEMENTATION DETAILS

The ReKV adaptation preserves its retrieval and decoding procedures while changing prefill context and selectively replacing cached KVs. QTS ranks retrieved frames for recomputation using layer-wise retrieval frequency and normalized retrieval scores.

A.1 REKV WITH FRAMEBLEND

  • FrameBlend changes ReKV's sliding-window prefill size while retaining its retrieval and decoding procedures.Original ReKV allows each token to attend to up to 15,000 preceding video tokens during idle-time prefill.Source: §A.1 · p. 13
  • QTS ranks frames by the number of layers retrieving each one, breaking ties with the sum of normalized retrieval scores across those layers.Because ReKV retrieves complete frame blocks, all tokens within a frame block share a retrieval score; selected frames are ordered temporally and jointly re-prefilled through all LLM layers.Source: §A.1 · p. 13
  • At each layer, only selected frames also present in that layer's original retrieval set have their KVs replaced, leaving ReKV's layer-wise decoding context and retrieval policy unchanged.The replacement uses recomputed KVs from the joint re-prefill.Source: §A.1 · p. 13

A.2 LIVEVLM WITH FRAMEBLEND

FrameBlend adapts its prefill window and query-time recomputation to the different cache and retrieval designs of LiveVLM and HERMES. In both, QTS selects frequently retrieved tokens for recomputation and replaces corresponding cached slots.

A.2 LIVEVLM WITH FRAMEBLEND

  • LiveVLM compresses its KV cache above a 12,000-token budget and retrieves 40% of 16-token pages independently for each layer and head at query time.Consequently, query-time context can differ across both layers and heads.Source: §A.2 · p. 13
  • For LiveVLM, QTS counts how many layer–head pairs retrieve each candidate token, breaks ties with normalized retrieval scores, then jointly re-prefills the most frequently retrieved tokens.A separate small KV buffer keeps the effective sliding prefill window independent of cache compression; only selected tokens' corresponding temporary query-time cache slots are replaced.Source: §A.2 · p. 13

A.3 HERMES WITH FRAMEBLEND

  • HERMES uses an auxiliary retrieval forward pass to select recomputation targets, ranking tokens by how many layers retrieve them and re-prefilling selected tokens in temporal order.The recomputed KVs replace only corresponding surviving slots; decoding continues to use HERMES's full compressed cache, and layer-specific position indices are maintained.Source: §A.3 · p. 13

B PREFILL WINDOW SIZE TUNING

FrameBlend's idle-time prefill window is tuned separately for each framework and benchmark, and is shorter than the frameworks' original attention ranges across evaluated settings.

C ADDITIONAL PREFILL WINDOW SIZE ANALYSIS

Under full recomputation, ReKV with FrameBlend generally performs better with a shorter prefill window than its 15K-token default, but the best size depends on the benchmark.

D ADDITIONAL EFFICIENCY ANALYSIS

FrameBlend reduces latency for HERMES and LiveVLM as video streams grow, with smaller reductions than ReKV; memory costs vary by window size and cache compression.

D ADDITIONAL EFFICIENCY ANALYSIS

E ADDITIONAL CPU MEMORY USAGE

  • On approximately one-hour RVS-Ego videos, query-time recomputation embeddings require 2.36 GiB for ReKV, 0.25 GiB for HERMES, and 1.01 GiB for LiveVLM in CPU memory.For compression-based frameworks, only embeddings for tokens retained in at least one layer can become recomputation candidates, lowering CPU memory overhead.Source: E ADDITIONAL CPU MEMORY USAGE · p. 14

Related papers

Loading 2610.29229v1…