LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrieval

1 Introduction

• Online Video LLMs must continuously process video before a query arrives and then respond in real time, balancing understanding quality, memory use, and speed.Most Video LLMs instead process the entire video after a question is posed.Source: §1 · p. 1
• Query-dependent compression risks discarding tokens relevant to future queries and delays processing until a query arrives; query-agnostic pruning can require O(n^2) response prefill and aggressive compression.Aggressive compression can degrade response accuracy.Source: §1 · pp. 1–2
• Existing query-agnostic cache approaches either retain full KVs on CPU at severe efficiency cost or discard KVs and lose long-term temporal details.The paper frames quality, memory overhead, and response speed as challenges existing methods do not address simultaneously.Source: §1 · p. 2
• VSB identifies sink vision tokens using vision-to-vision attention, then balances attention scores with context coverage to retain long-term information.Retaining sink tokens helps keep attention outputs of newly arriving tokens close to pre-compression levels.Source: §1 · p. 2
• PaR retrieves query-relevant KVs while retaining recent clips for short-term context, and decouples positional embeddings to support page-level retrieval.The framework uses precomputed KVs to reduce response attention computation from O(n^2) to approximately O(n).Source: §1 · p. 2

2.1 Video Large Language Models

• Video LLMs align visual features with text tokens, but long or streaming videos challenge them through memory bottlenecks and information complexity; offline strategies can lose information or be inefficient.The cited strategies include sparse sampling, instruction-guided token compression, and token merging.Source: §2.1 · p. 2

2.2 KV Cache Compression for Video LLMs

• KV compression methods exploit attention sparsity, but query-derived text scores can delay online responses; LiveVLM instead uses vision-token attention patterns for streaming-oriented compression.Prior approaches include discarding low-attention tokens, while MMInference explores unimodal and cross-modal sparse patterns for prefill acceleration.Source: §2.2 · pp. 2–3

2.3 Streaming Video Understanding

• Streaming systems trade spatial detail, latency, memory, or long-term information; LiveVLM uses VSB to preserve global context under a moderate cache budget and PaR to retrieve visual details efficiently.ReKV stores the full cache on CPU, while InfiniPot-V and StreamMem use constrained-memory compression.Source: §2.3 · p. 3

3.1 Sparse Attention Patterns of Vision Tokens

• Sink vision tokens attract sustained attention from other vision tokens, so retaining them helps limit compression-induced deviations during streaming inference.Source: §3.1 · p. 3
• VSB balances summed attention scores with the context range covered, avoiding redundant local high-attention tokens that crowd out new critical tokens.Source: §3.1 · p. 3
• VSB retains more potential answer tokens than TopK across all model layers, especially shallow layers, better adapting to query-agnostic online scenarios.Source: §3.1 · p. 3

3.2 Algorithm Design

• VSB estimates importance from the last r vision-token queries attending to cached keys; the added computation and memory overhead is less than 1% of full-scale attention.Source: §3.2 · p. 4
• VSB allocates the cache budget across equal-capacity context buckets, first greedily placing the top-R tokens and then filling remaining capacity by descending importance.Source: §3.2 · p. 4

4.1 Discontinuous Positions Degrade Key Similarity

• Discarding tokens creates discontinuous positions that weaken key similarity within retrieval pages, making representative keys unreliable and degrading response accuracy.Source: §4.1 · p. 4

4.2 Decouple Retrieval from Positional Embeddings

• PaR removes positional embeddings before retrieval and reintegrates them into retrieved keys, reducing positional interference while preserving positions for model reasoning.Source: §4.2 · pp. 4–5
• PaR averages position-agnostic keys within equal-sized pages, scores pages through simplified attention with question queries, and retrieves relevant KV pages.Source: §4.2 · p. 5

5.1 Implementation Details

• Experiments use LLaVA-OneVision-Qwen2-7B-OV with a 12k-token cache budget, page size 16, and 40% retrieval; benchmarks cover offline and online video QA.The setup uses FP16 mixed precision on an NVIDIA 4090D GPU with 24 GB of memory.Source: §5.1 · p. 5

5.2 Main Results

• LiveVLM improves MLVU by 3.4% and VideoMME long-form by 5.1% over the foundation model, and scores 57.0 on VideoMME medium-form.Source: §5.2 · p. 5
• LiveVLM achieves the strongest RVS-Ego and RVS-Movie performance under the stated constraints and improves StreamingBench average accuracy by 4.25% over LLaVA-OneVision-7B.Table 4 covers Real-Time Visual Understanding and Omni-Source Understanding accuracy.Source: §5.2 · p. 5, Table 4 · p. 6

5.3 Efficiency Analysis

• At 256 frames, LiveVLM uses 3.02× less peak GPU memory than Dispider and 1.19× less than ReKV, with 1.73× faster response latency than ReKV.Figure 8 compares memory and latency across frame budgets; Dispider's memory use and latency rise sharply as frame count increases.Source: §5.3 · p. 5, Figure 8 · p. 6

5.4 Ablation Studies

• VSB enables more frames under the same memory constraint, and adding PaR to VSB further improves overall accuracy by 1.9%.With both modules, performance improves by 2.6%, 4.1%, and 1.7% on the three subtasks.Source: §5.4 · p. 6, Table 5 · p. 6
• Accuracy across the three subtasks rises then falls as retrieval ratio increases; 0.4 is optimal, while inputting the entire cache gives the lowest overall accuracy.Table 6 reports the varying-retrieval-ratio ablation.Source: §5.4 · p. 6, Table 6 · p. 6

• LiveVLM is a training-free framework designed for streaming online video understanding and real-time interaction.Source: §6 · p. 6
• Vision Sink Bucketing retains critical video details within a fixed-size cache.Source: §6 · p. 6
• Position-agnostic KV Retrieval collects short-term and long-term visual information efficiently.Source: §6 · p. 6