PaperlayerView exact revision on arXiv

Source-linked AI summary

HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding

Haowei Zhang, Shudong Yang, Jinlan Fu, See-Kiong Ng, Xipeng Qiu

arXiv:2601.14724v4cs.CVcs.AIcs.CL

TL;DR

Streaming MLLMs struggle to maintain stable understanding, real-time responses, and low GPU memory overhead simultaneously. HERMES addresses this with a training-free hierarchical KV-cache framework and compact cache reuse, achieving competitive or improved accuracy with substantially lower token use and latency.

  • Problem

    Streaming video understanding requires stable long-video performance, low-latency responses, and memory-efficient deployment, which existing MLLMs struggle to provide simultaneously.

  • Method

    HERMES uses mechanistic layer-wise attention analysis to manage KV cache as hierarchical memory with cache management, cross-layer smoothing, and position re-indexing.

  • Results

    HERMES achieves competitive accuracy with up to 68% fewer video tokens, gains up to 11.4% on streaming benchmarks, and up to a 10× latency speedup over prior training-free SOTA.

  • Takeaways & Limitations

    HERMES supports accurate streaming video understanding with consistently low response latency and compact GPU memory usage, including across multiple streaming tasks.

  • Takeaways & Limitations

    The evaluation does not include 72B-scale MLLMs or integration with other orthogonal training-free techniques because of computational resource constraints.

Abstract

from arXiv · show

Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated significant improvement in offline video understanding. However, extending these capabilities to streaming video inputs, remains challenging, as existing models struggle to simultaneously maintain stable understanding performance, real-time responses, and low GPU memory overhead. To address this challenge, we propose HERMES, a novel training-free architecture for real-time and accurate understanding of video streams. Based on a mechanistic attention investigation, we conceptualize KV cache as a hierarchical memory framework that encapsulates video information across multiple granularities. During inference, HERMES reuses a compact KV cache, enabling efficient streaming understanding under resource constraints. Notably, HERMES requires no auxiliary computations upon the arrival of user queries, thereby guaranteeing real-time responses for continuous video stream interactions, which achieves 10$\times$ faster TTFT compared to prior SOTA. Even when reducing video tokens by up to 68% compared with uniform sampling, HERMES achieves superior or comparable accuracy across all benchmarks, with up to 11.4% gains on streaming datasets.

1 Introduction

Streaming video understanding requires stable long-video performance, low-latency responses, and memory-efficient deployment, but existing methods struggle to satisfy these requirements simultaneously. HERMES addresses this gap by treating the KV cache as hierarchical memory and reusing a compact cache during inference.

  • Existing streaming MLLMs struggle to simultaneously provide stable understanding, low-latency responses, and memory-efficient deployment.
  • External-memory methods perform retrieval and multimodal prefilling at query time, causing high latency and lacking end-to-end cohesion.
  • HERMES is a training-free, plug-and-play method that conceptualizes KV cache as hierarchical memory across multiple granularities.
  • HERMES reduces video tokens by up to 68% while achieving competitive accuracy, including gains of up to 11.4% on streaming benchmarks.
  • HERMES achieves up to a 10× latency speedup over the prior training-free SOTA method while maintaining a compact GPU memory footprint and avoiding auxiliary query-time computation.

2 Layer-wise Preference for Hierarchical Streaming Video Information

The paper investigates layer-wise attention preferences in streaming video KV caches and identifies three memory roles: sensory, working, and long-term memory. These roles motivate hierarchical cache management rather than uniform token eviction.

  • Streaming setup: Sliding-window streaming processes continuous video chunk by chunk and evicts tokens when the KV cache reaches its memory budget.
  • Attention investigation: The attention investigation samples short, medium, and long VideoMME videos and streams them into LLaVA-OV-7B with fixed-size chunks and a constant cache budget.
  • Shallow layers as sensory memory: Shallow layers act as sensory memory, concentrating attention on recent visual tokens and rapidly decaying attention to earlier tokens.
  • Deep layers as long-term memory: Deep layers act as long-term memory, using sparse rhythmic attention peaks spaced 196 tokens apart as frame-level anchor tokens.
  • Middle layers as working memory: Middle layers act as working memory by integrating recent and earlier information while transitioning toward deep-layer rhythmic patterns.

3 HERMES

HERMES manages streaming-video KV caches hierarchically according to layer-wise memory roles, then smooths importance across layers and re-indexes positions. These mechanisms retain compact, aligned memory while supporting streaming or offline inference trade-offs.

  • HERMES overview: HERMES is a training-free framework with hierarchical KV cache management, cross-layer memory smoothing, and position re-indexing.
  • Hierarchical KV cache management: Shallow-layer token importance follows recency-based exponential forgetting, whereas deep-layer importance uses query-related attention to retain stable frame-level anchors.
  • Hierarchical KV cache management: Middle-layer importance interpolates recency and attention with a layer-dependent weight to bridge shallow and deep memory behavior.
  • Cross-layer memory smoothing: Cross-layer memory smoothing propagates importance signals from deeper to shallower layers to reduce misaligned visual memory caused by independent eviction.
  • Cross-layer memory smoothing: Evicted tokens are aggregated into one summary token per layer, compactly encoding long-term memory while remaining in the KV cache.
  • Position re-indexing: Position re-indexing remaps indices into [0, |M|) to stabilize inference under continuously growing streams; lazy re-indexing lowers overhead, while eager re-indexing costs more but suits offline videos.

4 Experiments

HERMES is evaluated across streaming and offline video benchmarks, efficiency settings, and component ablations. It combines competitive understanding with stable resource use across long inputs.

  • Streaming Video Understanding: HERMES reaches 79.44% and 59.21% accuracy on StreamingBench and OVO-Bench using only 4K video tokens.These results improve over Qwen2.5-VL-7B by 6.13% and 6.93%, respectively.
  • Streaming Video Understanding: HERMES improves RVS streaming accuracy by up to 11.4% over the base model with uniformly sampled 64 frames.It consistently surpasses prior training-free methods on RVS-Ego and RVS-Movie.
  • Offline Video Understanding: HERMES achieves 60.29% on Egoschema and 58.85% on VideoMME, while its 56.92% MVBench accuracy is comparable to the base model’s 57.02%.The reported results cover long and short offline video benchmarks under a limited video-token budget.
  • Efficiency Analysis: 10× faster TTFT is achieved by HERMES compared to the prior SOTA method.The comparison spans GPU memory and TTFT latency across input frame numbers.
  • Efficiency Analysis: HERMES maintains stable GPU memory usage and TTFT as input frames increase, using a 4K-token memory budget with LLaVA-OV-7B.TTFT is measured from user query input to decoding the first output token.
  • Efficiency Analysis: Fixed memory budgets keep GPU memory from increasing with longer videos, while TTFT and TPOT remain consistently low across video lengths and chunk sizes.These measurements support real-time responsiveness in the tested streaming settings.
  • Ablation Study: Performance stabilizes once the total memory budget reaches 4K, whereas long offline-video performance degrades significantly below 4K.Streaming datasets tolerate a smaller memory budget than long offline datasets.
  • Ablation Study: Allocating more tokens to deep layers improves long-term memory preservation and long-video VideoMME performance.Overall performance remains comparable across layer-dependent budget configurations.

5 Related Work

Related work spans external-memory and internal KV-cache approaches for streaming video understanding. HERMES targets the underexplored hierarchical organization of KV-cache memory for low-latency streaming use.

  • Streaming Video Understanding: External-memory methods store captions or vision patches and retrieve them at query time, which incurs high latency and can require model-specific training.These methods perform ad-hoc retrieval and multimodal prefilling before answering queries.
  • KV Cache Compression for Video Input: Existing online KV-cache compression methods largely overlook the hierarchical storage structure of KV cache in streaming scenarios.Offline compression methods are poorly suited to unpredictable future frames and user queries.
  • KV Cache Compression for Video Input: HERMES addresses this gap through fine-grained memory utilization and low-latency responses.Its related-work distinction motivates hierarchical KV-cache management for streaming video.

6 Conclusion

HERMES is presented as a training-free framework for efficient streaming video understanding. It treats KV cache as hierarchical video memory and uses cross-layer smoothing and position re-indexing for long streaming inputs.

  • Conclusion: HERMES conceptualizes KV cache as a hierarchical video memory system across multiple granularities.The framework is guided by mechanistic attention analysis.
  • Conclusion: Cross-layer memory smoothing and position re-indexing enhance understanding performance for long streaming inputs.These mechanisms are introduced within the HERMES framework.

Limitations

The paper identifies a scale-related evaluation boundary and reports that its attention findings remain consistent across sliding-window lengths. It also describes layer-specific guidance for token compression.

  • Scope boundary: Experiments do not include the 72B model variant because of computation resource constraints.The limitation is illustrated with Qwen2.5-VL-72B.
  • Evaluation evidence: Attention patterns remain consistent across varying sliding-window lengths, supporting the reported generality of the findings.The additional visualization is provided in Fig. 6.
  • Method scope: The compression procedure uses global guidance for deep layers and hybrid guidance for middle layers when conversation history is available.The guidance strategy reflects the different attention behaviors of deep and middle layers.

C Configuration of Cross-Layer Memory Smoothing

Cross-layer memory smoothing assigns different smoothing weights to shallow, middle, and deep layer stages. The paper motivates these choices by contrasting stable long-term memory with diverse short-term perception and evaluates them through ablation.

  • Configuration rationale: Different λ values are assigned to layer stages because long-term memory is relatively stable while short-term memory supports diverse perception.The passage motivates stage-specific smoothing rather than a single shared value.
  • Memory interpretation: The configuration links layer-stage behavior to distinct memory characteristics: stable long-term memory and diverse short-term memory.This summarizes the stated basis for choosing different smoothing weights.
  • Configuration validation: The selected cross-layer smoothing hyperparameter configuration is evaluated in an ablation study.The paper refers to Table 6 for this evaluation.

D Details of evaluated benchmarks

The streaming benchmark statistics distinguish question formats and identify subset abbreviations used to describe benchmark coverage.

  • Question format: Streaming benchmark questions are categorized as either multiple-choice (MC) or open-ended (OE).These abbreviations are defined in the benchmark statistics table.
  • Subset labels: The benchmark column uses “rt” for real-time understanding and “bw” for backward tracing subsets.The abbreviations specify the evaluated streaming subsets.
  • Benchmark statistics: Table 11 provides key statistics for the streaming benchmarks.The table organizes the benchmark metadata alongside the defined labels.

D.1 Streaming Benchmarks

The paper evaluates streaming video understanding using several benchmark families covering real-time perception, backward tracing, and long-form online scenarios. Its displayed materials also include guidance prompts and sliding-window visualizations.

  • StreamingBench: StreamingBench evaluates real-time visual, omni-source, and contextual understanding, with 2,500 questions across 500 videos in its real-time subset.The real-time subset covers 10 tasks, including object perception and causal reasoning.
  • OVO-Bench: OVO-Bench contains approximately 2,800 fine-grained multiple-choice question-answer pairs across 644 videos and 12 tasks.The paper uses its real-time perception and backward tracing subsets, excluding proactive responding.
  • Visualization and guidance: The streaming-benchmark materials include sliding-window attention visualizations and local and global guidance prompts for token compression.The figures cover different window sizes and conversation-history conditions.
  • RVS datasets: RVS-Ego and RVS-Movie comprise 10 Ego4D videos and 22 MovieNet clips, totaling over 21 hours of streaming video.These datasets target real-time understanding in online streaming scenarios.

D.2 Offline Benchmarks

Offline evaluation spans short and long video benchmarks, while HERMES also uses positional re-indexing and cached-key correction to support compact video-token representations. Case studies and benchmark descriptions cover temporal, spatial, and real-time visual understanding tasks.

  • Offline benchmarks: Egoschema contains over 5,000 human-curated multiple-choice QA pairs linked to egocentric video clips for long-form understanding.
  • Offline benchmarks: VideoMME evaluates MLLMs across six domains, 900 videos, 2,700 multimodal QA pairs, and diverse durations.
  • Positional re-indexing: Left-compaction reassigns retained 1D video-token positions contiguously after the fixed system prefix while preserving temporal order.The compacted position uses the retained token’s rank and a fixed prefix offset.
  • Positional re-indexing: Rotary delta correction aligns cached keys with remapped positions, enabling direct KV-state reuse without regenerating keys.The correction uses the relative positional shift and RoPE frequency vector.
  • Positional re-indexing: For Qwen2.5-VL, dimension-wise compaction creates a dense contiguous temporal-spatial grid while preserving monotonic ordering and distinct index roles.The procedure independently compacts temporal, height, and width coordinates while keeping the system prefix fixed.
  • Case studies: HERMES case studies report finer-grained temporal and spatial understanding than LLaVA-OV-7B on streaming long videos.Figures 11 and 12 present representative temporal and spatial examples; the cited table captions enumerate real-time and backward-tracing task categories.
Loading 2601.14724v4…