PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-MemComp: Taylor-Guided Sensitivity-Aware Memory Management for Streaming Video Understanding

Yonghui Niu, Yuhao Ji, Chang Nie, Jiao Zhang, Yilong Li, Junlan Feng, Ran He, Caifeng Shan, Chaoyou Fu

arXiv:2610.32784v1cs.CV

TL;DR

Streaming video creates memory pressure for MLLMs, and existing compression methods may overlook accumulating approximation errors. Video-MemComp uses a Taylor-guided framework to reduce local error and allocate cache by layer sensitivity. It matches or surpasses full-cache baselines across benchmarks, including a reported accuracy increase on Qwen2.5-VL-3B with over 90% less KV-cache memory.

  • Problem

    Streaming video has unbounded input while MLLMs have finite memory, and existing compression methods can overlook layer-wise error accumulation and amplification.

    Source: §1 · pp. 1–2

  • Method

    Video-MemComp uses a first-order Taylor expansion to separate local approximation error from layer-wise structural sensitivity, guiding Pre-RoPE aggregation and sensitivity-aware cache budgeting.

    Source: §1 · pp. 2–3

  • Results

    Across benchmarks, Video-MemComp matches or surpasses full-cache baselines; Qwen2.5-VL-3B overall accuracy rises from 62.3% to 64.1% with average KV cache limited to 6,000 tokens per layer and memory reduced by over 90%.

    Source: §1 · p. 3, §5 · p. 12

  • Takeaways & Limitations

    The reported results support a training-free, model-agnostic approach to streaming video understanding with a strict O(1) memory footprint.

    Source: §5 · p. 12

    The Taylor decomposition may break down under extreme compression, offline sensitivity priors may degrade under distribution shifts, and Pre-RoPE aggregation adds moderate overhead.

    Source: §5 · p. 12

Abstract

from arXiv · show

The rapid growth of video data has created an urgent need for efficient video understanding systems that can handle long and continuous visual inputs, posing significant challenges for Multimodal Large Language Models (MLLMs) due to severe memory bottlenecks. Moreover, existing compression methods often overlook the explosive accumulation of approximation errors. To address this issue, we propose Video-MemComp, a training-free and sensitivity-aware context management framework for memory-constrained streaming video understanding. Our method theoretically decomposes compression error into orthogonal local approximation error and layer-wise structural sensitivity via a first-order Taylor expansion. To minimize local approximation error, we perform token aggregation in a pre-RoPE safe zone with an attention-conserving logit bias, mitigating geometric distortions caused by RoPE rotations. To manage layer-wise structural sensitivity, we dynamically allocate the KV cache budget based on offline gradient priors and online context entropy, prioritizing high-sensitivity layers. Video-MemComp outperforms or achieves performance comparable to full-cache and state-of-the-art compression methods on benchmarks including MLVU, VideoMME, and EgoSchema. Notably, it improves the overall accuracy of Qwen2.5-VL-3B from 62.3% to 64.1% while reducing the KV cache to an equivalent of 6,000 tokens per layer, saving over 90% of KV memory.

1 Introduction

Streaming video creates unbounded input under finite memory, while existing compression can miss layer-dependent error amplification and RoPE-related distortions. Video-MemComp decomposes compression error to guide local aggregation and sensitivity-aware memory allocation, and reports competitive or better performance than full-cache baselines.

1 Introduction

  • Streaming video combines temporally unbounded inputs with finite GPU memory, while KV-cache size grows linearly with sequence length and risks out-of-memory failures.The paper also identifies compression errors that can amplify across layers and RoPE rotations that distort merging similarities.Source: §1 · pp. 1–2
  • A first-order Taylor expansion decomposes output error into local approximation error El and layer-wise structural sensitivity Sl, which measures gradient-based amplification.The design objective is to minimize local error while explicitly managing structural sensitivity.Source: §1 · p. 2
  • The framework combines Pre-RoPE aggregation and sensitivity-aware budget allocation to target the two error components under an O(1) memory design.The paper characterizes the approach as training-free and model-agnostic.Source: §1 · pp. 2–3
  • On MLVU with Qwen2-VL-7B, Video-MemComp outperforms strong baselines across memory budgets and surpasses Full KV at 6K, 12K, and 24K budgets.The result is described as removing redundancy while preserving key semantics.Source: Figure 2 · p. 3
  • Pre-RoPE aggregation uses attention-conserving logit bias to preserve attention mass during token consolidation, avoiding RoPE-induced geometric barriers to merging.Budget allocation uses offline gradient priors to reclaim memory from low-sensitivity layers for high-sensitivity layers.Source: §1 · p. 3
  • 62.3% to 64.1% overall accuracy on Qwen2.5-VL-3B, with average KV cache limited to 6,000 tokens per layer and memory reduced by over 90%.The paper also reports matching or surpassing full-cache baselines across MLVU, Video-MME, and EgoSchema.Source: §1 · p. 3

2 Related Work

Prior work addresses visual-token redundancy and KV-cache growth through spatial or temporal compression, heuristic pruning, and dynamic budgeting. Video-MemComp distinguishes itself by combining a Taylor-based error decomposition with local-error reduction and structural-sensitivity-aware allocation under O(1) memory.

2 Related Work

  • Spatial token-compression methods reduce intra-frame redundancy, while spatio-temporal approaches exploit inter-frame similarity but mostly rely on heuristics.Some spatio-temporal methods partially violate attention conservation, and static-centric logit-bias compensation does not resolve O(T) growth.Source: §2 · pp. 3–4
  • The cited visual-token literature includes spatial methods that often overlook temporal redundancy, motivating approaches that account for both intra-frame and inter-frame structure.The distinction is stated in the related-work discussion rather than as a benchmark result.Source: §2 · pp. 3–4
  • KV-cache methods use fixed sparsity, stabilization, cumulative attention, importance metrics, or dynamic layer-wise budgeting based on signals such as attention entropy or feature norms.The cited discussion says these methods often overlook layer-wise structural sensitivity, where errors can accumulate and amplify.Source: §2 · p. 4
  • Streaming retrieval and store-then-retrieve schemes incur linear storage, while bounded-memory methods may ignore inter-frame similarity or rely on heuristics.The discussion notes that even O(1) approaches can suffer information loss.Source: §2 · p. 4
  • Video-MemComp uses Taylor expansion to decompose compression into local approximation error and layer-wise structural sensitivity while maintaining a strict O(1) memory footprint.Its overview maps local-error reduction to aggregation and pruning, and sensitivity management to tri-hybrid budget allocation.Source: §2 · p. 4, Figure 3 · p. 4

3 Methodology

Video-MemComp uses a first-order Taylor expansion to separate compression error into local approximation error and layer-wise structural sensitivity. It addresses these components through token aggregation and pruning, then sensitivity-aware allocation of the KV-cache budget.

3.1 Problem Reformulation via Taylor Expansion

  • A first-order Taylor expansion separates output error into local approximation error and layer-wise structural sensitivity, assuming sufficiently small layer perturbations.Structural sensitivity acts as a global amplification factor, so layers with larger sensitivity warrant more budget.Source: §3.1 · p. 5

3.2 Minimizing Local Approximation Error El

  • Pre-RoPE aggregation avoids RoPE-distorted similarity, while cosine similarity identifies redundant tokens and fusion-count logit bias preserves their attention mass.The bias is computed as log(f_i), where f_i tracks the number of raw tokens represented by a surviving token.Source: §3.2 · p. 5
  • A proxy query estimates cached-key importance without future knowledge, and each layer retains its top b(l) KV pairs under the allocated budget.The layer-specific budget is set by the inter-layer allocation strategy.Source: §3.2 · p. 6

3.3 Minimizing Local Approximation Error El

  • Under an average budget of 500 KV tokens per layer, the allocation assigns more budget to shallow layers and prunes deeper layers more aggressively.Figure 4 displays layer-wise budgets alongside normalized gradient norm, entropy, and feature magnitude.Source: Figure 4 · p. 7, §3.3 · p. 7
  • The budget objective minimizes expected sensitivity-weighted local error, but the exact optimization is NP-hard and is approximated with a composite importance score.The score combines a uniform minimum budget, offline gradient-based structural sensitivity, and online entropy and feature-magnitude measures; the resulting allocation is front-heavy.Source: §3.3 · pp. 6–8

4 Experiments

Across offline and streaming evaluations, Video-MemComp performs comparably to or better than baselines under restricted KV budgets. Ablations examine its compression components, layer-wise allocation, and threshold choices.

4.1 Experimental Setup

  • The evaluation covers four MLLM backbones, offline video benchmarks, and streaming QA datasets, with official development sets used for offline results.Streaming tests include timestamped RVS-Ego/Movie QA and choice-based OVO-Bench; detailed OVO-Bench configurations and results are in the supplement.Source: §4.1 · p. 8

4.2 Evaluation Results

  • At a restricted 6K KV budget, Video-MemComp scores 65.5 on Qwen-2-VL-7B, exceeding InfiniPot-V (64.7) and the full-input baseline (64.9).It also scores 64.1 vs. 62.3 on Qwen2.5-VL-3B and shows comparable or superior performance across four backbones.Source: §4.2 · p. 9, Table 1 · p. 9
  • 56.6% average accuracy makes Video-MemComp the top memory-constrained method on RVS streaming benchmarks, exceeding StreamingTOM (55.8%).On RVS-Ego, it reaches 60.1%, above StreamingTOM (58.3%) and InfiniPot-V (57.9%).Source: §4.2 · p. 9, Table 2 · p. 10
  • On LLaVA-OneVision-7B, Video-MemComp reduces GPU memory use and interaction latency versus ReKV, although ReKV has higher throughput.The efficiency evaluation uses an NVIDIA RTX A6000; reported measures include peak memory, visual encoder latency per frame, and time to first token.Source: §4.2 · p. 9, Table 3 · p. 10

4.3 Ablation Study

  • Hybrid intra-layer compression combining Pre-RoPE aggregation and sparsity-based KV pruning outperforms single-mechanism baselines; adding intra-frame aggregation raises MLVU from 67.68 to 68.19.The intra-frame gains extend to VideoMME Medium and Long, with minor drops on short clips.Source: §4.3 · p. 10, Table 4 · p. 10
  • Sensitivity-aware dynamic allocation consistently outperforms uniform allocation across similarity thresholds, with the largest gains on long-video and aggregate metrics.The comparison uses identical intra-layer compression to isolate inter-layer scheduling effects.Source: §4.3 · p. 11, Table 5 · p. 11
  • On OVO-Bench, allocations dominated by structural sensitivity outperform uniform-heavy configurations, while a small dynamic weight (λe ≈0.025) yields the global optimum.Excessive reliance on dynamic metrics degrades performance due to high variance in online activations.Source: §4.3 · p. 11, Figure 6 · p. 12
  • Accuracy peaks at θinter = 0.8 and θintra = 0.85; aggressive temporal pruning (θinter < 0.5) causes accuracy collapse.Overly conservative thresholds retain redundancy, exhaust the KV budget, and increase local approximation error El.Source: §4.3 · p. 11

5 Conclusion

Video-MemComp frames streaming-cache compression as managing local information loss and layer-wise sensitivity, then applies that distinction to aggregation and pruning. The conclusion also identifies boundaries of the approach, while the appendix roadmap previews its supporting analysis and evaluations.

5 Conclusion

  • Video-MemComp decomposes global output error into local approximation error and layer-wise structural sensitivity to guide compression under an O(1) memory budget.The conclusion describes the framework as training-free and model-agnostic.Source: §5 · p. 12
  • The Taylor-based decomposition is heuristic and may fail under extreme compression; offline sensitivity priors may degrade under distribution shifts, and Pre-RoPE aggregation adds moderate overhead.Future work includes higher-order approximations, online or adaptive sensitivity estimation, and more efficient aggregation.Source: §5 · p. 12

A Appendix Roadmap

  • The appendix roadmap assigns error decomposition to section B, hybrid-compression theory and implementation to C, sensitivity-aware allocation to D, and supplementary evaluations to E and F.Source: A Appendix Roadmap · p. 17

B.1 Problem Formulation

  • The compression objective maps the full KV cache to a sparse cache under a strict global memory budget while minimizing output deviation.Source: §B.1 · p. 17

B.3.1 Case 1: Similarity Redundancy (Minimization via Aggregation)

  • For semantically similar tokens with similar keys and values, aggregation uses a representative centroid and a log(N) bias to preserve their combined attention mass.The merging is performed in Pre-RoPE space to avoid RoPE’s geometric barrier.Source: §B.3.1 · p. 18

B.3.2 Case 2: Sparsity Redundancy (Minimization via Pruning)

  • For negligible-attention tokens, importance-guided pruning bounds the attention-output approximation error by the total pruned probability mass.Source: §B.3.2 · p. 18

B.2 Error Analysis via First-Order Taylor Approximation

The section uses first-order Taylor analysis to separate compression error into local attention-output deviation and layer-dependent amplification. It also motivates complementary approaches to token similarity and sparsity, with cosine similarity justified under stated assumptions.

B.2 Error Analysis via First-Order Taylor Approximation

  • A first-order Taylor expansion supplies a feasible proxy for output-error analysis by tracking compression-induced deviations in each layer’s attention output.Directly minimizing global output error is described as intractable because the model is high-dimensional and nonlinear.Source: §B.2 · p. 17
  • Local approximation error measures information loss within a layer, while structural sensitivity captures gradient-based amplification that motivates prioritizing higher-sensitivity layers.Source: §B.2 · p. 17

B.3 Minimizing Local Error El

  • The section describes similarity and sparsity as distinct forms of video redundancy addressed by the hybrid strategy.Source: §B.3 · p. 18

C Minimizing Local Approximation Error: Theoretical Justification and Hybrid Compression Implementation

  • The appendix presents a hybrid compression framework spanning cosine-similarity grounding, complementary temporal filtering and Pre-RoPE aggregation, and adaptive compression as variable-rate quantization.Source: C Minimizing Local Approximation… · p. 19

C.1 Optimality Proof of Cosine Similarity Under Generalized Attention Constraints

  • Token redundancy is framed as consistency of attention-logit responses to arbitrary queries after key projection and positional encoding.Source: §C.1 · p. 19

C.1.1 Formalization of Generalized Response Error

  • The response-error formalization assumes normalized input features on a hypersphere and treats RoPE as an orthogonal rotation.It also defines an arbitrary query as a potential attention query generated within the network.Source: §C.1.1 · p. 19

C.1.2 Local Equivariance and Effective Back-Projection

  • For locally positioned tokens, continuity of RoPE permits a shared local rotation approximation; the resulting effective back-projected query spans feature space when the key projection is full-rank.Source: §C.1.2 · p. 19

C.1.3 Worst-Case Error Bound Analysis

This section links similarity-based compression to attention-response consistency under local-position assumptions, then describes temporal filtering, spatial aggregation, and adaptive token budgets. The implementation uses a two-stage process to filter repeated frame content and aggregate surviving tokens.

C.1.3 Worst-Case Error Bound Analysis

  • Under local positional approximation and balanced feature magnitudes, cosine similarity in input space is a rigorous condition for attention-response consistency despite key projection and RoPE.Source: §C.1.3 · p. 20

C.2 Mechanisms of the Dual Compression Strategy

  • The dual strategy drops temporally redundant tokens without repair and merges spatially redundant tokens with repair.Source: §C.2 · p. 20

C.2.1 Inter-Frame: Implicit Temporal High-Pass Filtering

  • Discarding current-frame tokens already represented in history removes their softmax denominator term, increasing the relative probability mass of new changes and valid history.The authors describe this as a temporal high-pass filter that focuses attention on high-frequency change regions.Source: §C.2.1 · p. 20

C.2.2 Intra-Frame: Energy Conservation via Logit Bias

  • Merging spatially redundant tokens uses a log(f_i) bias to conserve attention mass and preserve the semantic weight of compressed background regions.The passage says this correction suppresses hallucinations associated with missing background information.Source: §C.2.2 · pp. 20–21

C.3 Adaptive Quantization: A Fixed-Distortion Variable-Rate Perspective

  • Under fixed distortion, the retained-token count varies with frame information entropy rather than remaining fixed per frame.The proposed allocation adapts the KV budget to content richness, such as static versus dynamic scenes.Source: §C.3 · p. 21

C.4 Algorithmic Implementation Details

  • The implementation first filters repetitive information between frames, then aggregates surviving tokens within each frame through matching and weighted reconstruction.The described steps include cosine-similarity filtering and mutual-best-match token merging.Source: §C.4 · p. 21

D Managing Structural Sensitivity: Rigorous Derivation of Budget Allocation Strategy

The derivation treats cache compression as a constrained error-minimization problem and motivates combining local-error signals with layer sensitivity. It develops robust dynamic scoring and a structural prior, then evaluates Video-MemComp across compression budgets.

D.1 Problem Formulation

D.2 Derivation of the Error Upper Bound

  • The error bound yields a greedy retention rule favoring tokens with the largest product of attention probability and value-vector norm.The bound accounts for Softmax renormalization after pruning.Source: §D.2 · pp. 22–23

D.3 Robust Logarithmic Surrogate

  • Direct linear scoring is numerically unstable under heterogeneous, heavy-tailed activations, so the method uses logarithmic utility with diminishing returns.The surrogate combines log feature magnitude with log(1 + attention entropy), tempering outliers and excessive investment in diffuse distributions.Source: §D.3 · p. 23

D.4 Derivation of Structural Sensitivity Prior

  • A first-order Taylor analysis separates local distortion from global layer sensitivity, indicating that allocation should account for both.Computing Jacobian norms online requires full backpropagation, so the method estimates sensitivity from calibration data under an assumption of relatively stable layer importance within a task domain.Source: §D.4 · p. 23, §D.4.1 · p. 24, §D.4.2 · p. 24

D.4.2 Gradient Surrogate and Structural Invariance Assumption

  • The structural prior uses expected gradient norms from LLaVA-Pretrain, a single-image calibration set chosen to avoid video-domain data leakage.The derivation argues that budgets should increase with layer sensitivity.Source: §D.4.2 · p. 24

D.5 Bayesian Interpretation of the Hybrid Strategy

  • The hybrid allocation is interpreted as MAP estimation, combining dynamic evidence, an offline structural prior, and a uniform smoothing term.The trade-off weights control confidence in static architectural features versus online observations.Source: §D.5 · pp. 24–25

E.1 Comparison with Various Compression Methods at Different Ratios

  • Evaluation on Qwen2-VL-7B uses MLVU’s official validation set and matches global cache capacity across 3,000, 6,000, 12,000, and 24,000 tokens per layer.The comparison includes visual-token, LLM-oriented KV, MLLM-specific KV, and uniform-sampling baselines.Source: §E.1 · p. 25

E.1.2 Experimental Results

  • 68.69% and 69.43% accuracy at 12K and 24K budgets surpass the 65.85% Full KV baseline on MLVU.The method consistently outperforms all baselines across the reported budget levels.Source: §E.1.2 · p. 25

E.2.3 Results Analysis

The analysis contrasts token compression, KV-cache compression, and their hybrid across memory budgets, then gives an information-theoretic interpretation of that trade-off. It also introduces an empirical analysis of layer-wise statistics.

E.2.3 Results Analysis

  • 52.89% accuracy at a 12,000-token budget lets token compression surpass the original model, whose accuracy is 50.56%.KV-cache-only compression stagnates above 3,000 tokens near the original-model level, while the hybrid offers a favorable trade-off across budgets.Source: §E.2.3 · p. 26

E.2.4 An Information-Theoretic Perspective

  • The framework is framed as joint source-channel coding: token compression aggregates similar inputs, while KV compression allocates storage to important features.The reported comparison says the hybrid combines token compression’s high-budget information retention with KV compression’s low-budget robustness.Source: §E.2.4 · p. 27

E.3 Empirical Analysis of Layer-wise Statistics

  • The accompanying empirical analysis examines layer-wise statistics of Qwen2.5-VL-3B to motivate combining structural priors with dynamic context metrics.The stated purpose is to justify integrating those two kinds of information.Source: §E.3 · p. 27

E.3.1 Data Collection Protocols

The data-collection protocol measures static structural sensitivity on single-image examples and dynamic context metrics during long-video inference.

E.3.1 Data Collection Protocols

  • Static sensitivity uses expected attention-output gradient norms from 100 LLaVA-Pretrain examples, while dynamic entropy and value-vector magnitudes are tracked on 10 VideoMME streams.The two datasets provide static and online measurements from independent data distributions.Source: §E.3.1 · p. 27

E.3.2 Static Prior Analysis: Gradient Norm Distribution

The reported gradient statistics show that shallow layers have high mean sensitivity but substantial sample-to-sample variation, motivating budget allocation that is not based on static priors alone.

E.3.2 Static Prior Analysis: Gradient Norm Distribution

  • Shallow layers 0–10 have extremely high mean gradients, consistent with the proposed error-avalanche effect and a need to skew budgets toward them.The passage describes minor shallow-layer perturbations as being amplified during deep propagation.Source: §E.3.2 · p. 28
  • Shallow-layer relative standard deviation generally exceeds 30%, indicating sample variation that static sensitivity priors alone do not capture.The analysis argues for adding online dynamic metrics to adapt to individual samples.Source: §E.3.2 · p. 28

E.3.3 Dynamic Behavior Analysis: Three-Stage Evolution

The analysis describes a three-stage distribution of entropy, gradients, and feature norms that informs layer-wise budget allocation. It also reports benchmark-specific model configurations and the evaluator used for streaming-video assessment.

E.3.3 Dynamic Behavior Analysis: Three-Stage Evolution

  • Layers 0–2 receive ample budget for high entropy and gradients; layers 3–30 show entropy near 10^-20 and redundancy; layers 31–35 get limited budget despite peak feature norm.The dynamic entropy metric identifies the middle-layer zone for aggressive pruning and budget savings.Source: §E.3.3 · pp. 28–29
  • Table 10 reports structural-sensitivity means, where higher values indicate greater sensitivity to pruning errors, and RSD, which reflects variation across samples.Source: Table 10 · p. 29

E.4 Model Adaptation and Hyperparameter Configuration

  • Configurations were optimized for each model–benchmark combination, including duration-adjusted compression thresholds for LLaVA-OV and resolution-aware frame sampling for InternVL.Specific hyperparameter configurations are listed in Table 11.Source: §E.4 · p. 29, Table 11 · p. 29

F.1 Evaluation Protocols on RVS-Benchmarks

  • RVS-Ego and RVS-Movie open-ended answers were judged by an LLM evaluator; the targeted gpt-3.5-turbo endpoint returned the backend identifier gpt-4o-mini-2025-07-18.The authors report the backend detail for reproducibility following API-provider updates.Source: §F.1 · p. 30

F.2 Extended Evaluation on OVO-Bench

On OVO-Bench, Video-MemComp approaches the full-input score on Qwen2.5-VL-7B with a restricted cache, and exceeds the baseline on Backward Tracing. The following subsections describe budget-allocation and token-threshold ablation studies.

F.2 Extended Evaluation on OVO-Bench

  • 50.8 overall score nearly matches the Qwen2.5-VL-7B full-input baseline’s 50.9, with under 4,000 KV-cache tokens; Backward Tracing scores 49.1 versus 47.4.The authors attribute the latter result to retaining key cues from the historical stream.Source: §F.2 · p. 30

F.3 Budget Allocation Hyperparameters Ablation Study Details

  • Table 13 reports OVO-Bench accuracy across 12 subtasks grouped into Backward Tracing, Real-time Perception, and Forward Active Responding, with configurations sorted by uniform-prior weight.The extreme pure-strategy configurations appear at the top and bottom; the global optimum is highlighted.Source: §F.3 · p. 30, Table 13 · p. 31

F.4 Token Compression Thresholds Ablation Study Details

  • Table 14 reports accuracy for Backward, Real-time, and Forward tasks across inter-frame and intra-frame similarity-threshold combinations.The best configuration is highlighted.Source: §F.4 · p. 30, Table 14 · p. 31
Loading 2610.32784v1…