PaperlayerView exact revision on arXiv

Source-linked AI summary

DeltaS: Reading the Gated Linear Attention State for KV Cache Eviction in Streaming Video

Taeyoun Kwon, Seungjin Kim, Hyeonyu Kim, Moon Hwan Kim

arXiv:2609.27470v1cs.CVcs.CLcs.LG

TL;DR

Streaming hybrid video models must evict KV entries before the question arrives, even though their full-attention cache grows with the stream. DeltaS uses normalized recurrent-state drift to retain chunks, and it outperforms query-agnostic bounded-memory baselines across six long-video benchmarks. Its results support using the two memories of hybrid architectures cooperatively.

  • Problem

    Hybrid models retain a fixed-size recurrent state but accumulate a growing full-attention KV cache, requiring query-agnostic eviction before the question arrives.

    Source: §1 · p. 1

  • Method

    DeltaS scores chunks by normalized recurrent-state change and prioritizes higher-drift chunks for full-attention KV retention.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    DeltaS leads query-agnostic bounded-memory methods on all six benchmarks at 8,192 tokens and five of six at 16,384 tokens, beating StreamMem by 2.1 points on average at 8,192 tokens.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    Controlled comparisons show that state drift outperforms position-, attention-, and key-value-based signals under a shared budget and retention policy.

    Source: §1 · p. 2

    Evaluation uses one hybrid backbone, and DeltaS’s utility under other state-update rules remains to be validated.

    Source: §6 · p. 8

Abstract

from arXiv · show

Recent video-language models increasingly adopt hybrid architectures that interleave linear and full attention layers for efficient long-context processing. While the recurrent state of linear attention remains fixed in size, the KV cache of full attention continues to grow with the video stream, making eviction necessary under a bounded memory budget. The key challenge in streaming is that eviction must occur before the question arrives, so what to retain has to be decided without the question. Existing eviction methods derive token scores from the KV cache itself, using position, attention, or key-value representations, and attention-based scores further require proxy queries or extra computation. Hybrid backbones offer another source of signal. In gated-delta linear attention, the recurrent state is updated by the residual between each input and what can already be retrieved from the state, so its change over a chunk of frames reflects how much new information the chunk brings. We propose DeltaS, a query-agnostic, training-free method that retains video chunks inducing larger normalized state change, or state drift. In a controlled comparison with the budget and retention policy held fixed, state drift outperforms position-, attention-, and key-value-based signals. With a signal costing only 1.9% of the forward pass, DeltaS surpasses the strongest query-agnostic bounded-memory baseline by 2.1 points on average across six long-video benchmarks and by 5.6 points on the longest benchmark. These results suggest that the two memories of hybrid architectures can work cooperatively. Code is available at https://github.com/MaumAI-Company/DeltaS.

1. Introduction

Hybrid backbones keep a fixed-size recurrent state but still accumulate a growing full-attention KV cache, so streaming requires query-agnostic eviction. DeltaS uses normalized recurrent-state change to rank chunks for retention and outperforms existing query-agnostic baselines across long-video benchmarks.

1. Introduction

  • The recurrent state stays fixed at 48 MiB, but the full-attention KV cache grows linearly and exceeds 10 GiB after one hour in the studied hybrid backbone.Because retention decisions precede the question, the eviction signal must be query-agnostic.Source: §1 · p. 1
  • Existing methods score cached tokens using position, attention, or key-value representations, and attention-based scoring may require proxy queries or extra attention computation.Hybrid backbones also provide recurrent states that summarize preceding context across multiple temporal scales.Source: §1 · pp. 1–2
  • DeltaS scores each video chunk by normalized recurrent-state change and prioritizes higher-drift chunks for retention in the full-attention KV cache.The method uses a signal already computed in the forward pass, without proxy queries or additional attention computation.Source: §1 · p. 2
  • At an 8,192-token budget, DeltaS is best among query-agnostic bounded-memory methods on all six benchmarks, beating StreamMem by 2.1 points on average.At 16,384 tokens, it is best on five of six benchmarks, with an average gain of 1.7 points.Source: §1 · p. 2
  • With the budget and retention policy fixed, state drift outperforms position-, attention-, and key-value-based signals.The signal costs 1.9% of the forward pass.Source: §1 · p. 2

2. Related Work

Prior work manages streaming memory through training-based approaches or inference-time compression and token selection. DeltaS differs by using changes in an existing recurrent state to guide KV retention without a new memory module, extra training, or query access.

Bounded-memory streaming video

  • Inference-time methods either compress stored representations or select tokens, with some making retention decisions before the question is known.DeltaS targets query-agnostic, training-free, bounded-memory streaming.Source: §2 · p. 2

Signals for KV cache eviction

  • KV eviction signals draw on position, attention, or key/value properties, but all derive from information associated with the KV cache itself.Examples include attention received by cached tokens, key norms, and value norms.Source: §2 · p. 2

Recurrent state as a retention signal

  • DeltaS reads changes in an existing gated-delta recurrent state to guide full-attention KV retention without a new memory module, extra training, or downstream-query access.Unlike HOLA, DeltaS does not adopt a trained memory architecture; it also differs from DLA, which uses normalized state differences to determine memory-state boundaries.Source: §2 · p. 2

3. Method

The method makes query-agnostic retention decisions as chunks arrive, using normalized changes in recurrent state to rank their corresponding KV entries. Temporal bucketing counters an early-stream bias in global drift ranking.

3.1. Streaming Setup and Hybrid Memory

  • The recurrent state remains fixed-size while full-attention layers store token-wise key-value pairs, so bounded memory requires eviction from those KV caches.Questions may arrive at any point, requiring preceding retention decisions to be made without query access.Source: §3.1 · p. 3
  • Across heads and layers, recurrent-state retention horizons span multiple temporal scales, providing a reference for evaluating incoming chunks.Some heads preserve earlier state contributions for more than half after one minute.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2. State Drift

  • Gated-delta updates write a gated residual between an incoming value and what the decayed state retrieves, so chunk-level net state change reflects the combined writes and decay.The residual is scaled by the write strength.Source: §3.2 · p. 3
  • DeltaS defines state drift as the normalized change between pre- and post-chunk states, averaged across linear-attention layers into one chunk score.Normalization by the pre-chunk state magnitude makes changes comparable across layers.Source: §3.2 · p. 3

3.3. DeltaS: State-Guided KV Eviction

  • Each chunk’s state-drift score is assigned to its tokens and shared across full-attention layers; in the experiments, each chunk represents one second of video.The previous score is reused during later eviction steps.Source: §3.3 · p. 4, Figure 4 · p. 5
  • DeltaS preserves sink tokens and the recent window, then retains higher-scoring candidates within the remaining budget; equal temporal buckets reduce the early-stream bias of global ranking.Buckets are recomputed at eviction using original arrival times, without advance knowledge of the final stream length.Source: §3.3 · p. 4

4. Experiments

Across fixed-budget evaluations, DeltaS leads query-agnostic bounded-memory baselines, while controlled comparisons, temporal-bucketing tests, and runtime measurements examine its signal and costs.

4.1. Experimental Setup

  • The evaluation uses frozen Qwen3.5-9B, two-frame streaming chunks, six long-video benchmarks, and per-layer budgets of 8,192 or 16,384 KV tokens.The budgets accommodate slightly over one or two minutes of video; some additional benchmarks do not consistently reward longer context, and recent-frame-only can outperform Full KV on several tasks.Source: §4.1 · p. 5

4.2. Main Results

  • At M = 8,192, DeltaS leads query-agnostic bounded-memory methods on all six benchmarks, beating the strongest average baseline by 2.1 points; at M = 16,384, it leads on five of six with a 1.7-point average gain.At the smaller budget, its average accuracy is 58.7, nearly matching StreamMem’s 58.8 at twice the KV budget.Source: §4.2 · p. 6, Table 1 · p. 6
  • 5.6 points on LVBench at M = 8,192 and 4.3 points at M = 16,384 mark DeltaS’s gains over the strongest bounded-memory baseline on the longest benchmark.Across benchmarks, gains generally increase at lower retention rates, though Video-MME v2 maintains an approximately 0.5-point advantage at both budgets.Source: §4.2 · p. 6, Figure 5 · p. 6

4.3. Controlled Comparison of Retention Signals

  • Under a shared budget and retention setup, DeltaS ranks first on LongVideoBench, MLVU, and LVBench, beating VaN by 0.7–2.1 points and β∥e∥ by 0.5–1.2 points.Both state-based signals outperform position-, attention-, and key-value-based alternatives in this comparison.Source: §4.3 · p. 7, Table 2 · p. 7

4.4. Effect of Temporal Bucketing

  • Temporal buckets reduce early-stream concentration toward more uniform coverage, and N = 4 yields the largest mean accuracy gain over N = 0 among tested settings.The bucket-count ablation holds total KV budget fixed and excludes the reserved recent window.Source: §4.4 · p. 7, Figure 6 · p. 7, Figure 7 · p. 8

4.5. Efficiency Analysis

  • 2.07 ms computes the state-drift signal, or 1.9% of chunk-forward time; total added cost is 6.4 ms per eviction, versus 15.4 ms for InfiniPot-V and 206.1 ms for β∥e∥.The runtime measurements use a single A100 and report median latency over eviction steps.Source: §4.5 · p. 8, Figure 8 · p. 8

5. Conclusion

DeltaS uses recurrent-state drift for query-agnostic, training-free KV eviction in gated-delta hybrid video-language models. The results support using the fixed-size recurrent state both to summarize context and to guide KV retention.

5. Conclusion

  • DeltaS uses recurrent-state drift as a query-agnostic, training-free KV eviction signal in gated-delta hybrid video-language models.Source: §5 · p. 8
  • Across six long-video benchmarks, state-based signals outperform position-, attention-, and key-value-based alternatives under a shared budget and retention policy.Source: §5 · p. 8
  • Temporal bucketing reduces the tendency to favor early stream content, while reuse of forward-pass states keeps additional computation low.Source: §5 · p. 8

6. Limitations

The evaluation and interpretation of DeltaS are bounded by the tested architecture, state-update rule, and scoring granularity.

6. Limitations

  • DeltaS is evaluated on a single hybrid backbone, so its effectiveness across other architectures and model scales remains unestablished.The authors identify broader architecture and scale testing as future validation.Source: §6 · p. 8
  • The state-drift interpretation relies on the gated delta rule’s residual update, and utility under other state-update rules requires validation.Source: §6 · p. 8
  • Because all tokens in a chunk share one score, DeltaS cannot distinguish more informative regions within that chunk through its scoring mechanism.The authors identify finer-granularity scoring as a natural extension.Source: §6 · p. 8

A.1. Method Adaptations for Table 1

The Table 1 comparison ports bounded-memory methods to a common Qwen3.5-9B streaming setup while preserving method-specific choices where possible. The appendix describes adaptations for chunking, scoring, and reference systems.

A.1. Method Adaptations for Table 1

  • Table 1 compares complete systems on Qwen3.5-9B with the same per-layer KV budget, while retaining method-specific choices when possible.Eviction applies only to the eight full-attention layers; substitutions are made when architecture prevents a direct port.Source: §A.1 · p. 11
  • Most methods compress whenever the cache exceeds budget; InfiniPot-V instead removes 25% of its budget and refills over about 18.5 steps at M = 8,192.The default path uses two-frame chunks; StreamMem and HERMES retain their original eight- and 16-frame chunks, respectively.Source: §A.1 · p. 11
  • InfiniPot-V uses the paper’s 0.125 recent-frame ratio, post-encoding keys for TaR, and includes inter-chunk separator text among compressible tokens.Its budget is split evenly between TaR and VaN; VaN is unchanged by the key-representation choice because it uses values.Source: §A.1 · p. 11
  • StreamMem adaptations use two-frame temporal-grid units for filtering and retain one representative when duplicate frames are found.The paper is the reference because no public implementation is available; remaining hyperparameters follow its ablation-optimal configuration.Source: §A.1 · p. 11
  • HERMES uses the released implementation’s approximate-attention path and enforces at least one shallow layer, leaving one of eight layer assignments different from the original.Its position re-indexing was verified as bit-identical to a direct transcription of the original operation.Source: §A.1 · p. 11
  • The simple controls vary only retention rules, while ReKV is query-aware and Full KV disables eviction as an unbounded-memory reference.Uniform samples over absolute stream time; recency keeps the most recent context, and ReKV adds retrieval after the question arrives.Source: §A.1 · p. 11
  • Where possible, key implementation components were checked against reference implementations rather than comparing published accuracy values across different backbones.HERMES re-indexing is bit-identical, and InfiniPot-V TaR scores are byte-identical when the complete frame grid is retained.Source: §A.1 · p. 11

A.2. Controlled Retention-Signal Comparison

The controlled comparison standardizes the retention setting to isolate scoring signals, while preserving signal-specific choices that cannot be separated from their definitions. It compares individual signals rather than reproducing each complete method.

A.2. Controlled Retention-Signal Comparison

  • All signals use M = 8,192, four sink tokens, a 1,024-token recent region, and two-frame chunks, with the remaining capacity selected by each score.The comparison replaces original pure top-k or frame-based conventions with a common token-count budget; method-specific scoring granularity is retained.Source: §A.2 · pp. 12–13, Table 4 · p. 13
  • SnapKV recomputes attention scores from the last 32 tokens of each incoming chunk without accumulating scores over time.The observation tokens remain in the common recent region; scores are pooled along cache index, and query-head scores sharing a KV head are aggregated before selection.Source: §A.2 · pp. 12–13
  • The key-norm signal scores the current cache directly, while KeyDiff recomputes its mean-direction anchor over the surviving cache at each eviction.Key-norm is unchanged by positional rotation; KeyDiff uses post-positional-encoding keys, consistent with its reference implementation.Source: §A.2 · p. 12
  • TaR compares retained keys with recent-region mean key directions at matching patch coordinates, using a linearized computation after token eviction destroys the full frame grid.For a complete grid, this formulation produces byte-identical scores to a transcription of the original method.Source: §A.2 · p. 12
  • VaN retains the value-norm score but disables layer-adaptive pooling, while the state-based score uses β∥e∥ as a shared chunk-level ranking.The state score averages per-head and per-layer residual-write magnitudes, then averages across chunk tokens; its residual write has rank at most one under the stated formulation.Source: §A.2 · pp. 12–13
  • Table 2 ranks each signal independently under the shared budget, unlike InfiniPot-V’s original combination of TaR, VaN, and a recent region.The controlled rows represent individual signals, not reproductions of complete methods.Source: §A.2 · p. 13

A.3. Reproducibility Details

The appendix specifies the model, streaming inputs, benchmark protocols, memory budgets, and runtime procedure used for reproducibility. It also notes protocol choices that limit direct comparison with some official benchmark results.

A.3. Reproducibility Details

  • Experiments use the public Qwen3.5-9B checkpoint in bfloat16, with 24 Gated DeltaNet layers and eight full-attention layers that each have four KV heads.The full-attention cache uses 32 KiB per token across layers, while recurrent linear-attention state remains fixed in size.Source: §A.3 · p. 13
  • Videos are sampled uniformly at 2 fps, and each two-frame chunk contains 98–112 visual tokens plus 20 timestamp tokens.All methods use the same chat template and timestamp convention; question and response KV entries are discarded before ingestion resumes.Source: §A.3 · p. 13
  • Video-MME and LongVideoBench use frames only, while Video-MME v2 reports per-question accuracy rather than the official group-based non-linear score.EgoSchema uses its public 500-item subset; the stated OVO-Bench regimes are backward tracing and real-time visual perception, excluding forward active responding.Source: §A.3 · pp. 13–14
  • Each full-attention layer receives M = 8,192 or 16,384 retained tokens, corresponding to approximately 256 MiB or 512 MiB of KV memory.At the sampling rate used, these budgets represent slightly over one or two minutes of video.Source: §A.3 · p. 14
  • Runtime is measured over eviction steps while streaming a 4,730-second MLVU video for 600 steps on one A100, with three repetitions and warm-up discarded.All methods use two-frame chunks for this measurement; accuracy is not evaluated under that modified setting, and the figure’s latency decomposition is less precise than total latency.Source: §A.3 · p. 14

B. Streaming QA Benchmarks

Short-context baselines lead aggregate scores on these streaming-QA benchmarks, but task-level results distinguish recent-answerable questions from those requiring earlier video information. DeltaS performs strongly on memory-dependent tasks, while HLD shows that more context can also reduce accuracy.

B. Streaming QA Benchmarks

  • The memoryless recency-4f and recency-8f baselines use 8 and 16 frames at 2 fps, respectively, and need no KV eviction.Their median cache lengths on StreamingBench are 1,552 and 2,911 tokens, respectively.Source: B. Streaming QA Benchmarks · p. 14
  • Short-context baselines achieve the highest aggregate scores across all three columns and outperform Full KV, which retains the complete video history.The table reports OVO-Bench’s two question-weighted multiple-choice regimes and StreamingBench; recency baselines are not bound by M.Source: Table 6 · p. 14, B. Streaming QA Benchmarks · p. 15
  • Across OVO-Bench’s nine tracks, recency-4f beats Full KV on seven, while Full KV leads on EPM by 2.7 points and ASI by 8.8 points.Five of ten StreamingBench tasks show a similar pattern; tracks favoring recency primarily ask about information near the stream’s end.Source: B. Streaming QA Benchmarks · p. 15
  • On HLD, recency-4f scores 43.0 versus 18.3 for Full KV, 24.2 for ReKV, and 19.9 for DeltaS; random-guess expectation is 27.6%.Every method processing the complete video scores below that expectation.Source: B. Streaming QA Benchmarks · p. 15
  • At M = 8,192, DeltaS reaches 66.7 on Backward excluding HLD, versus 64.5 for Full KV and 65.6 for the next-best bounded-memory method, InfiniPot-V.DeltaS also ranks first on EPM at this budget and scores 60.1 on Counting, compared with 39.4 for recency-4f and 39.9 for recency-8f.Source: Table 6 · p. 14, B. Streaming QA Benchmarks · p. 15
  • Aggregate performance is strongly influenced by tasks solvable from recent observations, and longer context can make some tasks harder.The authors therefore report these benchmarks separately and use memory-dependent tasks to assess whether eviction preserves information needed later.Source: B. Streaming QA Benchmarks · p. 15

Related papers

Loading 2609.27470v1…