PaperlayerView exact revision on arXiv

Source-linked AI summary

DensityKV: Density-Guided KV Cache Compression for Long Video Generation

Wenqu Zhao, Xuemin Chi, Xin Zhang, Guoqing Ma, Baorun Li, Jianjie Fang, Peizhi Tang, Chen Gao, Wei Wu

arXiv:2608.27922v1cs.CV

TL;DR

Long autoregressive video generation loses earlier context while accumulating redundant historical KV states, creating a consistency and storage challenge. DensityKV manages exact token-level KV banks per attention head using post-RoPE Soft-Riesz density and admission-relative growth constraints. Across three backbones, it improves long-horizon consistency under the same capacity bound and keeps persistent historical storage independent of rollout length.

  • Problem

    Long rollouts propagate appearance and motion errors while sliding-window attention discards earlier states, and storing all historical KV states grows memory with redundant coverage.

  • Method

    DensityKV is a training-free strategy that uses per-head token banks, post-RoPE key geometry, Soft-Riesz density, and insertion-relative density constraints to manage exact K/V states.

  • Results

    DensityKV achieves the best overall average rank and 31 of 54 single-metric wins across three backbones and 30–120-second generations under the same historical-capacity bound.

  • Takeaways & Limitations

    DensityKV preserves non-local subject and scene evidence while keeping historical storage bounded independently of rollout length and leaving retained states directly usable by native attention.

Abstract

from arXiv · show

Autoregressive video diffusion models enable streaming generation through sliding-window attention, but each generated block is conditioned on previously generated content, causing appearance and motion errors to propagate recursively over time. Historical key-value (KV) memory preserves earlier subject and scene states and helps maintain long-horizon consistency. However, retaining every generated state creates a historical archive that grows continuously with the rollout, while recurrent states repeatedly add redundant coverage. To address this problem, we propose DensityKV, a training-free historical KV bank management strategy. DensityKV maintains a separate token-level KV bank for each attention head and measures local redundancy among the post-RoPE keys that directly parameterize attention routing using Soft-Riesz density. By constraining neighborhood-density growth after states enter the bank, DensityKV limits repeated historical accumulation while preserving coherent states from each completed generation block. Experiments across three autoregressive video generation backbones and multiple generation lengths show that, at the same upper bound on historical KV capacity, DensityKV improves long-horizon consistency and generation stability while keeping persistent historical storage bounded independently of rollout length.

1 Introduction

DensityKV addresses long-video consistency by managing historical KV memory at token and attention-head granularity, using post-RoPE key density to limit redundant accumulation under bounded capacity.

  • Recursive conditioning propagates appearance and motion errors, while sliding-window attention discards earlier subject and scene states during long rollouts.
  • Storing all historical states makes the cache grow with generation length, and many retained states provide redundant coverage.
  • Token-level selection lets different regions compete for memory according to local key-space coverage, unlike coarse frame-level retention.
  • Post-RoPE key-space density provides query-independent redundancy estimates, while values remain paired exactly with their original keys.
  • DensityKV uses per-head token banks, Soft-Riesz crowding, and insertion-relative density constraints to admit and evict states without training or reconstruction.
  • Across three backbones and 30–120-second generations, DensityKV achieves the best overall average rank and 31 of 54 single-metric wins under matched capacity bounds.

2 Related Work

Prior long-video methods modify generation schedules, retain or retrieve historical context, or compress representations; DensityKV instead performs online, query-agnostic selection of exact native KV pairs.

  • Long-video approaches extend generation through noise rescheduling, FIFO denoising, or causal distillation beyond the training horizon.
  • Historical-memory methods preserve anchors, structured buffers, retrieved latent blocks, dynamic anchors, or scene memories to reintroduce discarded context.
  • Many recent methods rely on retrieval signals, predicted queries, head-role salience, future-query proxies, or transformed cache representations.
  • DensityKV updates online as clean blocks arrive, preserving exact K/V pairs through native post-RoPE key geometry rather than transformed cache representations.
  • Video differs from language-model KV compression because generated blocks arrive online while future queries remain unavailable.

3 Method

DensityKV maintains separate bounded historical banks for each layer and head, ranks incoming states by post-RoPE key crowding, and enforces density-growth and capacity constraints while preserving exact K/V pairs.

  • Problem Setup: Each finalized denoised block supplies token-level candidates, with each per-head post-RoPE key serving as both attention key and density descriptor.
  • Post-RoPE Attention Geometry: Post-RoPE key distance bounds normalized attention-logit discrepancy for any query, motivating query-agnostic coverage estimation in attention space.
  • Post-RoPE Attention Geometry: Squared Euclidean distance preserves neighborhood ordering and key-norm information, but the resulting measure estimates routing coverage rather than attention-output error.
  • Soft-Riesz Density: Soft-Riesz density uses regularized inverse-power interactions over post-RoPE keys, with σ setting neighborhood width, p controlling decay, and ϵ preventing singularities.
  • Soft-Riesz Density: High local density identifies crowded neighborhoods, and DensityKV constrains density growth after admission relative to each state’s frozen insertion baseline.
  • Online Density-Constrained Bank Update: Separate per-head banks reflect head-specific geometries, while synchronized admission shares occupancy across heads and preserves each state’s paired value.
  • Online Density-Constrained Bank Update: Candidates are ordered by mean-normalized disturbance, then the largest feasible cross-head prefix is selected under density-growth and capacity-driven eviction constraints.
  • Online Density-Constrained Bank Update: Evictions remove mandatory or highest-density states until capacity is released, preserving the density-growth invariant for surviving history.

4 Experiments

Experiments evaluate DensityKV across three autoregressive video backbones, multiple horizons, six VBench metrics, storage, and controlled ablations. Under matched historical-KV capacity, DensityKV improves overall performance and bounds persistent temporal-state storage independently of rollout length.

  • Persistent temporal-state storage: 3.28 GiB of total persistent temporal-state storage lets DensityKV reduce storage relative to LongLive-RAG by 9.8×, 19.6×, and 39.2× at 30, 60, and 120 seconds.LongLive-RAG storage grows from 32.1 to 128.5 GiB, whereas DensityKV remains bounded.
  • Comparison across backbones: 31 of 54 single-metric wins and an Avg. Rank of 1.89 make DensityKV the strongest overall method across three backbones and three horizons.DensityKV leads with 8, 13, and 10 wins at 30, 60, and 120 seconds, respectively.
  • Comparison across backbones: DensityKV improves Background Consistency over LongLive-RAG in all nine horizon–backbone comparisons.
  • Comparison across backbones: DensityKV ranks first at every Self-Forcing horizon and reaches an Avg. Rank of 1.33 on Causal-Forcing at 60 and 120 seconds.On Causal-Forcing, it achieves the best scores on five metrics, while ∞-RoPE has higher Dynamic Degree.
  • Comparison across backbones: At 60 and 120 seconds on LongLive, DensityKV leads Background Consistency, Dynamic Degree, and Imaging Quality, while LongLive-RAG retains Subject Consistency and Aesthetic Quality.The results indicate that gains vary with each generator’s error profile.
  • Parameter sensitivity: τ = 2.0, M = 9,360, σ = 8, and p = 2 provide the selected balanced default across consistency, dynamics, and visual quality.In the sensitivity sweeps, these settings obtain Avg. Ranks of 1.83, 1.67, 1.83, and 1.17, respectively.

5 Conclusion

DensityKV bounds historical storage while preserving exact key-value states and improving long-horizon consistency across frozen video backbones.

  • DensityKV controls post-admission crowding in post-RoPE key space while preserving exact key-value pairs.The training-free selector operates at token and head granularity without training or modifying the generator.
  • Across three frozen video backbones, DensityKV improves long-horizon consistency under the same historical-capacity bound.
  • Persistent historical storage remains independent of rollout length while retaining non-local subject and scene evidence.Retained values are neither merged nor reconstructed, so selected states remain directly usable by native attention.

Supplementary Material

The supplementary material develops the post-RoPE attention geometry, specifies the memory-key RoPE convention, and organizes evaluation, ablation, and qualitative analyses.

  • Organization: Appendices A–C derive post-RoPE key geometry and Soft-Riesz density, then provide the complete online bank update.
  • Organization: The supplementary protocols cover evaluation and ablations, while additional material extends qualitative comparisons across all three backbones.Token admission events are also visualized.
  • Post-RoPE Attention Geometry: Post-RoPE query and key vectors are the attention vectors produced after applying the RoPE transforms used at read time.
  • Post-RoPE Attention Geometry: The evaluated backbones zero the temporal RoPE coordinate for nonlocal memory keys while preserving spatial coordinates.DensityKV stores these same post-RoPE keys, matching the geometry consumed by native attention.

A.1 Attention-Logit Discrepancy

The appendix links post-RoPE Euclidean key distance to worst-case attention-logit discrepancy and explains why DensityKV uses squared Euclidean rather than cosine distance without per-head normalization.

  • A.1 Attention-Logit Discrepancy: Euclidean distance between post-RoPE keys exactly measures their worst-case normalized attention-logit discrepancy over nonzero queries.Equality is attained when the query is parallel to the key difference, and RoPE orthogonality preserves the result for native attention.
  • A.2 Expected Key-Response Geometry and Cosine Distance: Future-query-aware geometry would use an expected squared logit discrepancy under the future post-RoPE query distribution.This distribution is unavailable or potentially nonstationary in the online bank-update setting.
  • A.2 Expected Key-Response Geometry and Cosine Distance: When future query statistics are unavailable or nonstationary, tracking the query second-moment matrix would be required for a distribution-specific metric.
  • A.2 Expected Key-Response Geometry and Cosine Distance: Under isotropic queries, squared Euclidean distance is the resulting query-response geometry, whereas cosine distance discards key-norm information.
  • A.2 Expected Key-Response Geometry and Cosine Distance: Cosine distance cannot uniformly bound attention-logit differences because positively rescaled keys retain cosine similarity but change scaled dot-product responses.The evaluated backbones do not enforce independent unit norms for every per-head key.

B.1 From Riesz Energy to a Smooth Crowding Score

DensityKV turns regularized Riesz interactions into a token-level online bank policy that orders candidates, enforces shared density-growth feasibility, and updates exact per-head historical banks.

  • From Riesz Energy to a Smooth Crowding Score: Classical Riesz interactions penalize crowded key configurations but are singular at coincident keys and lack an attention-head-specific distance scale.
  • From Riesz Energy to a Smooth Crowding Score: DensityKV uses a regularized Soft-Riesz interaction whose parameters control near-duplicate regularization, neighborhood bandwidth, and decay rate.The resulting density is an unnormalized operational crowding score compared only within the same layer and head.
  • Online Bank Update: Each layer and head maintains exact token-level key-value pairs, with at most M historical pairs retained under unbounded rollout and unknown future queries.
  • Online Bank Update: Candidates are ordered by mean-normalized insertion disturbance, and the policy selects the largest prefix satisfying the density-growth constraint for every head.Synchronized admission requires evictions when occupancy plus the candidate prefix exceeds the per-head budget.
  • Online Bank Update: Co-admitted candidates establish insertion baselines together, so their mutual interactions do not reject one another during the same update.Only density growth introduced by later generation blocks contributes after admission.
  • Online Bank Update: The exact update costs O(NMdk), covering mutual interactions among admitted states and interactions with evicted states.

D Experimental Details

Experiments use frozen Wan2.1-T2V-1.3B autoregressive generators, matched nonlocal capacity, and fixed protocols for inference-time evaluation and persistent-storage accounting.

  • Experiments use three frozen backbones: LongLive, Self-Forcing, and Causal-Forcing, all built from Wan2.1-T2V-1.3B.
  • DensityKV runs only at inference time on 128 refined MovieGenBench prompts, with 30-, 60-, and 120-second trajectories generated using seed 0.The 30- and 60-second videos are packet-preserving truncations of the same 120-second trajectory.
  • 9,360 historical KV states per head provide the shared nonlocal capacity, equivalent to six latent frames of 1,560 spatial tokens each.DensityKV and LongLive-RAG also share an explicit first-frame sink and a five-frame local window.
  • 1.67 GiB is the DensityKV bank total with metadata, while persistent temporal-state storage reaches 3.28 GiB including the sink and local window.The bank payload alone is 1.61 GiB under L = 30, H = 12, dk = dv = 128, and n = M = 9,360.
  • The default policy uses post-RoPE keys, exact key-value pairing, independent layer-head banks, M = 9,360, τ = 2.0, and σ = 8.Each finalized block supplies N = 4,680 candidates per head, corresponding to three latent frames.
  • Evaluation follows six VBench dimensions, with higher scores better and average rank computed across metric ranks.The overall average spans 54 backbone–horizon–metric cells.

E Ablation Details

Ablations vary DensityKV’s threshold, capacity, bandwidth, ordering, baselines, eviction, synchronization, admission granularity, and retention geometry while isolating policy choices.

  • Parameter sweeps vary one scalar at a time: density-growth threshold τ, bank capacity M, Soft-Riesz scale σ, or inverse-power exponent p.ϵ remains fixed at 1 because changing it mainly rescales effective bandwidth.
  • Candidate-ordering ablations compare mean-normalized disturbance ordering with original spatiotemporal source order.Lower mean-normalized disturbance scores are considered earlier.
  • Baseline ablations compare fixed insertion-time density references against refresh baselines recomputed after every update.The comparison tests whether admission neighborhoods or evolving-bank densities should remain the reference.
  • Eviction ablations compare threshold-first removal with densest-only eviction and source-order completion of the eviction set.The default first removes states whose projected density ratio crosses the threshold.
  • Cross-head synchronization compares a shared feasible prefix length with independent per-head prefix selection, while preserving head-specific token identities.The default admits the same number of candidates per head within a layer.
  • Within-block competition changes admission to sequential density-gated updates, making it a mechanism comparison rather than a strict one-factor ablation.Earlier candidates immediately affect later candidates under this variant.
  • Adding values improves pre-RoPE average rank from 2.33 to 2.00 but worsens post-RoPE average rank from 1.83 to 3.17.The suite compares pre- and post-RoPE keys with [K; V] descriptors under the same ordering.

F Extended Long-Horizon Qualitative Comparisons

Extended qualitative comparisons show matched trajectories across 1–120 seconds for three generation configurations on MovieGenBench prompts.

  • Causal-Forcing: Figure 6 compares Native, LongLive-RAG, and post-RoPE K-only DensityKV on five Causal-Forcing prompts.Columns show matched trajectories at 1, 10, 20, 30, 60, and 120 seconds.
  • Self-Forcing: Figure 7 compares Native, LongLive-RAG, and DensityKV on five Self-Forcing prompts across 1–120 seconds.
  • LongLive: Figure 8 compares Native, LongLive-RAG, and DensityKV on five LongLive prompts across 1–120 seconds.

G Token Admission Frequency and Layerwise Coverage

Admission analysis resolves retained-token events across layer-head banks and source times, showing concentrated initialization followed by sparse, layer-specific admissions.

  • Figure 9 resolves admission events across all 30 layers and 12 attention heads rather than collapsing them into a binary union.
  • Figure 9 measures admission intensity, layer-and-source-time counts, and admitted-entry and unique-position fractions for a Causal-Forcing panda-guitar diagnostic.The orange intensity is logarithmic, and the gray region marks initial construction from source frames 1–6.
  • 99.77% of retained layer-head entries in the analyzed trace come from the first six source frames spanning 0–1.25 seconds at 4 fps.The passage attributes this concentration primarily to matched capacity M = 9,360 = 6×1,560, not intrinsic importance of the video beginning.
  • After initialization, admissions become sparse and layer-specific, indicating that the density-growth test can still admit under-covered states.
Loading 2608.27922v1…