PaperlayerView exact revision on arXiv

Source-linked AI summary

Forcing-KV: Hybrid KV Cache Compression for Efficient Autoregressive Video Diffusion Models

Yicheng Ji, Zhizhou Zhong, Jun Zhang, Qin Yang, XiTai Jin, Ying Qin, Wenhan Luo, Shuiyang Mao, Wei Liu, Huan Li

arXiv:2605.09681v1cs.CV

TL;DR

Autoregressive video diffusion models incur substantial attention complexity and memory overhead from redundant historical KV caches. Forcing-KV compresses these caches by tailoring pruning to static and dynamic attention heads, achieving up to 2.82× acceleration while maintaining output quality and reducing cache memory by 30%.

  • Problem

    Redundant historical KV caches impose substantial attention complexity and memory overhead in autoregressive video diffusion, limiting efficient long-horizon and high-resolution generation.

  • Method

    Forcing-KV profiles attention heads as static or dynamic, then applies structured pruning to static heads and segment-wise similarity pruning to dynamic heads.

  • Results

    30% cache memory reduction and up to 1.35× and 1.50× speedups at 480P scale to up to 2.82× acceleration at 1080P while maintaining output quality.

  • Takeaways & Limitations

    Forcing-KV provides a high-fidelity, efficient approach to KV cache utilization in autoregressive video generation across models, benchmarks, lengths, and resolutions.

  • Takeaways & Limitations

    The observations are primarily based on existing autoregressive video model families trained under the Self Forcing paradigm.

Abstract

from arXiv · show

Autoregressive (AR) video diffusion models adopt a streaming generation framework, enabling long-horizon video generation with real-time responsiveness, as exemplified by the Self Forcing training paradigm. However, existing AR video diffusion models still suffer from significant attention complexity and severe memory overhead due to the redundant key-value (KV) caches across historical frames, which limits scalability. In this paper, we tackle this challenge by introducing KV cache compression into autoregressive video diffusion. We observe that attention heads in mainstream AR diffusion models exhibit markedly distinct attention patterns and functional roles that remain stable across samples and denoising steps. Building on our empirical study of head-wise functional specialization, we divide the attention heads into two categories: static heads, which focus on transitions across autoregressive chunks and intra-frame fidelity, and dynamic heads, which govern inter-frame motion and consistency. We then propose Forcing-KV, a hybrid KV cache compression strategy that performs structured static pruning for static heads and dynamic pruning based on segment-wise similarity for dynamic heads. While maintaining output quality, our method achieves a generation speed of over 29 frames per second on a single NVIDIA H200 GPU along with 30% cache memory reduction, delivering up to 1.35x and 1.50x speedups on LongLive and Self Forcing at 480P resolution, and further scaling to 2.82x speedup at 1080P resolution. Code and demo videos are provided at https://zju-jiyicheng.github.io/Forcing-KV-Page.

1 Introduction

Forcing-KV addresses the attention complexity and memory overhead caused by redundant historical KV caches in autoregressive video diffusion by exploiting stable head-wise specialization. It combines static structural pruning with dynamic similarity pruning, achieving substantial speedups and 30% cache memory reduction while maintaining quality.

  • Motivation: Autoregressive video diffusion generates video chunk by chunk, conditioning each new chunk on previously generated content through a KV cache.This supports long-horizon, variable-length generation with interactive inputs.
  • Motivation: Historical KV caches create increasing attention complexity and memory overhead because new chunks attend to increasingly long, redundant visual context.The problem is especially severe for long-horizon and high-resolution videos.
  • Head specialization: Static heads focus on the current chunk and transition anchor frame, while dynamic heads sustain long-horizon consistency and motion through inter-frame attention.These roles form a universal head specialization pattern across mainstream autoregressive video diffusion models.
  • Method: Forcing-KV profiles heads offline using frame-wise attention mass, then preserves critical frames for static heads and applies segment-wise similarity pruning to dynamic heads.The hybrid strategy decouples static patterns from dynamic context utilization.

2 Related Work

Related work spans the shift from bidirectional, one-shot video diffusion to autoregressive streaming generation for long-horizon video. Efficiency efforts address heavy attention computation and multi-step denoising through model-specific acceleration techniques, increasingly tailored to autoregressive video diffusion.

  • Video Diffusion Models: Video diffusion models evolved from bidirectional, one-shot generation toward autoregressive, streaming generation.Early bidirectional models commonly use the Diffusion Transformer architecture, while autoregressive diffusion modeling supports long-horizon generation.
  • Video Diffusion Models: Autoregressive diffusion modeling was adopted to reduce bidirectional denoising costs and support long-horizon video generation.
  • Efficient Video Generation: Video diffusion models face heavy attention computation and multi-step denoising, motivating acceleration methods for inference.For bidirectional models, prior techniques include sparse attention, linear attention, quantization, and feature caching.

3 Observation

The paper identifies stable static and dynamic attention-head specializations in autoregressive video diffusion, linking their distinct KV-cache dependencies to a hybrid compression strategy. Static heads preserve local continuity through current-chunk and transition-anchor attention, while dynamic heads use globally informative, similarity-redundant context for motion and consistency.

  • 3.1 Attention Head Patterns: Across Wan2.1, SkyReels-V2, Self Forcing, and LongLive, attention heads consistently separate into static and dynamic patterns.The comparison spans bidirectional, autoregressive, many-step, and few-step video diffusion models.
  • 3.1 Attention Head Patterns: Static heads attend to the current chunk and most recent transition-anchor frame, preserving intra-frame fidelity and continuity across local autoregressive chunks.Their chunk-wise attention pattern emphasizes local spatial frames and the latest historical frame.
  • 3.1 Attention Head Patterns: Dynamic heads track corresponding regions across historical frames, capturing inter-frame evolution, motion, and object changes through diagonal-stripe KV-cache patterns.The fixed stride reflects fixed frames-per-chunk and tokens-per-frame, aligning generated regions with corresponding historical regions.
  • 3.2 Functional Roles: Static heads are insensitive to distant context but depend on the transition-anchor frame, whereas dynamic heads use global context for subject consistency and motion dynamics.Masking the transition-anchor frame sharply increases chunk discontinuity, while reducing dynamic-head context progressively lowers dynamic degree and consistency.
  • 3.2 Functional Roles: Adjacent frames show high, segment-dependent KV-cache similarity, supporting static pruning for static heads and similarity-based dynamic pruning for dynamic heads.This decouples static local patterns from dynamic context utilization in the proposed hybrid compression scheme.
  • 3.3 Stability and Generalizability: 0.16 average intra-head divergence is substantially smaller than 0.83 average inter-head divergence, indicating stable specialization across samples and denoising steps.Tightly clustered per-head features provide a basis for effective head classification.

4 FORCING-KV

Forcing-KV compresses autoregressive diffusion KV caches through offline classification of attention heads into static and dynamic types. It then applies structured pruning to static heads and adaptive segment-wise similarity pruning to dynamic heads.

  • 4.1 Head Profiling: Forcing-KV classifies attention heads offline as static or dynamic before inference, based on their distinct attention patterns.Static-head attention concentrates on the current chunk and transition anchor frame, whereas dynamic-head attention is distributed across the attention window.
  • 4.1 Head Profiling: The profiling criterion is model-specific but can be evaluated within a single prompt and is insensitive to α for most heads.This supports scalability and aligns with eviction methods that remove frames with lower accumulated attention mass.
  • 4.2 Static Pruning: Static heads retain sink frames, the transition anchor frame, and the current chunk to preserve autoregressive transitions and intra-frame spatial structure.The structured strategy is motivated by static heads’ sensitivity to the transition anchor and limited use of distant context.
  • 4.3 Dynamic Pruning: Dynamic heads partition latent frames into n segments, compute adjacent-frame cosine similarity, and evict the top-k most similar segments under compression ratio r.The method preserves (1 −r)% of all frame segments and keeps low-similarity segments as less redundant content.
  • 4.3 Dynamic Pruning: Dynamic segment similarity uses only key states from attention heads in the diffusion transformer’s first block as a computational proxy.This avoids computing similarity across all transformer blocks while exploiting redundancy between adjacent frames.

5 Experiments

Experiments on LongLive and Self Forcing show that FORCING-KV accelerates short- and long-video generation while preserving comparable quality and improving temporal dynamics. Scaling and ablation studies attribute its benefits to hybrid head-wise compression, stable head profiling, and dynamic similarity pruning.

  • Comparison with Full KV Cache: FORCING-KV achieves 1.30× and 1.50× inference speedups on LongLive and Self Forcing for long videos, versus 1.35× and 1.44× for short videos.The evaluation covers 5-second, 30-second, and 60-second videos on both models.
  • Comparison with Full KV Cache: Only ∼27% and ∼46% of the KV cache participate in self-attention for Self Forcing and LongLive, while VBench quality remains comparable or slightly improved at 80.43 vs. 80.23.The user study reports comparable visual quality at 45.0% vs. 50.0% and stronger temporal dynamics at 52.8% vs. 42.2%.
  • Comparison with Other Compression Strategies: FORCING-KV achieves a 1.50× vs. 1.23× speedup against StreamingLLM and substantially higher quality metrics than Dummy Forcing at comparable speedups.The comparison attributes the advantage over StreamingLLM to head-level decomposition and higher compression ratios.
  • Scaling with Attention Window and Resolution: 1.40× to 2.82× speedup is achieved as attention-window size and video resolution increase, because attention computation scales quadratically and memory consumption scales linearly with KV-cache growth.The scaling analysis is conducted on Self Forcing.

6 Conclusion

FORCING-KV is a hybrid KV cache compression framework for autoregressive video diffusion models, motivated by universal head specialization across mainstream models. It maintains output quality while improving generation speed and reducing cache memory.

  • FORCING-KV combines hybrid KV cache compression with a universal head specialization pattern shared across mainstream autoregressive video diffusion models.The observed specialization pattern motivates the proposed compression strategy.
  • 29 FPS is achieved while maintaining output quality and reducing cache memory by 30%.The reported generation speed exceeds 29 FPS.
  • 1.35× and 1.50× speedups are delivered by FORCING-KV.These speedups are reported together with 30% cache memory reduction.

7 Limitation and Future Works

The study’s observations are primarily grounded in existing Self Forcing-based autoregressive video diffusion model families, motivating validation on future autoregressive models. As a training-free method, it leaves training-stage KV cache reduction and fine-tuning for longer context windows to future work.

  • Limitations: The observations primarily derive from existing Self Forcing-based autoregressive video diffusion model families.These include Self Forcing, LongLive, SkyReels-V2, Krea-Realtime-14B, and the broader family of “forcing” models.
  • Future Works: Evaluating whether these observations hold in future autoregressive models remains an open direction.The authors believe the observations are closely tied to fundamental principles of autoregressive generation.
  • Future Works: Training-stage KV cache reduction is left for future work because Forcing-KV is training-free.Fine-tuning could potentially support longer context windows.

A Chunk Discontinuity · B Attention Patterns of Various Diffusion Models

The paper defines chunk discontinuity using adjacent-frame optical-flow differences to assess temporal smoothness in autoregressively generated videos. Across bidirectional and autoregressive diffusion models, attention heads consistently specialize in inter-frame or intra-frame functions, while autoregressive models concentrate transition information in an anchor frame.

  • A Chunk Discontinuity: Chunk discontinuity quantifies transitions across autoregressive chunks using optical-flow differences between adjacent frames.For a video with F frames generated in K chunks, the method computes adjacent-frame flow differences with RAFT.
  • A Chunk Discontinuity: δ_t = Δ_RAFT(I_t, I_{t+1}) measures the optical-flow difference for adjacent frames t = 1, . . . , F − 1.This is the metric definition given for adjacent-frame optical-flow differences.
  • A Chunk Discontinuity: For temporally smooth, continuous videos, adjacent-frame optical-flow differences should vary relatively uniformly, producing a uniformly lower metric value.The metric is designed for autoregressive diffusion models that generate continuous videos without scene cuts.
  • A Chunk Discontinuity: A low chunk-discontinuity value also indicates a lower average peak difference and milder temporal variation.This links lower metric values to smoother temporal behavior rather than merely uniform flow differences.
  • B Attention Patterns of Various Diffusion Models: Experiments across Wan2.1, SkyReels-V2, LongLive, and Self Forcing find spatiotemporal functional specialization of attention heads to be common across diffusion models.The experiments include bidirectional and autoregressive models, spanning many-step and few-step variants.
  • B Attention Patterns of Various Diffusion Models: Different attention heads are respectively responsible for inter-frame and intra-frame attention across the evaluated models.The reported specialization appears in both bidirectional and autoregressive video diffusion models.
  • B Attention Patterns of Various Diffusion Models: Autoregressive models learn smooth transitions from preceding frames through a Markovian generation process.The current block must obtain transition information from previous frames.
  • B Attention Patterns of Various Diffusion Models: Visualization shows that autoregressive transition information is concentrated primarily in a transition anchor frame rather than the full history.This distinguishes autoregressive attention behavior from the broader historical-frame context described for bidirectional models.

C User Study

The user study tested whether benchmark results matched human perception using 12 participants who compared videos from four methods across 5-second and 30-second generations. Participants evaluated videos on visual quality, dynamic degree, and consistency.

  • Setup: 12 participants evaluated 15 video groups containing outputs from Self Forcing, LongLive, Dummy Forcing, and Forcing-KV.The study included 5-second and 30-second generations from Self Forcing and LongLive.
  • Evaluation criteria: Consistency was one of the three criteria used to assess the generated videos.The supplied passage identifies consistency as an evaluation criterion but does not provide its full definition.
  • Evaluation criteria: Visual quality measured overall appeal through visual fidelity, coherence, motion quality, and subjective viewing experience.This criterion jointly considers multiple aspects of perceived video quality.
  • Evaluation criteria: Dynamic degree measured the naturalness, richness, and engagement of video motions and changes.Participants judged whether motion was realistic and diverse rather than static or repetitive.

D Results on Krea-Realtime-14B · E Interactive Video Generation · F Seamless Integration of Quantization

The method remains effective when scaled to Krea-Realtime-14B and extends to interactive video generation through segment-wise KV cache compression. It also integrates with FP8 quantization to further improve throughput with minimal performance loss.

  • D Results on Krea-Realtime-14B: Krea-Realtime-14B evaluation confirms that the method remains effective at the 14B model scale.The model is trained with Self Forcing, and evaluation uses only offline inference without additional optimizations.
  • D Results on Krea-Realtime-14B: The Krea-Realtime-14B experiment evaluates 5-second video generation using VBench.These results are reported in Table 4.
  • E Interactive Video Generation: Interactive generation applies KV cache compression separately to the video segment associated with each prompt.The approach is evaluated with LongLive interactive prompts and summarizes quality and efficiency in Table 5.
  • E Interactive Video Generation: Each LongLive prompt group contains 6 progressively evolving descriptions, each corresponding to a 10-second video segment.Quality is evaluated over the entire generated video using VBench-Long, covering 60-second interactive video generation.
  • F Seamless Integration of Quantization: FP8 quantization reduces attention-module computational cost by using FP8 attention kernels tailored to NVIDIA Hopper.The resulting quality and efficiency measurements are reported in Table 6.
  • F Seamless Integration of Quantization: FP8 integration further boosts throughput with minimal performance drop, demonstrating compatibility with other acceleration techniques.The quantized results are described as quality and efficiency results with FP8 quantization.

G More Ablation Study Results

Additional ablations on Self Forcing and 5-second LongLive videos show trends consistent with the main results. Simple head profiling closely approaches manual profiling and outperforms an unprofiled random baseline.

  • Ablation results: Ablations on Self Forcing and 5-second LongLive videos exhibit trends consistent with the 30-second LongLive results.Tables 7 and 8 extend the ablation analysis across these settings.
  • Head profiling: Simple head profiling closely approaches manual profiling while significantly outperforming the unprofiled random baseline.The passage describes simple profiling as sufficient for the reported ablation settings.
  • KV cache compression: Table 7 reports ablations of KV cache compression and head profiling strategies on 5-second LongLive videos.The supplied table caption identifies the evaluated strategies and video duration.

H Implementation Details … L Quality Examples

The paper details baseline and compression implementations, then reports quality, head-distribution, runtime-share, and qualitative analyses across the evaluated video-generation settings. These analyses support hybrid KV compression as a way to retain quality while reducing attention-related overhead.

  • H Implementation Details: The baselines are StreamingLLM, implemented with 3 sink and 4 recent frames, and Dummy Forcing, a representative AR-diffusion KV compression method.Dummy Forcing preserves local-region KV cache for local heads and adjusts the history cache length.
  • H Implementation Details: Forcing-KV uses offline head profiling with α = 0.8 and hybrid compression controlled by dynamic-head compression ratio r.Static heads retain transition frames, and profiling with a single prompt completes within a few minutes.
  • I VBench Scores Across All Dimensions: FORCING-KV improves dynamic degree, overall consistency, appearance style, and total VBench score over its base model under a reduced KV cache budget.The paper attributes this to preserving head-specific context without sacrificing local transition information or historical context.
  • J Where Are Static and Dynamic Heads Located?: Dynamic heads constitute approximately 60% of all heads and are more prevalent in middle layers, especially layers 13, 15, and 17.The paper hypothesizes that input and output layers prioritize structured information for local video quality, while middle layers handle inter-frame information.
  • K Trend of the Proportion of Self-Attention: Self-attention’s share of Transformer-block time rises from 24% to 61% and from 61% to 89% as sequence length grows, increasing the benefit of KV cache compression.The analysis also shows total runtime increasing with attention-window size and resolution.
  • L Quality Examples: The paper provides a quality example of a 60-second interactive video on LongLive.This example is presented in Figure 14.
  • L Quality Examples: Additional qualitative examples cover LongLive at 5s and 30s and Self Forcing at 5s and 30s.These examples are presented in Figures 15 and 16, respectively.
Loading 2605.09681v1…