Source-linked AI summary
Efficient Autoregressive Video Diffusion with Dummy Head
Hang Guo, Zhaoyang Jia, Jiahao Li, Bin Li, Yuanhao Cai, Jiangshan Wang, Yawei Li, Yan Lu
TL;DR
Existing autoregressive video diffusion models under-use historical context because some attention heads focus almost exclusively on the current frame. Dummy Forcing allocates context heterogeneously, classifies heads dynamically, and packs context for more aggressive cache compression. Without additional training, it achieves up to 2.0× speedup and supports generation above 24 FPS with a small quality drop.
Problem
Existing autoregressive video diffusion models face efficiency challenges on long visual token sequences despite using cached historical context.
Method
Dummy Forcing uses heterogeneous memory allocation, dynamic head programming, and packed attention forward to control and compress head-specific context.
Results
Up to 2.0× end-to-end acceleration over the baseline and video generation above 24 FPS are achieved without additional training.
Takeaways & Limitations
Dummy Forcing applies to efficient, high-resolution, and long-context video generation while reducing redundant contextual information.
Takeaways & Limitations
The method is currently training-free, and post-training is identified as a possible route to improve performance or compression rates.
Abstract
from arXiv · showhide
The autoregressive video diffusion model has recently gained considerable research interest due to its causal modeling and iterative denoising. In this work, we identify that the multi-head self-attention in these models under-utilizes historical frames: approximately 25% heads attend almost exclusively to the current frame, and discarding their KV caches incurs only minor performance degradation. Building upon this, we propose Dummy Forcing, a simple yet effective method to control context accessibility across different heads. Specifically, the proposed heterogeneous memory allocation reduces head-wise context redundancy, accompanied by dynamic head programming to adaptively classify head types. Moreover, we develop a context packing technique to achieve more aggressive cache compression. Without additional training, our Dummy Forcing delivers up to 2.0x speedup over the baseline, supporting video generation at 24.3 FPS with less than 0.5% quality drop. Project page is available at https://csguoh.github.io/project/DummyForcing/.
1. Introduction
Autoregressive video diffusion enables sequential generation with cached historical context, but existing models inefficiently use that context. Dummy Forcing compresses redundant context across attention heads and accelerates generation without additional training.
- Bidirectional video diffusion processes all frames together, delaying viewing and limiting interactive generation.
- About 25% of attention heads assign over 80% of their attention to the current frame despite access to historical frames.
- These current-frame-focused heads are termed dummy heads because they mainly refine the current frame rather than aggregate context.
- Dummy Forcing combines heterogeneous memory allocation, dynamic head programming, and packed attention forward to compress redundant historical context.
- Without training, Dummy Forcing achieves up to 2.0× end-to-end acceleration over the baseline and generates videos above 24 FPS.
2. Related Work
Prior video diffusion models commonly generate all frames with bidirectional attention, while autoregressive models enable KV caching but face long-sequence efficiency challenges. Related cache-compression work largely targets language-model token redundancy rather than video frame-level redundancy.
- Early video diffusion models use bidirectional attention to generate all frames simultaneously, but this incurs significant quadratic complexity for long videos.
- Autoregressive video diffusion models address sequential generation with KV caching, yet long visual token sequences remain computationally challenging.
- Language-model KV-cache methods retain important tokens using attention-based criteria such as sink-token or intermediate-token importance.
- Existing cache-compression approaches do not consider video frame-level redundancy, whereas dummy-head caches can potentially be removed entirely.
- Video-generation acceleration methods for bidirectional models often use sparse attention because those models do not support KV caching.
3. Method
The method analyzes how attention heads use sink, neighbor, and current-frame context, then compresses redundant KV caches through heterogeneous allocation, adaptive classification, and context packing.
- Preliminary: Autoregressive diffusion models maintain a sliding KV-cache window containing one sink frame and neighboring frames to aggregate historical context.The sink frame provides a global anchor, while neighboring frames capture temporal dependencies as the window moves.
- Motivation: The frame attention score measures how current-frame queries attend to sink, neighbor, and current-frame keys.It is computed from attention maps over the corresponding frame-token index groups.
- Motivation: 25% of attention heads assign over 80% of their attention weights to the current frame and are therefore classified as dummy heads.Dummy heads are selected by choosing the heads with the largest current-frame attention scores.
- Motivation: Removing dummy-head KV caches causes only a 0.26% performance drop, whereas random eviction severely degrades performance.This provides evidence that dummy heads contribute little to cross-frame context aggregation.
- Dummy Forcing: Dummy Forcing combines heterogeneous memory allocation, dynamic head programming, and packed attention forward to compress redundant context.HMA assigns head-specific cache lengths, DHP adaptively classifies head types, and PAF extends dummy-head context to mitigate classification-boundary errors.
4. Experiments
The experiments evaluate Dummy Forcing across short, long, high-resolution, and long-context video generation, alongside component ablations. The method improves generation efficiency while largely preserving quality, with performance depending on cache allocation and dummy-head count.
- Classic Video Generation: 1.4× speedup at 24.3FPS accompanies only a 0.1% quality drop on 5-second video generation with Self Forcing.The comparison uses efficiency and quality evaluation on short videos.
- Long Video Generation: 1.4× end-to-end speedup on 30-second videos comes with up to a 0.4% quality drop.The evaluation uses VBench-Long and reports faster generation with strong motion and high visual quality.
- High-Resolution Video Generation: 2.0× acceleration without quality drop is achieved for 1080P video with the LongLive model.The speedup becomes more pronounced as the number of processed tokens increases, including at higher resolutions.
- Long-Context Video Generation: Long-context experiments evaluate historical context length under similar runtime cost, targeting limitations from sliding-window methods.The setup considers narrative videos with scene transitions and compares cached-frame budgets.
- Ablation Studies: When the dummy-head count is below 240 of 360, VBench performance remains relatively stable, while pruning 300–360 heads causes significant degradation.The degradation occurs because KV-cache pruning harms neighbor heads that are important for context aggregation.
5. Discussion
The discussion examines how Dummy Forcing’s head classes are distributed and how the method can combine with other acceleration techniques. Its head allocation reflects different layer roles in processing current and historical information.
- Compatibility with Other Acceleration Methods: Combining Dummy Forcing with TeaCache yields generation speeds over 30FPS.The two methods are described as orthogonal and complementary acceleration approaches.
- Head Distribution across Layers: Dummy heads primarily occur in the first and last layers, while neighbor heads cluster in intermediate layers.The distribution is averaged across 100 prompts.
- Head Distribution across Layers: The layer distribution is consistent with early layers gating current-frame information, intermediate layers querying past frames, and final layers refining the current frame.This describes the paper’s proposed interpretation of the observed head specialization.
6. Conclusion
The conclusion presents Dummy Forcing as a cache-compression approach for efficient autoregressive video diffusion across multiple generation settings.
- 6. Conclusion: Dummy Forcing exposes attention heads to different context lengths through heterogeneous memory allocation and classifies them online with dynamic head programming.Packed Attention Forward extends dummy-head context for aggressive dummy-head settings.
- 6. Conclusion: The method is applied to efficient, high-resolution, and long-context video generation, with experiments demonstrating its generality.These applications span multiple downstream video-generation tasks.
Impact Statement
The work targets faster and more resource-efficient video generation without additional training. Its broader risks are inherited from synthetic video generation, including potential misinformation or misuse.
- Impact Statement: The primary goal is faster video generation and more effective computational-resource use without additional training.The work is developed on pre-trained video generation models.
- Impact Statement: Potential misinformation and misuse risks are inherited from the broader class of generative video models.The statement expects the broader impact to be largely positive while remaining aligned with existing ethical considerations.
A. Proof of Optimality for Greedy Strategy
The greedy strategy for assigning dummy heads is optimal because exchanging any selected head with a higher-value alternative never decreases the objective.
- Dynamic Head Programming formulates dummy-head assignment as a dynamic programming problem solvable in O(n log n) with a greedy algorithm.The section provides a proof of optimality.
- The value function f_h measures the contribution associated with the h-th attention head.
- The greedy solution I∗ selects dummy-head indices according to the derived head values.
- For any alternative assignment, swapping a selected head i for an unselected head j with ℓ_i ≥ ℓ_j yields I_swap and does not reduce the total value.
- Repeated swaps transform any feasible assignment into I∗ without decreasing the objective, proving greedy optimality.
B. Dummy Head in Other Models
Dummy heads also appear in CausVid and Rolling Forcing, indicating that head-specific context use extends across autoregressive video diffusion frameworks.
- CausVid exhibits heads with disproportionately high attention to the current frame under different noise-distribution conditions.
- CausVid’s attention analysis is evaluated across autoregressive steps and denoising timesteps.
- Rolling Forcing also contains dummy heads despite jointly denoising multiple frames with progressively increasing noise levels.
- Applying Dummy Forcing with 50% dummy heads to Rolling Forcing produces no noticeable video-quality degradation.
C. More Implementation Details
The implementation and profiling experiments use controlled attention-map sampling and compare self-attention runtime across context lengths and long-video evaluation settings.
- Observation experiments average attention maps over 100 VBench text prompts at the third autoregressive step and final denoising step.
- A single bicycle-acceleration prompt is used with the third autoregressive step and final denoising step for Tab. 1 attention maps.
- Sampling 25% of query tokens estimates the frame attention score in under 10ms while retaining good performance.
- Table 8 reports whole-video quality scores and CLIP semantic scores for each 10-second clip in 60-second interactive video generation.
- Figure 12 profiles averaged single-layer self-attention runtime across different context lengths, with HW denoting visual tokens per latent frame.
D. More Experimental Results
Additional experiments examine long-video generation, single-layer speed, large-model scaling, cache compression, attention patterns, and evaluation protocols, while identifying training and explanatory gaps for future work.
- More Experimental Results: 60-second interactive videos are generated from six prompts, each controlling a 10-second segment, and evaluated with VBench-Long dimensions supporting customized prompts.
- More Experimental Results: 1.7× speedup is achieved by Dummy Forcing when self-attention processes interactions across 15 frames.
- More Experimental Results: On RealTime-14B, pruning KV caches for 50% of heads achieves acceleration and an even better total score on VBench-Long.
- More Experimental Results: 27.8% of the baseline cache length remains under Dummy Forcing’s binary dummy/non-dummy compression lower bound.
- More Experimental Results: Sink, neighbor, and dummy heads show distinct attention distributions, with dummy heads focusing mostly on the current frame.
- More Experimental Results: The long-context task generates 15-second interactive videos from three prompts controlling separate 5-second segments.
- More Experimental Results: The authors identify training-free operation and the unexplained emergence of dummy heads as future improvement and investigation areas.
G. More Generation Results
This section presents additional visual and quantitative generation results, covering scene transitions, long-context generation, and short- and long-video tasks. It also describes the prompt setup used to evaluate scene transitions and long-term visual memory.
- Overview: The section organizes additional results across scene-transition evaluation, autoregressive diffusion pipeline comparisons, context probing, and single-prompt video-generation tasks.The reported tasks include 5s short-video generation and 30s long-video generation.
- Scene-transition evaluation: The scene-transition evaluation generates three sequential prompts to test scene transitions and preservation of long-term visual memory.The prompts include the original prompt, a new prompt introducing a required scene transition, and an identical copy of the original prompt.
- Quantitative comparisons: Figures 15 and 16 compare Dummy Forcing with CausVid and Rolling Forcing, respectively, using 50% head KV-cache pruning for dummy heads.The comparisons are applied to other autoregressive diffusion pipelines, including Diffusion Forcing and Rolling Forcing.