PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast Autoregressive Video Diffusion and World Models with Temporal Cache Compression and Sparse Attention

Dvir Samuel, Issar Tzachor, Matan Levy, Michael Green, Gal Chechik, Rami Ben-Ari

arXiv:2602.01801v2cs.CVcs.AI

TL;DR

Growing KV caches make autoregressive video diffusion increasingly slow and memory-intensive during long generation. FAST-AR combines temporal cache compression with ANN-based cross- and self-attention sparsification, achieving up to ×5–×10 speedups while preserving video quality and constant GPU memory over long generations.

  • Problem

    Growing KV caches increase attention latency and GPU memory use, limiting temporal context and harming long-range consistency in autoregressive video diffusion.

  • Method

    FAST-AR is a training-free framework combining TempCache temporal KV merging with ANN-based prompt pruning and semantically matched self-attention sparsification.

  • Results

    Up to ×5–×10 end-to-end speedups are achieved while preserving video quality and maintaining constant GPU memory during long video generations.

  • Takeaways & Limitations

    FAST-AR provides a plug-and-play attention approach for efficient long-horizon autoregressive video generation without retraining or fine-tuning.

Abstract

from arXiv · show

Autoregressive video diffusion models enable streaming generation, opening the door to long-form synthesis, video world models, and interactive neural game engines. However, their core attention layers become a major bottleneck at inference time: as generation progresses, the KV cache grows, causing both increasing latency and escalating GPU memory, which in turn restricts usable temporal context and harms long-range consistency. In this work, we study redundancy in autoregressive video diffusion and identify three persistent sources: near-duplicate cached keys across frames, slowly evolving (largely semantic) queries/keys that make many attention computations redundant, and cross-attention over long prompts where only a small subset of tokens matters per frame. Building on these observations, we propose a unified, training-free attention framework (FAST-AR) for FAST-AutoRegressive diffusion, consisting of three components: TempCache compresses the KV cache via temporal correspondence to bound cache growth; AnnCA accelerates cross-attention by selecting frame-relevant prompt tokens using fast approximate nearest neighbor (ANN) matching; and AnnSA sparsifies self-attention by restricting each query to semantically matched keys, also using a lightweight ANN. Together, these modules reduce attention, compute, and memory and are compatible with existing autoregressive diffusion backbones and world models. Experiments demonstrate up to x5 - x10 end-to-end speedups while preserving near-identical visual quality and, crucially, maintaining stable throughput and nearly constant peak GPU memory usage over long rollouts, where prior methods progressively slow down and suffer from increasing memory usage.

1. Introduction

FAST-AR addresses the growing attention bottleneck in autoregressive video diffusion by exploiting redundancy in cached keys, slowly evolving semantic queries and keys, and prompt-token relevance. Its training-free TempCache, AnnCA, and AnnSA components reduce cache growth and attention computation while enabling fast, memory-efficient long-form generation.

  • Motivation: Autoregressive video diffusion enables sequential streaming generation, but growing KV caches make attention latency and memory usage increase with each generated frame.Frames are produced online and can be consumed immediately, while 3D spatio-temporal attention scales with the number of cached keys.
  • Results: 5×–10× speedups on a single H100 GPU enable multi-minute generation while preserving high visual quality and keeping peak GPU memory nearly constant over long rollouts.The reported acceleration requires no further training or optimization.
  • Motivation: Three persistent redundancies motivate acceleration: near-duplicate cached keys, slowly evolving semantic queries and keys, and prompt attention concentrated on a few relevant tokens.These patterns support KV merging, reduced query–key score computations, and frame-specific prompt-token selection.
  • Method: FAST-AR unifies three training-free modules: TempCache compresses temporally corresponding KV entries, AnnCA selects frame-relevant prompt tokens, and AnnSA restricts queries to semantically matched keys.AnnCA and AnnSA use lightweight approximate nearest-neighbor matching for cross- and self-attention sparsification.

2. Related Work

Related work spans autoregressive video diffusion and world models, cache reuse and KV compression, sparse self-attention, and approximate nearest neighbor search. These approaches target the computational and memory costs of long-horizon video generation through chunked denoising, caching, structured sparsity, and approximate retrieval.

  • Autoregressive video diffusion models and video world models: Diffusion transformers scale well but make video generation expensive because of long spatiotemporal token sequences and repeated denoising steps.Recent work extends diffusion to long horizons with autoregressive chunked generation, repeatedly denoising the next segment conditioned on previously generated history.
  • KV compression and caching for autoregressive video diffusion: Training-free caching reuses intermediate computations when diffusion changes are small, while FLOWCACHE bounds long-horizon growth through chunkwise recomputation and importance-based KV compression.These methods address repeated denoiser evaluation across diffusion timesteps and autoregressive generation.
  • Sparse self-attention: Structured sparsity methods reduce the quadratic cost of full 3D self-attention over space–time tokens using head sparsity, semantic token selection, or static spatiotemporal masks.SVG, SVG2, and RADIAL ATTENTION represent distinct strategies for reducing sparse-kernel inefficiency or achieving sub-quadratic cost.
  • Approximate nearest neighbor search: Approximate nearest neighbor search accelerates high-dimensional similarity retrieval by trading exactness for speed.Common approaches include graph indices such as HNSW, hashing-based methods, and clustering-based indices; LSH provides lightweight hash-based retrieval with multi-table bucket probing.

3. Auto-regressive Video Diffusion Models

Autoregressive video diffusion computes attention over current-frame features and a KV cache containing prior frames or prompt tokens. Because the cache grows throughout generation, attention cost, cumulative work, latency, and memory usage increase with rollout length, despite substantial attention sparsity.

  • Attention formulation: Attention uses queries, keys, and values, with cached keys and values concatenating the current frame with all previously generated frames.In cross-attention, the cached keys and values correspond to projected prompt tokens for the current frame.
  • Attention sparsity: ≈85% attention recall is achieved at 30% density, indicating substantial sparsity in autoregressive video diffusion.The density retains only the highest-attention entries on Rolling-Forcing, averaged over transformer blocks.
  • Scaling bottleneck: KV-cache length grows with generated frames, making per-step attention cost linear in cache length and cumulative T-frame work O(T^2).This growth causes increasing inference latency and memory usage.

4. Motivation

The motivation examines whether autoregressive video diffusion contains exploitable attention sparsity and identifies redundancy in self- and cross-attention as generation proceeds. The analysis focuses on semantic structure, repeated cached keys, and the cost of attending to long prompts.

  • Attention sparsity: The study tests attention sparsity in the autoregressive setting, where frames are generated sequentially and the attention context grows over time.Prior work found natural self-attention sparsity in offline video diffusion, but its presence during autoregressive generation remained unclear.
  • Attention sparsity: Attention sparsity is quantified on 100 videos using attention recall, the dense-attention mass preserved after retaining only the largest attention entries.Recall is computed per transformer block, with mean and standard deviation reported across blocks.
  • Sources of sparsity: The representation analysis seeks redundant computations that can be reduced or removed without degrading generation quality.It examines attention-module features across different generation timesteps.
  • Self-attention redundancy: Self-attention features show semantic clustering and strong key repetition across frames, motivating compression of the KV cache.Foreground and background tokens form distinct semantic groups, while cached keys repeat across frames.
  • Cross-attention overhead: Cross-attention is costly because long, detailed prompts require every query to attend to all prompt tokens at every layer and generation timestep.These prompts describe objects, actions, and events spanning the entire video.

5. Method

FAST-AR reduces autoregressive video diffusion attention cost and memory through three training-free components: TempCache compresses the temporal KV cache, while AnnSA and AnnCA sparsify self- and cross-attention using approximate nearest-neighbor token selection. The method preserves attention accuracy through exact redundancy handling when possible and controlled approximation otherwise, while supporting efficient inference-time matching.

  • Overview: FAST-AR combines TempCache for temporal KV-cache compression, AnnSA for sparse self-attention, and AnnCA for sparse cross-attention.All three components use fast approximate nearest-neighbor matching to select a small candidate token set before attention.
  • ANN Attention: ANN attention retrieves keys with the largest query–key dot products and computes attention only over the resulting small candidate set.LSH and quantized similarity search reduce matching cost without offline preprocessing, followed by sparse attention kernels.
  • TempCache: TempCache identifies temporally corresponding keys with top-1 ANN matching and retains only recent representatives for groups of semantically corresponding tokens.Exact duplicate-key merging is lossless with a logit shift of + log m_t and mean-group values; similarity-threshold merging provides controlled approximation in practice.
  • AnnCA: AnnCA selects only frame-relevant prompt tokens, avoiding dense attention between all latent queries and all prompt keys.Cross-attention concentrates on different prompt tokens as frame content changes, such as cat, van, and dog.
  • AnnSA: AnnSA sparsifies self-attention by transferring semantic buckets from cross-attention pruning and computing attention with block-sparse kernels.The method exploits the tendency of video tokens to attend to semantically related and spatially or temporally proximate tokens.

6. Experiments

Experiments evaluate FAST-AR on autoregressive video diffusion and world-model backbones using fidelity, perceptual-quality, efficiency, and long-horizon scaling measures. Across these settings, FAST-AR achieves strong quality–efficiency trade-offs, including up to ×10.7–×10.8 speedups and stable scaling over long rollouts.

  • Models: Experiments use RollingForcing and LongVie2, representing autoregressive video diffusion and transformer-based world-model generation.RollingForcing targets real-time multi-minute synthesis, while LongVie2 frames long-video generation as world modeling.
  • Metrics: Quality is measured against Dense FlashAttention-3 using PSNR, SSIM, and LPIPS under matched prompts and random seeds.LongVBench and LongVGenBench scores additionally measure perceptual video quality.
  • Datasets: Evaluations use LongVBench and LongVGenBench to stress long-context generation and assess long-range quality under extended generation.LongVBench is based on VBench prompts, while LongVGenBench targets long-video generation.
  • Baselines: Baselines include Dense FlashAttention-3, TeaCache, and FlowCache, alongside prior attention-sparsification methods.TeaCache reuses intermediate computations across similar denoising steps, while FlowCache uses chunk-wise policies for autoregressive video generation.
  • Results: ×10.7–×10.8 end-to-end speedup is achieved by combining TempCache with ANN-based self- and cross-attention on Rolling-Forcing LongVBench.TempCache compresses the KV cache with the highest recall while matching dense-attention quality.
  • Long-horizon scaling: Nearly constant FPS is sustained over a 3K-frame rollout while attention cost remains effectively independent of cache length.Dense FA3 and prior sparsification baselines slow as context grows, whereas the proposed method maintains throughput.

7. Results

FAST-AR achieves a strong quality–efficiency trade-off on long-video generation, preserving dense-attention visual quality while maintaining stable throughput and nearly constant peak GPU memory during long rollouts.

  • Quantitative Results: ∼16% Min Density, ∼90–91% attention recall, and VBench ≈84.1 show TempCache-LSH/Quant compress the KV cache while preserving quality.These results are reported across LongVBench settings and match dense quality.
  • Long-Horizon Scaling: FPS for dense FA3 drops continuously over the 3K-frame rollout as its KV cache grows, while SVG1/2 and RadialAttention also fail to sustain throughput.The baselines incur substantial per-block preprocessing, including clustering or energy/decay estimation.
  • Long-Horizon Scaling: Peak GPU memory increases for dense attention and current approaches with the expanding KV cache, whereas FAST-AR remains constant.The same memory trend is also observed for world models, with LongVie2 results provided in the supplementary material.
  • Qualitative Results: Across diverse prompts, FAST-AR preserves Dense FlashAttention-3’s subject identity, fine details, coherent motion, and stable appearance over time.The qualitative comparison covers characters, landscapes, fast motion, and complex textures.

8. Summary

The paper introduces FAST-AR, a unified, training-free framework that addresses three attention redundancies in autoregressive video diffusion with TempCache, AnnCA, and AnnSA. Experiments achieve up to ×5– ×10 end-to-end speedups with constant GPU memory over long generations while preserving video quality.

  • Summary: KV-cache growth slows streaming generation and increases memory use, driven by duplicate keys across frames, slowly evolving semantic Q/K, and costly long-prompt cross-attention.These redundancies motivate the framework’s three components.
  • Summary: TempCache merges temporal KV entries, AnnCA prunes prompt tokens with ANN matching, and AnnSA sparsifies self-attention with ANN-based key selection.Together, the modules target temporal cache redundancy, cross-attention cost, and self-attention redundancy.
  • Summary: The framework is unified, training-free, plug-and-play, and requires no retraining or fine-tuning.It is designed to integrate with existing autoregressive video diffusion systems.
  • Summary: Up to ×5– ×10 end-to-end speedups are achieved with constant GPU memory over long video generations while preserving video quality.The reported gains address both runtime and memory growth during extended generation.

A. Proof of Lemma 5.1 · B. Additional Results

The proof establishes that attention over groups of identical keys can be reduced exactly to representative keys with averaged values and multiplicity biases. The additional results report efficient attention trade-offs, including strong compression, near-baseline quality, and substantial speedups.

  • A. Proof of Lemma 5.1: Lemma A.1 partitions positions into g disjoint groups whose keys are identical within each group.Each group G_t has multiplicity m_t = |G_t| and a representative key k′_t.
  • A. Proof of Lemma 5.1: Attention over n positions is exactly equivalent to attention over g representative keys and group-mean values.The equivalence follows by regrouping the attention numerator and denominator by identical-key groups.
  • A. Proof of Lemma 5.1: The exact reduction adds log m_t to each group’s attention logit to preserve the effect of multiplicity.Defining the shifted score as s̃_t = s_t + log m_t yields a softmax over groups.
  • A. Proof of Lemma 5.1: The proof derives the result by observing that identical keys produce identical attention scores, then regrouping numerator and denominator terms.Separate regrouping steps handle the numerator and denominator before exposing the group mean values.
  • B. Additional Results: ∼33.1% Min Density is achieved by TempCache with high recall, while ANN-based sparse attention preserves near-baseline quality at low density.Table 2 compares KV compression, sparse self-attention, sparse cross-attention, and the full system against dense FA3 on LongVie2 (LongVGenBench).
  • B. Additional Results: ×6.3–×6.9 speedup is achieved by the full method compared with dense FA3 in the reported LongVie2 trade-off.The full-system result combines TempCache with ANN-based sparse self-attention and cross-attention.
  • A. Proof of Lemma 5.1: Multiple positions with identical keys can be merged without approximation error by averaging values and applying a log m_t logit bias.When no duplicate keys exist, m_t = 1 for every group and the expression reduces to standard attention.

B.1. Quantitative Results · B.2. Qualitative Results

FAST-AR improves the quality–efficiency trade-off across video diffusion models while maintaining nearly constant attention time and peak GPU memory over long rollouts. Qualitatively, TempCache+ANN preserves visual fidelity and temporal consistency close to dense FlashAttention-3, unlike baselines that accumulate artifacts and drift.

  • B.1. Quantitative Results: TempCache reduces KV-cache density to ≈33% while retaining 91.3–92.4% recall, improving LongVGenBench to 63.10–63.16 at ×3.5–×3.7 speedup.TeaCache/FlowCache provide only ×1.1 speedup while retaining ∼90%+ density.
  • B.1. Quantitative Results: Our approach keeps attention time constant across the full 3K-frame generation, while dense FlashAttention-3 and baselines become progressively slower.SVG1/2 are orders-of-magnitude slower on the log-scale comparison, and RadialAttention remains slower than the proposed method.
  • B.1. Quantitative Results: Our method keeps peak GPU memory essentially flat throughout generation, indicating a constant-size KV cache, whereas FlashAttention-3 and other baselines grow with context length.The competing memory growth is attributed to KV-cache expansion.
  • B.1. Quantitative Results: TempCache-Quant lowers MAGI-1 compute to 110 PFLOPs and reaches 4.11× speed with VBench 78.99%, LPIPS 0.3156, SSIM 0.6555, and PSNR 23.12.Across MAGI-1 and SkyReels-V2, it reduces PFLOPs and latency while maintaining or improving perceptual quality.
  • B.1. Quantitative Results: AnnSA-LSH/Quant achieves quality comparable to or slightly better than sparse-attention baselines while remaining in a similar efficiency regime.On short clips, STA (FA3) still attains the lowest latency and highest speedup because sparse kernels add overhead and are less optimized than FlashAttention.
  • B.2. Qualitative Results: TempCache+ANN remains visually close to dense FlashAttention-3, preserving scene layout, subject identity, and lighting over time, including challenging thin structures and low-contrast textures.The qualitative comparisons are on LongVie2 rollouts.
  • B.2. Qualitative Results: SVG1/2 often accumulate blur, texture wash-out, and subject disappearance or distortion during long-horizon generation, while RadialAttention is more stable but still degrades in challenging scenes.The comparison identifies artifacts and drift as the main qualitative weaknesses of offline-designed sparsification baselines.

C. Ablation Study

The ablation study examines TempCache’s compression threshold and representative-selection choices alongside ANN quantization precision. Results expose trade-offs between attention recall and efficiency, while favoring conservative merging and recent-key representatives.

  • TempCache similarity threshold: As the TempCache similarity threshold decreases from 0.9 to 0.5, compression becomes more aggressive while attention recall drops from 0.90 to 0.10.Over-merging can remove keys that still contribute meaningfully to attention, motivating conservative thresholds.
  • Quantization precision for ANN matching: Increasing ANN quantization precision raises recall from 0.20 (2-bit) to 0.98 (32-bit), while FPS decreases from 32 to 10.The results show an accuracy–efficiency trade-off, with mid-precision settings such as 8-bit offering a compromise.
  • Representative selection in KV merging: Last-key achieves 90% attention recall, outperforming mean-key at 75% and medoid-key at 66% when selecting representatives for temporally corresponding key groups.Keeping the most recent key aligns with autoregressive generation, where current-frame queries are most compatible with recent keys.
Loading 2602.01801v2…