PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion

Hidir Yesiltepe, Jiazhen Hu, Tuna Han Salih Meral, Adil Kaan Akan, Kaan Oktay, Hoda Eldardiry, Pinar Yanardag

arXiv:2605.30351v1cs.CVcs.AI

TL;DR

Long-horizon video diffusion has largely left the memory-heavy per-head KV layout unchanged. VideoMLA replaces it with shared latent content and positional keys, reducing KV memory while achieving the best reported one-minute overall score among evaluated methods.

  • Problem

    Existing long-rollout video diffusion methods modify cached tokens or positions but generally preserve the memory-heavy per-head KV layout.

  • Method

    VideoMLA replaces per-head keys and values with a shared low-rank content latent and head-shared decoupled 3D-RoPE positional key.

  • Results

    0.859 highest 60s overall VBench score among evaluated methods, while VideoMLA also achieves the best dynamic degree at both 30s and 60s.

  • Takeaways & Limitations

    The MLA bottleneck, rather than pretrained video-attention spectra, determines the effective rank used by VideoMLA.

  • Takeaways & Limitations

    Experiments focus on Wan2.1-T2V-1.3B and minute-scale generation, leaving larger backbones, higher resolutions, longer horizons, and prompt switching for future work.

Abstract

from arXiv · show

Long-rollout causal video diffusion has converged on a fixed-size sliding-window KV cache, with recent progress innovating within this layout by changing which tokens occupy the window or how their positions are encoded. The per-head KV layout itself, a dominant contributor to streaming memory and latency, has been mostly left unchanged. In this paper, we present the first study of Multi-Head Latent Attention (MLA) in video diffusion. VideoMLA replaces per-head keys and values with a shared low-rank content latent and a shared decoupled 3D-RoPE positional key, reducing per-token KV memory by 92.7% at every cached layer. We further investigate why MLA succeeds in video diffusion even though the spectral assumption often used to motivate it in language models does not hold: pretrained video attention is not low-rank, with 99%-energy effective rank far above any practical latent dimension. VideoMLA retains quality at compression ratios where direct spectral approximation would predict large reconstruction error. We show that the MLA bottleneck, rather than the pretrained spectrum, determines the effective rank: both spectral and random initialization occupy nearly the full rank budget from initialization, and training preserves this budget while adapting within it. On VBench, VideoMLA matches short-horizon streaming video diffusion baselines, achieves the best overall score at long horizons among evaluated methods, and improves throughput by 1.23x on a single B200.

1 Introduction

VideoMLA targets the per-head KV-cache layout in causal video diffusion, where fixed-size rolling caches increasingly constrain long-horizon generation. It achieves substantial memory reduction while addressing why MLA works despite pretrained video attention being far from low-rank.

  • Causal video diffusion generates frames or chunks autoregressively with a rolling KV cache, enabling minute-long videos at interactive rates on a single GPU.As rollouts lengthen, the per-head KV cache increasingly defines the operating constraints.
  • Recent methods mainly retain fixed-size sliding-window caches while changing token selection, memory compression, training, or positional encoding.Complementary approaches instead alter attention computation, including constant-memory linear attention and encoder-only caching.
  • 92.7% lower per-token KV memory is achieved at every cached layer by replacing per-head keys and values with shared latent and positional representations.VideoMLA uses a shared low-rank content latent and a head-shared decoupled 3D-RoPE positional key.
  • VideoMLA retains quality despite Wan-1.3B video attention having 99%-energy effective rank far above practical latent dimensions.This challenges the usual MLA motivation based on low-rank pretrained attention projections.
  • r0.99 ≈0.98 dc at every budget shows the composed operator’s rank is set by the architectural bottleneck rather than dense-source spectral structure.The result holds across depth for budgets dc ∈{64, 128, 256, 512}.

2 Related Work

Prior causal video generation work develops streaming autoregressive diffusion with rolling KV caches, while efficiency methods restructure attention or memory. MLA was introduced and extended for language deployment, motivating its study in video diffusion despite different memory profiles and pretrained attention spectra.

  • Causal Video Generation: CausVid initiated causal video diffusion with DMD-based distillation, and Self-Forcing reduced train–test mismatch through self-generated rollout training.Later work targets long-horizon stability through joint denoising and attention sinks, teacher-guided correction, and related techniques.
  • Efficient Causal Video Generation: SANA-Video, SCD, and VideoSSM reduce temporal or layer-wise memory through linear attention, causal-encoder caching, or SSM-compressed global memory.These approaches do not compress the per-token, per-head KV state.
  • Multi-Head Latent Attention: DeepSeek-V2 introduced MLA’s shared low-rank latent and decoupled positional key, while DeepSeek-V3 scaled it and later works adapted pretrained LLM attention to MLA.MTLA compresses along time; MHA2MLA and TransMLA convert pretrained MHA or GQA LLMs, whereas this study examines MLA in video diffusion.

3 Method

VideoMLA compresses cached key-value content into a shared low-rank latent while storing positional information in a separate head-shared 3D-RoPE key. It reconstructs head-specific attention states on demand, reducing default cache storage to 224 scalars per token per layer, a 92.7% reduction.

  • Shared low-rank KV cache: VideoMLA jointly compresses each token’s key-value content into a shared latent cKV_t, replacing dense per-head content keys and values in the cache.The latent dimension dc is much smaller than the full per-head KV dimension and serves as the layer’s content-cache capacity knob.
  • On-demand reconstruction: Per-head keys and values are reconstructed from the shared cached latent through up-projections, while only the positional branch is shared across heads.NoPE queries, NoPE keys, and values remain head-specific after up-projection.
  • Cache efficiency: 92.7% reduction: the default cached state uses 224 scalars per token per layer instead of dense per-head KV storage.The general per-token state changes from 2n_h d_h scalars to d_c + d_rope scalars.
  • Decoupled positional encoding: VideoMLA keeps content position-free and carries positional information through one head-shared decoupled 3D-RoPE key, with rotation applied when assembling the active window.This keeps cached states independent of absolute rollout time and supports sliding-window re-indexing.
  • Integration with causal attention: VideoMLA preserves the surrounding chunkwise causal pipeline because attention consumes reconstructed per-head keys and values as in the dense module.Sink tokens, FlexAttention kernels, and causal block masks operate without structural changes outside the attention layer.

4 Experiments

Experiments implement VideoMLA by replacing only self-attention in Wan-2.1 T2V-1.3B, yielding a compact latent KV cache and evaluating quality, efficiency, and serving scalability. VideoMLA preserves long-horizon quality while improving throughput, latency, memory use, and batch capacity.

  • Implementation Details: VideoMLA replaces only self-attention in the 30-block Wan-2.1 T2V-1.3B backbone, using dc = 192 and dq = 768 by default.The model has hidden dimension 1536, 12 heads, and per-head dimension 128.
  • Quantitative Results: 0.859 overall score: VideoMLA achieves the highest 60s VBench score and the best dynamic degree at both 30s and 60s, with 0.981 and 0.958.It also obtains the best imaging quality and motion smoothness, indicating that latent KV compression preserves dynamic long-horizon generation.
  • Efficiency Results: Highest throughput and lowest latency: VideoMLA leads chunk-wise autoregressive models on both efficiency metrics and achieves the best CLIP-T score.LongSANA has a slightly higher CLIP-F score, but VideoMLA attains a higher HPSv3 score and produces sharper, more dynamic, temporally stable rollouts.
  • Batch Scaling Under Fixed Memory: 77–91% reduction: MLA lowers the per-request memory slope from 6.26 GB/batch for MHA to 0.57–1.43 GB/batch across dc ∈{64, 128, 192, 256, 512}.Dense MHA reaches the memory limit at B = 28, while MLA with dc = 64 remains within budget at B = 320 and supports 4.6× to at least 11.4× larger batch sizes.

5 Why MLA Works in Video Diffusion: Rank Budget vs. Spectral Structure

VideoMLA retains generation quality despite pretrained video attention being far from low-rank at practical latent dimensions. The evidence indicates that MLA’s architectural rank budget, rather than the dense operator’s spectrum, determines the learned effective rank.

  • 45.8% of spectral energy is preserved at dc = 192 in the median layer, while 99%-energy effective rank exceeds 1300 in every layer.A direct rank-dc spectral approximation would discard most dense key/value energy despite VideoMLA retaining generation quality.
  • This mismatch rejects recovering hidden low-rank structure in pretrained attention weights as the explanation for MLA’s success in video diffusion.Instead, MLA changes the optimization problem by imposing a rank bottleneck through its construction.
  • The composed key/value operator is constrained to rank at most dc, and its layer-wise 99%-energy rank remains close to 0.98dc across latent sizes.For dc ∈{64, 128, 256, 512}, normalized spectra share a common shape truncated at dc, showing the bottleneck sets effective rank.
  • Both SVD and random initialization nearly saturate the rank budget from initialization, while training preserves effective rank and spectral tail.Training adapts within the imposed budget rather than discovering a lower-rank solution or collapsing the spectrum.

6 Limitations and Broader Impact

VideoMLA’s latent budget presents a quality–efficiency trade-off, while the study’s scope is limited to one backbone and minute-scale generation. More efficient long-horizon generation may reduce deployment cost and broaden access to creative tools.

  • Limitations: d_c = 64 improves memory headroom but loses fine-grained details and degrades quality.The latent budget therefore cannot shrink arbitrarily without sacrificing quality.
  • Limitations: Experiments focus on Wan2.1-T2V-1.3B and minute-scale generation, leaving larger backbones, higher resolutions, longer horizons, and prompt switching for future work.These conditions define the current evaluation scope.
  • Broader Impact: More efficient long-horizon generation can reduce deployment cost and broaden access to creative tools.This is the paper’s stated broader-impact potential.

7 Conclusion … C.1 Wan2.1-T2V-1.3B Backbone

VideoMLA introduces a shared latent KV-cache design for autoregressive video diffusion, targeting the dense per-head cache used by Wan2.1-T2V-1.3B. The merged material also describes the user-study protocol and the backbone’s latent-token, rectified-flow, positional-encoding, and streaming-cache foundations.

  • 7 Conclusion: VideoMLA reduces per-token KV-cache memory by 92.7% by replacing dense per-head keys and values with shared content and positional representations.The design uses a shared low-rank content latent and a head-shared decoupled 3D-RoPE positional key while remaining compatible with standard chunk-causal generation.
  • B Details on User Study: The user study evaluates one-minute generations from nine models with ratings from 50 participants.Participants assess Prompt Adherence, Temporal Consistency, and Dynamic Consistency.
  • C Background: Wan2.1-T2V-1.3B operates as a latent video diffusion transformer over spatiotemporal latent tokens rather than RGB pixels.A 3D VAE compresses the temporal dimension by 4× and each spatial dimension by 8×.
  • C Background: The backbone uses rectified flow, linearly interpolating clean latent x_0 and Gaussian noise ε as x_t = (1 − t)x_0 + tε.The reverse process uses a neural velocity field and Euler integration at inference time.
  • C.1 Wan2.1-T2V-1.3B Backbone: Wan2.1-T2V-1.3B contains 30 transformer blocks, hidden dimension d = 1536, 12 attention heads, and per-head dimension d_h = 128.Its diffusion transformer applies multi-head self-attention over video latent tokens.
  • C.1 Wan2.1-T2V-1.3B Backbone: 3.02B cached scalars correspond to approximately 6.0GB in bf16/fp16 for a 21-latent-frame cache spanning 1,560 tokens per latent frame and 30 cached layers.This dense per-head KV layout is VideoMLA’s main memory target.
  • C.1 Wan2.1-T2V-1.3B Backbone: The backbone applies 3D-RoPE separately across temporal, height, and width coordinate subspaces before concatenation.Each RoPE dimension has a fixed maximum sequence length of 1024, although RoPE remains mathematically defined beyond that range.
  • C.1 Wan2.1-T2V-1.3B Backbone: During autoregressive long-video generation, latent frame chunks are generated sequentially using a rolling KV cache of previous chunks.Reusing previous key and value states enables streaming generation, while dense attention retains full per-head states for every retained token.

D Implementation Details … D.5 Long-Horizon RoPE Re-indexing

VideoMLA modifies Wan2.1-T2V-1.3B temporal self-attention with shared low-rank content and positional caches while preserving chunk-causal sliding-window inference. Long-horizon RoPE re-indexing keeps positions within the training regime for extended rollouts using the specified sink, window, and sampler.

  • D.1 Backbone and Tokenization: VideoMLA replaces only temporal self-attention layers in Wan2.1-T2V-1.3B, whose backbone has 30 transformer blocks, hidden dimension 1536, and 12 attention heads.The text-conditioning branch and all non-attention modules remain unchanged.
  • D.1 Backbone and Tokenization: Each 5-second, 480 × 832 clip at 16 fps contains 32760 self-attention tokens, generated autoregressively in chunks of 3 latent frames.The VAE representation has 21 latent frames, each producing 1560 visual tokens.
  • D.2 VideoMLA Block: 224 scalars per token per layer replace 3072 in dense MHA, yielding a 13.7× cache reduction through shared content and positional representations.The default cache stores 192 content-latent scalars plus 32 positional-key scalars.
  • D.3 NoPE/RoPE Split and 3D RoPE: VideoMLA uses a NoPE/RoPE split: positional keys are shared across heads, while NoPE keys, NoPE queries, and values remain head-specific after up-projection.The RoPE branch is decoupled from the content cache.
  • D.3 NoPE/RoPE Split and 3D RoPE: 3D-RoPE allocates 16 complex frequency pairs across temporal, height, and width axes as (6, 5, 5), using the highest-frequency bands from each axis group.This is the default positional-frequency allocation.
  • D.4 Chunk-Causal Sliding-Window Attention: Chunk-causal attention allows within-chunk attention and restricts earlier chunks from attending to later ones, while long-horizon inference retains one sink latent frame and the most recent six latent frames.The sink occupies 1560 cached token slots, and tokens outside it are evicted in FIFO order when the cache is full.
  • D.5 Long-Horizon RoPE Re-indexing: During long rollouts, cached positional keys are stored before RoPE, then assigned local temporal coordinates and rotated together with the current query chunk in the active sink-plus-window context.This bounded, window-relative coordinate system remains usable after cache eviction.
  • D.5 Long-Horizon RoPE Re-indexing: All reported long-video rollouts use one sink latent frame, a six-latent-frame local window, and the 4-step student sampler, keeping RoPE phases within the training regime beyond 21 latent frames.The re-indexing enables extension beyond the original 5-second clip length.

D.6 Training Pipeline

VideoMLA is trained in three stages: teacher-forced flow matching, 4-step consistency distillation, and distribution matching distillation. The reported results, including long-horizon evaluations, use the Stage-3 checkpoint.

  • Training setup: Training uses three stages on 8 NVIDIA B200 GPUs with FSDP full sharding, bf16 mixed precision, AdamW, and rectified-flow denoising.AdamW uses β1 = 0 and β2 = 0.999.
  • Stage 1: Teacher Forcing: Stage 1 initializes MLA projections from an SVD-style decomposition of pretrained Wan dense attention matrices and trains a chunk-causal flow-matching student.The student uses clean previous-block context from teacher-encoded latents, learning rate 5 × 10−6, total batch size 2, 1000 training timesteps, and timestep shift 5.0.
  • Stage 2: Consistency Distillation: Stage 2 starts from the Stage-1 checkpoint and distills VideoMLA to a 4-step sampling schedule [1000, 750, 500, 250].The stage uses timestep shift 5.0, classifier-free guidance scale 3.0, generator learning rate 2 × 10−6, critic learning rate 4 × 10−7, and total batch size 2.
  • Stage 3: Distribution Matching Distillation: Stage 3 initializes from the Stage-2 checkpoint at iteration 2500 and fine-tunes with distribution matching distillation on the same 4-step schedule.The frozen teacher provides the real score, the fake score is learned online, and training uses five critic updates per generator update with EMA weight 0.99.
  • Final checkpoint: The Stage-3 checkpoint is used for all reported VideoMLA results, including long-horizon evaluations.This makes the final distillation checkpoint the basis for the paper’s reported evaluation results.

E Inference-Time Reparameterization

VideoMLA uses an inference-time reparameterization that performs attention directly in latent form, avoiding dense per-head key/value reconstruction. Learned content projections are absorbed into fixed matrices, while the position-dependent RoPE branch remains separate for sliding-window re-indexing.

  • Latent-form inference: Inference avoids reconstructing dense per-head keys and values by computing attention directly from cached content latents.Dense reconstruction after each cache read would partially erase latent-caching benefits.
  • Content-path absorption: The content score uses a learned matrix Ah that can be precomputed once, then applied directly to the query latent without materializing per-head vectors.Ah is independent of the sequence, cache contents, diffusion timestep, and rollout position.
  • Value-path absorption: The value path accumulates weighted cached latents and projects them with Bh, rather than reconstructing dense values before applying the output projection.This folds the value up-projection into the output mixer.
  • RoPE branch: RoPE remains separate: unrotated head-shared positional keys are rotated with RoPE3D using current window indexing before computing positional scores.Because RoPE is position-dependent, it cannot be folded into a fixed parameter matrix; unrotated storage preserves sliding-window re-indexing.
  • Memory and bandwidth: 224 cached scalars per token per layer are used with h = 32, versus 3072 scalars for dense MHA.The reparameterization preserves the training-time attention computation while realizing latent-cache memory and bandwidth savings at inference.

F Additional Ablations

Additional ablations show that latent KV dimension and the NoPE/RoPE channel split govern VideoMLA’s quality–efficiency trade-offs. Moderate compression and content-heavy channel allocation outperform overly restrictive or RoPE-heavy alternatives.

  • Latent dimension: Latent KV dimension controls VideoMLA’s main quality–efficiency trade-off.At dc = 64, compression is largest but semantic and quality scores drop; increasing to dc = 128 largely recovers quality while retaining substantial compression.
  • Latent dimension: At dc = 64, overly aggressive compression reduces semantic and quality scores by removing fine-grained visual details.This setting provides the largest cache compression and memory headroom, but its budget is too restrictive.
  • NoPE/RoPE split: The 96/32 NoPE/RoPE split performs best, indicating that streaming video benefits from allocating most channels to cached content.The balanced 64/64 setting improves over the extremes but remains below the content-heavy default.
  • NoPE/RoPE split: Only 16 RoPE channels provide insufficient positional capacity, while the RoPE-heavy 32/96 split leaves too little capacity for cached content and hurts semantic fidelity.The resulting weaknesses affect temporal and spatial anchoring in the first case and semantic fidelity in the second.
Loading 2605.30351v1…