PaperlayerView exact revision on arXiv

Source-linked AI summary

Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation

Haocheng Xi, Yiming Xie, Hexu Zhao, Yiwen Zhang, Michael Liu, Thomas Creavin, Kurt Keutzer, Xiuyu Li, Zhaoyang Lv, Chenfeng Xu, Haiwen Feng

arXiv:2609.20744v2cs.LG

TL;DR

Directly replacing Softmax with linear attention can degrade video-generation quality by losing fine-grained interactions, while attention is a major computational cost. Video DeltaNet combines local Softmax with bidirectional linear memory; on a 14.3-second, 768p video, eight-step VDN-H3 reduces denoising time by 14.5× relative to 50-step dense H3 on eight B200 GPUs.

  • Problem

    Directly applying linear attention to video can degrade generation quality because its compressed state cannot preserve the fine-grained interactions available to Softmax.

    Source: §1 · p. 2

  • Method

    Video DeltaNet combines local Softmax attention and global boundary anchors with bidirectional linear memory, whose Video Delta Attention updates memory once per frame using its spatial tokens jointly.

    Source: §1 · p. 2

  • Results

    14.5× reduction in denoising time relative to the 50-step dense H3 baseline: VDN-H3 completes denoising for a 14.3-second, 768p video in 6.70 seconds on eight NVIDIA B200 GPUs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Eight-step VDN-H3 remains close to 50-step Dense H3 across the reported quality evaluations while accelerating video generation.

    Source: §8 · p. 12

    When residuals use the same inherited state, overlapping keys can produce conflicting writes without accounting for one another.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

Video diffusion models repeatedly process long spatiotemporal token sequences during denoising, making attention a major computational bottleneck. Linear attention offers an appealing alternative and has been widely adopted in recent large language models, but directly applying it to video models often fails to preserve the fine-grained interactions required for high-quality generation. We present Video DeltaNet (VDN), which combines local Softmax attention with bidirectional linear memory for long-range video context. Its linear branch introduces Video Delta Attention (VDA), which updates memory once per frame by jointly incorporating its spatial tokens. Separate output projections and learnable gates calibrate the two branches, while a staged teacher-alignment recipe progressively introduces the new pathway into pretrained models. We instantiate VDN on MiniMax H3, applying the hybrid to video-to-video interactions while retaining Softmax for interactions involving text or audio. With eight-step distillation and an optimized SGLang serving stack, VDN-H3 completes DiT denoising for a 14.3-second, 768p video in 6.70 seconds on eight NVIDIA B200 GPUs, corresponding to a 14.5x speedup over the 50-step dense H3 baseline on the same GPU count. GitHub code available at: https://github.com/OpenVDN/vdn-minimax-h3. Weights available at: https://huggingface.co/OpenVDN/vdn-minimax-h3

1 INTRODUCTION

Video generation’s quadratic Softmax cost motivates a hybrid that preserves local interactions while using linear memory for distant video context. VDN combines this design with staged adaptation and reports faster denoising while maintaining Dense H3-level video quality.

Motivation

  • Attention dominates long-sequence video-generation cost, but replacing Softmax directly with linear attention meaningfully degrades quality by losing fine-grained interactions.In the profiled MiniMax H3 workload, Softmax accounts for more than 85% of denoiser runtime.Source: §1 · p. 2

Contributions

  • VDN combines local Softmax attention and global boundary anchors with bidirectional linear memory for distant video context.Video Delta Attention updates memory once per frame by jointly incorporating spatial tokens and key correlations.Source: §1 · p. 2
  • A staged recipe aligns the new linear pathway with the pretrained teacher, then uses low-rank updates to co-adapt the hybrid layer while preserving the backbone.The complete recipe also includes few-step distillation and optimized inference without training a new foundation model from scratch.Source: §1 · p. 2

Results

  • 14.5× reduction relative to the 50-step dense H3 baseline on the same GPU count; eight-step VDN-H3 matches or slightly exceeds Dense H3 across video-quality metrics.For a 14.3-second, 768p video, denoising takes 6.70 seconds on eight NVIDIA B200 GPUs.Source: §1 · p. 2

2 VIDEO DELTANET: HYBRID ATTENTION ARCHITECTURE

Video DeltaNet assigns nearby video interactions to explicit Softmax attention and distant context to bidirectional linear memory. It adds global boundary anchors, text-aware memory, and separately calibrated branch outputs.

2 VIDEO DELTANET: HYBRID ATTENTION ARCHITECTURE

  • Video DeltaNet allocates nearby-frame interactions to token-to-token Softmax and distant video context to linear attention.Source: §2 · p. 2

2.1 SLIDING-WINDOW SOFTMAX ATTENTION

  • Nearby frames use exact Softmax within a bidirectional temporal window; two boundary anchors provide global references by connecting all frames to the first and last latent frames.The window follows the tokenizer’s five-frame temporal chunks, yielding a 15-frame window except at sequence boundaries.Source: §2.1 · p. 3

2.2 BIDIRECTIONAL LINEAR ATTENTION

  • Forward and reverse scans summarize video context before and after the Softmax window, excluding boundary anchors already available through Softmax.At readout, VDN uses the state immediately outside each window boundary, applies accumulated channel-wise decay across the skipped span, and sums the two disjoint readouts.Source: §2.2 · p. 3
  • Each directional scan starts with half of a text-token summary, so the summed linear readouts count the prompt once while Softmax retains direct text access.Source: §2.2 · p. 4

2.3 COMBINING OUTPUTS FROM TWO BRANCHES

  • The branches use separately gated and projected outputs: Softmax has a content-dependent sigmoid gate, while the linear readout is RMS-normalized and sigmoid-gated.The projected outputs are added to the pretrained residual stream; the feed-forward sublayer remains unchanged.Source: §2.3 · p. 4

3 VIDEO DELTA ATTENTION: FRAME-WISE DELTA RULE

Video Delta Attention (VDA) adapts the delta rule to video by updating memory jointly over a frame’s spatial tokens. The section motivates this frame-wise update, derives its joint solution, and establishes a non-expansive inherited-state transition under fixed prepared features and gates.

3.2 VIDEO DELTA ATTENTION: A FRAME-WISE UPDATE

  • VDA updates recurrent memory jointly from all key–value pairs in a video frame, so interactions among its spatial tokens shape one frame-level write.Unlike token-wise corrections evaluated independently from the same decayed state, the joint update lets overlapping key directions be resolved together.Source: §3 · p. 4, §3.2 · p. 6

3.1 PRELIMINARIES OF LINEAR ATTENTION

  • The token-wise delta rule reads a value associated with a key and writes a prediction-residual correction, with decay and write gates controlling memory retention and correction strength.Video diffusion instead makes all spatial tokens of a frame available together, motivating a frame-level rather than token-ordered update.Source: §3.1 · pp. 4–5

3.2 VIDEO DELTA ATTENTION: A FRAME-WISE UPDATE

  • When several patches address similar key directions, independent corrections evaluated at the same frozen state can conflict or reinforce one another.The joint solution evaluates residuals at the shared updated state; its inverse couples writes through key overlap.Source: §3.1 · p. 5, §3.2 · p. 6

3.3 STABILITY AND CORRELATION AWARENESS OF VIDEO DELTA ATTENTION

  • For fixed prepared features and gates, VDA’s inherited-state transition is non-expansive: changing the entering state changes its carried contribution by at most the same Frobenius norm.The proposition assumes βt,u ≥0 and 0 ≤αt,j ≤1; the bound is about inherited state, not new writes or the full network.Source: §3.3 · p. 6

4 END-TO-END TRAINING PIPELINE

The training pipeline progressively integrates a Linear branch into the pretrained Softmax backbone, then distills 50-step VDN-H3 into an eight-step sampler. Architecture adaptation freezes pretrained base weights while first aligning branches separately and later co-adapting the hybrid.

4 END-TO-END TRAINING PIPELINE

  • The pipeline first adapts the architecture in three stages, then distills 50-step VDN-H3 into an eight-step sampler.The released student is trained against VDN-H3’s own 50-step sampler, isolating step reduction from architecture conversion.Source: §4 · p. 6, §4.2 · p. 7

4.1 STAGED ARCHITECTURE ADAPTATION

  • A1 aligns each Linear branch independently, A2 trains the assembled hybrid end to end, and Stage B jointly trains the Linear pathway, Softmax gates, and QKVO LoRA adapters.Pretrained base weights remain frozen throughout; A1 and A2 also keep Softmax gates frozen.Source: §4.1 · pp. 6–7, Figure 4 · p. 7

4.2 FEW-STEP DISTILLATION

  • The released student uses a DMD2-style objective without the GAN term and is initialized from community MiniMax-H3-Turbo-LoRA.Generator, real-score, and fake-score roles share one FSDP backbone through separate adapters, with three fake-score updates per generator update; the checkpoint uses 250 generator steps.Source: §4.2 · p. 7

5 EFFICIENT INFERENCE

Inference optimizations fuse VDA operations, scan chunk-level transitions, and specialize kernels and serving for the hybrid attention paths. The described implementation keeps recurrent states and small-matrix inverses in FP32 while applying MXFP8 to wide GEMMs.

5 EFFICIENT INFERENCE

  • Four fused Triton kernels combine VDA preparation, per-frame statistics, boundary-state gathering, and output normalization, gating, and layout conversion.QK normalization and rotary embedding are fused separately on the Softmax path.Source: §5 · p. 7
  • Chunk-wise scans compose each VAE chunk into an affine transition and scan the shorter chunk sequence in both directions, reducing scan depth and launch overhead by roughly the chunk size.The prompt state is included as a leading virtual frame.Source: §5 · p. 7
  • A single CUDA kernel computes each frame’s (I + At)−1 by blocked Gauss–Jordan elimination, while recurrent states and small-matrix inverses remain in FP32.The serving stack also packs window-Softmax queries by visible-key pattern, shards VDA by attention head, and uses MXFP8 for wide GEMMs.Source: §5 · pp. 7–8

6 EXPERIMENTS

The experiments compare eight-step VDN-H3 with 50-step Dense H3 and fast-model baselines on fixed evaluation prompts, then measure quality, efficiency, kernel optimizations, and qualitative outputs. Across reported quality evaluations, VDN-H3 remains close to Dense H3 while the optimized system reduces denoising latency.

6.1 SETTINGS

  • The evaluation uses 103 fixed third-party prompts, with all models rendered at the same resolution and duration, and compares final eight-step VDN-H3 against 50-step Dense H3 and four-NFE FastH3.Training uses 10,015 pre-encoded video clips; reported metrics include five video-quality measures, RAFT flow magnitude, FIRM-Video scores, and FL2VA endpoint metrics.Source: §6.1 · p. 8

6.2 QUALITY RESULTS

  • Across five no-reference quality metrics, eight-step VDN-H3 matches or exceeds 50-step Dense H3 by +0.06 to +1.00; its FIRM-Video scores and FL2VA measures also remain close.For FL2VA, VDN-H3 has PSNR 28.67 versus 28.85, SSIM 0.826 versus 0.833, and LPIPS 0.1156 versus 0.1044 (lower is better).Source: §6.2 · pp. 8–9

6.3 EFFICIENCY RESULTS

  • For 14.3-second, 768p video, the system-optimized backbone reduces latency from 35.35 to 11.16 seconds on one H200 (3.2×) and from 16.0 to 6.2 seconds on one B200 (2.6×).As latent frames increase from 42 to 102, measured Softmax attention density falls from 42.1% to 20.0%; eight-step distillation and eight-GPU inference bring B200 denoising to 6.7 seconds.Source: §6.3 · p. 9, Table 2 · p. 9, Table 3 · p. 9

6.4 ABLATION STUDIES

  • Kernel optimizations improve VDA-Prep latency to 1.6 ms on H200 and 3.4 ms on B200, while a fused inverse yields a 4.6–5.0× speedup.Chunk-wise scans fall from 4.6 to 1.1 ms on H200 and from 3.0 to 0.8 ms on B200 with seven heads per rank in eight-GPU inference; Window Softmax improves on B200 but remains nearly unchanged on H200.Source: §6.4 · pp. 9–10, Figure 6 · p. 10

6.5 GALLERY

  • The gallery compares 50-step Dense H3 with eight-step VDN-H3 across visual styles, scene transitions, and motion patterns, with reported visual quality on par.The figure presents six synchronized frames from five prompts in each paired comparison.Source: §6.5 · p. 10

7 RELATED WORK

Related work spans efficient attention for video, recurrent linear memory and hybrid diffusion architectures, and methods for few-step distillation and distributed inference. The paper also notes concurrent analysis of frame-level delta updates.

Efficient video attention algorithms

  • Video attention acceleration includes exact and low-precision kernels, sparse methods, hybrid sparse-linear approaches, and quantization, alongside video-specific sparse methods.The cited work ranges from optimized exact kernels to methods targeting video diffusion or autoregressive video caches.Source: §7 · p. 10

Linear attention and recurrent memory

  • Linear attention and recurrent-memory work includes DeltaNet variants and state-space models, while video methods apply these ideas to understanding, generation, and long-term consistency.The passage also situates linear and hybrid diffusion backbones alongside LoGeR’s sliding-window attention and parametric long-term memory.Source: §7 · p. 10
  • Reflections on Video DeltaNet are identified as concurrent analysis of frame-level delta updates.The cited concurrent work is Zhu (2026).Source: §7 · p. 11

Few-step distillation and distributed inference

  • Few-step distillation methods reduce denoising evaluations, while multi-GPU approaches shard sequences or distribute patch and pipeline work across diffusion steps.The passage also identifies xDiT as composing parallel strategies and StreamDiffusionV2 as targeting low-latency distributed video streaming.Source: §7 · pp. 11–12

8 CONCLUSION

Video DeltaNet accelerates video generation while largely preserving Dense H3 quality. Its optimized backbone and combined distillation and multi-GPU inference reduce denoising latency on the reported workload.

8 CONCLUSION

  • On the 14.3-second, 768p workload, the optimized backbone is 2.6× faster on one B200 and 3.2× faster on one H200.With eight-step distillation and eight-GPU inference, DiT denoising takes 6.70 seconds on B200s and 12.5 seconds on H200s; eight-step VDN-H3 remains close to 50-step Dense H3 across reported quality evaluations.Source: §8 · p. 12

A.1 PROOF OF PROPOSITION 1

The proof analyzes how inherited state changes under fixed frame inputs. It establishes a bound for that inherited state, with the stated scope excluding new writes and the full network.

A.1 PROOF OF PROPOSITION 1

  • Fixing frame inputs lets the proof separate inherited state from the new write by substituting the decayed prior state into Equation (5).The excerpt introduces this decomposition as the basis for the proposition’s argument.Source: §A.1 · p. 16
  • The Gram matrix is positive semidefinite, so the eigenvalues of its regularized inverse lie in (0, 1]; together with bounded decay, this supports the inherited-state bound.The proof states 0 ≤ α_t,j ≤ 1 and eigenvalues 1/(1 + λ_i) ∈ (0, 1].Source: §A.1 · p. 16
  • The proposition does not require decay and the Gram matrix to commute, and its bound covers inherited state at fixed frame inputs rather than new writes or the full network.Source: §A.1 · p. 16

A.2 FRAME-SIZE SCALING AND CORRELATION AWARENESS

The appendix explains how VDA adjusts frame-wise updates to key correlations, describes bidirectional state handling and adaptation settings, and compares quality across denoising budgets.

A.2 FRAME-SIZE SCALING AND CORRELATION AWARENESS

  • With αt = 1, aligned keys can approach this bound, so per-token normalization alone does not ensure stability; eigenvalues of I − At leave [−1, 1] if any λi(At) > 2.Source: §A.2 · p. 16
  • VDA adjusts its joint frame update to key geometry: overlapping keys receive a saturating weight, while orthogonal keys receive separate weights, unlike SANA-WM’s shared U-based scaling.The inverse-based update needs no frame-size scale.Source: §A.2 · p. 16

A.3 BOUNDARY GATHER AND DECAY BRIDGE

  • The bidirectional boundary gather uses forward and reverse interior-frame states, while its decay bridge avoids local frame writes and bypasses linear attention for full-clip windows.Clips containing only anchor frames also receive no linear video output.Source: §A.3 · p. 17

A.4 ARCHITECTURE ADAPTATION HYPERPARAMETERS

  • Architecture adaptation uses a 10−4 peak learning rate in Stages A1, A2, and B, while Stage D uses 10−5 for the generator and 2 × 10−5 for the fake-score model.In Stage B, the Linear branch uses 0.4× the LoRA learning rate; Table 4 covers A1, A2, and B, not few-step distillation.Source: §A.4 · p. 17, Table 4 · p. 18

A.5 RELEASE CONFIGURATION AND REPRODUCIBILITY

  • The release configuration preserves the vdn_solve settings, including chunk size 5, radius 1, anchors in both directions, and the alpha decay bridge.Release artifacts are intended to link each result to model and execution configuration through hashes, manifests, seeds, hardware, software, and timing metadata.Source: §A.5 · p. 18

A.6 QUALITY ACROSS DENOISING BUDGETS

  • At 50 steps, Stage-B VDN-H3 is broadly on par with Dense H3 before few-step distillation, though two endpoint-fidelity measures show small degradations.This comparison isolates the hybrid architecture and its adaptation from sampling-step reduction.Source: §A.6 · p. 18, Figure 8 · p. 19
  • At eight steps, VDN-H3 scores highest on all five no-reference quality measures and its VQAT is 10.78 points higher than FastH3 v2.It remains close to the dense-plus-Larry reference on FIRM and FL2VA; FastH3 v2 has no reported FL2VA scores.Source: §A.6 · p. 19, Figure 9 · p. 19
  • At four steps, VDN-H3 exceeds Dense H3 plus Larry and FastH3 v1 on all five no-reference quality measures, with VQAT 12.64 points above FastH3 v1.It also has better FL2VA endpoint fidelity than FastH3 v1; the significance flags do not directly compare those two variants.Source: §A.6 · p. 19, Figure 10 · p. 20

Related papers

Loading 2609.20744v2…