Source-linked AI summary
ReHyAt: Recurrent Hybrid Attention for Video Diffusion Transformers
Mohsen Ghafoorian, Amirhossein Habibian
TL;DR
Long-video diffusion is constrained by quadratic attention costs, while retraining powerful softmax models with alternative attention can be prohibitively expensive. ReHyAt combines local softmax and global linear attention in a recurrent, chunk-wise design; it reports near-state-of-the-art quality at sharply reduced training cost, though some videos remain temporally incoherent.
Problem
Long-video generation faces a memory bottleneck, while converting compute-heavy bidirectional softmax models through retraining from scratch is prohibitively expensive.
Source: §1 · p. 2Method
ReHyAt uses temporally chunked hybrid attention, combining local softmax with global linear attention, and reformulates it as a constant-memory recurrent model.
Source: §1 · p. 2Results
Takeaways & Limitations
The approach supports scalable long-duration video generation with constant memory and linear compute requirements.
Source: §6 · p. 9A small fraction of videos, especially from the most efficient variants, still show temporal incoherence.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Recent advances in video diffusion models have shifted towards transformer-based architectures, achieving state-of-the-art video generation but at the cost of quadratic attention complexity, which severely limits scalability for longer sequences. We introduce ReHyAt, a Recurrent Hybrid Attention mechanism that combines the fidelity of softmax attention with the efficiency of linear attention, enabling chunk-wise recurrent reformulation and constant memory usage. Unlike the concurrent linear-only SANA Video, ReHyAt's hybrid design allows efficient distillation from existing softmax-based models, reducing the training cost by two orders of magnitude to ~160 GPU hours, while being competitive in the quality. Our light-weight distillation and finetuning pipeline provides a recipe that can be applied to future state-of-the-art bidirectional softmax-based models. Experiments on VBench and VBench-2.0, as well as a human preference study, demonstrate that ReHyAt achieves state-of-the-art video quality while reducing attention cost from quadratic to linear, unlocking practical scalability for long-duration and on-device video generation. Project page is available at https://qualcomm-ai-research.github.io/rehyat.
1. Introduction
ReHyAt targets the compute and memory limits of long video diffusion by converting bidirectional softmax models into a recurrent hybrid form. It combines local softmax with global linear attention and supports chunk-wise generation with constant memory.
1. Introduction
- Self-attention scales quadratically in sequence length, making videos beyond roughly 10 seconds difficult within typical GPU memory and latency budgets.Source: §1 · pp. 1–2
- Linear attention offers linear complexity and constant memory, but its weaker fine-grained dependency modeling limits quality, while existing hybrid designs remain quadratic.Source: §1 · p. 2
- ReHyAt applies softmax attention to critical local dependencies and linear attention globally, combining high-fidelity local modeling with long-range dependencies.Source: §1 · p. 2
- ReHyAt converts a state-of-the-art bidirectional softmax video diffusion model into a chunk-wise recurrent model in fewer than 200 GPU-hours, with negligible quality impact.Source: §1 · p. 2
- The hybrid formulation can be reformulated as a chunk-wise RNN with constant memory complexity for efficient inference.Source: §1 · p. 2
2. Related Work
Related work reduces attention costs through efficient architectures and token-selection strategies, but many approaches require costly retraining or sacrifice full-token attention. ReHyAt instead distills pretrained softmax models into a recurrent hybrid design.
2. Related Work
- ReHyAt distills and fine-tunes pretrained softmax models, avoiding the heavy retraining or from-scratch training often required by efficient-attention approaches.Source: §2 · p. 2
- ReHyAt preserves the transformer block structure for lightweight distillation, unlike SSM and RWKV alternatives that are costly to derive from DiT weights.Source: §2 · p. 2
- Token merging, downsampling, tiling, and sparsity gain efficiency by discarding attention for most tokens, whereas ReHyAt attends to the full token set.Source: §2 · p. 3
- Compared with Attention Surgery [11], which retains quadratic complexity, ReHyAt uses a temporally non-uniform hybrid layout that enables linear complexity and recurrent inference.Source: §2 · p. 3
- ReHyAt combines linear attention for long-range dependencies with softmax attention for adjacent tokens and is obtained in ∼160 GPU-hours, two orders of magnitude more efficiently than SANA-Video [4].Source: §2 · p. 3
3. Methods: ReHyAt
ReHyAt combines local softmax attention with global linear attention in temporally chunked sequences, then reformulates the design for chunk-wise recurrent generation. Its training pipeline distills a bidirectional softmax teacher and lightly fine-tunes the resulting model.
3.1. Preliminaries: Linear Attention
- Linear attention enables precomputation and linear-time complexity, but its kernel mapping is less expressive than the softmax exponential kernel.ReHyAt uses distinct learnable query and key feature maps with per-head embeddings and polynomial-degree expansion to approximate the exponential kernel’s dynamic range.Source: §3.1 · p. 3, §3.2 · p. 4
3.2. Hybrid Attention Formulation
- Each chunk uses softmax attention for tokens within its temporal slices and linear attention for the other tokens; overlapping key/value slices support message passing across chunks.Overlap is intended to mitigate episodic motion or appearance incoherence at latent-chunk transitions.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
3.3. Recurrent HyAt
- The chunk-wise recurrent formulation keeps computation O(N) in video length and memory constant, while generating each full chunk at once.Training can use the causal non-recurrent form and rearrange the trained model into an RNN at sampling time.Source: §3.3 · p. 5
3.4. Two-stage Training
- ReHyAt distills existing bidirectional full-softmax attention into a recurrent hybrid model, achieving competitive quality in fewer than 200 GPU-hours.The pipeline comprises attention distillation followed by lightweight fine-tuning.Source: §3.4 · p. 5
3.4.1. Attention Distillation
- The first stage trains each block independently, learning only ϕq and ϕk to match teacher activations across prompts, noise samples, and denoising steps.This distillation requires no prompt/video training pairs.Source: §3.4.1 · p. 5
3.4.2. Lightweight Fine-tuning
- Fine-tuning the whole DiT on a modest prompt/video-pair set for a small number of iterations recovers generation quality, especially chunk-transition smoothness.The optimization uses the normal flow-matching objective.Source: §3.4.2 · p. 5
4. Experimental Setup
The experiments assess generation quality against benchmark and human evaluations, profile compute and mobile performance, and specify the datasets and model variants used for training.
4.1. Evaluation of generation quality
- Quality is evaluated at Wan2.1's original 81 × 480 × 832 resolution using extended VBench and VBench-2.0 prompts, plus 500 blinded paired-video preferences from 50 VBench prompts.Participants could choose a preferred video or report no significant difference; pair order was randomized.Source: §4.1 · p. 5
4.2. Assessment of compute complexity
- Compute profiling compares attention FLOPs and video-duration scaling; mobile tests measure latency and memory reads and writes on a Snapdragon 8 Gen 4.The mobile measurements use 320 × 480 frames at the original five-second length and longer durations.Source: §4.2 · pp. 5–6
4.3. Training specification
- Low-resolution fine-tuning uses 350K Open-Sora Plan videos, while high-resolution fine-tuning uses 22K synthetic Wan2.1 14B videos; variants convert 15, 20, or 25 of 30 blocks.Hybrid variants test chunk sizes Tc ∈ {1, 2, 3, 5, 7} and overlap sizes To ∈ {0, 1, 2, 3}; each converted block adds approximately 2.4M parameters.Source: §4.3 · p. 6
5. Results
ReHyAt remains competitive in generation quality while reducing attention compute and memory burden. Ablations examine chunk size, overlap, and causality as design choices affecting quality and recurrent execution.
5.1. Generation Quality
- ReHyAt is competitive with efficient video diffusion models on VBench, remains competitive with larger models such as CogVideoX1.5 5B on VBench-2.0, and shows no significant human-preference difference from Wan2.1 1.3B.The human preference comparison covers 500 paired videos.Source: §5.1 · p. 6
5.2. Sampling Compute Burden
- Up to 4× fewer attention operations than FlashAttention; the Tc=3, To=1 variant is approximately 2× more efficient than quality-preserving uniform hybrid attention at R=2.The comparison uses five-second videos at 480 × 832 resolution.Source: Figure 4 · p. 7, §5.2 · p. 7
- At 121 frames, ReHyAt is approximately 16× faster than Wan2.1 FlashAttention and is the only tested method that readily extends beyond 10 seconds without out-of-memory errors.The mobile latency comparison spans durations from 5 seconds (81 frames) to 10 seconds (161 frames).Source: Table 4 · p. 7, §5.2 · p. 7
- At 121 frames, ReHyAt uses approximately 11× less total memory read/write than FlashAttention, while its peak-memory usage remains constant as video duration grows.The reported total memory read/write is expected to grow linearly with duration.Source: §5.2 · p. 7, Table 5 · p. 8
5.3. Ablations Studies
- Increasing Tc generally improves quality, with the larger gain from Tc=1 to 2 than from further increases to 3 and 4.The authors suggest the first extension of softmax attention from spatial to spatiotemporal may explain the larger gain.Source: §5.3 · pp. 7–8, Figure 5 · p. 8, Table 6 · p. 8
- The first spatial-to-spatiotemporal softmax extension yields a more significant improvement than further increases in To to 3 and 4; Table 7 measures ReHyAt hybrid quality on VBench with 25× converted blocks and Tc=3.Source: §5.3 · p. 8, Table 7 · p. 8
- Causality does not deteriorate VBench quality, and removing forward-looking linear attention yields little compute saving; its main advantage is RNN reformulation with lower, constant peak memory.That memory property supports on-device generation of longer videos.Source: Table 8 · p. 8, §5.3 · p. 8
6. Conclusion and Future Work
ReHyAt is a recurrent hybrid attention mechanism for video diffusion transformers, targeting scalable long-duration generation with constant memory and linear compute. Its lightweight distillation pipeline achieves near-state-of-the-art quality, though some videos—especially from the most efficient variants—show temporal incoherence.
6. Conclusion and Future Work
- ReHyAt enables video diffusion transformers to use constant memory and linear compute for scalable, long-duration generation.The conclusion presents it as a recurrent hybrid attention mechanism.Source: §6 · pp. 8–9
- The lightweight distillation pipeline achieves near-state-of-the-art video quality with dramatically reduced training cost.Source: §6 · p. 9
- A small fraction of videos, especially from the most efficient variants, still show some temporal incoherence.The conclusion identifies this as an area for future improvement.Source: §6 · p. 9
7. Appendix
The appendix reports training and sampling settings, qualitative comparisons, benchmark results, and a compute-scalability comparison for ReHyAt variants. It also states that Microsoft Copilot was used only to improve clarity and readability.
7.1. Training Details and Hyperparameters
- Distillation trains only the key and query transformations per block, while fine-tuning updates all hybrid DiT parameters for 1,000 iterations.Distillation uses batch size 1 and learning rate 10−3; fine-tuning uses batch size 16 and learning rate 10−5.Source: §7.1 · p. 12
7.2. Qualitative Samples
- Qualitative comparisons show frames from the original Wan2.1 1.3B model and three recurrent hybrid variants across 18 prompts at 480×832.The corresponding full video sequences are included in the supplementary materials.Source: §7.2 · p. 12
7.3. Detailed VBench Comparison
- Figure 7 compares selected hybrid models with Wan2.1 1.3B across all VBench dimensions on the full benchmark at 480×832.Source: §7.3 · p. 12, Figure 7 · p. 13
7.4. Detailed VBench-2.0 Comparison
- Both ReHyAt variants, 15×Tc=3 and 15×Tc=5, perform on par with Wan2.1 1.3B on the VBench-2.0 overall Total score.Tables 9–11 provide the full VBench-2.0 results at 480×832 resolution, including comparisons with attention surgery (15×R2).Source: §7.4 · p. 12, Table 9 · p. 14, Table 10 · p. 14
7.5. Compute complexity vs Attention Surgery
- Figure 6 compares compute scalability with video length for ReHyAt, attention surgery, and Wan2.1 flash attention, showing FLOPs and latency.Source: Figure 6 · p. 12, §7.5 · p. 12
7.6. Use of Large Language Models
- Microsoft Copilot was used exclusively to improve clarity and readability; the authors state that the technical content, experimental design, and conclusions are their own.Source: §7.6 · p. 12