Source-linked AI summary
RhymeFlow: Training-Free Acceleration for Video Generation with Asynchronous Denoising Flow Scheduling
Chensheng Dai, Shengjun Zhang, Yifan Li, Zhang Zhang, Zheng Zhu, Yueqi Duan
TL;DR
Video diffusion inference is costly because quadratic 3D attention and dense denoising process every frame at every timestep. RhymeFlow uses content-aware asynchronous schedules, densely denoising keyframes while projecting skipped non-keyframe states; the paper reports reduced redundant computation with maintained visual fidelity and compatibility with other accelerators.
Problem
Video diffusion models incur high inference cost from quadratic 3D attention and dense, full-timestep denoising of every frame.
Method
RhymeFlow selects pivotal semantic keyframes for dense denoising, progressively skips non-keyframe steps, and projects their missing latent states for temporally consistent attention.
Results
RhymeFlow substantially reduces redundant computation while maintaining high visual fidelity across the reported video-generation evaluations.
Takeaways & Limitations
Content-aware frame scheduling offers an acceleration dimension orthogonal to sparse attention, quantization, and token pruning, enabling potential compounded improvements.
Abstract
from arXiv · showhide
Video generation models based on Diffusion Transformers (DiTs) have achieved remarkable performance in video synthesis, yet they suffer from high inference latency and computational costs due to the quadratic complexity of 3D attention. Existing acceleration methods primarily reduce computational complexity within each individual denoising steps through techniques such as sparse attention and KV-caching. However, they rigidly adhere to the inherent constraint of the standard diffusion pipeline: every frame in the target video sequence must be subjected to a complete, dense denoising process across all diffusion timesteps. We observe that due to the corresponding contents and motions among adjacent frames, when keyframes with critical semantic transitions are anchored, the intermediate states of others often follow more predictable trajectories, which indicates that such uniform, dense denoising process is inherently redundant for natural video data. To this end, we introduce \textbf{RhymeFlow}, a training-free framework that decouples the denoising trajectories of different frames. Specifically, we first identify a sparse set of pivotal key frames that dominate the latent semantic evolution. Then, only these keyframes undergo dense, step-by-step denoising to ensure structural integrity, while non-keyframes progressively skip denoising steps to minimize computational cost. Since skipped intermediate states of non-keyframes break the temporal coherence in keyframe denoising steps, leading to visual degradation, we further introduce a latent trajectory projection module, which enables keyframes to interact with a complete and temporally consistent sequence representation. Extensive experiments on current DiT-based video generation models demonstrate our method outperforms existing baselines with higher inference speed and better visual quality.
1 Introduction
Video diffusion models face prohibitive inference costs, while existing training-free methods reduce computation within each denoising step but still process every frame densely across all timesteps. RhymeFlow instead assigns heterogeneous schedules to keyframes and non-keyframes, using latent trajectory projection to preserve temporal coherence.
- Motivation: DiT-based video diffusion is expensive because 3D spatiotemporal attention is quadratic and generation requires many denoising steps.These costs hinder practical deployment.
- Motivation: Existing acceleration methods lower the complexity of individual denoising steps, but every frame still undergoes dense, stepwise denoising across all diffusion timesteps.The cited approaches include KV-cache management, compression, and sparse attention.
- Motivation: Neighboring frames exhibit spatiotemporal coherence, making non-keyframe latent trajectories predictable after pivotal keyframes are densely denoised.The paper associates pivotal keyframes with critical structural or semantic transitions.
- RhymeFlow: RhymeFlow identifies pivotal keyframes, densely denoises them, and lets non-keyframes skip designated steps to reduce computation.The framework is training-free and assigns heterogeneous denoising schedules.
- RhymeFlow: Latent trajectory projection estimates skipped non-keyframe states so keyframes can attend to a complete, temporally consistent sequence representation.This addresses the temporal-coherence problem introduced by skipped intermediate states.
2 Related Works
Related work addresses video diffusion cost through fewer denoising steps, model compression, caching, and sparse attention. These approaches target sampling steps or per-pass computation, motivating RhymeFlow’s frame-specific scheduling perspective.
- Video Diffusion Models: DiT video models generate high-quality videos but remain slow and costly because 3D attention scales quadratically with pixel-frame tokens.The cost becomes especially prohibitive for long, high-resolution videos.
- Decreasing the Denoising Steps: Denoising-step reduction uses ODE solvers, consistency models, and progressive distillation, but some approaches require retraining or have limitations for large video models.The supplied passage begins describing two major limitations, with the second truncated.
- Diffusion Model Compression: Model-compression methods reduce per-forward-pass cost through quantization, efficient architectures, or compact autoencoders.Examples include W8A8 and 4-bit quantization schemes.
- Training-free Acceleration via Caching: Training-free caching methods reuse similar intermediate features or KV-cache entries from adjacent denoising steps to avoid redundant computation.DeepCache and FasterCache are cited examples.
- Sparse Attention in LLMs and Video: Sparse-attention methods reduce attention computation by focusing on concentrated subsets of tokens, including heavy-hitter or local tokens.The supplied passage identifies video adaptations such as SVG and SAP, but their description is truncated.
3 Method
RhymeFlow selects semantically important keyframes, begins with synchronous warm-up, then applies progressive asynchronous skipping to non-keyframes while preserving keyframe context through projection and periodic synchronization.
- Method Overview: RhymeFlow classifies frames by latent semantic evolution and assigns different denoising schedules to keyframes and non-keyframes.The method is training-free and combines progressive skipping with latent trajectory projection.
- Sequential Keyframe Selection: Clean-latent proxies support sequential keyframe selection by providing structurally clearer representations for evaluating frame-wise correlations.A single-step denoising prediction estimates each frame’s clean latent before selection.
- Sequential Keyframe Selection: The selection procedure traverses candidate frames chronologically and compares each with its nearest preceding keyframe using cosine similarity and a threshold.The initial frame is used to initialize the keyframe set.
- Warm-up Stage: All frames receive synchronous full-attention updates during the initial Tw warm-up steps to establish global composition, motion priors, and color palette.The warm-up uses the standard unmodified denoising function.
- Progressive Asynchronous Scheduling: Progressive asynchronous scheduling uses smaller skips during noisy early stages and larger skips during smoother, more predictable later stages.The piecewise schedule uses midpoint Tmid and strides nsmall < nlarge, such as 2 and 3.
- Asynchronous Denoising Step: Keyframes advance one denoising step while non-keyframes jump multiple steps, reducing computation through heterogeneous update frequencies.Non-keyframes advance directly from an earlier state while bypassing intermediate denoising steps.
- Asynchronous Denoising Step: Rhythmic Points periodically synchronize all frames with full 3D attention and recalibrate non-keyframe trajectories against keyframe anchors.These checkpoints bound error accumulation from prolonged step-skipping.
- Latent Trajectory Projection: Latent trajectory projection estimates missing non-keyframe states so keyframe attention can use true keyframe states and projected non-keyframe states.The projected representations provide transient context for the current attention operation.
4 Experiments
Experiments evaluate RhymeFlow across multiple video diffusion models, metrics, baselines, parameter settings, qualitative examples, ablations, and user preferences. The method achieves favorable quality–efficiency trade-offs while its design choices improve fidelity and control memory overhead.
- Experimental Setup: RhymeFlow is evaluated on Wan 2.1 and CogVideoX-v1.5, producing 81-frame videos at 720p and using fidelity, visual-quality, and efficiency metrics.The evaluation includes PSNR, SSIM, LPIPS, VBench dimensions, and computational efficiency.
- Experimental Setup: The experiments compare RhymeFlow with SpargeAttn, MInference, PAB, SVG, and SAP using official task configurations and architecture-compatible settings.SAP is evaluated on Wan 2.1, while MInference and PAB are evaluated on CogVideoX-v1.5.
- Trade-off Analysis: At Tw = 8, increasing M from 3 to 5 raises PSNR from 23.742 to 27.707 and SSIM from 0.721 to 0.812, while speedup falls from 1.60× to 1.39×.The ablation demonstrates the efficiency–fidelity trade-off controlled by warm-up duration and keyframe budget.
- Trade-off Analysis: The default Tw = 8, M = 4 configuration records ImgQual. 0.6706, SubCon. 0.8831, and a 1.53× speedup, reducing runtime from 993.5s to 650.4s.This setting is used for subsequent benchmark comparisons.
- Comparison with State-of-the-Art Methods: Across Wan 2.1 and CogVideoX-v1.5, RhymeFlow reaches 26.291 PSNR and 0.783 SSIM with 1.53× speedup on Wan 2.1, and 1.78× speedup with 0.986 SubConsistency on CogVideoX-v1.5.On HunyuanVideo, the base method achieves the best visual quality across reported metrics, while Ours+SAP achieves the lowest latency and highest speedup.
- Ablation Studies: Semantic keyframe selection improves SSIM from 0.643 to 0.783 and lowers LPIPS from 0.183 to 0.168 versus Uniform, while Random and First produce PSNR below 21.The results support selecting anchors according to latent semantic changes rather than fixed placement.
- Ablation Studies: Removing progressive scheduling reduces speedup from 1.53× to 1.40× and PSNR from 26.291 to 25.399, while KV-cache management lowers peak VRAM from 49.9 GB to 42.6 GB.Trajectory projection is also reported as important for limiting latent error and preserving PSNR.
- User Study: In user comparisons against SVG, RhymeFlow is preferred for visual quality by 51.2% versus 30.5%, while temporal coherence favors RhymeFlow by 53.7%.The reported comparisons use binomial tests after excluding ties.
5 Conclusion
RhymeFlow accelerates video diffusion by assigning content-aware asynchronous denoising schedules to keyframes and predictable non-keyframes. The framework reduces redundant computation while maintaining high visual fidelity and suggests compatibility with further acceleration methods.
- RhymeFlow differentiates semantically important keyframes from predictable non-keyframes through a content-aware asynchronous schedule.It assigns heterogeneous update frequencies and approximated trajectories to different frame groups.
- The framework substantially reduces redundant computation while maintaining high visual fidelity.
- RhymeFlow could combine with sparse attention, quantization, and token pruning for compounded improvements.The paper identifies learned continuous scheduling as a future direction for adapting schedules to scene dynamics and model behavior.
A.1 Additional Visualization Results
Additional visualizations compare RhymeFlow with the original dense model and encourage motion-video inspection for assessing temporal behavior. The reported results emphasize preservation of complex visual and motion details without noticeable artifacts.
- The additional results compare sequences generated by the original full-dense model and RhymeFlow.The accompanying multimedia materials are recommended for assessing temporal coherence and motion dynamics beyond static image sequences.
- RhymeFlow preserves intricate details and complex motion dynamics without noticeable artifacts.
A.2 Evaluation on Long-Duration Videos
Long-duration evaluations examine RhymeFlow's stability across parameter settings and its trade-off against training-free baselines. On extended videos, RhymeFlow and its SVG combination reach up to 1.91× acceleration while preserving high visual fidelity.
- A.2 Evaluation on Long-Duration Videos: The long-duration evaluation extends experiments to 240-frame videos to assess scalability and robustness.
- A.2 Evaluation on Long-Duration Videos: RhymeFlow maintains high structural similarity and perceptual quality across warm-up-duration and keyframe-budget configurations.The sensitivity analysis varies Tw and M on Wan 2.1.
- A.2 Evaluation on Long-Duration Videos: 1.91× acceleration is achieved by RhymeFlow and Ours+SVG on extended videos while preserving high visual fidelity versus masking-based counterparts.
A.3 Ablation Study on Progressive Scheduling Parameters
The progressive-scheduling ablation studies update strides and analyzes asynchronous speed-up using FLOPs-related variables. Larger strides favor speed over generation fidelity, supporting the default nsmall = 2 and nlarge = 3 balance.
- A.3 Ablation Study on Progressive Scheduling Parameters: Larger update strides prioritize inference speed over generation fidelity in the progressive-scheduling ablation.The study validates nsmall = 2 and nlarge = 3 as the default balance between visual quality and speed.
- A.3 Ablation Study on Progressive Scheduling Parameters: The FLOPs analysis uses Wan2.1-T2V-1.3B and focuses on basic asynchronous scheduling without progressive scheduling.
- A.3 Ablation Study on Progressive Scheduling Parameters: The analysis defines L, H, d, F, N, C, T, and Twarmup as model, video, context, and denoising variables.
- A.3 Ablation Study on Progressive Scheduling Parameters: Twarmup denotes the number of warmup steps using dense attention.
B.1.1 Dense Baseline FLOPS Analysis.
The dense baseline applies full attention across all frames, layers, and denoising timesteps, with quadratic attention dominating computation. RhymeFlow reduces this cost by using warmup, selective skipping, and heterogeneous frame participation.
- Dense Baseline FLOPS Analysis: A single self-attention layer computes QKV projections, attention scores, softmax weighting, value aggregation, and output projection.The sequence contains S tokens with feature dimension D; softmax is treated as negligible relative to matrix multiplications for large D.
- Dense Baseline FLOPS Analysis: The dense layer cost is 4SD^2 + 2S^2D + 3S^2, with the quadratic term 2S^2D typically dominant for large S and D.This dominance motivates reducing the number of tokens participating in attention.
- Dense Baseline FLOPS Analysis: Across L layers and T timesteps, dense denoising costs L × T × 2D(2SD + S^2) FLOPs.The formulation multiplies per-layer attention cost by the full denoising depth and number of steps.
- Selective Step Skipping: RhymeFlow decomposes computation into warmup, skip-step, and full-denoising FLOPs rather than applying dense attention uniformly.During warmup, all frames retain dense computation; afterward, only selected frame groups participate at each step.
- Selective Step Skipping: With T = 50, Twarmup = 10, M = 5, n = 2, and F = 21, skip steps use only keyframes while rhythmic points update all frames.The supplied numerical setup uses 5 keyframes and 21 total frames.
B.1.4 Theoretical Speedup Analysis
The theoretical analysis attributes RhymeFlow’s acceleration to asynchronous frame scheduling, while practical overheads and reduced GPU utilization narrow the measured gain. Rolling KV-caching and synchronized layer updates support memory efficiency and temporal consistency.
- Theoretical Speedup: The theoretical FLOP reduction predicts a 1.545× speedup, while the measured speedup is 1.53×.The difference is attributed to hardware efficiency and algorithmic overheads absent from pure FLOP counting.
- Practical Overheads: Algorithmic overheads include keyframe identification and latent trajectory projection for skipped frames.Similarity computation, selection, and vector operations add costs that are not included in the idealized FLOP analysis.
- Hardware Efficiency: Reducing skip-step processing from 21 frames to 5 keyframes lowers sequence length from 75,600 to 18,000 and reduces GPU occupancy.The workload can shift from compute-bound to memory-bound, lowering effective TFLOPs/s.
- Rolling KV-Cache: A per-layer, per-frame rolling KV-cache projects intermediate states from the two most recent denoising steps.The cache stores post-attention hidden states and updates them at rhythmic points for subsequent skipped steps.
- Skipped-Step Attention: At skipped steps, non-keyframes provide projected key-value representations while keyframes perform full attention over all frames.This preserves the contextual inputs required by keyframe attention without computing non-keyframe query outputs.
- Cross-Layer Consistency: Shared keyframe indices, a global step counter, and synchronized cache updates enforce a common timestep across layers.The design requires τ(ℓ) = τ(ℓ′) = tcurrent for all layers.
C Detailed Implementation of RhymeFlow
RhymeFlow combines content-aware keyframe selection with asynchronous denoising. Keyframes follow dense updates, non-keyframes use progressive strides and latent projection, and the scheduling dimension can complement token-level sparse attention.
- Implementation Overview: RhymeFlow consists of content-aware sequential keyframe selection followed by asynchronous denoising flow scheduling.Algorithm 1 selects frames, and Algorithm 2 applies heterogeneous update schedules.
- Asynchronous Scheduling: The method begins with synchronous warmup, then assigns progressive skip strides and activates keyframes plus periodically scheduled non-keyframes.The stride changes from nsmall to nlarge at the midpoint, while active frames are updated according to the schedule.
- Latent Projection: Latent Trajectory Projection synthesizes missing temporal context while keyframes retain step-by-step denoising for structural integrity.This combination keeps keyframe updates globally coherent despite non-keyframe skipping.
- Combination with Sparse Attention: RhymeFlow operates across denoising steps, whereas SVG and SAP optimize token interactions within individual attention computations.The distinct dimensions make the scheduling approach orthogonal to intra-step sparse attention.
- Combination with Sparse Attention: A combined mask first removes skipped non-keyframe computation and then sparsifies attention among the remaining active frames.This hierarchical sparsity can compound inter-step and intra-step efficiency gains.
D.2 Analysis of Failure Cases
Extreme skipping exposes a quality boundary for RhymeFlow: large strides produce shimmering and temporal aliasing in non-keyframes. The authors identify learned scheduling and improved interpolation as future directions.
- Failure Cases: With nskip = 7, aggressive skipping causes shimmering artifacts and loss of high-frequency textures in non-keyframes.The stress test highlights the need to balance progressive scheduling rather than maximize the stride.
- Failure Cases: At very large strides, linear latent trajectory projection fails to capture high-frequency motion because its locally smooth ODE-path assumption breaks down.The resulting degradation appears particularly in non-keyframe regions.
- Future Directions: The current progressive strategy is heuristic-based, motivating future learned or uncertainty-based scheduling policies.Such policies could determine which frames to skip and the appropriate intervals in a data-dependent manner.
- Future Directions: Combining RhymeFlow with PTQ or LoRA is proposed as a possible route to 3× to 5× speedups without compromising generative capabilities.The passage presents this as a potential extension, not an achieved result.