FasterCache: Training-Free Video Diffusion Model Acceleration with High Quality

1 INTRODUCTION

• Video diffusion attention increases inference cost, while cache-based reuse reduces redundant computation but raises concerns about feature variation and limited exploration beyond attention modules.A 6-second 480P video typically takes 2–5 minutes to synthesize.Source: §1 · p. 2, Figure 2 · p. 2
• CFG nearly doubles inference time because of unconditional outputs, whose features are highly similar at the same timestep but less similar across adjacent timesteps.Conditional–unconditional differences shift from low- to mid-frequency features during mid-sampling to high-frequency features late in sampling, evolving gradually.Source: §1 · p. 2
• FasterCache dynamically reuses attention features to preserve adjacent-step distinction and continuity, while CFG-Cache stores conditional–unconditional residuals and adjusts frequency components before reuse.The approach is training-free and aims to accelerate inference while preserving video details.Source: §1 · pp. 2–3
• 1.67× speedup on Vchitect-2.0, with VBench baseline 80.80% → FasterCache 80.84%; FasterCache also outperforms existing approaches in speed and video quality.Evaluations cover Open-Sora 1.2, Open-Sora-Plan, Latte, CogVideoX, and Vchitect-2.0.Source: §1 · p. 3

2.1 PRELIMINARY

• CFG computes conditional and unconditional outputs during sampling, adding inference latency to the iterative denoising process.The reverse process predicts each next sample from the current one; CFG’s extra unconditional computation increases cost.Source: §2.1 · pp. 3–4

2.2 RETHINKING ATTENTION FEATURE REUSE

• Adjacent-step attention features are highly similar but retain subtle differences essential for fine detail, so direct reuse can produce smoother, less detailed videos.The paper’s example is a smoother sky with fewer visible stars under vanilla feature reuse.Source: §2.2 · p. 4, Figure 5 · p. 4, Figure 4 · p. 4

2.3 FEATURE REDUNDANCY IN CFG

• In mid-to-late sampling, conditional and unconditional outputs are more similar at the same timestep than across adjacent steps, while reusing a previous unconditional output degrades video quality.The findings point to redundancy within CFG that can be targeted without assuming adjacent-step outputs are interchangeable.Source: §2.3 · pp. 4–5, Figure 6 · p. 5

2.4 FASTERCACHE FOR VIDEO DIFFUSION MODEL

• Dynamic Feature Reuse computes attention outputs at alternate timesteps and uses differences between cached features, weighted by w(t), to approximate intermediate features.In the experiments, w(t) increases as sampling progresses, emphasizing feature differences more at later stages.Source: §2.4 · p. 5
• CFG-Cache stores high- and low-frequency biases between conditional and unconditional outputs, then adaptively enhances them when reconstructing unconditional outputs during reuse.The frequency weights prioritize mid-low frequencies during the mid-sampling phase and high-frequency components later.Source: Figure 8 · p. 6, §2.4 · p. 6

3.1 EXPERIMENTAL SETTINGS

• The evaluation covers five video diffusion models, compares FasterCache with PAB and ∆-DiT, and measures efficiency and visual quality with MACs, latency, VBench, LPIPS, PSNR, and SSIM.Experiments use full spatial and temporal attention every two timesteps; CFG output reuse runs every five timesteps starting at one-third of sampling steps.Source: §3.1 · p. 7

3.2 MAIN RESULTS

• FasterCache shows stable acceleration efficiency and superior visual quality across base models, sampling schedulers, resolutions, and video lengths.Table 1 compares efficiency and visual quality against ∆-DiT and PAB on a single GPU.Source: Table 1 · p. 7, §3.2 · p. 8

3.3 ABLATION STUDY

• Dynamic Feature Reuse and CFG-Cache each reduce inference costs, and combining them further minimizes overhead with negligible efficiency impact versus vanilla feature reuse.Table 2 reports the impact of the acceleration components on inference efficiency.Source: §3.3 · p. 8, Table 2 · p. 8
• Vanilla feature reuse lowers VBench and LPIPS, while Dynamic Feature Reuse improves VBench from 78.34% to 78.69% and alleviates detail loss.The reported examples include reduced moon and snowflake detail with vanilla reuse; adding the bias term lowers feature MSE.Source: §3.3 · p. 8, Table 3 · p. 8
• CFG-Cache without enhancement reduces visual quality, while low- or high-frequency enhancement improves the corresponding components and their combination gives the best visual quality.Dynamic enhancement also helps mitigate error accumulation and improve visual fidelity.Source: §3.3 · p. 8, Figure 10 · p. 9

3.4 SCALABILITY AND GENERALIZATION

• With Dynamic Sequence Parallelism, FasterCache consistently improves inference speed as GPU count increases and surpasses the compared methods.Table 4 reports scaling to multiple GPUs across base models.Source: Table 4 · p. 8, §3.4 · p. 9
• FasterCache maintains stable acceleration as video resolution and frame count increase, and the evaluation also includes image-to-video and image synthesis models.Figure 11 reports acceleration efficiency across resolutions and lengths; Figure 12 presents results for DynamiCrafter and PixArt-sigma.Source: §3.4 · p. 9, Figure 11 · p. 9

4.1 DIFFUSION MODELS FOR VIDEO SYNTHESIS

• Video synthesis research moved from diffusion UNets toward diffusion transformers after their scalability was validated in image synthesis.The cited video models include both architectural approaches.Source: §4.1 · p. 9

4.2 EFFICIENCY IMPROVEMENTS IN DIFFUSION MODELS

• Efficiency methods reduce sampling steps or per-step inference costs, but existing improvements remain insufficient for diffusion transformers in video synthesis.Per-step approaches include pruning, quantization, architecture search, token reduction, and caching intermediate features.Source: §4.2 · p. 10

5. CONCLUSION AND DISCUSSION

• FasterCache is training-free and combines dynamic feature reuse with CFG-Cache to accelerate video synthesis while preserving quality and detail.The method addresses adjacent-step feature reuse in attention modules and redundancy between same-timestep conditional and unconditional features.Source: §5 · p. 10
• The method may produce unsatisfactory results when base-model synthesis quality is suboptimal or may occasionally degrade results in complex scenes with substantial motion.The paper says the latter can currently be remedied through manual hyperparameter adjustments.Source: §5 · p. 10

A.1 FURTHER DETAILS OF BASE MODELS

• Open-Sora 1.2 combines 2D-VAE and 3D-VAE compression with ST-DiT blocks for diffusion.The VAE components are described as enhancing video compression.Source: §A.1 · p. 15
• Open-Sora-Plan uses CausalVideoVAE for visual compression and a 3D full-attention architecture to capture joint spatial and temporal features.Source: §A.1 · p. 15
• Latte models video distributions in latent space using spatio-temporal tokens and a series of transformer blocks.Source: §A.1 · p. 15

• PAB propagates attention outputs across subsequent steps using a pyramid-style broadcasting mechanism.Source: §A.2 · p. 15
• PAB varies its broadcast strategy by attention-layer variance and also uses broadcast sequence parallelism for distributed inference.Source: §A.2 · p. 15
• Δ-DiT caches block residuals from later DiT blocks during early sampling and from earlier blocks during later sampling.Source: §A.2 · p. 15

A.3.1 MORE DISCUSSION ON DYNAMIC FEATURE REUSE

• DFR treats layer output features as a timestep-dependent function and estimates the nonzero error incurred by Vanilla Feature Reuse’s adjacent-step substitution.Source: §A.3.1 · p. 15
• The second-order term only marginally reduces L1 distance versus the first-order term, while adding memory and latency costs, so DFR uses first-order error estimation.Source: §A.3.1 · p. 16
• The feature bias is scaled by w(t) to approximate the reuse error, reducing information loss while maintaining computational efficiency.Source: §A.3.1 · p. 16
• A linearly increasing w(t), rising from 0 to 1 during DFR application, is a simple and effective way to capture missing features dynamically.Source: §A.3.1 · p. 16
• Feature-bias methods outperform reuse without bias; linear w(t) matches optimized learnable weights and outperforms constant weights, so DFR adopts linear interpolation.Source: §A.3.1 · p. 16, Table 5 · p. 16
• Compared with the original features, DFR introduces smaller feature differences than Vanilla Feature Reuse, whose synthesized images show noticeable detail degradation.Source: §A.3.1 · p. 16

A.3.2 FURTHER DISCUSSION ON CFG-CACHE

• After the early stage, conditional and unconditional outputs at the same timestep are modeled as differing by a bias Δ, whose change across nearby timesteps is treated as negligible when Δt is sufficiently small.Source: §A.3.2 · pp. 16–17
• From CFG-Cache onset to sampling end, output differences shift from low-frequency dominance toward high-frequency dominance.Source: §A.3.2 · p. 17
• CFG guidance emphasizes perceptual shapes and layouts in early and middle sampling, then contributes primarily to high-quality details in later sampling, motivating stage-specific frequency weights.Source: §A.3.2 · p. 17

A.3.3 FASTERCACHE UNDER DIFFERENT CFG SCALES AND NEGATIVE PROMPTS

• The authors compare empty and non-empty negative prompts and evaluate LPIPS, SSIM, and PSNR against the original model’s videos.Source: §A.3.3 · p. 18
• FasterCache performs similarly with empty and non-empty negative prompts, consistent with CFG-Cache biases being little affected by negative-prompt changes.Source: §A.3.3 · p. 18, Figure 15 · p. 18

A.4 ADDITIONAL QUALITATIVE EXPERIMENTS

• FasterCache shows reliable fidelity while accelerating text-to-video generation across Open-Sora 1.2, Open-Sora-Plan, Latte, CogVideoX-2B, and Vchitect-2.0.Source: §A.4 · p. 18, Figure 17 · p. 20, Figure 18 · p. 21
• FasterCache achieves 1.63 times speedup (206s →126s) on CogVideoX-5B and 1.74 times (320s →184s) on Mochi-10B while maintaining video fidelity.Source: §A.4 · p. 19, Figure 21 · p. 24
• On DynamiCrafter image-to-video generation, FasterCache achieves a 1.52× speedup while demonstrating good fidelity.Source: §A.4 · p. 19, Figure 22 · p. 25

A.5.1 USER PREFERENCE STUDY

• In a preference study using 30 videos per model and five raters per sample, raters strongly prefer FasterCache videos over those from other acceleration methods.Source: §A.5.1 · p. 19, Table 6 · p. 19

A.5.2 HYPERPARAMETER SELECTION

• As the Dynamic Feature Reuse caching interval increases, synthesized-video fidelity gradually decreases, and the interval can be adjusted as needed.Source: §A.5.2 · p. 19, Table 7 · p. 19
• CFG-Cache intervals exceeding 5 timesteps significantly reduce fidelity, so the method uses an interval of 5; α1 = 0.2 and α2 = 0.2 also work effectively.Source: Figure 16 · p. 18, §A.5.2 · p. 19, Table 8 · p. 19