PaperlayerView exact revision on arXiv

Source-linked AI summary

Heuristics are Not Enough: Calibrated and Corrected Caching for Video Diffusion Acceleration

Jiyue Jiang, Zhiyu Huo, Yuhan Chen, Yanruisheng Shao, Gong Zhang, Lei Deng

arXiv:2610.16545v1cs.CVcs.AI

TL;DR

Adaptive feature caching for video diffusion lacks criteria aligned with generation quality, phase-specific thresholds, and validation against actual approximation error. AlignCache combines phase-adaptive routing, guided-output counterfactual calibration, and momentum-augmented drafts; experiments report the best speed-quality Pareto frontier across three video models.

  • Problem

    Adaptive caching methods use heuristic criteria misaligned with generation quality, a global threshold insensitive to phase dynamics, and budget estimates not validated against actual approximation error.

    Source: §1 · p. 1

  • Method

    AlignCache combines phase-specific routing with staleness guards, online calibration from counterfactual CFG-guided errors, and momentum-augmented drafts with anchor-guided unconditional correction.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    Across Wan2.1-1.3B, Open-Sora-1.2, and HunyuanVideo, AlignCache achieves the best speed-quality Pareto frontier, with higher speedup and better fidelity than prior methods.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The results support AlignCache as a feedback-driven, self-correcting scheduler that consistently improves acceleration-quality trade-offs across the evaluated video generation models.

    Source: §1 · p. 2, §6 · p. 9

Abstract

from arXiv · show

Feature caching has become a mainstream training-free strategy for accelerating video diffusion models by reusing intermediate outputs across denoising steps. However, existing adaptive methods rely on heuristic difference measures misaligned with actual generation quality, employ a single global threshold that ignores heterogeneous phase dynamics, and never validate their budget estimates against true approximation error, allowing systematic bias to propagate uncorrected. We propose \textbf{AlignCache}, a closed-loop caching framework that directly aligns scheduling decisions with their true influence on the final generated video through three tightly coupled components: (1) phase-adaptive budget routing that partitions the denoising trajectory into structurally distinct phases, each governed by an independent threshold and a staleness-aware guard; (2) closed-loop threshold calibration that exploits counterfactual error measurement at the classifier-free guided output level to adjust decision boundaries online via an EMA feedback loop; and (3) momentum-augmented draft synthesis combining direction-magnitude gated residual extrapolation with anchor-guided unconditional correction. By transforming scheduling from a static risk accumulator into a feedback-driven, self-correcting controller, AlignCache consistently achieves superior acceleration-quality trade-offs across state-of-the-art video generation models. Our code is available at https://github.com/jiangjyjy/AlignCache.

1 Introduction

Video diffusion inference is costly because iterative denoising requires many sequential network passes, while existing adaptive caching criteria do not reliably reflect generation quality or phase-specific dynamics. AlignCache uses counterfactual feedback, phase-aware routing, and improved skip drafts to pursue better acceleration-quality trade-offs.

1 Introduction

  • Iterative denoising requires dozens of sequential forward passes through billion-parameter networks, making efficient video diffusion inference a deployment bottleneck.The paper situates this cost in state-of-the-art Diffusion Transformer video systems.Source: §1 · p. 1
  • Existing adaptive cache schedulers use heuristic differences misaligned with generated-video quality, one global threshold, and budgets unchecked against actual approximation error.The unchecked estimates can let systematic bias propagate during sampling.Source: §1 · pp. 1–2
  • AlignCache combines phase-specific thresholds and staleness guards, classifier-free-guided counterfactual error calibration, and momentum-augmented drafts with anchor-guided unconditional correction.The calibration adjusts decision boundaries online through an exponential moving average feedback loop.Source: §1 · p. 2
  • Experiments across diverse video generation models report superior acceleration-quality trade-offs for AlignCache.The introduction also reports that its scheduling preserves fine visual details that competing methods tend to distort.Source: §1 · p. 2

2 Related Works

Video generation has shifted toward scalable Diffusion Transformers, but iterative denoising remains computationally costly. Feature-caching approaches exploit temporal redundancy, yet adaptive methods still face a gap between their scheduling criteria and generation quality.

2.1 Video Diffusion Models

  • DiT-based systems including Sora, Open-Sora, CogVideoX, HunyuanVideo, and Wan2.1 have advanced video fidelity and coherence, while iterative denoising remains costly.Earlier video generation used U-Net and latent diffusion frameworks; recent DiT systems benefit from Rectified Flow formulations that use fewer sampling steps.Source: §2.1 · p. 2

2.2 Feature Caching for Diffusion Acceleration

  • Feature caching exploits temporal redundancy, but fixed-interval methods ignore timestep importance and adaptive methods use heuristic metrics that can trade limited speedup for quality loss or vice versa.Frequency decomposition and higher-order forecasting address aspects of the misalignment, while the gap to perceptual quality remains under-explored.Source: §2.2 · p. 3

3 Preliminaries

Cache-based acceleration routes each denoising step to either exact computation or reuse, balancing quality against efficiency. The section motivates AlignCache by identifying phase-varying residual behavior and limitations in open-loop budget scheduling.

3 Preliminaries

  • Residual magnitudes differ across early, middle, and late denoising phases, while direct-reuse error varies across conditional, unconditional, and CFG-guided branches.Properly designed correction reduces relative error by 30–50% over direct reuse, with the largest gap where direct reuse is most error-prone.Source: §3 · p. 3
  • Cache-based methods choose between a full forward pass and cached approximation at each timestep, making the routing policy the determinant of the quality-efficiency trade-off.The existing budget rule tests accumulated risk against a fixed threshold.Source: §3 · p. 4
  • Open-loop schedulers accumulate heuristic risk and skip below a fixed threshold, but a global threshold misses phase dynamics, no staleness limit bounds skip runs, and unchecked budgets can propagate bias.AlignCache addresses these limitations with phase-adaptive routing, staleness guards, and closed-loop calibration.Source: §3 · p. 4

4 Methodology: AlignCache

AlignCache replaces a fixed open-loop routing boundary with a phase-aware, self-correcting threshold. It combines phase-adaptive routing, guided-output counterfactual calibration, and momentum-augmented skip drafts.

4 Methodology: AlignCache

  • AlignCache couples phase-adaptive routing, counterfactual threshold calibration, and momentum-augmented draft synthesis to improve skip-mode fidelity.Its dynamic threshold is phase-aware and self-correcting rather than a fixed boundary.Source: §4 · p. 4

4.1 Phase-Adaptive Budget Routing

  • The scheduler partitions denoising into early, mid, and late phases with independent base thresholds, accumulates motion-amplified risk, and forces exact computation when a phase-specific staleness cap is exceeded.Warm-up and tail timesteps are also computed exactly; an online calibration scale modifies each phase's base threshold.Source: §4.1 · p. 5

4.2 Closed-Loop Threshold Calibration

  • At budget-triggered exact steps, AlignCache compares counterfactual skip drafts with exact outputs at the CFG-guided level, measuring distortion that determines generation quality.A per-phase EMA calibration ratio tightens the threshold when estimated error is too low and relaxes it when error is overestimated; feedback excludes warm-up, bootstrap, and staleness-triggered exact steps.Source: §4.2 · p. 5

4.3 Momentum-Augmented Draft Synthesis

  • Draft synthesis extrapolates residuals from recent exact anchors with gated momentum to reduce drift, then uses anchor-guided conditional-unconditional coupling to refine unconditional drafts.The direction gate suppresses extrapolation when residual directions diverge, while the magnitude gate attenuates it during rapid norm changes.Source: §4.3 · pp. 5–6

5 Experiments and Results

Across three video generators, AlignCache achieves the highest speedup and best fidelity, while ablations examine scheduler and predictor choices and case studies compare visual details under matched or higher speedups.

5.1 Setup

  • Evaluation compares PAB, TeaCache, FasterCache, and EasyCache across Wan2.1-1.3B, Open-Sora-1.2, and HunyuanVideo using latency, speedup, fidelity metrics, and VBench Total.Source: §5.1 · pp. 6–7

5.2 Main Results

  • Across all three backbones, AlignCache attains the highest speedup and best fidelity; Wan2.1-1.3B reaches 2.29× with 29.80 dB PSNR and 0.038 LPIPS.Against EasyCache, the Wan2.1-1.3B result improves PSNR by +1.42 dB and LPIPS by −25% while running at 2.29× versus 2.19×.Source: §5.2 · p. 7
  • AlignCache achieves a 2.15× speedup; on Open-Sora-1.2, it delivers the largest speedup among methods (1.86×) while leading on PSNR, SSIM, and LPIPS.Source: §5.2 · p. 7

5.3 Ablation Studies

  • More warm-up raises fidelity while reducing speedup, and each 0.0025 θp increment buys roughly +0.05× to +0.10× speedup with graceful fidelity degradation.Increasing warm-up from W=5 to W=14 raises PSNR by +2.66 dB and reduces LPIPS by 37%, while speedup falls from 2.29× to 1.97×.Source: §5.3 · p. 8
  • Across an 11-point phase-ratio grid, PSNR varies by less than 0.55 dB and SSIM by less than 0.015, as online calibration absorbs small boundary misspecifications.Source: §5.3 · p. 8
  • Raising β(p) from 0.50 to 0.85 lifts PSNR by +11.1 dB and reduces LPIPS by 83%, while β(p)=0.90 collapses speedup to 1.00×.Source: §5.3 · p. 8
  • Disabling residual extrapolation costs 0.10 dB PSNR and 16% LPIPS for only +0.09× extra speedup; second-order momentum captures residual curvature during extended skips.Source: §5.3 · p. 8

5.4 Case Studies

  • In representative shark and shoe videos, competing caches distort silhouettes, geometry, textures, and edges, while AlignCache preserves details at comparable or higher speedups.The comparisons cover Wan2.1-1.3B and Open-Sora-1.2.Source: §5.4 · p. 9, Figure 4 · p. 9

6 Conclusion and Future Works

AlignCache reports the best speed-quality Pareto frontier across three video-generation models. The authors identify future directions in transferring its calibration principle, reducing prompt warm-up, combining accelerators, and extending to other generation settings.

Conclusion

  • AlignCache achieves the best speed-quality Pareto frontier on Wan2.1-1.3B, Open-Sora-1.2, and HunyuanVideo, with higher speedup and better fidelity than prior cache-based accelerators.Source: §6 · p. 9

Future Works

  • Counterfactual error measurement could also calibrate training-free accelerators such as token pruning or sparse attention.Source: §6 · p. 9
  • A prompt-conditioned prior over ¯rp,t could reduce warm-up overhead and increase speedups on short trajectories.Source: §6 · p. 9
  • Combining AlignCache with quantization, distillation, or compiler-level kernel fusion could yield compounded speedups while the controller absorbs added approximation error and preserves stable quality across heterogeneous hardware.Source: §6 · p. 10
  • Extending AlignCache to long-form video and multimodal diffusion is proposed as a next step because their redundancy patterns differ from text-to-video.Source: §6 · p. 10

A.1 Implementation Details

The implementation selects hyperparameters by sweeping five prompts with two seeds per prompt, then uses model-specific thresholds and phase boundaries. Evaluation uses different sample counts for principal metrics and ablations.

Hyperparameter tuning

  • Hyperparameters were tuned by sweeping 5 prompts with 2 seeds per prompt to seek a better quality-speed trade-off.Source: §A.1 · p. 12

Final hyperparameters

  • Wan2.1-1.3B uses base threshold 0.065, 12 warm-up exact CFG pairs, and phase thresholds 0.07475, 0.065, and 0.052; Open-Sora-1.2 uses 0.11, 8 pairs, and 0.1045, 0.1265, and 0.11.Source: §A.1 · p. 12
  • HunyuanVideo uses base threshold 0.024, 10 warm-up pairs, and phase thresholds 0.0252, 0.0288, and 0.0324.Source: §A.1 · p. 12

Evaluation setup

  • Phase boundaries are 15% and 86% for Wan2.1-1.3B and Open-Sora-1.2, and 18% and 72% for HunyuanVideo, unless otherwise specified.Source: §A.1 · p. 12
  • Principal VBench evaluations use 5 or 10 samples per prompt on NVIDIA RTX PRO 6000-96GB GPUs, while ablations use 2 videos per prompt with seeds 42 and 954 on Wan2.1-1.3B.Source: §A.1 · p. 12

A.2 Detailed Open-Loop Budget Scheduling

The open-loop scheduler accumulates a heuristic risk budget and skips a timestep when it remains below a fixed global threshold. Exact computations reset the budget, but the design lacks phase-specific thresholds, staleness limits, and error validation.

Open-loop scheduling

  • The scheduler accumulates a heuristic risk increment into Bt and skips a timestep whenever Bt is below the fixed global threshold τ.Source: §A.2 · p. 12

Risk budget update

  • The risk increment uses gain factor ga(t), estimated from the two most recent anchors, and the budget resets after each exact computation.Source: §A.2 · p. 12

Limitations

  • The design has a single global threshold, no staleness constraint, and no validation of its budget against approximation error.Source: §A.2 · p. 12

A.3 Detailed Phase-Adaptive Budget Routing

The routing module partitions denoising progress into phases and accumulates a risk budget from normalized input variation and adaptive motion gain. Its phase-specific threshold is adjusted by an online scale factor.

Phase Partition

  • The routing module partitions the denoising trajectory by normalized progress ρt into early, mid, and late phases, each with an independent base threshold θp.Source: §A.3 · p. 12

Cumulative Risk Budget Update

  • At timestep t, input variation δt is the mean absolute change between xt and xt−1, and the budget increment is based on that variation.Source: §A.3 · p. 13
  • The budget adds a normalized input perturbation amplified by motion gain ga(t), then resets to zero after an exact computation.Source: §A.3 · p. 13

Motion Gain Update

  • Motion gain ga(t) is updated after exact computations from output variation relative to input variation across the two most recent exact anchors, adapting sensitivity to local trajectory dynamics.Source: §A.3 · p. 13

Dynamic Threshold

  • The dynamic routing threshold combines phase threshold θp with online scale factor sp,t maintained by closed-loop calibration.Source: §A.3 · p. 13

A.4 Detailed Closed-Loop Threshold Calibration

AlignCache calibrates thresholds using per-phase feedback that compares observed error with the budget. A clipped exponential moving average smooths this ratio, and its inverse relationship to the threshold scale adjusts caching decisions.

A.4 Detailed Closed-Loop Threshold Calibration

  • The calibration ratio compares observed error with the budget, and a per-phase clipped exponential moving average tracks it.The ratio is defined as Eobs_t/Bt.Source: §A.4 · p. 13
  • The EMA decay factor λ controls estimate smoothness, while clipping bounds prevent outlier observations from destabilizing the controller.Source: §A.4 · p. 13
  • When the smoothed ratio exceeds 1, the threshold scale decreases and triggers more exact computations; below 1, the threshold relaxes to permit more skips.Source: §A.4 · p. 13

A.5 Detailed Momentum-Augmented Draft Synthesis

The supplied leads situate AlignCache’s momentum-augmented draft synthesis within the appendix and point to Sections 4 and A.1 for the method description. They also mention related case studies, broader impacts, and limitations.

A.5 Detailed Momentum-Augmented Draft Synthesis

  • Momentum gating suppresses extrapolation when consecutive anchor residuals diverge in direction or change rapidly in magnitude, and decays toward zero when either condition fails.Negative cosine similarity is clamped to zero; aligned residuals of similar magnitude receive the full momentum correction, while the fallback is the first-order scaled estimate.Source: §A.5 · pp. 13–14
  • The unconditional draft blends the extrapolation output with an anchor-transferred estimate, using conditional–unconditional coupling in CFG to improve guided approximations.The passage says this applies during both skip execution and counterfactual error measurement.Source: §A.5 · p. 14

A.7 Case Studies

  • At 2.38× speedup, AlignCache preserves facial geometry, lip contour, and skin texture on Wan2.1-1.3B, while competing methods show distortions or smoothed expression cues.The comparison uses the prompt “A person is kissing.”Source: §A.7 · p. 14, Figure 5 · p. 15
  • At the highest compared speedup of 1.64×, AlignCache recovers woven twig structure and fine feather details on Open-Sora-1.2, unlike competing methods’ blurring, flattening, or color drift.The comparison uses the prompt “A bird building a nest from twigs and leaves.”Source: §A.7 · pp. 14–15, Figure 6 · p. 16

A.8 Broader Impacts

  • AlignCache lowers video-diffusion inference cost and may improve accessibility, while faster generation can lower barriers to deepfakes or disinformation and preserves base-model biases.The paper recommends deploying it with upstream content moderation, provenance watermarking, and the original generators’ release policies.Source: §A.8 · p. 15

NeurIPS Paper Checklist

  • The paper identifies prompt-independent calibration, warm-up overhead, and extension to long-form video and multimodal diffusion as limitations or open directions.The checklist also notes that the work is scoped to training-free cache-based acceleration for video diffusion models.Source: NeurIPS Paper Checklist · p. 16

10. Broader impacts

  • The authors report releasing AlignCache code through an anonymized repository with documentation and reproduction instructions, and refer to Sections 4 and A.1 for the method description.Source: §10 · p. 21

Related papers

Loading 2610.16545v1…