PaperlayerView exact revision on arXiv

Source-linked AI summary

From Reusing to Forecasting: Accelerating Diffusion Models with TaylorSeers

Jiacheng Liu, Chang Zou, Yuanhuiyi Lyu, Junjie Chen, Linfeng Zhang

arXiv:2503.06923v2cs.CVcs.AI

TL;DR

Diffusion-model feature caching loses effectiveness across distant timesteps because feature similarity declines and reuse errors increase. TaylorSeer forecasts future features with higher-order finite-difference derivatives and Taylor expansion, achieving high-ratio acceleration across image and video models while preserving generation quality. Its approach assumes features evolve smoothly with bounded higher-order derivatives, and direct reuse otherwise accumulates error as acceleration increases.

  • Problem

    Existing cache-then-reuse methods degrade as timestep gaps grow because feature similarity decreases, limiting high-ratio acceleration.

  • Method

    TaylorSeer uses multi-step features and finite differences to approximate higher-order derivatives, then applies Taylor expansion to forecast future features without training.

  • Results

    TaylorSeer achieves 2.5×, 4.99×, and 5.00× acceleration on DiT, FLUX, and HunyuanVideo, respectively, while maintaining high-quality generation.

  • Takeaways & Limitations

    TaylorSeer supports training-free, high-ratio diffusion acceleration, with particular effectiveness at larger intervals between full activations.

  • Takeaways & Limitations

    The approach assumes diffusion-model features evolve smoothly as a differentiable function with bounded higher-order derivatives.

Abstract

from arXiv · show

Diffusion Transformers (DiT) have revolutionized high-fidelity image and video synthesis, yet their computational demands remain prohibitive for real-time applications. To solve this problem, feature caching has been proposed to accelerate diffusion models by caching the features in the previous timesteps and then reusing them in the following timesteps. However, at timesteps with significant intervals, the feature similarity in diffusion models decreases substantially, leading to a pronounced increase in errors introduced by feature caching, significantly harming the generation quality. To solve this problem, we propose TaylorSeer, which firstly shows that features of diffusion models at future timesteps can be predicted based on their values at previous timesteps. Based on the fact that features change slowly and continuously across timesteps, TaylorSeer employs a differential method to approximate the higher-order derivatives of features and predict features in future timesteps with Taylor series expansion. Extensive experiments demonstrate its significant effectiveness in both image and video synthesis, especially in high acceleration ratios. For instance, it achieves an almost lossless acceleration of 4.99$\times$ on FLUX and 5.00$\times$ on HunyuanVideo without additional training. On DiT, it achieves $3.41$ lower FID compared with previous SOTA at $4.53$$\times$ acceleration. %Our code is provided in the supplementary materials and will be made publicly available on GitHub. Our codes have been released in Github:https://github.com/Shenyi-Z/TaylorSeer

1. Introduction

Diffusion Transformers deliver high-quality visual generation but impose substantial computational demands, motivating training-free feature caching. TaylorSeer replaces direct reuse with forecasting from stable feature trajectories, improving quality at high acceleration ratios.

  • Diffusion Transformers improve visual-generation quality but substantially increase computational demands, limiting practical use.
  • Feature Caching-based Acceleration: Existing feature-caching methods store features from previous timesteps and reuse them later to skip substantial computations without additional training.These methods explore caching from token and residual perspectives.
  • TaylorSeer: TaylorSeer introduces cache-then-forecast, using multi-step features and finite differences to estimate higher-order derivatives for Taylor-series prediction of future features.The method exploits continuity in feature changes rather than directly reusing cached features.
  • Results: TaylorSeer reduces quality loss versus the previous SOTA by 36× and performs promisingly beyond 6× acceleration, where previous methods fail.
  • Results: TaylorSeer achieves 2.5×, 4.99×, and 5.00× acceleration on DiT, FLUX, and HunyuanVideo, respectively, while maintaining high-quality generation.The reported results cover image and video synthesis without training or search costs.
  • Feature Caching-based Acceleration: As timestep gaps increase, feature similarity decreases, causing quality degradation that limits acceleration gains for cache-then-reuse methods.This motivates treating feature caching as a sequential prediction problem.

3. Method

TaylorSeer accelerates diffusion models by forecasting future features rather than directly reusing cached features. It exploits smooth feature trajectories, approximating derivatives with finite differences and applying Taylor expansion to balance accuracy and computational savings.

  • Motivation: Diffusion features evolve smoothly over timesteps, supporting predictable variation despite numerical discretization.The method assumes differentiable feature transformations with bounded higher-order derivatives.
  • Limitations of Feature Reuse: Naïve caching reuses features from earlier timesteps, but larger timestep gaps cause exponential error accumulation by ignoring temporal dynamics.This limits quality as the caching interval increases.
  • TaylorSeer: TaylorSeer caches feature values and finite differences, then forecasts intermediate features using Taylor series expansion.Higher-order finite differences approximate derivatives without explicitly computing them.
  • TaylorSeer: The method requires only (m + 1) fully computed timesteps to predict features at intermediate timesteps.This design balances efficiency and prediction accuracy across the selected caching interval.
  • Forecasting Orders: Forecasting order controls temporal modeling: m = 0 performs direct caching, m = 1 captures linear trends, and m ≥2 models nonlinear long-range trajectories.Higher-order forecasting reduces errors while introducing sampling-interval-dependent approximation terms.

4. Experiments

Experiments evaluate TaylorSeer across text-to-image, text-to-video, and class-conditional image generation, including quantitative, qualitative, and ablation studies. Results show strong quality preservation at high acceleration, with higher-order Taylor expansions improving long-interval forecasting.

  • Text-to-Image Generation: 4.99× acceleration with 1.0039 ImageReward outperforms competing methods near 5× speedup, whose ImageReward declines to 0.9470 or 0.9086.The comparison uses TaylorSeer (N=6, O=2), DuCa (N=6), and ToCa (N=8).
  • Qualitative Comparisons: TaylorSeer preserves text, visual consistency, motion blur, and trajectory smoothness better than competing methods in qualitative image and video comparisons.Examples include correctly rendering “START,” “STOP,” a suitcase, and motion details during a motorcycle turn.
  • Text-to-Video Generation: 5.00× speedup with a 79.93% VBench score outperforms ToCa and DuCa on HunyuanVideo.At N=6 and O=1, TaylorSeer reaches 5.56× speedup while maintaining a 79.78% VBench score.
  • Class-Conditional Image Generation: At 4.53× acceleration, TaylorSeer maintains an FID of 2.65 on DiT-XL/2, while other methods degrade substantially beyond 3.5× acceleration.TaylorSeer (N=5, O=3) is compared with ToCa, DuCa, FORA, and reduced DDIM steps.

5. Conclusion

The paper reframes feature caching from directly reusing past features to forecasting future features with Taylor expansion, targeting quality preservation at high acceleration ratios.

  • 5. Conclusion: Traditional feature caching directly reuses features from previous timesteps and suffers severe generation-quality drops at high acceleration ratios.This limitation motivates the forecasting formulation.
  • 5. Conclusion: TaylorSeer replaces “cache-then-reuse” with “cache-then-forecast,” formulating feature caching as a sequential prediction problem solved with Taylor expansion.The approach is motivated by stable feature trajectories across timesteps.
  • 5. Conclusion: High-order Taylor expansions capture complex feature trajectories more rigorously than direct feature reuse.This design is intended to preserve generation quality at high acceleration ratios.
  • 5. Conclusion: 4.53×–5.56× acceleration was demonstrated across DiT, FLUX, and HunyuanVideo without quality degradation.The experiments span class-conditional image, text-to-image, and text-to-video generation architectures and tasks.

6. Experimental Details

The experimental details cover three models and tasks, then specify baseline-specific caching configurations and layer-skipping schedules.

  • 6. Experimental Details: Experiments use FLUX for text-to-image generation, HunyuanVideo for text-to-video generation, and DiT for class-conditional image generation.The models come from different architectures and generation tasks.
  • 6. Experimental Details: FORA uses a uniform activation interval with N=3 on FLUX, while ToCa uses N=4, 90% caching, non-uniform activation, and attention-based token selection.These settings describe baseline configurations for comparison.
  • 6. Experimental Details: DuCa applies conservative caching after even-numbered fresh steps and aggressive caching after odd-numbered steps.The supplied passage presents this as the FLUX baseline configuration.
  • 6. Experimental Details: ∆-DiT skips layers 14–27 during steps 49–25 and layers 0–13 during steps 24–0.The skipped layer ranges change between early and later generation stages.

7. Supplementary Results for Ablation Studies

The supplementary ablation material examines TaylorSeer’s interval parameter and Taylor expansion order on ImageNet with DiT-XL/2, emphasizing their speed–quality trade-off.

  • 7. Supplementary Results for Ablation Studies: The ablation study evaluates TaylorSeer on ImageNet with DiT-XL/2 using different configurations.The referenced table is titled “Ablation Study with Different Configurations on ImageNet with DiT-XL/2.”
  • 7. Supplementary Results for Ablation Studies: The study varies the interval parameter N and Taylor expansion order O to assess computational efficiency and generation quality.These are the stated design choices examined in the ablation.
  • 7. Supplementary Results for Ablation Studies: The ablation results indicate that N and O matter for balancing performance and speed.The passage states that these design choices are important to the balance between generation quality and computational efficiency.

8. Anonymous Page for Video Presentation

The paper provides an anonymous GitHub page and website for a detailed video-generation demonstration, with additional videos in the supplementary material.

  • 8. Anonymous Page for Video Presentation: An anonymous GitHub page was created to showcase TaylorSeer’s advantages in video generation.The page is presented as a demonstration resource.
  • 8. Anonymous Page for Video Presentation: The detailed demonstration is available at https://taylorseer.github.io/TaylorSeer/.Videos are also available in the Supplementary Material.

9. Supplementary Visualization Examples

Visualizations on FLUX and HunyuanVideo show TaylorSeer generating high-quality outputs with superior fidelity and consistency across diverse scenarios.

  • TaylorSeer produces high-quality image and video outputs on FLUX and HunyuanVideo.The visualizations are presented as qualitative evidence for the method's generation quality.
  • The examples highlight superior fidelity and consistency across diverse generation scenarios.
  • The supplementary visualizations provide qualitative comparisons of TaylorSeer on both models.

10. Supplementary Visualization of Feature Trajectories in Diffusion Models

PCA visualizations show stable feature trajectories and consistent derivative patterns across timesteps, supporting the predictability of diffusion-model feature dynamics and future feature evolution.

  • PCA projections from zeroth- through fourth-order features and derivatives display consistent patterns across timesteps.Together, these visualizations illustrate the stability and predictability of feature dynamics.
  • PCA projections show that diffusion-model features form stable trajectories across different timesteps.This supports the predictability of feature evolution over time.
  • First-order feature derivatives exhibit consistent patterns across timesteps.The consistency further supports predictable feature dynamics.
  • Second-order derivatives reveal higher-order dynamics and smooth feature transitions.
  • Third-order derivatives capture more complex temporal patterns in feature dynamics.
  • Fourth-order derivatives expose fine-grained temporal variations and further validate predictable feature evolution.

11. More Results for TaylorSeer

Additional experiments show that TaylorSeer generalizes across multiple video DiT models and tasks while maintaining strong quality at substantial acceleration ratios. The method can also surpass baseline scores, with the authors attributing this enhancement to parameter redundancy and a possible regularization-like inference effect.

  • TaylorSeer consistently achieves superior performance across HiDream, FramePack, and WAN2.1 text-to-video DiT models.
  • The FLUX and HunyuanVideo visualizations provide qualitative evidence for the method's advantages.
  • 4.0 times acceleration lets TaylorSeer outperform TeaCache on HiDream across PSNR, SSIM, LPIPS, and ImageReward.
  • TaylorSeer extends beyond core image and video acceleration to super-resolution and audio generation tasks.
  • TaylorSeer improves FLUX ImageReward from 0.9898 to 1.0181 and HunyuanVideo VBench from 80.66 to 80.74.These results are reported as improvements over the corresponding baseline performance.
  • The authors attribute performance gains to parameter redundancy and hypothesize a regularization-like effect during inference.They relate this effect to the Taylor approximation simplification used at each inference step.
Loading 2503.06923v2…