Source-linked AI summary
Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model
Feng Liu, Shiwei Zhang, Xiaofeng Wang, Yujie Wei, Haonan Qiu, Yuzhong Zhao, Yingya Zhang, Qixiang Ye, Fang Wan
TL;DR
Diffusion video models are slow because sequential denoising and uniform caching fail to account for changing output differences across timesteps. TeaCache estimates those differences from timestep-modulated noisy inputs, refines them by rescaling, and uses them for selective output caching. Across diverse generation settings, it delivers robust efficiency and visual quality, including up to 4.41x acceleration over Open-Sora-Plan with negligible quality degradation.
Problem
Sequential denoising slows diffusion-model inference, while uniform timestep caching neglects nonuniform output differences and can create redundant cached outputs.
Method
TeaCache is a training-free approach that estimates output differences from timestep-embedding-modulated noisy inputs and refines the estimates with polynomial rescaling for selective caching.
Results
TeaCache demonstrates robust efficiency and visual quality across diverse models, schedules, video lengths, and resolutions, achieving up to 4.41x acceleration over Open-Sora-Plan with negligible visual-quality degradation.
Takeaways & Limitations
Input-based difference estimates enable dynamic caching of model outputs while targeting faster inference without substantial visual-quality loss.
Takeaways & Limitations
At extreme reduction rates, reference-based metrics such as PSNR and SSIM decline, although qualitative outputs remain satisfactory.
Abstract
from arXiv · showhide
As a fundamental backbone for video generation, diffusion models are challenged by low inference speed due to the sequential nature of denoising. Previous methods speed up the models by caching and reusing model outputs at uniformly selected timesteps. However, such a strategy neglects the fact that differences among model outputs are not uniform across timesteps, which hinders selecting the appropriate model outputs to cache, leading to a poor balance between inference efficiency and visual quality. In this study, we introduce Timestep Embedding Aware Cache (TeaCache), a training-free caching approach that estimates and leverages the fluctuating differences among model outputs across timesteps. Rather than directly using the time-consuming model outputs, TeaCache focuses on model inputs, which have a strong correlation with the modeloutputs while incurring negligible computational cost. TeaCache first modulates the noisy inputs using the timestep embeddings to ensure their differences better approximating those of model outputs. TeaCache then introduces a rescaling strategy to refine the estimated differences and utilizes them to indicate output caching. Experiments show that TeaCache achieves up to 4.41x acceleration over Open-Sora-Plan with negligible (-0.07% Vbench score) degradation of visual quality.
1. Introduction
Diffusion models face slow inference because sequential denoising prevents parallel decoding, while uniform caching overlooks timestep-varying output differences. TeaCache addresses this with training-free, input-based difference estimation, timestep modulation, and rescaling for selective caching.
- Sequential denoising makes diffusion-model inference slow, especially as model size, video resolution, and duration increase.
- Uniformly caching outputs can leave redundant cached results because output differences fluctuate across timesteps and are unknown before computation.
- TeaCache is a training-free strategy that estimates output differences from readily accessible model inputs instead of computing time-consuming outputs.
- TeaCache modulates the gradually updated noisy input with timestep embeddings to better represent input differences across denoising timesteps.
- A polynomial fitting procedure corrects scaling bias between input and output differences, enabling the corrected input difference to indicate whether outputs should be cached.
- TeaCache uses a two-stage strategy with slow and fast variants and accelerates OpenSora, Open-Sora-Plan, and Latte with negligible quality cost.
2. Related Work
Related work accelerates diffusion inference through fewer sampling steps, efficient solvers, token reduction, and feature caching. Existing caching methods target U-Net or DiT features, residuals, and content-adaptive schedules.
- Diffusion transformers address U-Net scalability constraints by using transformer architectures to increase model capacity for image and video generation.
- Inference-acceleration research includes fewer sampling steps, efficient ODE or SDE solvers, and training-based methods requiring additional resources.
- Training-free approaches reduce redundant image-synthesis computation by reducing input tokens or reusing intermediate features across successive timesteps.
- DeepCache and Faster Diffusion modify U-Net diffusion through feature caching, while FORA and △-DiT cache DiT residuals between attention layers.
- PAB caches and broadcasts intermediate video features at timestep intervals, and AdaCache dynamically adjusts caching according to content complexity.
3. Methodology
Diffusion models iteratively denoise noisy inputs, with timestep embeddings modulating transformer inputs and outputs. TeaCache estimates output differences from timestep-embedding-modulated noisy inputs, refines them with polynomial fitting, and uses the estimates to reuse cached outputs at dynamically selected timesteps.
- 3.1. Preliminaries: Diffusion models progressively denoise random noise through forward and reverse processes, with timestep t controlling the noise strength.During inference, the reverse process reconstructs data from noisy representations.
- 3.1. Preliminaries: Timestep embeddings transform scalar timesteps through sinusoidal embedding and an MLP, then modulate self-attention and FFN inputs and outputs.This modulation can significantly affect model-output magnitude.
- 3.2. Analysis: Relative L1 distance measures consecutive output changes: larger differences indicate outputs to cache, while smaller differences permit reusing a cached output.This criterion is conceptually direct but requires estimating output differences before computing the current output.
- 3.2. Analysis: Across Open Sora, Latte, and OpenSora-Plan, timestep-embedding-modulated noisy inputs correlate strongly with output differences and capture model-specific timestep patterns.The patterns differ across models, including a U shape, a flipped L shape, and multiple peaks caused by repeated scheduler timesteps.
- 3.3. TeaCache: TeaCache reuses cached outputs at redundant timesteps by accumulating relative input differences against a threshold, refreshing the cache when the threshold is exceeded.A smaller threshold refreshes more often; a larger threshold accelerates generation but may adversely affect visual appearance.
- 3.3. TeaCache: TeaCache applies polynomial fitting to rescale timestep-embedding-modulated noisy-input differences before using them as output-difference estimates.The polynomial maps input difference x to estimated output difference y and corrects scaling bias.
- 3.3. TeaCache: Unlike uniformly reducing inference timesteps, caching dynamically selects timesteps with large differences and reuses outputs over following timesteps.The supplied comparison also states that TeaCache maintains visual quality while reducing timesteps suffers deterioration.
4. Experiment
TeaCache is evaluated across video-generation models, resolutions, lengths, schedulers, and competing acceleration methods using efficiency and visual-quality metrics. It consistently improves the quality–efficiency trade-off, while its indicator and rescaling ablations support the proposed design.
- Experimental Setup: The evaluation compares efficiency using FLOPs and latency, and visual quality using VBench, LPIPS, PSNR, and SSIM.Experiments use NVIDIA A800 80GB GPUs; polynomial fitting is calibrated with 70 T2V-CompBench prompts.
- Main Results: TeaCache consistently delivers higher efficiency and better visual quality than other training-free methods across base models, schedulers, resolutions, and video lengths.The comparison includes Open-Sora 1.2, Open-Sora-Plan, and Latte against methods including PAB, T-GATE, and ∆-DiT.
- Main Results: 6.83× is TeaCache’s highest speedup on Open-Sora-Plan, while 4.41× provides the highest quality, compared with PAB’s previous best 1.49× speedup.For Latte, TeaCache-slow reaches 1.86× versus PAB’s 1.34×; for OpenSora, the optimal speedup is 2.25× versus 1.40× previously.
- Main Results: TeaCache outperforms PAB in visual quality at lower latency and maintains acceleration as video resolution and frame count increase.The visual comparison uses Open-Sora, Latte, and Open-Sora-Plan specifications on a single A800 GPU.
- Quality-Efficiency Trade-off: Across latency configurations, TeaCache outperforms PAB on all quality metrics, although PSNR and SSIM decline at extreme reduction rates.Qualitative outputs remain satisfactory despite reduced agreement with the reference at these settings.
- Ablation Studies: Timestep-embedding-modulated noisy input correlates more strongly with model-output differences than timestep embedding and selects timesteps dynamically by prompt.The input indicator consistently surpasses timestep embedding across evaluated models.
- Ablation Studies: A first-order polynomial fit improves VBench by 0.24% over the original data, with gains saturating at fourth-order fitting.The fitted rescaling also improves LPIPS, SSIM, and PSNR.
5. Conclusion
TeaCache accelerates video synthesis by using timestep-embedding-modulated noisy-input similarity to guide dynamic output caching and polynomial rescaling to refine that estimate. The approach is reported as robust across models, schedules, video sizes, and related generation settings, with broader scaling and quality-efficiency evidence supporting its practical scope.
- Conclusion: TeaCache is a training-free approach that accelerates video-synthesis inference while maintaining high-quality output.The method is designed to be compatible with diffusion-transformer models.
- Conclusion: TeaCache uses timestep-embedding-modulated noisy-input similarity as an indicator of output similarity for dynamic model-output caching.The indicator exploits the reported correlation between model-input and model-output similarity.
- Conclusion: Polynomial rescaling refines the estimated output similarity and improves timestep selection for caching.The rescaling addresses bias between input-difference and output-difference scales.
- Conclusion: TeaCache shows robust efficiency and visual quality across video and image generation models, sampling schedules, video lengths, and resolutions.It also improves inference speed when scaled across multiple GPUs and sustains acceleration for longer and higher-resolution videos.