Source-linked AI summary
Memory-Efficient Training-Free Acceleration of Diffusion Transformers with BaryCache
Chengjie Lu, Tianchi Deng, Zhengqi He, Zhijian Gao, Huisi Wu, Xueliang Li
TL;DR
DiT sampling is computationally expensive, while cache-based acceleration can increase VRAM usage and constrain batch size. BaryCache uses training-free stepwise forecasting with a stabilized Barycentric Extrapolator, achieving a quality–memory trade-off across image and video generation and up to 3.30× speedup over baseline DiT inference.
Problem
DiT sampling requires repeated large matrix operations, while existing caching methods can increase VRAM usage by storing intermediate states and constrain inference batch size.
Method
BaryCache performs training-free stepwise feature forecasting with a Barycentric Extrapolator whose schedule-dependent weights can be constructed separately from feature aggregation.
Results
Across C2I, T2I, and T2V generation, BaryCache provides a favorable memory–quality trade-off and achieves up to 3.30× end-to-end sampling speedup over baseline DiT inference.
Takeaways & Limitations
BaryCache offers a practical, memory-efficient approach for accelerating diffusion-model inference while maintaining high perceptual quality across image and video tasks.
Takeaways & Limitations
BaryCache can occasionally produce unexpected patterns, and its empirically selected decay factors have not been systematically studied across models and tasks.
Abstract
from arXiv · showhide
Diffusion Transformers achieve high-fidelity image and video generation, but their iterative sampling remains expensive, for each denoising step requires large matrix operations. Existing cache-based acceleration reduces redundant computation yet increases the VRAM footprint by storing intermediate states, which can directly constrain inference batch size. In this work, we propose a training-free acceleration method that performs stepwise forecasting for DiT sampling using a Barycentric Extrapolator. By leveraging barycentric extrapolation, our predictor is numerically stable and alleviates oscillatory artifacts analogous to the Runge phenomenon during forward forecasting. Across extensive experiments on both image and video generation, our approach provides a favorable trade-off between memory usage and perceptual quality, while delivering up to 3.30x end-to-end sampling speedup compared with baseline DiT inference.
1. Introduction
DiT inference is expensive, and existing acceleration methods face trade-offs among computation, memory, and quality. BaryCache addresses this trade-off through training-free stepwise feature forecasting with a stabilized Barycentric Extrapolator.
- Scaling DiTs to higher resolutions and capacities increases the tension between generation quality and computational efficiency.
- Existing acceleration strategies reduce memory, training, or sampling costs, but several require retraining, finetuning, or face high-dimensional memory challenges.
- Training-free caching reuses intermediate activations across diffusion steps, but quality can degrade substantially at high acceleration ratios.
- BaryCache predicts future stepwise DiT features with a training-free Barycentric Extrapolator to balance memory usage and generation quality.
- Barycentric Extrapolator uses stabilized weight construction and post-processing to mitigate Runge-oscillation-like instabilities during stepwise extrapolation.
- 3.30× speedup is achieved across C2I, T2I, and T2V generation while maintaining high perceptual quality.
2. Related Works
Diffusion acceleration includes step reduction and activation caching, with caching methods targeting redundancy across steps or within steps. BaryCache pursues stepwise prediction to retain fidelity while keeping inference-time memory manageable.
- Sampling-step reduction methods use deterministic trajectories, higher-order solvers, flow-based models, distillation, or consistency modeling to accelerate diffusion inference.
- Barycentric Extrapolator predicts stepwise features from recently computed outputs, providing the core predictor used by BaryCache.
- BaryCache belongs to stepwise caching methods that exploit temporal redundancy across neighboring diffusion steps.
- Blockwise caching reuses features within a diffusion step across layers, modules, or tokens, including attention, MLP, and token-level activations.
- BaryCache implements stepwise prediction while aiming to retain high fidelity and keep inference-time memory manageable.
3. Method
BaryCache forecasts stepwise DiT features from a dynamic cache using barycentric extrapolation, with stabilized weights and schedule-dependent reuse to reduce memory-intensive computation.
- Stepwise feature prediction: Stepwise features evolve smoothly across diffusion steps, motivating their prediction from prior evaluations.The method treats layer and module outputs as incremental contributions whose trajectories are predictable.
- Barycentric Extrapolator: The Barycentric Extrapolator constructs predictions from cached features using timestep schedules and a precomputable Diff Key.Weights depend on relative timestep differences and decay rather than on the feature values themselves, enabling weight computation to be decoupled from feature aggregation.
- Weight construction: Trapezoidal-like barycentric weights provide a stable, pole-free interpolation choice for the history buffer.The raw coefficients are related to Floater-Hormann rational interpolation with degree d = 0 or d = 1.
- Stabilization: Weight clipping, decay, and blending with the newest cached feature stabilize extrapolation and control prediction consistency.The suggested clipping constant is C = 10; decay is empirically tied to history length, and λ controls the blend with the latest cached feature.
- BaryCache: BaryCache predicts future stepwise DiT features from cached feature maps instead of recomputing every diffusion step.The cache stores a history of previous feature tensors and supports either full computation or cache-forward prediction.
- Preliminary evaluation: In preliminary evaluation, stepwise prediction outperforms blockwise variants, while BaryCache surpasses TaylorSeer at later steps despite higher error at step 2.The comparison indicates reduced Runge-phenomenon-like instability and lower memory usage for BaryCache.
4. Experiment
Experiments across C2I, T2I, and T2V show that BaryCache balances memory usage, generation quality, and sampling speed. Ablations indicate that stepwise caching, regular schedules, and tuned coefficients support this trade-off.
- Experimental Setup: BaryCache is evaluated on DiT-XL/2, PixArt-Σ, and HunyuanVideo across C2I, T2I, and T2V settings.The evaluation uses 50 sampling steps with DDIM for C2I/T2I and Euler for T2V.
- Quantitative Results: Under N = 6 on T2I, BaryCache improves ImageReward from 0.5001 to 0.6654 and CLIPScore from 31.04 to 31.10 versus Δ-DiT.It remains only 0.5× slower than the fastest caching method under the same setting.
- Quantitative Results: On T2V at skip ratio seven, BaryCache achieves PSNR 17.9988, SSIM 0.6496, and LPIPS 0.3463 with low VRAM usage.On C2I, it reduces inference memory by nearly 3GB, although TaylorSeer achieves stronger FID/IS.
- Qualitative Results: BaryCache preserves semantic structure, spatial details, motion dynamics, and sharper structure better than competing methods in qualitative image and video comparisons.The comparisons cover PixArt-Σ and HunyuanVideo under aggressive acceleration.
- Ablation Studies: A cache history of H = 2 achieves the best ImageReward, making the most efficient tested configuration also the most effective perceptually.The ablation adjusts decay ratios for H = 2, 3, and 4 to maintain numerical stability.
- Ablation Studies: Equal-spaced schedules consistently outperform adaptive schedules under matched full-computation budgets, while Br = 1.0 performs best among tested blockwise settings.Increasing λ enables more aggressive forecasting, whereas δ controls fine details; both can be tuned to balance fidelity and efficiency.
5. Discussion
The authors report occasional unexpected patterns and identify local object-information loss during skipped forward passes as a possible source. Decay-factor sensitivity across models and tasks remains insufficiently studied.
- Limitations: BaryCache can occasionally produce unexpected image or video patterns that the current extrapolator cannot fully correct.The authors associate these artifacts with possible local object-information loss during skipped forward passes.
- Limitations: Decay factors are empirically selected, and their sensitivity across different models and tasks has not been systematically studied.The paper identifies more adaptive decay-factor strategies as future work.
6. Conclusion
BaryCache accelerates DiT inference through stepwise feature extrapolation, balancing generation quality with manageable memory usage. Experiments on image and video tasks show strong performance against existing acceleration methods.
- BaryCache predicts stepwise DiT activations with high fidelity while maintaining manageable memory usage.
- Extensive image and video generation experiments show a strong balance between quality and memory usage.
- BaryCache outperforms existing caching and step-reduction methods on most evaluated tasks.
- Future work may investigate more adaptive extrapolation techniques to further improve generation quality.