PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Caching for Faster Video Generation with Diffusion Transformers

Kumara Kahatapitiya, Haozhe Liu, Sen He, Ding Liu, Menglin Jia, Chenyang Zhang, Michael S. Ryoo, Tian Xie

arXiv:2411.02397v2cs.CV

TL;DR

Video DiTs can be computationally expensive, particularly for high-resolution or long-video generation. AdaCache reuses residual computations with a schedule tailored to each video and motion content. Across multiple open-source video DiTs, it achieves comparable generation quality at a fraction of the latency.

  • Problem

    Video DiTs can demand substantial memory and computation, especially with many input tokens, while fewer acceleration techniques transfer readily from UNet-based pipelines.

    Source: §1 · p. 1

  • Method

    AdaCache is training-free and plug-and-play, reusing transformer residual computations on a per-video schedule guided by representation change and motion regularization.

    Source: §1 · p. 2, §4.2 · p. 6, §4.3 · p. 6

  • Results

    Across multiple open-source video DiTs, AdaCache achieves comparable generation quality at a fraction of the latency.

    Source: §6 · p. 12

  • Takeaways & Limitations

    The authors present AdaCache as widely applicable with minimal effort for high-fidelity long-video generation.

    Source: §6 · p. 12

Abstract

from arXiv · show

Generating temporally-consistent high-fidelity videos can be computationally expensive, especially over longer temporal spans. More-recent Diffusion Transformers (DiTs) -- despite making significant headway in this context -- have only heightened such challenges as they rely on larger models and heavier attention mechanisms, resulting in slower inference speeds. In this paper, we introduce a training-free method to accelerate video DiTs, termed Adaptive Caching (AdaCache), which is motivated by the fact that "not all videos are created equal": meaning, some videos require fewer denoising steps to attain a reasonable quality than others. Building on this, we not only cache computations through the diffusion process, but also devise a caching schedule tailored to each video generation, maximizing the quality-latency trade-off. We further introduce a Motion Regularization (MoReg) scheme to utilize video information within AdaCache, essentially controlling the compute allocation based on motion content. Altogether, our plug-and-play contributions grant significant inference speedups (e.g. up to 4.7x on Open-Sora 720p - 2s video generation) without sacrificing the generation quality, across multiple video DiT baselines.

1 Introduction

Video Diffusion Transformers offer scalability and generalization but are costly to run, while acceleration methods do not readily transfer from UNet pipelines. AdaCache adapts caching to each video generation, and the paper reports faster inference with comparable quality.

1 Introduction

  • DiTs can demand substantial memory and computation, and fewer acceleration techniques transfer readily from UNet-based pipelines.These demands become especially critical with many input tokens, such as in high-resolution long-video generation.Source: §1 · p. 1
  • 4.7× speedup with comparable VBench quality is reported for AdaCache on Open-Sora 720p, 2-second generation.The comparison measures average latency on a single A100 GPU.Source: Figure 1 · p. 2
  • Videos vary in texture and motion complexity, motivating a generation-specific process to balance quality and latency.The paper notes that this content-dependent approach had not been sufficiently explored for video generation.Source: §1 · p. 2
  • AdaCache is a training-free, plug-and-play method that reuses transformer residual computations according to a per-video caching schedule.Motion Regularization allocates computation based on motion content.Source: §1 · p. 2

2 Related Work

Related work spans video diffusion architectures, inference acceleration, and content-adaptive generation. AdaCache is distinguished as a training-free caching approach tailored to each video generation rather than a fixed schedule.

2 Related Work

  • Video diffusion architectures have evolved from UNets with spatio-temporal attention to transformer-based DiTs, alongside increasing denoising latency in larger models.The passage connects the increased latency especially to long-video generation.Source: §2 · p. 3
  • Inference acceleration includes retraining methods and training-free methods such as token reduction and caching; the latter avoid additional training costs.The paper explores caching tailored for video DiTs, unlike prior fixed caching schedules.Source: §2 · p. 3
  • Content-adaptive generation covers approaches that selectively process latents, switch compute budgets, or adapt denoising steps.The cited AdaDiff and Block caching methods address image generation tasks.Source: §2 · p. 3
  • AdaCache selectively caches layers or modules and tailors its schedule to each video, unlike fixed architecture-level schedules.The paper characterizes this flexibility as providing more control over compute allocation.Source: §2 · p. 3

3 Not All Videos Are Created Equal

The section motivates content-dependent denoising by showing that video step requirements, representation changes, and quality at similar compute budgets vary. These observations support adapting compute both across videos and across denoising stages.

3 Not All Videos Are Created Equal

  • Videos differ in complexity and in the minimum denoising steps needed for reasonable quality, so a content-dependent schedule can exploit that variation.The paper relates complexity to changes in information across space and time, including detail and motion.Source: §3 · p. 4
  • Representation changes vary across videos and denoising stages, motivating a non-uniform compute schedule within and across generations.Smaller changes indicate greater redundancy and an opportunity to reuse representations.Source: §3 · p. 4
  • At similar compute cost, generation quality varies across configurations using the same pretrained DiT, motivating per-video resource allocation.The study considers base schedules of 30 or 100 denoising steps.Source: §3 · p. 5

4 Adaptive Caching for Faster Video DiTs

AdaCache adapts video DiT computation by reusing cached residuals according to their rate of change, while Motion Regularization adjusts caching using estimated video motion. A shared schedule across layers is used to maintain denoising stability.

4.1 Preliminaries: Video Diffusion Transformers

  • Video DiT blocks repeat spatial-temporal attention, cross-attention, and MLP computations across layers and denoising steps, with equal layer-step costs in streamlined architectures.AdaCache caches residual computations while the denoised representation continues to update using computed or reused residuals.Source: §4.1 · p. 5, Figure 4 · p. 5

4.2 Adaptive Caching

  • AdaCache selects when to recompute residuals using their rate of change, mapping higher distances to lower cache-rates and more frequent recomputation.A cache-rate is the number of subsequent steps that reuse a computed representation; the method uses a codebook of rates and metric thresholds.Source: §4.2 · p. 6
  • AdaCache reuses cached representations between scheduled computations, applying a common distance metric and cache-rate across DiT layers because layer-specific schedules make generations unstable.The distance metric may be averaged across layers or computed at a selected layer, and a lightweight metric such as L1 avoids additional latency overheads.Source: §4.2 · p. 6

4.3 Motion Regularization

  • Motion Regularization scales the caching distance using motion-score and motion-gradient estimates, so higher estimated motion increases the distance and selects a lower cache-rate.The motion-score is estimated from residual frame differences but is unreliable early; motion-gradient can predict later motion, and a common motion regularization is used across layers for stability.Source: §4.3 · pp. 6–7

5 Experiments

Experiments compare AdaCache with other training-free accelerators across three video DiT backbones, measuring quality and latency. AdaCache offers strong speedups with comparable quality, while motion regularization improves quality and the multi-GPU and resolution studies test its broader operating range.

5.1 Implementation details

  • The main experiments generate 900+ videos across Open-Sora, Open-Sora-Plan, and Latte, comparing AdaCache with training-free accelerators using quality and complexity metrics.Unless otherwise stated, ablations and qualitative tests use Open-Sora prompts for 720p, 2s videos at 100 steps.Source: §5.1 · p. 7

5.2 Main results

  • AdaCache-fast reaches 2.24× on Open-Sora; AdaCache reaches 3.70× versus PAB’s 1.45× on Open-Sora-Plan and 2.74× versus the prior 1.33× on Latte.The baselines are Open-Sora at 480p, 2s, 30 steps; Open-Sora-Plan at 512×512, 2.7s, 150 steps; and Latte at 512×512, 2s, 50 steps. Speedups were measured on a single A100 GPU.Source: §5.2 · p. 8, Table 1 · p. 8
  • In randomized user tests, 70% preferred AdaCache over PAB, while users preferred MoReg over vanilla AdaCache 25% to 14% and tied them 60% of the time.The study collected 1,800 responses from 36 users across 50 questions.Source: §5.2 · pp. 8–9, Figure 5 · p. 8

5.3 Ablation study

  • For Open-Sora 720p, 2s videos at 100 steps, MoReg reduces temporal inconsistencies while retaining a 4.5× speedup versus AdaCache’s 4.7×; the authors select it for the best quality-latency trade-off.MoReg allocates more computation in proportion to motion.Source: Figure 7 · p. 9, §5.3 · p. 10
  • Across multi-GPU settings, AdaCache outperforms PAB, and its additional speedup over similarly parallelized baselines increases with GPU count.The study uses Open-Sora 480p, 2s, 30-step generations and Open-Sora-Plan 512×512, 2.7s, 150-step generations on A100 GPUs.Source: Figure 8 · p. 10, §5.3 · p. 10
  • AdaCache provides consistent speedups across resolutions without affecting quality, including 480p 2s, 480p 4s, and 720p 2s generations at 100 steps.The ablation evaluates 32 Open-Sora videos using the VBench custom dataset.Source: Table 3 · p. 10, §5.3 · p. 11
  • L1/L2 distance better estimates representation change than cosine distance; the mid-layer metric avoids extra overhead, while temporal attention gives the best cache-metric trade-off.The study also finds that measuring a single layer is not significantly different from aggregating multiple layers.Source: §5.3 · p. 11

5.4 Qualitative results

  • Across Open-Sora, Open-Sora-Plan, and Latte, AdaCache gives comparable generation quality at 4.49×, 3.53×, and 2.46× speedups, respectively, versus PAB’s 1.26×, 1.45×, and 1.33×.The qualitative comparison reports prompt fidelity and no significant artifacts, while noting that pixel-space alignment with the baseline is not required.Source: Figure 9 · p. 11, §5.4 · p. 12

6 Conclusion

AdaCache accelerates video diffusion transformers without retraining by caching residual computations on a schedule tailored to each generation. The authors report comparable quality at a fraction of latency across multiple open-source video DiTs.

6 Conclusion

  • AdaCache combines cached residual computations with a generation-dependent schedule to improve video DiT inference speed without retraining.The paper also proposes Motion Regularization to allocate computation based on motion content and improve the quality-latency trade-off.Source: §6 · p. 12
  • Across multiple open-source video DiTs, AdaCache achieves comparable generation quality at a fraction of latency.Source: §6 · p. 12
  • The authors describe AdaCache as widely applicable with minimal effort for high-fidelity long-video generation.Source: §6 · p. 12

Ethics Statement

The work evaluates a generic, training-free video DiT acceleration method on publicly available open-source models. It reports public prompts, reproducible steps, and released code.

Ethics Statement

  • The method is evaluated on publicly available open-source video DiTs and is not tied to a specific model or commercial application.Source: Ethics Statement · p. 12
  • The authors state that the method poses no unique risk requiring special consideration beyond risks associated with other video generation models.Source: Ethics Statement · p. 12

Reproducibility Statement

  • Experiments use public benchmark prompts and pretrained open-source models; the training-free method requires no pretrained-weight updates, and the authors release code and reproduction steps.Source: Reproducibility Statement · p. 12

A Appendix

The appendix figure tracks Motion Regularization metrics over diffusion steps. Motion-score reflects latent frame differences but is unreliable early, while motion-gradient helps predict later motion.

A Appendix

  • Figure A.1 plots histograms of motion-score and motion-gradient across diffusion steps.Source: Figure A.1 · p. 13
  • Motion-score is estimated from latent frame differences and correlates with perceived video motion, but noisy early latents can make it unreliable.Source: Figure A.1 · p. 13
  • Motion-gradient across steps helps predict later, pixel-space motion early; together, the metrics provide a reasonable caching regularization.Source: Figure A.1 · p. 13

A.1 Design decisions

Motion-score and motion-gradient jointly guide caching based on video motion, while a codebook maps distance thresholds to cache-rates for a chosen denoising schedule.

A.1 Design decisions

  • Motion-score can be misleading early because it is derived from noisy latents; motion-gradient across steps predicts later motion to guide caching regularization.Together, the metrics allocate computation according to the generated video's motion content.Source: §A.1 · p. 13
  • A predefined codebook pairs basis cache-rates with distance thresholds and is specific to the denoising schedule's step count.Source: §A.1 · p. 13
  • Thresholds may require tuning for each video DiT baseline, while cache-rates can be adjusted to provide faster or slower AdaCache variants.Source: §A.1 · p. 13

A.2 Additional qualitative results

The qualitative results show that motion regularization stabilizes AdaCache generations while retaining speedups, and AdaCache compares favorably with prior-art at comparable inference speed.

A.2 Additional qualitative results

  • For 480p, 2-second, 30-step clips, MoReg stabilizes quality toward the ends of long videos and better follows baseline generations.The reported speedups are 2.10× with MoReg and 2.24× without it.Source: §A.2 · p. 13, Figure A.2 · p. 14

A.3 Additional qualitative comparisons with prior-art

  • AdaCache consistently shows better generation quality than PAB at comparable speedup in 720p, 2-second, 100-step Open-Sora comparisons.The passage reports this at 2.61× speedup and also at 1.66× speedup.Source: §A.2 · p. 14, Figure A.2 · p. 14

A.3 Text prompts used in qualitative examples

The qualitative examples draw on prompts from several publicly available sources and include comparisons with prior-art.

A.3 Text prompts used in qualitative examples

  • The prompts come from the Open-Sora gallery, VBench, and Sora, and all are publicly available.The supplementary material provides prompts used for the paper’s qualitative results.Source: §A.3 · p. 14
  • On baseline Open-Sora, qualitative prior-art comparisons at 720p for 2-second generations with 100 steps evaluate AdaCache against training-free inference accelerator PAB at comparable speedup; AdaCache consistently shows better generation quality.Source: Figure A.3 · p. 15

Related papers

Loading 2411.02397v2…