PaperlayerView exact revision on arXiv

Source-linked AI summary

Motion-Aware Caching for Efficient Autoregressive Video Generation

Jing Xu, Yuexiao Ma, Xuzhe Zheng, Xing Wang, Shiwei Liu, Chenqian Yan, Xiawu Zheng, Rongrong Ji, Fei Chao, Songwei Liu

arXiv:2605.01725v2cs.CVcs.AI

TL;DR

Autoregressive video generation is hindered by sequential denoising costs, while existing coarse chunk-level caching misses fine-grained motion differences. MotionCache links caching error to residual instability and uses inter-frame differences for coarse-to-fine, motion-aware token reuse. It reports substantial speedups on SkyReels-V2 and MAGI-1 while preserving generation quality.

  • Problem

    Autoregressive video generation faces prohibitive iterative-denoising costs, and existing coarse-grained caching overlooks heterogeneous update requirements across frames and tokens.

  • Method

    MotionCache combines a structural warm-up phase with motion-characteristics-weighted token accumulation, using frame differences as a proxy for token importance.

  • Results

    MotionCache achieves significant speedups while delivering superior perceptual quality and temporal coherence on SkyReels-V2 and MAGI-1.

  • Takeaways & Limitations

    Dynamic resource allocation prioritizes high-motion regions while reusing residuals for static backgrounds, moving beyond rigid all-or-nothing chunk caching.

  • Takeaways & Limitations

    The initial coarse structure-construction phase continues until K full computations have established the structural foundation.

Abstract

from arXiv · show

Autoregressive video generation paradigms offer theoretical promise for long video synthesis, yet their practical deployment is hindered by the computational burden of sequential iterative denoising. While cache reuse strategies can accelerate generation by skipping redundant denoising steps, existing methods rely on coarse-grained chunk-level skipping that fails to capture fine-grained pixel dynamics. This oversight is critical: pixels with high motion require more denoising steps to prevent error accumulation, while static pixels tolerate aggressive skipping. We formalize this insight theoretically by linking cache errors to residual instability, and propose MotionCache, a motion-aware cache framework that exploits inter-frame differences as a lightweight proxy for pixel-level motion characteristics. MotionCache employs a coarse-to-fine strategy: an initial warm-up phase establishes semantic coherence, followed by motion-weighted cache reuse that dynamically adjusts update frequencies per token. Extensive experiments on state-of-the-art models like SkyReels-V2 and MAGI-1 demonstrate that MotionCache achieves significant speedups of $\textbf{6.28}\times$ and $\textbf{1.64}\times$ respectively, while effectively preserving generation quality (VBench: $1\%\downarrow$ and $0.01\%\downarrow$ respectively). The code is available at https://github.com/ywlq/MotionCache.

1 Introduction

Autoregressive video generation remains costly despite memory-saving architectures and caching methods, because coarse chunk-level reuse overlooks heterogeneous motion. MotionCache addresses this with theoretically grounded, motion-aware token caching and reports faster generation with preserved quality.

  • Motivation: Iterative denoising and attention scaling make high-resolution, long-duration video generation costly and difficult to deploy in real time.Autoregressive chunking bounds memory growth, but generating a 7-second 540 × 540 SkyReels-V2 video still takes approximately 27 minutes on an A800 GPU.
  • Limitations of Existing Methods: Existing caching methods reduce redundant computation, but FlowCache applies an all-or-nothing decision to entire chunks.This overlooks token-level redundancy and spatial or content differences among frames within the same timestep.
  • Theoretical Basis: MotionCache links caching error to residual instability and uses frame differences as a lightweight proxy for token motion and importance.The proposed proxy supports precise identification of dynamic regions.
  • MotionCache: MotionCache combines structural warm-up with motion-weighted token accumulation to prioritize high-motion details and allocate computation dynamically.Its framework is described as coarse-to-fine and targets autoregressive video generation.
  • Results: 7.26× and 2.07× speedups are reported on SkyReels-V2 and MAGI-1, respectively, alongside superior visual fidelity and temporal coherence.These results are presented as outperforming existing methods.

2 Related Work

Autoregressive video generation adapts sequential next-token prediction to video chunks, while caching methods exploit temporal redundancy to reduce inference cost. Prior work includes fixed schedules, content-aware policies, trajectory prediction, and FlowCache for autoregressive settings.

  • Autoregressive Video Generation: Autoregressive video generation partitions continuous streams into sequential latent chunks conditioned on preceding context.Causal Diffusion-Forcing and fixed attention windows help avoid full-sequence quadratic complexity while supporting high-fidelity synthesis.
  • Feature Caching-based Acceleration: Feature caching accelerates inference by reusing intermediate features or residuals when temporal redundancy makes repeated computation unnecessary.The surveyed methods are training-free and target inference efficiency.
  • Feature Caching-based Acceleration: Prior caching methods use fixed reuse schedules, dynamic content-aware policies, or Taylor expansions to estimate when and how features should be reused.Examples include FORA, ∆-DiT, TeaCache, ERTACache, AdaCache, and TaylorSeer.
  • Autoregressive Caching: FlowCache extends caching concepts to autoregressive video generation by exploiting temporal characteristics of video chunks.The passage identifies it as an autoregressive-domain extension of feature caching.

3 Preliminaries

The paper frames autoregressive video generation as chunked conditional denoising with Euler updates, and feature caching as residual reuse driven by temporal redundancy. These mechanisms reduce computation by bypassing forward passes when local features remain stable.

  • Diffusion Model: Flow-matching diffusion models reverse a noise process by approximating a time-dependent velocity field and numerically integrating it during denoising.The reverse process uses solvers such as Euler’s method.
  • Feature Caching Strategies: Residual differences across adjacent timesteps and frames show heterogeneous temporal behavior, motivating more selective caching than uniform chunk-level reuse.The distributions are described as long-tailed and variable within chunks.
  • Autoregressive Video Generation Model: Autoregressive generation decomposes a long video into latent chunks, with each chunk denoised conditionally on preceding context.Each chunk has temporal, spatial, and channel dimensions, and its update is indexed by timestep t.
  • Autoregressive Video Generation Model: The denoising timestep lies within a window determined by the total discretization steps and the maximum permitted window size.The passage defines t, T, and l for the chunked denoising process.
  • Feature Caching Strategies: Feature caching uses temporal redundancy to reuse residuals instead of recomputing the forward pass when consecutive inputs are sufficiently similar.The cached residual approximates the velocity, reducing computational overhead through local feature stability.

4 Analysis of Caching Error

The analysis identifies residual inconsistency as the source of caching error and shows that motion-related frame differences can serve as a proxy for residual instability. Empirical distributions and ranking agreement support token-wise adaptive caching.

  • Caching Error: Caching error is bounded by inconsistency between feature residuals across adjacent timesteps.The analysis motivates moving from coarse chunk skipping to fine-grained token-wise caching.
  • Residual Stability: Larger deviations between reused and current residuals directly produce larger caching errors, so unstable tokens should be recomputed selectively.Tokens with negligible residual differences can instead retrieve cached values.
  • Heterogeneous Temporal Redundancy: Residual differences have a long-tailed distribution, with median 2.078 and a tail reaching 9.878, indicating non-uniform update requirements.This heterogeneity makes uniform chunk-wise caching inefficient for static and dynamic regions alike.
  • Intra-Chunk Frame Discrepancy: Residual differences across frames within one autoregressive chunk vary substantially, reaching a maximum difference of 5.9219.Distinct latent frames represent different temporal segments and therefore are not uniformly redundant.
  • Residual Stability and Motion Dynamics: Under a Lipschitz assumption on the residual field, intra-chunk frame differences bound residual differences across timesteps.The resulting frame-difference proxy is mathematically grounded rather than merely heuristic.
  • Motion Proxy Validation: NDCG scores consistently exceed 0.94 when frame-difference token rankings are compared with rankings from actual residual differences.This indicates that the proxy preserves token-importance ordering across denoising timesteps.

5 Methodology

MotionCache uses motion-aware token importance and weighted accumulation to allocate computation to dynamic regions while reusing cached residuals for static tokens. A dual-stage coarse-to-fine schedule first stabilizes global structure, then refines details selectively.

  • 5.1 Motion-Aware Token Importance: MotionCache prioritizes high-motion tokens for computation while reusing residuals for static backgrounds.It derives token importance from intra-chunk frame differences and dynamically allocates computational resources.
  • 5.1 Motion-Aware Token Importance: Frame-specific min-max normalization maps motion importance to weights in [α, 1], preserving a baseline update probability for static tokens.The floor value α prevents static background tokens from being completely frozen.
  • 5.2 Importance-Weighted Accumulation Policy: Tokens with high motion accumulate update error rapidly, whereas static tokens suppress accumulation and are computed only after their accumulator exceeds threshold τ.Selected tokens undergo a forward pass and their accumulators are reset to 0.
  • 5.3 Dual-Stage Coarse-to-Fine Inference Schedule: The initial coarse-grained phase synchronizes computation across each chunk until K full computations establish a stable structural foundation.During this phase, the entire chunk is either fully updated or fully skipped.
  • 5.4 Experimental Results: MotionCache achieves superior efficiency-quality trade-offs compared with TeaCache and FlowCache on SkyReels-V2 and MAGI-1.The supplied comparison passage identifies higher generation quality and speedups relative to other baselines, while the table caption covers both evaluated models.
  • 5.3 Dual-Stage Coarse-to-Fine Inference Schedule: After global structure stabilizes, fine-grained inference computes only active tokens and retrieves inactive tokens from the residual cache.Active tokens are gathered into a compact batch, then scattered back to update the residual cache.

6 Experiments

Experiments evaluate MotionCache on MAGI-1 and SkyReels-V2 using perceptual-quality and efficiency metrics, including VBench, FLOPs, and latency. Across ablations and main comparisons, motion-aware caching achieves strong speed-quality trade-offs, with α = 0.6 and K = 6 identified as effective settings.

  • Experimental Setup: Experiments use MAGI-1-4.5B-distill and SkyReels-V2-1.3B on A800 GPUs, evaluating perceptual quality and computational efficiency.Metrics include LPIPS, PSNR, SSIM, VBench, FLOPs, and practical inference latency.
  • Main Results: 2.07× speedup with VBench 74.59%: MotionCache-fast on MAGI-1 preserves robust visual quality while outperforming faster TeaCache and FlowCache variants.TeaCache-fast and FlowCache-fast reach VBench scores of 68.81% and 73.42%, respectively.
  • Main Results: 1.64× acceleration: MotionCache-slow on MAGI-1 delivers nearly lossless quality and preserves fine-grained semantic details.The passage contrasts this with details lost in coarse-grained schemes.
  • Main Results: 6.28× acceleration with VBench 82.84%: MotionCache-slow on SkyReels-V2 outperforms FlowCache-slow and TeaCache-slow in speed and structural alignment.FlowCache-slow achieves 6.26× and VBench 82.70%, while TeaCache-slow achieves 1.89× and VBench 82.67%.
  • Main Results: 7.26× speedup with VBench 82.75%: MotionCache-fast on SkyReels-V2 maintains excellent quality at state-of-the-art acceleration.Existing methods show texture drifting and structural misalignment at lower acceleration ratios.
  • Ablation Studies: α = 0.6 balances quality and efficiency, while K = 6 is optimal because longer warm-up increases latency for only marginal quality gains.Increasing α improves background preservation but raises latency; increasing K benefits global structure but adds computational overhead.

7 Conclusion

MotionCache accelerates autoregressive video generation through motion-aware, fine-grained residual reuse. Its experiments report substantial speedups while preserving perceptual quality and temporal coherence on SkyReels-V2 and MAGI-1.

  • Conclusion: MotionCache accelerates autoregressive video generation by prioritizing high-motion regions and reusing residuals for static backgrounds.The framework uses a theoretical connection between residual instability and intra-chunk frame discrepancies to allocate computation dynamically.
  • Conclusion: MotionCache replaces rigid all-or-nothing chunk skipping with fine-grained, motion-centric resource allocation.The conclusion describes this as a promising direction for efficient video synthesis and potential real-time deployment.
  • Conclusion: Experiments on SkyReels-V2 and MAGI-1 show significant speedups alongside superior perceptual quality and temporal coherence.

A Detailed Proof of Proposition 4.1

The proof analyzes caching error under the flow-matching Euler update by comparing a fully computed residual with an update that reuses a cached residual. It concludes that the error scales with timestep size and adjacent-timestep residual instability.

  • Proof: The proof starts from the standard Euler discretization for the i-th video chunk and timestep t −1, using the model’s true velocity for the full update.The ground-truth residual is derived from full computation.
  • Proof: When caching is active, computation at t −1 is bypassed and the system uses a residual stored from the preceding timestep.The approximated update therefore reuses cached information instead of recomputing the residual.
  • Proof: Caching error is defined as the Euclidean distance between the fully computed output latent and the output latent obtained through residual reuse.
  • Result: Caching error is linearly dependent on step size ∆t and strictly determined by residual-vector instability between adjacent timesteps.

B Detailed Proof of Lemma 4.2

The proof connects residual variation across denoising timesteps to local frame differences under a Lipschitz assumption on the residual field’s temporal gradient. It then establishes that caching error is upper-bounded by intra-chunk frame difference.

  • Assumption: Lemma 4.2 assumes the temporal gradient ∇tR of the continuous residual function satisfies a Lipschitz condition with respect to the input latent.
  • Proof: A first-order Taylor expansion approximates the residual at timestep t −1, with higher-order terms ignored in the local analysis.The residual difference is then dominated by its partial derivative with respect to time.
  • Interpretation: The temporal derivative ∂R/∂t corresponds to curvature of the generative ODE trajectory, which is theoretically zero for straight-line trajectories from static data.
  • Motion Relation: The proof models curvature as a Lipschitz-continuous function of signal motion and compares adjacent frames as a local staticity reference.
  • Result: Caching error is strictly upper-bounded by intra-chunk frame difference after combining the intermediate bounds.

C.1 Video Configuration and Model Implementation

MotionCache uses model-specific autoregressive configurations and a coarse-to-fine schedule with an initial global warm-up. Evaluation aggregates selected VBench metrics using the benchmark’s official normalization and weighting.

  • Hyperparameters: SkyReels-V2 uses α = 0.5, K = 6, and m = 4 warm-up timesteps, while MAGI-1 uses α = 0.5, K = 9, and m = 5.Cache reuse is disabled during the first m timesteps to ensure trajectory stability.
  • Model execution granularity: MAGI-1 operates at inter-chunk granularity with a sliding window, whereas SkyReels-V2 uses hierarchical intra-chunk blocks and staggered denoising.
  • Evaluation: Evaluation averages selected VBench metrics using the benchmark’s official normalization and weighting methodology.

D Detailed Ablation Study Results

The ablations support MotionCache’s parameter choices, theoretical approximations, scalability, latent-space design, and motion-aware qualitative advantages. Across long-video settings and visual comparisons, the framework maintains lower computation while preserving fine-grained content more effectively than baselines.

  • Soft-mapping floor α: α values above 0.5 produce stable quality, whereas lower values under-update static background tokens and degrade fine-grained details.The ablation sweeps α from 0.0 to 1.0 in increments of 0.1.
  • Phase 1 duration K: K values above 5 yield stable evaluation scores, indicating that global semantic structure is sufficiently established before spatial-mask alignment.The ablation sweeps K from 0 to 17; K = 17 corresponds to the FlowCache baseline.
  • Theoretical approximation: For 50 denoising steps, Δt² = 0.0004 while the residual difference is at least 1.175, making the omitted higher-order term negligible in practice.This supports omitting the O(Δt²) term in the Taylor expansion.
  • Long-video efficiency: MotionCache’s peak-memory increase remains stable from 7s to 10s, while its overall computation stays lower than all baselines at both lengths.Attention operations dominate FLOPs because they scale quadratically with sequence length.
  • Motion-cue efficiency: RGB-domain optical flow is impractical because it requires repeated latent-to-RGB decoding; one SkyReels-V2 VAE decode takes 9.7 seconds.The cumulative decoding overhead across dozens of timesteps would negate caching acceleration.
  • Motion-cue efficiency: Sparse and dense optical flow are 429× and 925× slower than frame differences, respectively, making their overhead exceed caching savings.The qualitative comparisons further show MotionCache preserving fine-grained details and avoiding artifacts that affect accelerated baselines.
Loading 2605.01725v2…