MORCA: Offline-to-Online Reinforcement Learning for Adaptive Cache Reuse in Video Diffusion Acceleration

1 INTRODUCTION

• Caching accelerates denoising by reusing results from earlier steps, offering a lightweight alternative with flexible speed–quality trade-offs.Dynamic caching methods commonly use estimates of per-step reuse error to guide threshold-based decisions.Source: §1 · p. 1
• Step error does not directly predict terminal error, and terminal error varies substantially even when step-error patterns are similar across prompts.Source: §1 · p. 2, Figure 1 · p. 2
• Adding lightweight features from current and previous input latents substantially improves terminal-error prediction accuracy.The reported prediction results are for Wan2.1 and measure terminal error using PSNR.Source: §1 · p. 2, Table 1 · p. 2
• MORCA uses a latent-aware scheduler and reuse-budget constraint to guide cache decisions by terminal error under user-specified acceleration targets.Source: §1 · p. 2
• Offline IQL initializes the scheduler from limited data, and online fine-tuning further improves its performance.The paper motivates IQL partly by its higher data utilization than on-policy methods.Source: §1 · p. 2
• MORCA achieves better speed–quality trade-offs than state-of-the-art caching methods across different video generation models.Source: §1 · p. 2

2.1 DIFFUSION INFERENCE ACCELERATION

• Distillation reduces generation to few or one steps but requires additional training; quantization and sparse attention lower per-step computation but typically rely on specialized kernels.Source: §2.1 · p. 3

2.2 CACHING-BASED DIFFUSION MODEL ACCELERATION

• Caching methods vary in granularity, reuse strategy, and schedule, with dynamic methods using estimated cache-reuse error and threshold rules to adapt decisions.Source: §2.2 · p. 3

2.3 REINFORCEMENT-LEARNING-BASED CACHE SCHEDULING

• RAPID3 formulates scheduling as an MDP; BAG distills schedules from expensive offline search, while OnlineCache uses on-policy REINFORCE and cannot guarantee user-specified acceleration targets.Source: §2.3 · p. 3

3 METHOD

• MORCA replaces SeaCache’s threshold rule with a learned scheduler that uses latent features and reuse-budget dynamics for terminal-error-guided decisions under user-specified acceleration targets.The scheduler retains SeaCache’s caching granularity and reuse strategy.Source: §3 · p. 3

3.1 PRELIMINARIES

• At each denoising step, cache inference either reuses a residual from the most recent recomputation or computes the DiT blocks normally.Under classifier-free guidance, the two branches share the reuse decision but keep separate residual caches.Source: §3.1 · p. 4

3.2 BUDGET-CONSTRAINED AND LATENT-AWARE SCHEDULER

• MORCA maps a target acceleration ratio to a reuse budget and formulates scheduling as a constrained MDP that maximizes expected video-quality reward while satisfying that budget.The augmented MDP incorporates the reuse count and restricts actions so the prescribed budget remains attainable.Source: §3.2 · p. 5
• The scheduler combines step-error proxies and statistics, pooled features from current, previous, and last-recompute latents, and information about the remaining reuse budget.The compact latent representation uses 3D pooling and pooled temporal means and variances to limit computational overhead.Source: §3.2 · p. 6

3.3 OFFLINE-TO-ONLINE REINFORCEMENT LEARNING PARADIGM

• Offline IQL initializes the scheduler from randomly sampled cache trajectories, then online fine-tuning explores and updates the policy while retaining offline data for stability.IQL reuses collected data across optimization updates; online training includes a critic-only warm-up before joint training.Source: §3.3 · pp. 6–7

4.1 EXPERIMENTAL SETTINGS

• Evaluation uses two pretrained Wan models, four caching baselines, and a fixed set of 200 VBench prompts, comparing quality against no-cache references and reporting latency.The compared speedups are approximately 1.8×, 2.4×, and 3.0× for Wan2.2, and 2.4×, 3.0×, and 4.0× for Wan2.1; lower targets are used for Wan2.2 to preserve fidelity.Source: §4.1 · p. 7

4.2 QUANTITATIVE COMPARISON

• MORCA achieves the best PSNR, SSIM, and LPIPS on Wan2.2 at all three target speedups, with PSNR gains over SeaCache of 2.22, 1.86, and 1.78 dB.On Wan2.1, it has the highest PSNR and SSIM at all three targets, improves LPIPS over SeaCache at all three settings, and has the best LPIPS at 3.0× and 4.0×.Source: Table 2 · p. 8, §4.2 · p. 8

4.3 QUALITATIVE COMPARISON

• MORCA better preserves original video structure and fine details than other caching methods in the qualitative comparisons.Wan2.2 video spatial structure is described as more sensitive to cache reuse, consistent with its relatively lower fidelity metrics.Source: §4.3 · p. 8, Figure 4 · p. 9

4.4 ABLATION AND ANALYSIS

• Mean absolute speedup errors are 0.03 on Wan2.2 and 0.05 on Wan2.1; mean absolute relative inference-time errors are 1.15% and 1.55%, respectively.Discrete reuse-step counts make attainable acceleration ratios discrete, leaving unavoidable speedup-control error.Source: §4.4 · p. 9
• Incorporating latent features into the scheduler state enables better scheduling decisions than omitting them.Table 3 reports offline training results on Wan2.1 at approximately 3.0× speedup.Source: §4.4 · p. 9, Table 3 · p. 9
• With comparable training data, IQL outperforms REINFORCE, offline IQL outperforms online IQL from scratch, and online fine-tuning further improves the offline-initialized scheduler.Table 4 reports results on Wan2.1 at approximately 3.0× speedup.Source: §4.4 · p. 9, Table 4 · p. 9

5 CONCLUSION

• MORCA formulates cache scheduling as an MDP and uses a budget-constrained, latent-aware scheduler trained through offline-to-online IQL for terminal-error-guided decisions at user-specified acceleration targets.Experiments across video-generation models show better speed–quality trade-offs than state-of-the-art caching methods.Source: §5 · p. 10
• As a schedule-level approach, MORCA can potentially be combined with different caching granularities and cache reuse strategies for further gains.The authors identify exploring these combinations as future work.Source: §5 · p. 10

A MODEL CONFIGURATIONS

• Both Wan models use 50 denoising steps at 832 × 480 resolution and 16 fps; Wan2.2 generates 45-frame videos with DPM++, while Wan2.1 generates 81-frame videos with UniPC.Model-specific flow shift, CFG scale, and offloading settings are also reported.Source: A MODEL CONFIGURATIONS · p. 15, Table 5 · p. 15

B.1 NETWORK ARCHITECTURE

• The scheduler processes pooled latent features through 3D and 2D CNN branches, projects their concatenated 768-dimensional output to a 128-dimensional embedding, and appends seven scalar state features.The resulting 135-dimensional input feeds an MLP with two 256-unit hidden layers; only the policy network is needed at inference.Source: §B.1 · p. 15
• Separate policy, twin Q, and value networks output two action logits, two action values per Q-network, and a scalar value; target Q-networks are updated through EMA.Table 6 summarizes shared training and IQL hyperparameters; γ is the discount factor, τ the expectile level, and β the advantage-weighting coefficient.Source: §B.1 · p. 15, §B.2 · p. 15

B.3 DATASET PREPARATION

• The pipeline randomizes a smooth threshold curve across denoising steps instead of independently sampling reuse decisions, to avoid rollouts with limited learning value.Curve mean is calibrated from a sampled target acceleration ratio; amplitude and shape are randomized, with cubic smoothstep interpolation and bound and smoothness adjustments.Source: §B.3 · pp. 15–16
• Each rollout becomes N transition tuples using cumulative reuse counts, step-error signals, step statistics, latent features, and the defined reward to construct states.The cumulative reuse count is the reuse budget at step t.Source: §B.3 · p. 16
• The collected data comprise 2,500 Wan2.2 trajectories from 100 prompts and 4,000 Wan2.1 trajectories from 1,000 prompts, covering target speedups of 1.5×–3.5× and 2.0×–4.5×, respectively.Source: §B.3 · p. 16
• Actions forced by the exact reuse-budget constraint or mandatory recomputation are excluded from policy loss but retained for Q- and V-function updates.Mandatory recomputation occurs at the first and last denoising steps and at the Wan2.2 expert-switching step.Source: §B.3 · p. 16

C BASELINE IMPLEMENTATION DETAILS

• Wan2.2 baseline configurations include separate fourth-degree TeaCache polynomials for high- and low-noise stages, fitted with 70 calibration prompts.The configurations cover three approximate acceleration regimes; Wan2.1 uses the official implementations, with method-specific settings for TeaCache, SeaCache, and DiCache.Source: C BASELINE IMPLEMENTATION DETAILS · p. 16, Table 7 · p. 17

D REDUCTION OF THE HARD-CONSTRAINED MDP

• The augmented MDP enforces exactly K reuse actions almost surely while preserving the optimal feasible value.Predetermined recomputation steps are handled by restricting reuse actions and adjusting the budget counts.Source: D REDUCTION OF THE HARD-CONSTRAINED MDP · p. 17
• The augmented process is Markov, and backward induction yields an optimal time-dependent deterministic Markov policy among admissible history-dependent policies.The maximizing rule is measurable, with ties broken by choosing the smaller action.Source: D REDUCTION OF THE HARD-CONSTRAINED MDP · pp. 17–18
• Augmenting the state with cumulative and remaining reuse counts makes budget-feasible policies correspond to policies in the original process without changing returns.The augmented process preserves the original state transitions and rewards.Source: D REDUCTION OF THE HARD-CONSTRAINED MDP · p. 17

E.1 ADDITIONAL QUALITATIVE COMPARISONS

• Figures 7 and 8 present additional qualitative comparisons between MORCA and baseline methods on Wan2.2 and Wan2.1, respectively, at three target acceleration ratios.Source: §E.1 · p. 18

E.2 SCHEDULER OVERHEAD

• Approximately 1% of total Wan2.2 inference time is spent on scheduler execution.Table 8 reports this proportion as acceptable overhead.Source: §E.2 · p. 18, Table 8 · p. 18

E.3.1 SCHEDULE VISUALIZATIONS AT THE SAME REUSE BUDGET

• Under the same reuse budgets, MORCA learns schedules that differ from SeaCache’s and achieves better generation fidelity in both examples.Figure 5 visualizes the schedules and corresponding generation results.Source: §E.3.1 · p. 18, Figure 5 · p. 19

E.3.2 ADDITIONAL TARGET ACCELERATION RATIOS

• MORCA outperforms SeaCache in PSNR, SSIM, and LPIPS across nine Wan2.2 target acceleration ratios from 1.5× to 3.5×.The ratios increase in 0.25× increments; Figure 6 shows the speed–quality trade-offs.Source: Figure 6 · p. 20, §E.3.2 · p. 20

• Differences in denoising dynamics mean MORCA’s learned scheduler cannot be directly transferred between models.Source: F LIMITATIONS · p. 20
• MORCA requires separate scheduler training for each model, adding costs for collecting cache-scheduling trajectories and optimizing the scheduler.Source: F LIMITATIONS · p. 20
• These additional training costs may limit practical adoption when training resources are constrained.Source: F LIMITATIONS · p. 20