PaperlayerView exact revision on arXiv

Source-linked AI summary

ForcingBooster: Adaptive Patch-wise Denoising for Fast Autoregressive Video Diffusion

Zekai Gu, Peng Wang, Jiahao Lu, Yongsen Mao, Cheng Lin, Yuan Liu

arXiv:2610.12625v1cs.CV

TL;DR

Uniform denoising budgets in autoregressive video diffusion can spend steps on already-stable patches while leaving harder patches in need of refinement. ForcingBooster uses a lightweight router to stop easy patches early and continue denoising difficult ones. It reduces patch-level denoising steps by about 50% while keeping VBench scores comparable to four-step baselines.

  • Problem

    Uniform denoising budgets across patches can waste computation on stable regions or leave difficult regions insufficiently refined when the whole chunk uses fewer steps.

    Source: §1 · p. 2

  • Method

    ForcingBooster uses a lightweight router to stop sufficiently converged patches early and continue denoising uncertain or condition-sensitive patches.

    Source: §1 · p. 2

  • Results

    About 50% fewer denoising steps across video patches while maintaining VBench scores comparable to corresponding four-step baselines.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The framework is compatible with most existing few-step autoregressive diffusion models and needs only a lightweight router and modest training, without fine-tuning the heavyweight generator or using additional video data.

    Source: §5 · p. 10

Abstract

from arXiv · show

Autoregressive video diffusion models have become a leading approach to real-time interactive video generation and video world modeling. However, existing methods typically allocate computation uniformly across space when generating the next video chunk, updating all patches with the same denoising schedule and number of function evaluations. This ignores the spatial heterogeneity of video content: static, simple, or recurring regions are often easier to denoise, while other regions require further refinement or additional context. A uniform budget thus leads to redundant computation on easy patches, whereas reducing the fixed step count can leave difficult patches insufficiently refined. We introduce \methodname{}, a general framework that dynamically allocates denoising steps to individual patches: easy patches stop early to improve efficiency, while difficult patches receive more refinement steps. This design accelerates generation with negligible quality degradation. Our method is compatible with most existing few-step autoregressive diffusion models, requiring only a lightweight routing module with modest training. Experiments demonstrate that \methodname{} reduces the total number of denoising steps applied across video patches by about 50\% while maintaining visual quality comparable to the four-step baseline.

1 INTRODUCTION

ForcingBooster addresses redundant uniform denoising by routing computation patch by patch in few-step autoregressive video generators. Across varied video-generation and world-model settings, it cuts patch-level denoising steps by about 50% while maintaining VBench scores comparable to four-step baselines.

1 INTRODUCTION

  • Autoregressive video diffusion generates successive temporal chunks conditioned on earlier ones, while distillation reduces the per-chunk denoising schedule to four steps or fewer.Causal attention supports KV caching so prior chunks need not be repeatedly processed.Source: §1 · p. 2
  • Uniform patch budgets waste computation on already-stable regions, while reducing steps for an entire chunk can leave regions needing refinement under-denoised.Examples include static backgrounds receiving fewer steps than moving or textured regions, and newly revealed structures receiving more than familiar regions.Source: §1 · p. 2
  • ForcingBooster adds a lightweight router that stops sufficiently converged patches and continues refining uncertain or condition-sensitive regions without modifying the generator.The router assesses patches using generated history and conditioning signals such as camera poses or text controls.Source: §1 · p. 2
  • Early-exited patches can output an intermediate clean-latent estimate available at each denoising step in diffusion and flow-matching generators.The estimate may be predicted directly or recovered from the generator’s native parameterization.Source: §1 · p. 2
  • Router training balances stopping quality against computation using on-policy rollouts and full-step outputs from the frozen generator, without extra video data or manually defined stopping labels.A stopping surrogate penalizes stopping when the current clean-latent prediction deviates from the full-step result, while a computational loss encourages fewer denoising steps.Source: §1 · pp. 2–3
  • About 50% fewer denoising steps across video patches, with VBench scores comparable to corresponding four-step baselines, using approximately 500 router training steps on average.The evaluation spans video generation and world models with different scales and architectures.Source: §1 · p. 3

2 RELATED WORK

Related work covers autoregressive chunk generation, few-step distillation, and approaches that adapt computation. ForcingBooster differs by learning patch-wise stopping in a frozen few-step autoregressive generator while retaining context for active patches.

2.1 AUTOREGRESSIVE VIDEO DIFFUSION

  • Autoregressive video diffusion generates temporal chunks conditioned on preceding chunks, with causal attention enabling KV caching of earlier context.Prior work also addresses long-video history management, generated-history training, multiplayer world modeling, and flexible chunking.Source: §2.1 · p. 3

2.2 DIFFUSION DISTILLATION AND POST-TRAINING

  • Distillation and post-training methods reduce denoiser evaluations, but typically update the generator and use a shared denoising-step budget across patches within each frame or chunk.The subsection also surveys one- or two-step generation, flexible sampling budgets, diversity preservation, reward supervision, and on-policy self-distillation.Source: §2.2 · p. 3

2.3 ADAPTIVE COMPUTATION FOR DIFFUSION

  • Adaptive computation includes per-sample or spatially varying schedules, caching, and token or region recomputation, while DSA still shares a budget across patches within each frame or chunk.ForcingBooster instead trains a lightweight router in a frozen few-step autoregressive generator, balancing reconstruction error against denoising cost using full-step references and histories generated under current routing. Stopped patches’ clean latents are fixed and removed from later Transformer updates while their KV context remains available.Source: §2.3 · pp. 3–4

3 METHOD

ForcingBooster uses intermediate clean-latent predictions to stop denoising individual patches when they are sufficiently refined, while other patches continue. Its method covers patch-wise routing and training, motivated by evidence that patches converge at different rates.

3.1 PRELIMINARY AND OVERVIEW

  • Intermediate clean-latent predictions let a patch stop early and serve as its final output; the router assesses patches in the context of generation history and conditioning.Regions reliably inferred from history may converge earlier than newly revealed or condition-dependent regions.Source: §3.1 · pp. 4–5

3.2 DENOISING STEPS REQUIRED BY DIFFERENT PATCHES

  • 30.60% of patches reach 35 dB PSNR after one step, while 47.45% do not reach it within the first three steps against the four-step reference.The controlled experiment measures agreement with the generator’s four-step output, not ground-truth reconstruction.Source: §3.2 · p. 5, Figure 3 · p. 5
  • With fixed schedules, one-step sampling poorly defines the hand, while two and three steps still leave visible hand and wrist distortions versus four steps.The sky and hillside look broadly similar across the fixed-step results.Source: §3.2 · p. 5

3.3 ADAPTIVE PATCH-WISE DENOISING ROUTER

  • After each step, the router predicts patch stop probabilities; patches above the stopping threshold leave the active set, use their current clean-latent predictions, and cache their KV values for remaining patches.As the active set shrinks, later steps compute representations only for patches still being denoised.Source: §3.3 · p. 6

3.4 ON-POLICY ROUTER TRAINING

  • The router is trained with a prediction-error-based stopping surrogate and an expected denoising-cost objective, using on-policy rollouts and full-schedule outputs from the frozen generator as training targets.Replay constructs router inputs and supervision from the pretrained generator’s trajectory without additional video data or stopping annotations.Source: §3.4 · pp. 6–7, Figure 4 · p. 7

4 EXPERIMENTS

Experiments compare adaptive patch-wise denoising with uniform fixed-step sampling across video generators and world models. ForcingBooster preserves quality near four-step baselines while reducing computation, and analyses examine model coverage, allocation strategies, router settings, and video dynamics.

4 EXPERIMENTS

  • Experiments compare adaptive patch-wise denoising with uniform one- to four-step sampling across video generators and world models, then analyze allocation and router settings.The fixed-step baselines use prefixes of the original schedule for subsequent chunks.Source: §4 · p. 7, §4.2 · p. 8

4.1 IMPLEMENTATION DETAILS

  • The main Causal Forcing 1.3B evaluation generates 81-frame, 480 × 832 videos at 16 FPS and trains only the router for 500 steps, keeping pretrained components and the denoising schedule fixed.Comparisons use the four-step sampler as baseline, with all methods using four steps for the first chunk and matched conditions and noise.Source: §4.1 · p. 7, §4.2 · p. 8

4.2 MAIN RESULTS

  • 1.3× end-to-end speedup and about 50% fewer patch denoising steps, with Quality and Total close to the four-step baseline and Semantic almost unchanged.At similar runtime to uniform two-step sampling, it scores higher on all three metrics; it also avoids complex-texture artifacts seen in uniform one-, two-, and three-step baselines.Source: Table 1 · p. 8, §4.2 · p. 8

4.3 FURTHER EVALUATION

  • Across additional models, ForcingBooster keeps Quality and Total close to four-step baselines and exceeds uniform one-step sampling on both metrics in every case.The qualitative comparison reports visual quality comparable to four-step results and visible artifacts in uniform one-step sampling.Source: Table 2 · p. 8, §4.3 · p. 8, Figure 5 · p. 9

4.4 PATCH-WISE ALLOCATION

  • At comparable runtimes, ForcingBooster achieves a higher Total score than Random and Chunk-wise router.Random indicates that exit locations matter beyond step-count distribution; Chunk-wise results support varying steps within a chunk to preserve quality.Source: Table 3 · p. 9, §4.4 · p. 9

4.5 ROUTER ANALYSIS

  • Router analysis finds that cost weights from 0.015 to 0.030 enable substantial savings with low LPIPS, while larger weights yield smaller gains and greater deviation; higher stopping thresholds raise runtime without consistently lowering LPIPS.Figure 7 reports LPIPS against the four-step output and full-video runtime normalized by the four-step baseline.Source: Figure 7 · p. 10, §4.5 · p. 10
  • Lower-motion videos achieve larger speedups, approaching one-step runtime, with up to approximately 1.5× speedup or about 75% fewer patch denoising steps.Videos are grouped by optical-flow magnitude between adjacent frames.Source: §4.5 · p. 10

5 CONCLUSION

ForcingBooster dynamically allocates denoising steps by patch, stopping simple patches early while refining complex ones. It accelerates generation with negligible quality degradation and is compatible with most few-step autoregressive diffusion models using lightweight routing and modest training.

5 CONCLUSION

  • ForcingBooster dynamically allocates denoising steps by patch, stopping simple regions early while giving complex regions additional refinement.The framework is compatible with most existing few-step autoregressive diffusion models and requires a lightweight router and modest training, without fine-tuning the heavyweight generator or using additional video data.Source: §5 · p. 10

Related papers

Loading 2610.12625v1…