Source-linked AI summary
Speculative Decoding for Autoregressive Video Generation
Yuezhou Hu, Jintao Zhang
TL;DR
Autoregressive video speculative decoding lacks token-level verification because video blocks are continuous tensors. SDVG uses image-quality routing between a 1.3B drafter and 14B target, achieving near-target quality with substantial speedups while accepting distributional shift and proxy limitations.
Problem
Video blocks lack token-level distributions, leaving it unclear how to adapt exact speculative decoding verification from LLMs to autoregressive video generation.
Method
SDVG is a training-free, architecture-preserving framework that routes each video block between a drafter and target using a fixed image-quality threshold.
Results
98.1% target-only VisionReward quality (0.0773 vs. 0.0788) is achieved at 1.59× speedup, while 2.09× speedup retains 95.69% quality and remains 17.1% above draft-only.
Takeaways & Limitations
A fixed threshold provides a smooth quality–speed Pareto frontier, enabling reward-guided inference-time compute allocation for autoregressive video generation.
Takeaways & Limitations
SDVG accepts distributional shift toward the drafter, and ImageReward misses temporal consistency and motion quality because it evaluates frames independently.
Abstract
from arXiv · showhide
Autoregressive video diffusion is emerging as a promising paradigm for streaming video synthesis, with step distillation serving as the primary means of accelerating inference. Whether speculative decoding, the dominant acceleration strategy for large language models, can be effectively adapted to autoregressive video generation remains an open question, because video blocks are continuous spatiotemporal tensors with no token-level distribution for exact rejection sampling. We introduce SDVG, which brings speculative decoding to block-based autoregressive video diffusion by replacing token verification with an image-quality router. A 1.3B drafter proposes candidate blocks via four denoising steps; each block is VAE-decoded and scored by ImageReward using worst-frame aggregation--taking the minimum per-frame reward to catch single-frame artifacts that averaging would mask. Blocks scoring above a fixed threshold tau are accepted into the 14B target's KV cache; the rest are regenerated by the target. Two additional design choices prove critical: the first block is always force-rejected to anchor scene composition, and tau serves as a single knob that traces a smooth quality-speed Pareto frontier. On 1003 MovieGenVideoBench prompts (832x480), SDVG retains 98.1% of target-only VisionReward quality (0.0773 vs. 0.0788) at a 1.59x speedup with tau=-0.7, and reaches 2.09x at 95.7% quality retention--while consistently outperforming draft-only generation by over +17%. The framework is training-free, requires no architectural changes, and can be seamlessly integrated into existing autoregressive video generation pipelines.
1 Introduction
Autoregressive video generation enables streaming but remains costly with large models, motivating SDVG to route video blocks between small and large models despite the lack of token-level verification.
- Autoregressive models generate video block by block through a shared KV cache, enabling frames to stream before the full sequence finishes.
- 14B autoregressive video models remain computationally demanding, while 1B-scale models cost less but produce lower quality.
- Qualitative comparisons place SDVG between draft-only speed and target-only quality, while closely matching the target visually at 1.59× faster inference.
- SDVG addresses the central challenge by using a small drafter and large target to route each video block according to image quality.
- 98.1% target-only VisionReward quality (0.0773 vs. 0.0788) is achieved with a 1.59× speedup on 1003 MovieGenVideoBench prompts.
- Three video-specific choices guide routing: a fixed threshold, mandatory first-block regeneration, and worst-frame scoring.
2 Background
SDVG builds on autoregressive video generation, hierarchical model cooperation, and speculative decoding, replacing exact token verification with image-quality routing for continuous video blocks.
- Diffusion video inference has primarily been accelerated through step distillation and GPU kernel optimization, which are orthogonal to SDVG.
- Autoregressive video models generate future blocks causally, with recent methods addressing training–inference mismatch and exposure bias.
- Prior hierarchical methods split denoising steps or route whole requests, but were not designed for block-level autoregressive video.
- Classical speculative decoding pairs a small drafter with a large target that verifies tokens while preserving the target distribution exactly.
- Because video blocks are continuous tensors without token distributions, SDVG replaces exact rejection sampling with an image-quality model as a block-level proxy.
3 Method: SDVG
SDVG routes drafter-proposed video blocks to either acceptance or target regeneration using a quality score, while preserving scene consistency through fixed thresholding, worst-frame scoring, and first-block rejection.
- SDVG maps each block’s quality score to accept or reject, balancing video quality against inference speed.The thresholded routing policy commits accepted drafts to the target cache and regenerates rejected blocks with the target.
- Rejected drafts are regenerated by the target from the same initial noise, while the VAE decode cache is restored to preserve temporal consistency.Accepted drafts are committed to the target KV cache and emitted directly.
- A fixed threshold τ provides a simple, calibration-free quality–speed control for routing decisions.In SDVG, τ is calibrated offline and remains a fixed scalar during inference.
- Worst-frame aggregation scores each block by its minimum per-frame reward, exposing severe single-frame artifacts that mean scoring can hide.The reward is computed over decoded frames, with R(f, p) denoting the reward for frame f under prompt p.
- The first block is always regenerated by the target because it establishes scene composition, subjects, and visual style without prior KV context.This prevents potentially irreversible layout errors from propagating through subsequent blocks.
4 Experiments
SDVG is evaluated on MovieGenVideoBench using paired 14B target and 1.3B drafter models, with ImageReward-based routing across autoregressive video blocks. Experiments show a smooth quality–speed tradeoff, while reward-guided and minimum-frame routing outperform their ablated alternatives.
- Experimental Setup: Experiments use Wan2.1-based 14B target and 1.3B drafter models, with four denoising steps per block and ImageReward scoring of decoded draft frames.Both models share a causal attention backbone with KV caching and are evaluated at 832 × 480 resolution.
- Experimental Setup: 1003 MovieGenVideoBench prompts measure VisionReward quality and wall-clock efficiency against draft-only and target-only baselines.Speedup is reported relative to target-only generation, while acceptance excludes the always-rejected first block.
- Main Results: 98.10% target-only VisionReward retention (0.0773 vs. 0.0788) is achieved at 1.59× speedup with τ=−0.7.Relaxing the threshold reaches 2.09× speedup at 95.69% quality retention, with diminishing speed returns beyond τ=−1.5.
- Main Results: 73.1% to 78.0% draft acceptance reduces VisionReward only from 0.0773 to 0.0764, identifying τ ∈[−1.0, −0.7] as a favorable operating regime.Quality drops more noticeably beyond 78% acceptance as lower-scoring drafts are admitted.
- Ablation Studies: Random routing lowers VisionReward to 0.0706, compared with 0.0773 for reward-guided SDVG at τ=−0.7.The matched-acceptance ablation indicates that the ImageReward signal is load-bearing for selective regeneration.
- Ablation Studies: Average-frame scoring reaches 0.0755 at 78.4% acceptance, below min-frame scoring at 0.0773 and 73.1% acceptance.Minimum aggregation catches single degraded frames that averaging can mask.
5 Limitations
SDVG has limitations involving distributional shift, proxy-based image quality scoring, and wasted computation for rejected drafts.
- Limitations: SDVG accepts a distributional shift toward the drafter, and stricter thresholds reduce the gap only at the cost of speedup.Unlike exact-rejection LLM speculative decoding, the framework does not preserve the target distribution exactly.
- Limitations: ImageReward scores frames independently and misses temporal consistency and motion quality, motivating a dedicated video-block quality model.The current router therefore relies on a text-image reward proxy for video-block decisions.
- Limitations: Rejected blocks waste drafter computation and VAE decoding, including forced rejection of block 0.Batching or speculative VAE decoding could reduce this overhead.
6 Conclusion
SDVG uses reward-guided speculative decoding for autoregressive video diffusion, combining forced first-block regeneration and worst-frame scoring to achieve target-like quality with faster inference.
- 98.1% target quality is achieved at 1.59× speedup with SDVG.
- Forced first-block regeneration and worst-frame quality scoring address block-level video speculative decoding challenges.
- A single fixed threshold traces a smooth quality–speed Pareto frontier, reaching up to 2.09× speedup while remaining above draft-only quality.
- SDVG applies to Self-Forcing-style autoregressive video models with drafter–target pairs, enabling reward-guided inference-time compute allocation.