PaperlayerView exact revision on arXiv

Source-linked AI summary

Speculative Forcing for Efficient Autoregressive Video Generation

Hyunwook Choi, Seungryong Kim, Dongyoon Han, Sangdoo Yun

arXiv:2610.22347v1cs.CV

TL;DR

Few-step autoregressive video generators run every token through the full denoising schedule despite tokens needing different amounts of refinement. Speculative Forcing predicts which intermediate token estimates can be accepted early, using training-free or learned verifiers. It maintains quality across short-video, long-video, and world generation while reducing computation, though the acceleration depends on the generator and domain.

  • Problem

    Few-step autoregressive video generators denoise every token to the final step even though different regions may require different amounts of refinement.

    Source: §1 · p. 2

  • Method

    Speculative Forcing treats intermediate clean estimates as drafts and uses training-free or learned verifiers to accept converged tokens early while retaining their cached keys and values as context.

    Source: §1 · p. 2

  • Results

    Across short videos, long videos, and controllable world rollouts, Speculative Forcing maintains quality while reducing computation; on LongLive, it matches four-step quality with approximately 1.7 denoising steps per token.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The results support convergence-based acceptance instead of using a shared denoising depth for every token.

    Source: §5 · p. 9

    Acceleration depends on the generator and domain: sensitive backbones or challenging domains may allow fewer early acceptances and limit speedup.

    Source: §D.1 · p. 19

Abstract

from arXiv · show

Few-step autoregressive video generators enable streaming generation, yet they run every token through the final denoising step, regardless of how much refinement it needs. We find this uniform computation is often unnecessary: many tokens already closely match their eventual full-step outputs in the initial steps, while others require refinement until the end. We introduce Speculative Forcing, a framework that treats intermediate clean estimates as drafts and predicts when individual tokens can be safely accepted early. Once accepted, a token is frozen as its final output and undergoes no further denoising, while its cached keys and values remain available as context for the remaining tokens. We explore two alternative verifier designs: a training-free one based on similarity to clean content and stability across steps, and a learned one that reads the frozen generator's intermediate features to estimate each token's remaining gap to its full-step output. Across short-video, long-video, and world generation, Speculative Forcing preserves the quality of full-step sampling while substantially reducing computation. On LongLive, it matches the four-step generator with ~1.7 denoising steps per token, running over 1.5x faster at 41.7 and 37.8 fps. On Lingbot-World 2.0, where camera motion continually changes the visible content, it maintains quality with ~2.5 denoising steps per token and runs 1.35x faster at 24.5 fps.

1 INTRODUCTION

Few-step autoregressive generators apply the full denoising schedule to every token, although tokens can require different amounts of refinement. Speculative Forcing predicts which token estimates can be accepted early; experiments report maintained quality with reduced computation across video and world generation.

1 INTRODUCTION

  • Different video regions and world-model tokens converge at different rates, raising whether every token needs denoising through the final step.Familiar content may settle early, while newly appearing structures or fine details can require further refinement.Source: §1 · p. 2
  • Speculative Forcing treats intermediate clean estimates as drafts, accepts sufficiently converged tokens early, and freezes them while retaining their cached keys and values as context.The remaining tokens continue denoising, so the verifier must predict whether a draft is close to its full-step output without running the remaining steps.Source: §1 · p. 2
  • Training-free and learned verifiers predict token-wise convergence using clean-content similarity and draft stability, or intermediate generator features, without modifying generator weights.The learned verifier uses a shallow head to estimate each token’s remaining gap to the full-step output.Source: §1 · p. 2
  • On LongLive, Speculative Forcing maintains four-step quality on short and long videos with approximately 1.7 denoising steps per token, running over 1.5× faster at 41.7 and 37.8 fps.On Lingbot-World 2.0, it maintains quality with approximately 2.5 denoising steps per token and runs 1.35× faster at 24.5 fps.Source: §1 · p. 2

2 RELATED WORK

Related work establishes few-step autoregressive video generation and adaptive denoising as the setting for Speculative Forcing. The method differs from approaches that vary computation by frame, chunk, region, or schedule step by committing individual tokens at different steps.

2 RELATED WORK

  • LongLive, Rolling Forcing, and Self-Forcing++ extend few-step autoregressive generation to long streaming videos, while related work also develops interactive world models conditioned on actions or camera trajectories.The cited long-video methods are LongLive, Rolling Forcing, and Self-Forcing++ (Yang et al., 2025a; Liu et al., 2025b; Cui et al., 2025).Source: §2 · pp. 2–3
  • Caching and spatially adaptive methods reuse or redistribute denoising computation, but still carry every token to the schedule’s end and bound skipped-computation error through periodic full updates.The passage also notes that substituting a previous output weakens as schedules shorten.Source: §2 · p. 3
  • Unlike methods assigning one denoising depth to a frame or chunk, Speculative Forcing commits intermediate clean drafts per token while retaining their cached keys and values as context.It draws on speculative decoding draft verification without adopting its exact acceptance rule.Source: §2 · p. 3

3 METHOD

Speculative Forcing uses token-wise draft acceptance to allocate denoising only to estimates that have not converged. It combines a training-free verifier based on content similarity and draft stability with a learned verifier that predicts each token’s remaining gap from generator features.

3 METHOD

  • The method accepts individual token drafts at different steps and uses two verifiers to predict convergence from latent estimates or intermediate generator features.Committed tokens retain cached keys and values as context for tokens that continue denoising.Source: §3 · p. 3

3.1 PRELIMINARIES

  • Few-step autoregressive generators denoise each chunk from Gaussian noise conditioned on clean history, producing clean estimates at each step before returning the final estimate.In Self Forcing, intermediate clean estimates receive distribution-matching supervision, although standard inference runs all tokens through all K steps.Source: §3.1 · pp. 3–4

3.2 TOKEN-WISE COMMITMENT

  • At each step, accepted tokens are fixed as output and excluded from later denoising forwards, while their cached keys and values remain available to active tokens.Rejected tokens are re-noised and refined; tokens still active at step K are committed then.Source: §3.2 · p. 4
  • Commitment reduces denoising token evaluations from KN to the sum of token commit steps, with an additional N-token clean pass.The completed chunk combines token estimates accepted at potentially different steps.Source: §3.2 · p. 4
  • The convergence gap compares an intermediate estimate with the full-step output under the same initial noise and conditioning, normalizing the distance by the median token norm.A draft meets the oracle convergence criterion when its gap is at most ϵ; the full-step output is unavailable during early commitment.Source: §3.2 · pp. 4–5

3.3 CONVERGENCE VERIFICATION

  • Verifier scores are smoothed across neighboring tokens before thresholding to encourage coherent acceptance decisions across space and time.The method uses evidence in the current draft and the features that produced it to assess convergence before the full-step output is available.Source: §3.3 · p. 5

3.3.1 TRAINING-FREE VERIFIER

  • The training-free verifier uses similarity to clean history or reference content at the first step, then uses changes between consecutive drafts at later steps.Draft similarity and stability are proxies for convergence, and drafts meeting the corresponding thresholds are accepted.Source: §3.3.1 · p. 5

3.3.2 LEARNED VERIFIER

  • The learned verifier predicts the convergence gap from frozen-generator features using a shallow head, and accepts tokens whose smoothed predicted gap meets a tolerance.At inference, the head predicts gaps for active tokens; adjusting ϵ controls the speed-quality trade-off without retraining or fine-tuning.Source: §3.3.2 · pp. 5–6

4 EXPERIMENT

The experiments test whether token-wise speculative acceptance can reduce denoising while preserving generation quality across video and world-model settings. Ablations examine verifier signals, acceptance granularity, and tolerance.

4.1 EXPERIMENTAL DETAILS

  • Both generators are frozen four-step models; evaluations use LongLive video benchmarks and held-out Lingbot navigation scenes with quality, Drift, and camera-pose measures.Verifier tolerances are calibrated against an oracle convergence criterion, and throughput is measured on one NVIDIA H100.Source: §4.1 · p. 6

4.2 EXPERIMENTAL RESULTS

  • On 5-second LongLive videos, both verifiers maintain the four-step generator’s overall VBench performance at fewer than 1.7 average denoising steps per token; on long videos, training-free retains competitive quality with a small consistency loss, while learned maintains quality and consistency.On 30-second videos evaluated with VBench-Long, both verifiers use approximately 1.7 denoising steps per token on average.Source: §4.2 · p. 7, Table 1 · p. 7, Table 2 · p. 7
  • In camera-changing world rollouts, the learned verifier preserves four-step quality and consistency with comparable camera-pose error; the training-free verifier remains quality-competitive with low Drift.World generation requires more denoising than standard video generation because camera control substantially changes the visible scene.Source: §4.2 · p. 7, Table 3 · p. 7
  • Qualitative comparisons show visual quality comparable to four-step generation without noticeable degradation over time, including coherent Lingbot regions under large camera motion.On LongLive, subject identity and image quality remain intact in scenes with dynamic objects.Source: §4.2 · p. 7, Figure 4 · p. 8

4.3 ABLATION STUDIES

  • On Lingbot, tolerances ϵ = 0.20 and ϵ = 0.24 preserve four-step quality with lower Drift; larger tolerances reduce steps but degrade quality.The reported threshold sweep identifies ϵ ∈ [0.20, 0.24] as exceeding the uniform four-step baseline on quality metrics.Source: §4.3 · p. 8, Table 4 · p. 9
  • At similar mean steps, adapted motion and noise-dispersion signals yield lower quality than the learned verifier and uniform sampling; camera changes can confound temporal-difference scores.The qualitative ablation also shows the learned verifier handling distortions that challenge training-free verification.Source: §4.3 · p. 8, Figure 5 · p. 9
  • The learned verifier achieves quality comparable to oracle verification without computing a full-step reference at inference.The oracle measures true residuals by running all steps for each chunk.Source: §4.3 · p. 8, Table 5 · p. 9
  • At comparable mean denoising steps, token-wise commitment achieves higher quality and lower Drift than chunk-wise acceptance, which gives all tokens the same exit step.Premature chunk-wise acceptance can propagate errors to later chunks.Source: §4.3 · p. 9, Table 5 · p. 9, Figure 5 · p. 9

5 CONCLUSION

The conclusion presents Speculative Forcing as token-wise allocation of denoising computation, with early acceptance for converged estimates and continued refinement for others.

5 CONCLUSION

  • Speculative Forcing treats intermediate clean estimates as drafts and allocates denoising according to token-wise convergence, using verifiers without changing generator weights.Accepted tokens stop receiving denoising updates while remaining tokens continue to be refined.Source: §5 · p. 9
  • Experiments across short and long videos and controllable world-model rollouts report faster generation while maintaining quality, with comparable or lower quality variation and preserved camera-following accuracy.The conclusion states that a shared denoising depth is not necessary for every token.Source: §5 · p. 9

A IMPLEMENTATION DETAILS

The implementation preserves the generators’ configurations and denoising schedules while applying token-wise acceptance, verifier scoring, and execution optimizations.

A IMPLEMENTATION DETAILS

  • The implementation retains released backbone configurations except for denoising policies, and uses four latent frames per chunk for Lingbot versus three for LongLive.The two generators use different attention sinks and local attention windows; Lingbot also retains its camera-conditioning pathway.Source: A IMPLEMENTATION DETAILS · p. 16
  • Both generators use the same shifted four-step schedule, and a token accepted at step k receives exactly the first k schedule steps.Uniform-step baselines use the same schedule prefixes for comparison.Source: A IMPLEMENTATION DETAILS · p. 16
  • The learned verifier reads final-block hidden states without another backbone forward and uses a shared 4.11M-parameter head trained across denoising steps.The head uses layer normalization, three 3D convolutions, and average pooling followed by nearest-neighbor expansion.Source: A IMPLEMENTATION DETAILS · p. 16
  • Accepted tokens retain the keys and values computed at acceptance, while a clean pass after each chunk refreshes all tokens’ cache states.Subsequent chunks therefore attend to history states computed from the completed clean output.Source: A IMPLEMENTATION DETAILS · p. 16
  • Verifier score maps receive spatial Gaussian smoothing and temporal smoothing over the current and subsequent latent frames within each chunk.Spatial smoothing is applied independently to each latent frame.Source: A IMPLEMENTATION DETAILS · p. 16
  • Only unaccepted tokens enter later denoising steps; shared indexing, in-place KV updates, and merged chunk-clean/next-step passes reduce overhead.Baselines use the same applicable execution optimizations for throughput comparisons.Source: A IMPLEMENTATION DETAILS · p. 16

B.1 EVALUATION METRICS

The evaluation uses benchmark-specific quality and motion measures, alongside denoising-cost and throughput metrics. Verification accuracy is measured by AUROC for identifying tokens that require further refinement.

B.1 EVALUATION METRICS

  • LongLive quality uses all 16 VBench dimensions for 5-second videos and VBench-Long dimensions for 30-second videos.Lingbot-World 2.0 aggregates six quality dimensions using VBench normalization and weights, excluding temporal flickering because camera motion can change pixels between frames.Source: §B.1 · p. 16
  • Verification accuracy is AUROC for identifying tokens needing further refinement, defined by relk(i) > ϵ.Throughput is generated frames per DiT forward time on one H100, while denoising cost is average evaluations per token excluding the clean KV-cache refresh pass.Source: §B.1 · p. 17
  • Lingbot-World 2.0 camera-following accuracy is measured by relative pose error against the commanded ground-truth trajectory.Rot is mean angular error in degrees; Trans is summed translation error divided by summed commanded displacement.Source: §B.1 · p. 17

B.2 BENCHMARK CONSTRUCTION

The Lingbot-World 2.0 benchmark fixes scenes, prompts, initial images, camera trajectories, and sampling seeds across methods. Its camera-control segments span translation, rotation, and stationary motion.

B.2 BENCHMARK CONSTRUCTION

  • The benchmark contains 200 scenes with an initial image, text prompt, and camera trajectory, using identical conditioning inputs and sampling seeds across methods.Each generated video has 253 frames and five camera-control segments.Source: §B.2 · p. 17
  • Scene descriptions and initial images are generated with Qwen2.5-7B-Instruct and Z-Image-Turbo, respectively, using shared environment, subject, and perspective prompts.Verifier training scenes are separate from benchmark scenes, with potential overlap filtered by text and image similarity.Source: §B.2 · p. 17
  • The 1,000 control segments comprise 593 translation, 276 rotation, and 131 stationary segments, with actions sampled across five consecutive segments per video.Translation speeds and yaw rates are sampled from specified ranges, and the camera intrinsics remain fixed across methods.Source: §B.2 · p. 17

C.1 ANALYSIS OF VERIFICATION ACCURACY

The verification analysis compares learned and latent-based signals under oracle-labeled rollouts and matched denoising-step budgets. The learned head leads the latent signals, while token-wise redundancy and stability support the training-free verifier.

C.1 ANALYSIS OF VERIFICATION ACCURACY

  • Verification AUROC is evaluated on oracle-commitment rollouts at ϵ = 0.24, with complete four-step denoising used to obtain convergence labels.AUROC is computed within each chunk and averaged across chunks, matching the runtime ranking scope; Table B covers verifier ablations calibrated to ϵ = 0.24.Source: §C.1 · pp. 17–18
  • A per-token MLP achieves verification accuracy comparable to OursH, indicating that contextualized backbone features provide useful convergence cues.Training on commitment rollouts is no better, even when measurements use that same distribution.Source: §C.1 · p. 18
  • The learned verifier leads every latent signal, while motion-based scheduling remains below redundancy at every step and tolerance.Redundancy is useful at the first step but less so as tokens are refined; stability is stronger later but unavailable at the first step, requiring two thresholds for the training-free verifier.Source: §C.1 · p. 18

C.2 ADDITIONAL ABLATION ON OUR HEAD

The additional ablations examine head design, LongLive quantitative evaluations, and supplementary qualitative results. Spatiotemporal smoothing is associated with more stable rollouts and improved imaging quality.

C.2 ADDITIONAL ABLATION ON OUR HEAD

  • Spatiotemporal smoothing achieves lower Drift and higher imaging quality, supporting more stable Lingbot-World 2.0 rollouts.Training on commitment rollouts or replacing the convolutional head with a per-token MLP yields similar generation quality and throughput at comparable denoising steps.Source: §C.2 · p. 18

C.3 ADDITIONAL QUANTITATIVE RESULTS

  • LongLive quality and semantic dimensions are reported across 5-second and 30-second evaluations, comparing uniform schedules with Speculative Forcing at ϵ = 0.24 and ϵ = 0.28.The Surprise Forcing policy uses the maximum adjacent-frame latent cosine distance over the preceding 20 chunks; chunks below rank threshold 0.4 use only steps one and four.Source: §C.3 · p. 18

C.4 ADDITIONAL QUALITATIVE RESULTS

  • Table D reports LongLive semantic evaluation using VBench, and supplementary video materials provide additional qualitative results.Source: §C.4 · p. 18, Table D · p. 19

D.1 LIMITATIONS

The discussion bounds acceleration by generator and domain difficulty and practical overhead, while identifying condition-rich generation settings as potential applications.

D.1 LIMITATIONS

  • Acceleration varies with backbone sensitivity and domain difficulty; the method does not guarantee equal speedups across generators or domains.Source: §D.1 · p. 19

D.2 OTHER DISCUSSIONS

  • The ideal speed-up is 5/(k̄ + 1) under linear token-evaluation scaling, but a clean KV-cache refresh adds a pass and chunk-level overhead tracks the last active token.Source: §D.2 · p. 19

D.3 POTENTIAL APPLICATIONS

  • Condition-rich settings may permit early acceptance of tokens matching references, while changing regions receive further refinement; proposed applications include augmented and virtual reality.Source: §D.3 · pp. 19–20

Related papers

Loading 2610.22347v1…