PaperlayerView exact revision on arXiv

Source-linked AI summary

TGuide: Faster Autoregressive Video Diffusion via Historical Trajectory Guidance

Zhuo Chen, Bin Zou, Fanyue Wei, Lixin Duan, Angela Yao, Wen Li

arXiv:2610.14657v1cs.CV

TL;DR

Four-step autoregressive video diffusion remains costly, and large denoising gaps challenge feature caching and lightweight prediction. TGuide uses historical trajectories to guide velocity prediction, combining spatial retrieval and correction with self-rollout training. Across three generators it preserves quality while accelerating generation, including longer-than-training Self Forcing videos.

  • Problem

    Even after distillation to four denoising steps, each step requires an expensive DiT evaluation, while large denoising gaps make feature reuse and lightweight prediction challenging.

    Source: §1 · p. 2, §3.2 · p. 5

  • Method

    TGuide predicts velocity with an Anchor-Transport-Correct predictor that retrieves and corrects historical features, trained through self-rollout on-policy distillation.

    Source: §1 · pp. 2–3

  • Results

    Across Self Forcing, Causal Forcing, and HY-WorldPlay, TGuide preserves generation quality; on 333-frame Self Forcing generation it achieves a 1.89× denoising speedup and exceeds the original 4-step model’s VBench Score.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    On Self Forcing, TGuide generalizes from 81-frame training videos to 333-frame generation, approximately four times the training length.

    Source: §5 · p. 10

Abstract

from arXiv · show

Can autoregressive video diffusion be accelerated further when only four denoising steps remain? Existing feature caching approaches struggle in this regime, as large feature changes between denoising timesteps undermine both direct reuse and lightweight predictors. We observe that features across video chunks remain strongly correlated at matching denoising timesteps, offering a complementary source of guidance. Building on this insight, we introduce TGuide, a framework that accelerates autoregressive video diffusion through a lightweight Anchor-Transport-Correct (ATC) predictor guided by historical sampling trajectories. Our ATC combines two complementary references: the current chunk's preceding-step features preserve its spatial structure, while the previous chunk's same-timestep features provide information at the target denoising stage. By integrating these references through spatial retrieval and adaptive correction, ATC improves velocity prediction accuracy despite the large gaps between denoising steps. To mitigate error accumulation across both denoising steps and video chunks, we train the predictor on its own autoregressive rollouts, allowing it to adapt to states shaped by its own earlier predictions. An optional confidence-aware scheduler further adjusts the quality-efficiency trade-off. Experiments on Self Forcing, Causal Forcing, and HY-WorldPlay show that TGuide preserves generation quality while achieving denoising speedups of up to $1.89\times$ over the original 4-step generators. On Self Forcing, TGuide generalizes from 81-frame training videos to 333-frame generation, approximately four times the training length, while exceeding the original model's VBench score.

1 INTRODUCTION

Four-step autoregressive video diffusion remains costly, while large denoising gaps make feature reuse and lightweight prediction difficult. TGuide combines current-chunk features with retrieved historical trajectories, and experiments report quality-preserving acceleration across three generators.

1 INTRODUCTION

  • Existing caching approaches struggle with four-step sampling because large denoising gaps weaken direct reuse and reduce lightweight predictors’ accuracy.Direct cross-chunk reuse also misses motion-induced feature differences.Source: §1 · p. 2
  • Self-rollout on-policy distillation trains ATC on states shaped by its own earlier approximations, while an optional confidence-aware scheduler adjusts the quality–efficiency trade-off.Ablations support the effectiveness of historical feature retrieval, correction, and self-rollout training.Source: §1 · pp. 2–3
  • 1.89× denoising speedup on 333-frame Self Forcing generation, after 81-frame training, while exceeding the original 4-step model’s VBench Score.The generation length is approximately four times the training length.Source: §1 · pp. 2–3
  • Experiments across Self Forcing, Causal Forcing, and HY-WorldPlay report acceleration while preserving generation quality, with favorable quality–efficiency trade-offs across three generators.The reported results also include generalization to videos longer than those used for training.Source: §1 · pp. 2–3
  • Four-step generators retain lower feature similarity across denoising timesteps than 50-step teachers, while cross-chunk similarity at matching timesteps remains largely preserved.Optical-flow alignment further improves cross-chunk similarity.Source: Figure 2 · p. 3
  • TGuide’s Anchor-Transport-Correct predictor combines current-chunk preceding-step features with retrieved, same-timestep features from the previous chunk to refine velocity prediction.Transport retrieves spatially relevant historical information, and Correct selectively incorporates it into the anchor prediction.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work reduces autoregressive video-generation cost through few-step distillation and feature caching, including reuse across chunks. TGuide instead retrieves spatially corresponding historical information to guide prediction when direct reuse is unreliable.

Autoregressive Video Diffusion

  • Autoregressive video diffusion research spans few-step distillation, longer-horizon context management, and action-controlled interactive generation.The cited work includes CausVid, Self Forcing, Causal Forcing, LongLive, and HY-WorldPlay.Source: §2 · p. 3

Feature Caching for Diffusion Transformers

  • Feature-caching methods reuse or predict intermediate features, but four-step sampling’s large denoising gaps make direct reuse and lightweight prediction more challenging.DisCa uses a lightweight predictor, while C3ache and X-Cache reuse residuals at matching denoising stages.Source: §2 · pp. 3–4
  • Unlike direct residual reuse across chunks, ATC retrieves and adapts spatially corresponding historical information to guide current velocity prediction.Matching denoising stages alone does not ensure spatial correspondence when motion displaces historical content.Source: §2 · p. 4

3 METHOD

TGuide accelerates four-step autoregressive video diffusion by predicting selected DiT outputs with historical guidance. Its ATC predictor combines current-chunk anchors with previous-chunk features, while optional scheduling and self-rollout training address prediction reliability.

3.1 PRELIMINARIES

  • Generating K chunks ordinarily requires K · N full DiT evaluations; this work considers already-distilled autoregressive generators with N = 4 and replaces selected evaluations with approximate velocities.Feature caching can use direct reuse, extrapolation, or a lightweight predictor, but larger denoising gaps make these approximations less accurate.Source: §3.1 · p. 4

3.2 PROBLEM SETUP AND MOTIVATION

  • Four-step sampling creates large denoising gaps that weaken within-chunk feature reuse, while previous-chunk features at matching timesteps remain comparable and complement the current chunk’s anchor.The previous chunk’s features cannot simply be copied because motion and viewpoint changes may displace content or reveal new regions.Source: §3.2 · p. 5

3.3 ANCHOR-TRANSPORT-CORRECT PREDICTOR

  • ATC evolves the current chunk’s anchor, retrieves spatially relevant features from the previous chunk, and applies a token-wise gated correction to predict hidden states.On Self Forcing, ATC uses 5.3% of full-DiT FLOPs and 6.8% of its latency; its predicted hidden states are mapped to velocity through the frozen output head.Source: §3.2 · p. 5, §3.3 · pp. 5–6

3.4 CONFIDENCE AWARE DYNAMIC SCHEDULING

  • The optional scheduler estimates prediction risk from local error, denoising span, and chunk position, accepting predictions below a threshold and otherwise invoking the full DiT.It favors more conservative acceptance decisions earlier in the rollout.Source: §3.4 · p. 6

3.5 TWO STAGE TRAINING

  • Self-rollout training fine-tunes ATC on trajectories shaped by its own predictions, reducing train-test mismatch across denoising steps and video chunks while keeping the base generator frozen.Offline regression on frozen full-DiT trajectories initializes this second training stage.Source: §3.5 · p. 7

4 EXPERIMENTS

Across Self Forcing, Causal Forcing, and HY-WorldPlay, TGuide reduces denoising computation while preserving generation quality. The experiments evaluate text-to-video and camera-conditioned generation, alongside predictor, training, and scheduling ablations.

4.1 EXPERIMENTAL SETUPS

  • Experiments compare three models with original 4-step and 2-step generators and four acceleration baselines at 832 × 480 on NVIDIA A100 GPUs.The evaluation reports TFLOPs, latency, speedup, and quality scores.Source: §4.1 · p. 7

4.2 QUALITATIVE COMPARISON

  • Across the three models, qualitative examples show TGuide better preserving vehicle structure, cleaner player-and-court depictions, and facial contours under camera motion.The comparisons are shown in Figure 4.Source: §4.2 · p. 7, Figure 4 · p. 8

4.3 QUANTITATIVE EVALUATION

  • At 333 frames, Self Forcing reaches Total Score 82.71 at a 1.89× speedup, exceeding the original 4-step model’s 82.55.This is approximately four times the 81-frame training length.Source: Table 1 · p. 8, §4.3 · p. 8
  • In camera-conditioned generation, TGuide achieves a 1.73× speedup and the best PSNR, SSIM, and LPIPS among accelerated methods against original outputs and under self-comparison.Self-comparison pairs return-path frames with corresponding frames from the initial pass.Source: §4.3 · p. 8, Table 2 · p. 9

4.4 ABLATION STUDIES

  • Replacing direct reuse with learned prediction raises the Total Score from 83.14 to 83.86 in the ATC ablation.The comparison uses 81-frame videos and a common Stage 1 offline regression protocol.Source: §4.4 · p. 8, Table 3 · p. 9
  • A suitable confidence threshold in optional dynamic scheduling achieves better quality at lower latency than fixed FPFF and FFPF schedules.The main comparisons use a fixed FPPF schedule.Source: §4.4 · p. 9
  • Stage 2 self-rollout distillation improves both DisCa and ATC, while ATC retains higher Total Scores at both video lengths and gains more at 333 frames.The passage reports generalization despite training only on 81-frame sequences.Source: §4.4 · p. 9

5 CONCLUSION

TGuide accelerates 4-step autoregressive video diffusion through historical trajectory guidance, and experiments report quality-preserving acceleration across three generators. On Self Forcing, it generalizes from 81-frame training videos to 333-frame generation while exceeding the original model’s VBench Score.

5 CONCLUSION

  • TGuide accelerates 4-step autoregressive video diffusion through historical trajectory guidance, selectively incorporating previous-chunk trajectory information to refine current predictions.Self-rollout training adapts the predictor to states shaped by its own approximations; optional confidence-aware scheduling offers flexible quality–efficiency trade-offs.Source: §5 · p. 9
  • On 333-frame Self Forcing generation, TGuide achieves a 1.89× denoising speedup while exceeding the original 4-step model’s VBench Score.Training used 81-frame videos, approximately one quarter of the generation length.Source: §5 · p. 10
  • Generative AI tools assisted with language editing, literature retrieval, and software development, but not methodology, experiment design, mathematical claims, or synthetic data.The authors say they reviewed and verified AI-assisted text, literature information, and code.Source: AI USE STATEMENT · p. 10

A.1 TEXT-TO-VIDEO EVALUATION

The evaluation specifies text-to-video and image-to-video protocols, including VBench scoring choices and image-revisit measures. Efficiency is assessed through computation, model-path latency, and speedup under matched conditions.

A.1 TEXT-TO-VIDEO EVALUATION

  • Text-to-video evaluation uses Self Forcing at 81 and 333 frames and Causal Forcing at 81 frames, with VBench scores based on Self Forcing’s extended prompts.The protocol uses 832 × 480 resolution and 16 FPS; generation prompts also serve as text references for text-dependent scoring.Source: §A.1 · p. 15
  • Full VBench aggregates 16 dimensions into Quality, Semantic, and Total Scores, whereas the eight-dimension VBench subset score is not directly comparable to full VBench scores.The Total Score combines Quality and Semantic Scores at weights of 4:1; subset experiments use 251 prompts.Source: §A.1 · p. 15

A.2 IMAGE-TO-VIDEO EVALUATION

  • HY-WorldPlay is evaluated on 50 held-out image-prompt pairs at 125 frames, using five VBench dimensions and four cyclic camera trajectories.PSNR, SSIM, and LPIPS measure either fidelity to matched original-model outputs or consistency when revisiting views within a model’s own sequence.Source: §A.2 · p. 15

A.3 EFFICIENCY METRICS

  • Efficiency is measured by average per-video TFLOPs, DiT model-path latency, and speedup, with measurements including rejected predictor attempts and full-evaluation fallbacks.KV-cache refresh, VAE decoding, and text encoding are excluded; speedup is mean baseline latency divided by mean method latency.Source: §A.3 · p. 16

A.4 COMPARISON METHODS

The comparison methods are implemented within the three autoregressive pipelines under matched conditions, using full-denoising, step-reduction, caching, and learned-prediction baselines. Additional comparisons report TGuide’s VBench and speedup standing among accelerated methods.

A.4 COMPARISON METHODS

  • Methods share evaluation prompts, random seeds, and generation settings within each base model; the first chunk uses full 4-step denoising and later chunks retain causal KV caching.C3ache and X-Cache are excluded because official implementations were unavailable at experiment time.Source: §A.4 · p. 16
  • The original 4-step baseline evaluates the full DiT at every step, while 3- and 2-step baselines reduce denoising steps without additional training.Naive Cache 2-step instead retains four updates but reuses the first-step velocity prediction at the two middle steps.Source: §A.4 · p. 16
  • TeaCache reuses block-stack residuals below a content-dependent change threshold, while TaylorSeer extrapolates cached block outputs and MotionCache selectively recomputes tokens.TaylorSeer resets feature histories at each chunk; MotionCache uses motion and accumulated feature changes to select tokens for recomputation.Source: §A.4 · pp. 16–17
  • DisCa predicts the current chunk’s hidden state from its noisy latent and preceding-step hidden state, without ATC’s cross-chunk feature-fusion branch.Its predictor capacity, training data, losses, and main optimization settings are controlled to match ATC for this comparison.Source: §A.4 · p. 17

B.1 ADDITIONAL QUANTITATIVE COMPARISONS

  • TGuide achieves the highest VBench score among accelerated methods and the largest speedup in the Self Forcing slow setting; on Causal Forcing it has the highest score at latency comparable to DisCa.The comparison uses dynamic K08 scheduling for Self Forcing and fixed FPPF for Causal Forcing.Source: Table 5 · p. 17, §B.1 · p. 17

B.2 FULL VBENCH RESULTS ON SELF FORCING

The full VBench results report per-dimension and aggregate scores for Self Forcing at two video lengths. DisCa and TGuide use fixed FPPF in this comparison.

B.2 FULL VBENCH RESULTS ON SELF FORCING

  • DisCa and TGuide are both evaluated with the fixed FPPF schedule in the reported comparison.The table includes results for both video lengths.Source: §B.2 · pp. 17–18
  • Table 6 reports individual dimension scores and three aggregate scores for Self Forcing at 81 and 333 frames.Dimension scores are raw values multiplied by 100; aggregate scores are normalized weighted percentages.Source: §B.2 · p. 17, Table 6 · p. 18
  • Quality, Semantic, and Total Scores are normalized weighted percentages, and higher values are better across the reported metrics.Bold marks the best accelerated method for each frame setting; aggregate rankings use scores before final rounding.Source: Table 6 · p. 18

B.3 ADDITIONAL QUALITATIVE RESULTS

On 20-second Self Forcing sequences, TGuide maintains recognizable subjects and coherent scenes at 1.89× denoising speedup, while several competing methods show visual degradation in later frames.

B.3 ADDITIONAL QUALITATIVE RESULTS

  • At 1.89× denoising speedup, TGuide preserves recognizable subjects and coherent scene structure in later frames of 20-second Self Forcing sequences.The bear retains detailed fur and a clear outline, while the forest path remains well defined as the viewpoint advances.Source: §B.3 · p. 18
  • Several competing methods develop texture artifacts, lose detail, or drift in appearance by 20 seconds, most visibly in workshop, bear, and nighttime street examples.Figure 5 shows frames at 0, 10, and 20 seconds under matched generation conditions, with denoising speedups indicated for each method.Source: §B.3 · p. 18, Figure 5 · p. 19

C CONFIDENCE-AWARE DYNAMIC SCHEDULING

Confidence-aware scheduling adjusts the quality-efficiency trade-off by estimating prediction risk and falling back to the full DiT when needed. The predictor is trained first on offline teacher features and then on student rollouts.

C CONFIDENCE-AWARE DYNAMIC SCHEDULING

  • At inference, the confidence head converts estimated candidate error into propagation-weighted risk; candidates above threshold trigger full-DiT fallback, whose added cost is included in efficiency measurements.The risk weighting accounts for the affected denoising span and gives greater weight to errors with more subsequent chunks in which to propagate.Source: C CONFIDENCE-AWARE DYNAMIC SCHEDULING · p. 20
  • K08 improves quality over FPFF and FFPF while reducing latency; against fixed FPPF, it gives higher quality at a lower acceleration level.K10 offers a faster operating point with a modest quality trade-off.Source: Table 7 · p. 21, C CONFIDENCE-AWARE DYNAMIC SCHEDULING · p. 21

D.1 STAGE 1: OFFLINE FEATURE REGRESSION

  • Stage 1 trains the predictor against full-DiT final hidden states and velocities, combining hidden-state and velocity MSE with weights 0.1 and 1, respectively.For Self Forcing and Causal Forcing, offline supervision uses teacher rollouts; for HY-WorldPlay, it uses 125-frame teacher rollouts.Source: §D.1 · p. 21

D.2 STAGE 2: SELF-ROLLOUT ON-POLICY DISTILLATION

  • Self Forcing and Causal Forcing Stage 2 varies the rollout exit step so ATC encounters different prediction depths and historical features produced by its earlier predictions.DMD supervision uses a frozen non-causal real-score teacher and a trainable non-causal fake-score critic on generated videos.Source: §D.2 · pp. 21–22
  • HY-WorldPlay Stage 2 fine-tunes ATC on student rollouts using direct full-teacher supervision, avoiding score-network training and alternating updates.The teacher evaluates the same noisy latent reached by the student, while student outputs drive subsequent sampling updates.Source: §D.2 · p. 22

D.3 DATASET

Training data and rollout procedures differ across Self Forcing and Causal Forcing versus HY-WorldPlay.

D.3 DATASET

  • Self Forcing and Causal Forcing use VidProM prompts, with offline full-model trajectories for Stage 1 and online trajectories from the current predictor for Stage 2.Source: §D.3 · p. 22
  • HY-WorldPlay training uses VBench I2V image-prompt pairs disjoint from evaluation pairs, combined with camera trajectories covering four opposing motion patterns.The patterns move forward then backward, left then right, tilt up then down, or pan left then right.Source: §D.3 · p. 22

E DETAILS OF THE MOTIVATING EXPERIMENTS

The motivating experiments compare feature similarity across denoising steps and chunks, while predictor selection evaluates source layers and block initializations.

E DETAILS OF THE MOTIVATING EXPERIMENTS

  • Optical-flow alignment increases the 4-step cross-chunk mean by 0.0138 for Self Forcing and 0.0164 for Causal Forcing.The diagnostic compares token cosine similarity across consecutive denoising timesteps and across chunks at matching timesteps; a separate 165-frame rollout experiment applies FRRF to one chunk at a time and measures full-video PSNR, SSIM, and LPIPS.Source: E DETAILS OF THE MOTIVATING EXPERIMENTS · p. 23

F PREDICTOR BLOCK SELECTION ON SELF FORCING

F.1 SOURCE LAYER AND INITIALIZATION

  • Layer 17 has the lowest validation velocity MSE (0.046459) among single-block candidates.The study compares source layers using validation velocity MSE on held-out teacher states; layer 17 motivates subsequent Self Forcing predictor training.Source: §F.1 · p. 23, F PREDICTOR BLOCK SELECTION ON SELF… · p. 23
  • With layer-17 K/V fixed, initializing from the complete teacher block yields the lowest validation velocity MSE among four tested initialization schemes.Table 9 reports these initialization controls.Source: §F.1 · p. 23

Related papers

Loading 2610.14657v1…