PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Aligned Forcing: Streaming Video Diffusion with Differentiable Noisy History

Weiqiang Wang, Zhuokun Chen, Yusheng Dai, Boying Li, Yi Zhang, Hossein Rahmani, Qiuhong Ke, Jianfei Cai

arXiv:2609.38114v1cs.CV

TL;DR

Long-horizon streaming video diffusion must handle error accumulation beyond finite training rollouts, while history noise trades visual quality against motion. SAF aligns history noise with the current block so history remains differentiable without reconstruction. It improves long-horizon generation and efficiency, though noisy local history can still cause abrupt changes under camera motion.

  • Problem

    Finite self-rollouts do not fully capture long-horizon error accumulation, and existing methods detach history or require costly reconstruction to restore its gradients.

    Source: §1 · pp. 2–3

  • Method

    SAF aligns history noise with the current block so preceding-block K/V remain noisy and differentiable in a single causal forward pass.

    Source: §1 · p. 3

  • Results

    SAF performs best in chunkwise evaluation and competitively in framewise evaluation, with up to 1.8× faster framewise training than SGF and 49.1 FPS in chunkwise four-GPU pipelined inference.

    Source: §1 · p. 3

  • Takeaways & Limitations

    SAF improves long-horizon generation and prompt switching while reducing training time and memory relative to SGF.

    Source: §5 · pp. 9–10

    Noisy local history can still cause abrupt changes when new content enters the frame under camera motion.

    Source: §5 · p. 10

Abstract

from arXiv · show

Autoregressive video diffusion enables interactive streaming generation, but suffers from error accumulation over long rollouts. Self-rollout training reduces exposure bias, yet finite rollouts leave long-range drift unresolved. We observe that the noise level of the history key-value (K/V) representations trades visual quality against motion, and that restoring gradients through the history aligns causal training far more closely with bidirectional training. Motivated by these observations, we introduce Self-Aligned Forcing (SAF), a training scheme that aligns the history of each block with the noise level of the block being denoised. Specifically, the history is the K/V produced by preceding blocks at the same denoising stage, so all blocks at a stage can be denoised in a single forward pass under a causal mask. This keeps the noisy history differentiable, allowing future losses to optimize how it is encoded. SAF therefore avoids a separate no-gradient rollout and per-block timestep-zero recaching, training up to 1.8x faster than prior methods with lower memory. At inference, SAF achieves the highest single-GPU throughput among existing methods and keeps one history bank per stage for a multi-GPU pipeline, reaching 49.1 FPS on 4 GPUs. Experiments show superior long-horizon generation with a better balance between visual quality and motion. Project page: https://anonymous.4open.science/w/self-aligned-forcing/.

1 INTRODUCTION

Long-horizon streaming remains difficult because self-rollout does not fully capture errors accumulated beyond finite training rollouts, while history noise and gradient flow each affect the quality of future predictions. SAF aligns history and current-block noise levels so noisy history remains differentiable, avoiding separate history reconstruction while supporting efficient training and inference.

1 INTRODUCTION

  • Finite self-rollouts do not fully account for errors accumulated over long horizons, while existing partially denoised-history methods still detach history K/V.HiAR finds that aligned history noise minimizes error accumulation but lowers overall VBench quality.Source: §1 · p. 2
  • Noisy history increases motion dynamics while reducing imaging quality and temporal consistency, leaving the model unable to learn a balance between these effects.The authors link both effects to lower history SNR weakening the history’s constraint on the current block.Source: §1 · p. 2
  • Restoring gradients through history raises gradient cosine similarity to a matched-weight bidirectional reference from 0.643 to 0.972.With detachment, the key projection Wk cosine similarity drops to about 0.2, suggesting weak learning of history encoding for future retrieval.Source: §1 · p. 2
  • SAF aligns history noise with the current block, enabling one causal forward pass to keep noisy history differentiable without reconstruction.All blocks are processed at each denoising stage, and gradients are taken at one randomly sampled stage; this avoids a separate no-gradient rollout and per-block recaching.Source: §1 · p. 3
  • SAF performs best in chunkwise evaluation and competitively in framewise evaluation, with up to 1.8× faster framewise training than SGF and 49.1 FPS in chunkwise four-GPU pipelined inference.The paper also reports the highest training and inference efficiency in both settings.Source: §1 · p. 3

2 RELATED WORK

Prior approaches improve long-horizon robustness through rollout, resampling, or partially denoised histories, but generally leave history K/V detached or require extra supervision or reconstruction. SAF’s overview highlights stage-aligned differentiable history and parallel rollout as an alternative.

2 RELATED WORK

  • Prior methods improve robustness through self-generated rollouts, resampling, corrective trajectories, or intermediate denoising states, but their later-block losses generally cannot optimize earlier history encoding.Live Avatar also matches history and current-block timesteps, yet retains serial rollout and detached history.Source: §2 · p. 3
  • Video-Mirai adds future-aware auxiliary supervision, while SGF restores future-to-history gradients by re-encoding clean history from a separate no-gradient rollout.These approaches leave efficient future-to-history optimization of noisy history K/V unresolved.Source: §2 · p. 3
  • SAF’s overview shows rollout parallel across blocks but sequential across stages, with stage alignment enabling differentiable history in one block-causal pass and K/V reuse supporting multi-GPU pipelining.Reusing denoising K/V also removes per-block recaching.Source: Figure 3 · p. 4

3 METHOD

SAF aligns each block’s history K/V with its current denoising stage, allowing causal, block-parallel training with differentiable history. A clean sink anchors persistent appearance, while per-stage K/V banks support streaming inference and GPU pipelining.

3.1 OVERVIEW

  • SAF aligns history noise with the current block at every denoising step, enabling parallel self-rollout across blocks within each stage.It complements this design with a clean sink for appearance and per-stage K/V banks for GPU-pipelined inference.Source: §3.1 · p. 4

3.2 STAGE-ALIGNED DIFFERENTIABLE HISTORY

  • Stage alignment lets one block-causal forward pass produce each block’s prediction and same-stage predecessor K/V, so noisy history remains differentiable.Gradients through the history update how predecessors encode it for later blocks, providing future-to-history supervision.Source: §3.2 · pp. 4–5

3.3 BLOCK-PARALLEL SELF-ROLLOUT

  • SAF retains self-rollout to address the train–test mismatch between ground-truth inputs in bidirectional training and histories derived from the model’s earlier outputs at inference.It samples an exit stage and computes gradients only there, with earlier stages run without gradients.Source: §3.3 · p. 5
  • Block-parallel self-rollout reduces sequential depth from N × d block-level to d stage-level forward passes, making training faster and more memory-efficient.The supervised pass also supplies differentiable history, avoiding a separate rollout to construct it.Source: §3.3 · p. 5

3.4 CLEAN SINK AS A PERSISTENT ANCHOR

  • A clean sink anchors subject identity and scene layout, while noisy local history conveys scene evolution; the clean anchor stabilizes appearance without suppressing motion.All stages share the clean sink, whose K/V remains in the computation graph at the exit stage.Source: §3.4 · pp. 5–6

3.5 STREAMING INFERENCE

  • SAF streams blocks using one history K/V bank per stage, reusing denoising K/V so each block needs S forward passes rather than the S+1 required by clean-history recaching.In multi-GPU stage pipelining, each GPU stores one history bank; single-bank SAF saves memory but cannot be pipelined and can mismatch training history noise levels.Source: §3.5 · p. 6

4 EXPERIMENTS

The experiments test whether 5-second training transfers to longer and interactive video generation, comparing SAF with existing methods and examining training and inference choices. Results show strong chunkwise quality and throughput, while ablations expose trade-offs among motion, visual stability, and efficiency.

4.1 EXPERIMENTAL SETUP

  • Models trained on 5-second clips are evaluated on Interactive 60-second videos with six changing prompts and MovieGen-100s 100-second videos, alongside VBench short-clip evaluation.The long-video benchmark tests generation at 20× the training length.Source: §4.1 · pp. 6–7

4.2 COMPARISON WITH EXISTING METHODS

  • SAFM leads nearly all aggregate chunkwise metrics across short-video, interactive, and long-video generation, and is the fastest single-GPU method.LongLive has a higher Interactive ViCLIP score, but SAFM has higher Interactive Quality and MovieGen-100s Quality; SAF also supports four-GPU pipelined inference.Source: §4.2 · pp. 7–8
  • SAFM tracks changing Interactive prompts and sustains table-wiping motion and scene layout over 100 seconds, while competing methods show semantic errors or visual instability.LongLive can duplicate a cat or produce inconsistent reflections; SGF can miss prompt-conditioned motion or duplicate the human subject.Source: §4.2 · p. 8, Figure 5 · p. 8, Figure 6 · p. 9

4.3 ABLATION STUDIES

  • Removing history K/V gradients mildly reduces Quality but substantially lowers Dynamic Degree; removing sink recaching causes the largest degradation in both metrics.For Fixedt single-bank inference, larger t generally improves Quality and Dynamic Degree but reduces Motion Smoothness and Imaging Quality; Mix is chosen as a more balanced SAFS policy.Source: Table 2 · p. 8, §4.3 · p. 9
  • Across SDF schedules, aligned inference gives the highest Quality, and SAF achieves the highest Quality under every inference policy.With aligned inference, SAF has the best overall Quality and dynamic degree, although its imaging quality is lowest in the table.Source: Figure 4 · p. 8, §4.3 · p. 9
  • SAF trains 1.8× faster than SGF in the framewise setting, where its 137-second cycle nearly matches its 133-second chunkwise cycle.Block-parallel rollout reduces sequential depth to the number of stages, while the supervised forward pass provides differentiable history without a construction rollout or per-block recaching.Source: §4.3 · p. 9, Table 3 · p. 9

5 CONCLUSION

The conclusion presents stage-aligned, differentiable history as trainable memory for streaming video diffusion, with improved long-horizon generation and faster training. The appendices describe SDF as a two-pass schedule generalization and detail inference cache and bank operations.

5 CONCLUSION

  • SAF aligns each block’s history with its denoising stage, enabling a single causal forward pass per stage with noisy, differentiable history and no separate history rollout or per-block timestep-zero recaching.The conclusion reports improved long-horizon generation and prompt switching, training up to 1.8× faster than SGF with lower memory, and efficient multi-GPU pipelined inference.Source: §5 · pp. 9–10

A SELF-DIFFUSION FORCING

  • SDF generalizes SAF by decoupling the history-write stage from the supervised query stage; aligned, fixed, and random policies specify how history stages are chosen.Unlike SAF, SDF runs all rollout stages and uses the sampled query stage to select recorded states for gradient-enabled replay.Source: A SELF-DIFFUSION FORCING · p. 15
  • When history and query stages differ, SDF uses a detached serial rollout followed by a differentiable replay with a dual-stream causal mask.The clean sink is recached for both passes; gradients pass through sink, context, and query K/V, but not the recorded trajectory.Source: A SELF-DIFFUSION FORCING · pp. 15–16

B ALGORITHM DETAILS

  • Inference preserves the sink in each local history window and evicts the oldest non-sink entries; multi-bank execution processes blocks through stage-specific histories.The single-GPU multi-bank schedule respects the same dependencies as the multi-GPU pipeline and produces identical videos from the same noise.Source: B ALGORITHM DETAILS · p. 16
  • SAFM keeps a separate history K/V bank per stage, whereas SAFS uses one bank and commits K/V from a sampled stage or the Mix policy.The Mix policy samples noise levels 250, 500, and 750 with probabilities 0.5, 0.25, and 0.25; the bank retains one entry per block.Source: B ALGORITHM DETAILS · p. 17

C.1 IMPLEMENTATION

Implementation uses a causal Wan2.1 generator trained with DMD, fixed video and denoising settings, and a shared optimization schedule. Inference and attention settings differ between chunkwise and framewise generation.

C.1 IMPLEMENTATION

  • The generator is causal Wan2.1-T2V-1.3B trained with DMD; a 14B model supplies the real score and a bidirectional 1.3B model supplies the fake score.Training samples contain 21 latent frames corresponding to 5 seconds at 832 × 480 and 16 FPS, with four denoising stages at noise levels 1000, 750, 500, and 250.Source: §C.1 · p. 17
  • Chunkwise blocks contain three latent frames with a 12-frame attention window and three-frame sink; framewise blocks contain one latent frame with a 21-frame window and four-frame sink.The sink stays fixed and is recached at timestep zero; remaining history uses noisy K/V from the denoising trajectory. SAFM uses separate stage banks, while SAFS uses one Mix-policy bank.Source: §C.1 · pp. 17–18
  • Training runs for 800 steps on eight GB300 GPUs with batch size four per GPU, using AdamW and an exponential moving average from step 200.The generator learning rate is 2 × 10^-6; the fake score model uses 4 × 10^-7 and is warmed up for 10 updates before generator updates begin once every five fake-score updates.Source: §C.1 · p. 18

C.2 BENCHMARKS AND METRICS

The evaluation spans short-video quality, interactive prompt switching, and long-horizon generation, using established protocols and deterministic seeds. Baselines are re-evaluated with official weights where available, with framewise coverage limited by released checkpoints.

C.2 BENCHMARKS AND METRICS

  • Interactive and MovieGen-100s test generation at 12× and 20× the 5-second training length, respectively.The benchmarks differ in prompt structure: Interactive has six prompts per video, while each MovieGen-100s video uses one extended prompt.Source: Table 4 · p. 18
  • VBench evaluates five videos for each of 944 prompts; Interactive evaluates 60-second videos across six successive 10-second prompts, and MovieGen-100s evaluates 100-second videos for 128 prompts.Interactive prompt alignment is segment-level ViCLIP similarity; Interactive and MovieGen-100s Quality use VBench-Long scores over 2-second clips.Source: Table 4 · p. 18, §C.2 · p. 19

C.3 BASELINES AND THROUGHPUT

  • Existing methods are re-evaluated with official weights and released inference settings; only Causal-Forcing and SGF have framewise checkpoints.Baseline throughput comes from previously reported results, while SAF throughput is measured on H100 GPUs for single-GPU and four-GPU pipelined inference.Source: §C.3 · p. 19, Table 5 · p. 19

D ADDITIONAL QUANTITATIVE RESULTS

The additional results show that SAFM performs strongly across framewise, per-dimension, interactive, and long-video evaluations, while consistency and motion often trade off. Its strengths vary by benchmark and inference setting.

D ADDITIONAL QUANTITATIVE RESULTS

  • SAFM has the highest throughput, VBench Quality, and Interactive ViCLIP in the framewise comparison, while SGF leads several other aggregate scores.SAFM also outperforms SAFS on all six aggregate scores; the inference-policy gap is larger than in chunkwise evaluation.Source: D ADDITIONAL QUANTITATIVE RESULTS · p. 19
  • Consistency and motion generally trade off: Causal-Forcing has the highest dynamic degree in five of six benchmark–setting pairs but the lowest background consistency in all six.In framewise evaluation, SGF shows the opposite pattern, with highest background consistency and lowest dynamic degree on all three benchmarks.Source: D ADDITIONAL QUANTITATIVE RESULTS · p. 19
  • In chunkwise VBench, SAFM leads or ties in temporal flickering and object class; in framewise VBench, it leads motion smoothness, aesthetic quality, and object class.Other methods lead some chunkwise dimensions, while SGF leads framewise subject consistency, background consistency, and imaging quality.Source: D ADDITIONAL QUANTITATIVE RESULTS · p. 19, Table 6 · p. 20
  • On Interactive, chunkwise SAFM reaches a dynamic degree of 73.97 versus the best baseline’s 56.20, and framewise SAFM has the best ViCLIP after the first segment.On MovieGen-100s, chunkwise SAFM is best in background consistency and aesthetic quality and nearly matches Causal-Forcing’s dynamic degree, 65.61 versus 65.69.Source: D ADDITIONAL QUANTITATIVE RESULTS · p. 20

E.1 COMPONENT AND K/V BANK ABLATIONS

The ablations test history gradients, sink recaching, and inference-bank policies. They show that removing key training components changes motion and prompt following, while mixed-bank inference balances several quality measures.

E.1 COMPONENT AND K/V BANK ABLATIONS

  • Removing history K/V gradients lowers dynamic degree under both inference policies: from 64.10 to 51.30 for SAFM and from 48.00 to 44.50 for SAFS.The accompanying increases in consistency and imaging quality come with reduced motion.Source: §E.1 · p. 20
  • Without sink recaching, dynamic degree falls below 7 and second-segment ViCLIP drops by more than 3 points, with nearly static videos and failure to follow the new prompt.The comparison evaluates the training ablations with the same SAFM and SAFS inference policies; Interactive generation covers the first two prompts.Source: §E.1 · p. 20
  • SAFS Mix achieves a higher ViCLIP score than any fixed bank, 25.99 versus at most 25.71.Its Quality and dynamic degree fall between Fixed750 and Fixed500, while motion smoothness and imaging quality fall between Fixed500 and Fixed250.Source: §E.1 · p. 21

E.2 TRAINING SCHEDULE ABLATIONS

The schedule ablation compares SDF history-sampling policies with SAF’s single-pass training on 20-second MovieGen-100s generations. SAF with aligned inference achieves the highest reported Quality and dynamic degree overall, while other schedule-policy combinations expose a consistency–motion trade-off.

E.2 TRAINING SCHEDULE ABLATIONS

  • SAF with Aligned inference achieves the best overall Quality and dynamic degree, 85.22 and 67.97, although its imaging quality is the table’s lowest at 69.84.SAF also has the highest Quality under each inference policy: 83.97 with Fixed250, 84.59 with Mix, and 85.22 with Aligned.Source: §E.2 · p. 21, Table 10 · p. 22
  • Aligned inference gives the highest Quality and dynamic degree under every training schedule.The comparison evaluates each schedule with Fixed250, Mix, and Aligned inference on 20-second generations from the 128 MovieGen-100s prompts.Source: §E.2 · p. 21, Table 10 · p. 22
  • With Fixed250 or Mix inference, the 60/20/20 schedule has the highest subject consistency, background consistency, temporal flickering, and motion smoothness, but the lowest dynamic degree.Mix inference outperforms Fixed250 in Quality under every schedule except the fixed-only 0/100/0 schedule.Source: §E.2 · p. 22, Table 10 · p. 22

F ADDITIONAL QUALITATIVE RESULTS

The supplementary figures compare SAFS and SAFM with baselines using matched prompts or prompt sequences, sample seeds, and evaluation settings. Interactive and MovieGen-100s visualizations present different time slices of the generated videos.

F ADDITIONAL QUALITATIVE RESULTS

  • Chunkwise figures compare SAFS and SAFM with all five baselines, while framewise figures compare them with Causal-Forcing and SGF.The framewise comparisons are limited to those two baselines because they release framewise checkpoints.Source: F ADDITIONAL QUALITATIVE RESULTS · p. 23
  • Interactive figures show the middle frame from each 10-second prompt interval and list the six prompts below the grid.Methods appear at the left of each row, and comparisons use the same prompt sequence and sample seed.Source: F ADDITIONAL QUALITATIVE RESULTS · p. 23
  • MovieGen-100s figures show frames at 0, 20, 40, 60, 80, and 100 seconds.Methods use the same sample seed and inference settings as in the quantitative evaluation.Source: F ADDITIONAL QUALITATIVE RESULTS · p. 23

G LIMITATIONS

SAF has memory and visual-stability boundaries: shared K/V banks can accumulate errors, while camera motion and prompt switches can produce inconsistent changes. The reported visual issues are specific to content outside the local window or absent from the sink.

G LIMITATIONS

  • SAFM uses more memory than SAFS on one GPU, but pipelined inference keeps each GPU's memory close to SAFS by storing only its stage's K/V bank.SAFM can run on one H100; SAFS is needed only when GPU memory is limited.Source: G LIMITATIONS · p. 23
  • SAFS's shared bank departs from stage-aligned history and allows long-rollout errors to accumulate, with a more visible effect framewise than chunkwise.The authors recommend SAFM with multi-GPU pipelined inference.Source: G LIMITATIONS · p. 23
  • With camera motion, newly entering boundary content can change abruptly because only noisy K/V in the local window and the sink remain available.The authors suggest selectively retaining informative K/V beyond the window as a possible remedy, left to future work.Source: G LIMITATIONS · p. 23
  • At prompt switches, new objects, scenes, or characters can destabilize appearance; SAF is trained only on single-prompt clips, though action changes are handled smoothly.The authors suggest training with prompt switches as a possible remedy, left to future work.Source: G LIMITATIONS · p. 23

Related papers

Loading 2609.38114v1…