Connected Self Forcing: Beyond Local Learning in Video Autoregression

1 INTRODUCTION

• Self Forcing trains on its own autoregressive rollouts, but detaching reused history blocks later predictions’ gradients to earlier chunk generation.Each chunk is both an output and causal context for later predictions, while the video-level DMD objective gives chunks direct output gradients.Source: §1 · p. 2
• Shortcut Gradient Replay reconnects selected cross-chunk gradient paths without retaining the full autoregressive computation graph or changing inference.CSF also adds no model parameters.Source: §1 · p. 2
• CSF improves long-horizon generation across 60- and 240-second evaluations, with stronger visual quality and temporal consistency.Reported gains include better imaging quality and background and subject consistency, with reduced subject and scene drift over extended rollouts.Source: §1 · p. 2
• The introduction identifies a forward–backward gap: chunks affect later predictions, but detachment blocks downstream gradients into earlier context generation.A controlled 1D Gaussian autoregressive experiment motivates future-to-history gradients without retaining the full rollout graph.Source: §1 · p. 2

Bidirectional video generation

• Bidirectional video systems use temporal modeling, hierarchical designs, and scaling, but remain less suitable for low-latency streaming.Lumiere generates a full temporal duration in one Space-Time U-Net pass; other cited approaches extend temporal range or resolution.Source: §2 · p. 3

Autoregressive video generation

• Autoregressive video methods generate frames or chunks from prior content, using tokenized video, causal tokenizers, independently noised tokens, or few-step distribution matching.CausVid extends distribution matching to few-step causal video generation (Yin et al., 2025).Source: §2 · p. 3
• Self Forcing directly trains on self-generated history with a video-level objective, while gradient truncation keeps training tractable.Related work also addresses train–test mismatch and improves few-step causal initialization or long-horizon training.Source: §2 · p. 3
• Concurrent methods train context KV writing or replay differentiable re-noised trajectories; CSF instead reconnects gradients through historical KV writing into earlier-chunk generation.Self Gradient Forcing keeps generated context stop-gradient, while Self-Replay Forcing enables cross-block gradients within replayed trajectories.Source: §2 · p. 3

History modeling for long video generation

• Long-video history methods preserve or retrieve context through co-denoising, windowed attention, KV recaching, memory allocation, compression, and retrieval.Ring Forcing uses ring training and scales memory with compression and sparse RoPE.Source: §2 · p. 3
• Other approaches alter temporal supervision or guidance through history conditioning, cross-clip control, foresight features, or predictors over future horizons.CSF instead propagates gradients from later-chunk losses into earlier context generation.Source: §2 · p. 3

3.1 AUTOREGRESSIVE VIDEO GENERATION WITH SELF FORCING

• Self Forcing generates chunks from prior generated history, detaches reused history and KV caches, and applies direct DMD supervision to every chunk.The rollout generates a chunk, re-encodes it at the clean context timestep, and uses it to update the causal cache.Source: §3.1 · p. 4

3.2 CROSS-CHUNK GRADIENT ASSIGNMENT

• Detachment preserves forward dependencies but removes later predictions’ additional supervision of earlier chunks through their influence on subsequent predictions.The missing backward dependency is the cross-chunk gradient gap; later supervision should reflect each earlier chunk’s downstream influence.Source: §3.2 · pp. 4–5

3.3 SHORTCUT GRADIENT REPLAY

• Shortcut Gradient Replay treats selected historical KV states as differentiable leaves, then propagates their gradients through KV writing and into the earlier chunk’s generation.Replay stops at the targeted chunk rather than recursively replaying intervening generation branches; writer-only feedback keeps generated-latent inputs detached.Source: §3.3 · pp. 5–6

3.4 TRAINING

• CSF accumulates direct DMD gradients and replayed cross-chunk feedback on the shared generator before one optimizer update.The feedback is summed from later chunks that directly read the historical cache, with λcross = 1 by default; inference removes replay and follows the original Self Forcing rollout.Source: §3.4 · p. 6

4.1 EXPERIMENTAL SETUP

• The experiments use DMD post-training with Wan2.1-T2V-1.3B and compare CSF against SF and SGF under matched training and inference configurations.Long-horizon evaluation uses approximately 60-second videos from 40 VBench-Long prompts and 240-second videos from a fixed 128-prompt Movie Gen Video Bench subset.Source: §4.1 · pp. 6–7

4.2 QUANTITATIVE RESULTS

• CSF achieves the highest Avg.(6) point estimate in all six 60- and 240-second chunk-wise settings.At 240 seconds, paired bootstrap intervals are above zero against SF for all three initializations and against SGF for Causal CD and TF; comparisons are within duration because prompt sets differ.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · p. 7
• At 60 seconds, CSF leads Aesthetic Quality and Background Consistency under all three initializations; at 240 seconds, it leads Background Consistency, Imaging Quality, and Subject Consistency.The reported metrics are especially relevant to long-horizon generation, where subject and scene drift can accumulate.Source: §4.2 · pp. 7–8
• CSF’s Dynamic Degree depends on initialization: it is near SGF and above SF under Causal CD, but below both baselines under TF and Causal ODE.The paper interprets Avg.(6) alongside Dynamic Degree rather than claiming uniform improvement.Source: §4.2 · pp. 8–9

4.3 QUALITATIVE COMPARISON

• In 60- and 240-second examples, CSF better preserves subject appearance and scene composition or semantics than SF and SGF across the reported rollouts.The comparisons cover Causal CD and TF initialization; examples include Tokyo street, cloud-reading, kangaroo, and pianist scenes.Source: Figure 4 · p. 8, Figure 5 · p. 8, §4.3 · p. 9

4.4 ABLATION STUDY

• In the 60-second Causal CD ablation, serial recursion stays close to CSF on quality and consistency but has substantially lower Dynamic Degree; halving λ raises Dynamic Degree while degrading several quality and consistency metrics.Full BPTT does not outperform CSF on Avg.(6) or Dynamic Degree.Source: §4.4 · p. 9

• Shortcut Gradient Replay reconnects downstream supervision to historical generation without retaining the full autoregressive computation graph or changing inference.Across several initializations, CSF improves long-horizon generation in 60- and 240-second videos.Source: §5 · p. 10

A VIDEO DEMO

• The supplementary video compares CSF with SF and SGF under matched prompts and inference settings in chunk-wise and frame-wise generation, with synchronized excerpts and source-time indicators.It also includes a four-way qualitative ablation of cross-chunk gradient propagation under Causal CD initialization.Source: A VIDEO DEMO · p. 15, OVERVIEW · p. 15

B.1 TRAINING CONFIGURATION

• Chunk-wise training divides 21 latent frames into seven three-frame chunks and uses a bounded causal context of one sink chunk and two recent chunks.A stochastic exit among four denoising steps is sampled per training video and shared across its rollout chunks.Source: §B.1 · p. 15

B.2 INFERENCE CONFIGURATION

• Inference uses the same four-step causal sampling procedure across methods, with no Shortcut Gradient Replay or additional inference-time network or forward pass.Chunk-wise generation uses a maximum 12-latent-frame cache; reported videos are generated at 832 × 480 resolution and 16 fps.Source: §B.2 · pp. 15–16

B.3 EVALUATION PROTOCOL

• Long-video evaluation uses seven VBench-Long dimensions, fixed 2-second clips, and source-video aggregation; the 60-second set has 40 prompts and the 240-second set has 128 fixed prompts.The 240-second prompts are sampled from Movie Gen Video Bench; frozen prompt sets and generation settings are shared across methods.Source: §B.3 · p. 16

B.4 INITIALIZATIONS AND BASELINE REPRODUCTION

• SF, SGF, and CSF are trained independently from the same initialization checkpoint within each setting, with matched prompts, training budget, and inference configuration.The three initialization schemes are TF, Causal CD, and Causal ODE.Source: §B.4 · p. 16

B.5 SHORTCUT GRADIENT REPLAY IMPLEMENTATION

• Shortcut Gradient Replay stores states needed to reconstruct local generation and KV-writing computations, then traverses chunks in reverse while releasing each temporary graph after gradient accumulation.This reconstructs required local gradient paths without materializing the full autoregressive computation graph.Source: §B.5 · pp. 16–17

C CONTROLLED GAUSSIAN TIME-SERIES TOY EXPERIMENT

• The toy experiment provides intuition about cross-step gradient propagation, but its observed behavior need not carry over to real-world video generation.The study matches architectures, initialization, sampling, and neural DMD training while varying backward paths through generated history.Source: C CONTROLLED GAUSSIAN TIME-SERIES TOY… · p. 17
• The generator predicts from current external input, generated history, and fresh sampling noise; paired rollout MSE measures error because the target continuation is deterministic given context and subsequent inputs.Evaluation uses held-out conditions and shared sampling-noise draws across ten paired seeds.Source: C CONTROLLED GAUSSIAN TIME-SERIES TOY… · p. 17
• The comparison varies gradient paths: SF detaches generated history, CSF reconnects selected paths through shortcut replay, and Full BPTT backpropagates through all 20 autoregressive positions.Methods receive 500 neural DMD generator updates from shared initialization; MSE is reserved for evaluation.Source: C CONTROLLED GAUSSIAN TIME-SERIES TOY… · p. 17
• 12.27% lower mean MSE than SF over positions 21–124; CSF improves in all ten paired seeds, while remaining 3.21% above Full BPTT.Mean MSE is 0.03088 for SF, 0.02709 for CSF, and 0.02625 for Full BPTT.Source: C CONTROLLED GAUSSIAN TIME-SERIES TOY… · p. 18

D.1 TRAINING EFFICIENCY AND MEMORY

• 0.48× the measured SF peak memory and 1.41× the time for CSF in the frame-wise stress test; Full BPTT ran out of memory on its first generator update.In chunk-wise training, CSF used 0.85× the SF peak memory at 1.27× the time, while replay-based Full BPTT reached 1.43× the SF peak.Source: §D.1 · p. 19
• CSF’s lower measured peak memory is consistent with replaying selected paths and releasing each local graph after backward, not with cross-chunk gradients intrinsically reducing memory.The Full BPTT control retains reconstructed historical connections until a single backward pass; its reported time includes replay cost.Source: §D.1 · pp. 19–20

D.2 SHORT-HORIZON CHUNK-WISE EVALUATION

• At short horizons, SF, SGF, and CSF remain broadly comparable; CSF achieves the highest Avg.(6) under Causal CD and TF, while under Causal ODE it remains closely matched with SF.Individual VBench dimensions exhibit mixed rankings across methods.Source: §D.2 · p. 20

D.3 ADDITIONAL CHUNK-WISE QUALITATIVE RESULTS

• In the shown chunk-wise examples, CSF tends to preserve defining subject characteristics and overall composition more steadily, while drift becomes more apparent at 60 and 240 seconds.At 5 seconds, the methods generally show comparable visual quality with modest variations.Source: §D.3 · pp. 21–22

D.4 FRAME-WISE EXPERIMENTS

• Under Causal CD, CSF achieves the highest Avg.(6) at both 60 and 240 seconds; under TF, it slightly improves over SF at 60 seconds and remains closely matched at 240 seconds.For TF, paired bootstrap intervals include zero at both horizons.Source: Table 7 · p. 22, §D.4 · p. 22

D.5 PAIRED BOOTSTRAP ANALYSIS

• Paired bootstrap intervals estimate CSF-minus-baseline Avg.(6) differences by resampling matched full-length videos, conditional on fixed checkpoints rather than training-seed variation.The analysis uses 20,000 resamples and reports 95% percentile intervals.Source: §D.5 · p. 22, Table 8 · p. 24

D.6 EXPLORING GRADIENT CONFLICT

• Removing negatively aligned cross-chunk gradient components raises Dynamic Degree but slightly lowers Avg.(6) at 5, 60, and 240 seconds, so the default retains unprojected feedback.The authors suggest some opposing feedback may act as a damping signal, but projection does not consistently improve visual quality or long-term consistency.Source: Table 9 · p. 24, §D.6 · p. 24

E DISCUSSION AND LIMITATIONS

• CSF selectively reconstructs paths to historical chunks rather than reproducing exact full-sequence backpropagation; whether full-history gradients are generally worse remains unestablished.The authors leave open which historical states should receive feedback and from which future states.Source: E DISCUSSION AND LIMITATIONS · p. 24
• Replay trades memory consumption for computation, and its cost grows with the number of replayed historical states or writer dependencies, potentially mattering more for larger models or much longer sequences.Source: E DISCUSSION AND LIMITATIONS · p. 24
• More selective replay or sparse historical gradient assignment is proposed as a direction for limiting unnecessary recomputation and gradient interference.The authors frame these mechanisms as potential approaches, not established results.Source: E DISCUSSION AND LIMITATIONS · p. 25
• Removing cross-chunk gradient components opposing direct DMD raises Dynamic Degree while slightly reducing aggregate quality and consistency metrics; CSF does not always maximize Dynamic Degree in unconstrained generation.The authors identify interaction with controllable generation as a direction for future study.Source: E DISCUSSION AND LIMITATIONS · p. 25