PaperlayerView exact revision on arXiv

Source-linked AI summary

Self Gradient Forcing: Native Long Video Extrapolation

Junhao Zhuang, Shiyi Zhang, Yuxuan Bian, Yaowei Li, Yawen Luo, Yijun Liu, Weiyang Jin, Songchun Zhang, Xianglong He, Xuying Zhang, Haoran Li, Haoyang Huang, Zeyue Xue, Nan Duan

arXiv:2607.20368v2cs.CV

TL;DR

Frozen-cache Self Forcing leaves a historical context-gradient gap: future losses cannot supervise how self-generated histories are written into K/V memory for later generation. SGF restores this supervision through bounded parallel reconstruction, improving long-video extrapolation while preserving short-horizon quality, identity, layout consistency, and temporal stability.

  • Problem

    Frozen-cache Self Forcing does not let future losses supervise how self-generated context is written into K/V memory, limiting native long-video extrapolation.

  • Method

    SGF combines a no-gradient serial self-rollout with parallel reconstruction of a sampled exit computation, allowing gradients through context K/V representations without full rollout backpropagation.

  • Results

    SGF preserves short-horizon quality while improving long-video identity, layout consistency, and temporal stability across frame-wise and chunk-wise generation and multiple horizons.

  • Takeaways & Limitations

    Using only a 5-second training window, SGF supports extrapolation to minute-scale videos while improving long-horizon consistency over matched Self Forcing baselines.

  • Takeaways & Limitations

    Direct differentiable-cache training is impractical for long-horizon self-rollout because its backward graph grows with rollout length.

Abstract

from arXiv · show

Recent autoregressive video diffusion methods are increasingly built upon Self Forcing, where the student is trained on histories produced by its own rollout rather than ground-truth video contexts. This reduces exposure bias, but the historical key-value cache is still used by future frames only as frozen rollout state. As a result, future losses cannot supervise how earlier generated latents should be written into more useful keys and values for later video-latent generation. We call this the historical context-gradient gap. We propose Self Gradient Forcing (SGF), a two-pass training strategy that restores this missing supervision signal without backpropagating through the full serial rollout. Pass 1 performs a no-gradient autoregressive rollout matching inference and, at a sampled denoising exit step, records both the self-generated context and the noisy latents fed to the model. Pass 2 performs parallel context-gradient reconstruction for the recorded exit step. The generated context is used as stop-gradient clean-latent input, while the model recomputes the context KV representations and future-to-context causal attention. Thus, SGF provides the missing memory-writing supervision within the native autoregressive training objective, using losses on future video latents to train the model to encode context into more effective causal memory. Across extensive long-horizon frame-wise and chunk-wise experiments under different initializations, SGF achieves stronger native long-video extrapolation than Self Forcing, especially in subject identity, background/layout consistency, and temporal stability. Remarkably, using only a 5-second training window, SGF can extrapolate to videos lasting several minutes. Code and models will be released to advance research on autoregressive video generation.

1 INTRODUCTION

Native long-video extrapolation is difficult because autoregressive video diffusion must condition on self-generated histories, while frozen-cache Self Forcing leaves future losses unable to supervise how those histories are written into causal memory. Self Gradient Forcing addresses this historical context-gradient gap with bounded two-pass recomputation and improves long-horizon consistency.

  • Motivation: Autoregressive video generation must preserve scenes, objects, layouts, and dynamics as generation extends beyond the training window using previously generated content.At inference, each new frame or chunk is conditioned on the prompt and previously generated content.
  • Problem: Teacher-Forcing conditions on ground-truth contexts, whereas inference uses self-generated histories, creating exposure bias that hinders native long-video extrapolation.Self Forcing mitigates this mismatch by training on histories produced by its own autoregressive rollout.
  • Historical context-gradient gap: Self Forcing lets future losses supervise cached-history reading but not the clean-timestep computation that writes self-generated history into causal K/V memory.This missing credit-assignment path is termed the historical context-gradient gap.
  • Limitation: Retaining differentiable historical KV caches would require autograd graphs across rollout writes, whose size grows with rollout length, transformer depth, and sequential cache updates.This makes direct KV-gradient backpropagation difficult to scale.
  • Self Gradient Forcing: SGF restores memory-writing supervision through two passes: a serial no-gradient rollout records self-generated context and noisy latents, then parallel context-gradient reconstruction recomputes the relevant representations.The strategy avoids full backpropagation through the sampled rollout and is evaluated across multiple initializations with frame-wise and chunk-wise generation.

2 RELATED WORK

Related work spans autoregressive video diffusion foundations, forcing objectives that train on self-generated histories, and long-horizon methods that modify available or retained context. SGF is positioned as complementary to context and cache design because it improves how self-generated content is written into future-readable KV representations.

  • Autoregressive video diffusion: Autoregressive video diffusion builds on diffusion objectives, accelerated samplers, latent diffusion, and diffusion transformers, extending them with 3D denoising and space-time architectures.The passage also identifies cascaded generation, latent video modeling, and motion modules as extensions for video systems.
  • Forcing objectives for self-generated histories: Teacher-forcing uses ground-truth prefixes, whereas Self Forcing trains on self-generated rollout histories to reduce exposure bias through distribution-matching distillation.Causal Forcing and subsequent methods build on this self-rollout training paradigm.
  • Long-horizon context and cache design: Long-horizon generation methods alter which context is available, how it is positioned, or how it is retained, including cache retention, streaming long tuning, sparse attention, and KV compression.The cited line of work also includes overlapping temporal co-denoising, rotary position embeddings, attention sinks, long-context supervision, head-wise cache behavior, retrieval-augmented latent history, and gated recall.
  • Long-horizon context and cache design: SGF is orthogonal to context and cache design, improving how self-generated content is written into future-readable KV representations for a given context and cache configuration.The distinction is that these prior methods change the memory exposed to the generator, whereas SGF changes memory writing.

3 METHOD

SGF identifies the historical context-gradient gap in frozen-cache Self Forcing and closes it with a two-pass reconstruction strategy. It keeps self-rollout states fixed while allowing future losses to supervise clean-context K/V writing without backpropagating through the full serial rollout.

  • Historical context-gradient gap: Autoregressive blocks condition on clean-timestep K/V entries written from earlier predicted clean latents, rather than directly on those latents.The cache-writing computation recurrently reads existing memory and appends each historical entry for later blocks.
  • Historical context-gradient gap: Frozen-cache Self Forcing detaches historical K/V entries, so future losses train history reading but cannot supervise how earlier self-generated latents are encoded into memory.This missing supervision path is called the historical context-gradient gap.
  • Two-pass training: SGF uses Pass 1 for a no-gradient self-rollout that records a sampled exit state, noisy inputs, predicted clean latents, and the serial cache trajectory.The recorded states are treated as fixed data for the subsequent reconstruction pass.
  • Two-pass training: In Pass 2, SGF stop-gradients the context latents but keeps reconstructed clean-context K/V representations differentiable when future target tokens attend to them.Future DMD losses therefore supervise both target-side denoising and clean-context K/V writing.
  • Computational design: The bounded reconstruction avoids a recurrent autograd graph through the full self-rollout by combining serial no-gradient rollout with fixed-window parallel gradient computation.Gradients flow through the Pass-2 clean-context forward, K/V projections, future-to-context attention, and target denoising computation.

4 EXPERIMENTS

Experiments evaluate SGF against matched Self Forcing baselines at 5s, 60s, and 240s using identical training windows and controlled inference settings. SGF generally improves long-horizon quality and consistency, preserves coherent motion and visual structure, and restores context-gradient supervision with modest computational overhead.

  • Experimental setup: All models use a 5-second training window, while matched SGF and Self Forcing comparisons test 5s, 60s, and 240s native extrapolation.Matched pairs share initialization, prompts, seeds, memory policy, sliding window, chunking, and sampling configuration.
  • Quantitative results: Across frame-wise and chunk-wise generation at both long horizons, SGF improves most quality and consistency metrics over matched Self Forcing baselines.The most consistent gains occur in aesthetic quality, background consistency, imaging quality, motion smoothness, subject consistency, and flickering.
  • Quantitative results: Dynamic degree is the main exception, because Self Forcing can score higher through incoherent scene jumps, broken camera geometry, and object deformation rather than better motion quality.SGF instead maintains more stable image quality and more plausible camera evolution.
  • Qualitative results: Qualitative comparisons show Self Forcing accumulating identity, crop, scene-layout, and camera drift, whereas SGF better preserves subject identity, camera relations, textured backgrounds, and temporal stability.Comparisons use the same prompts, seeds, horizons, initializations, and inference geometry, isolating the training objective.
  • Human preference: More than 1,900 blind paired judgments across 10 matched comparisons evaluate human preference for SGF against Self Forcing under identical conditions.The reported GSB score assigns positive values to preference for SGF.
  • Training feasibility: 87.01GB peak memory and 11.71s per five training steps are measured for SGF, versus 79.01GB and 10.39s for frozen-cache Self Forcing, while direct differentiable-cache training runs out of memory.SGF bounds gradients to Pass 2 after a no-gradient Pass-1 rollout, avoiding recurrent cache-formation graphs.

5 CONCLUSION

The paper presents Self Gradient Forcing (SGF), a two-pass strategy that closes the historical context-gradient gap in frozen-cache Self Forcing. It preserves no-gradient serial rollout while enabling future losses to supervise self-generated history representations without full rollout backpropagation.

  • Contribution: SGF closes the historical context-gradient gap in frozen-cache Self Forcing through a two-pass training strategy.The method targets supervision of how self-generated histories are written into K/V memory.
  • Method: The method keeps serial self-rollout no-gradient and reconstructs the sampled exit computation in parallel.This design avoids backpropagating through the full serial rollout.
  • Evaluation: Future losses supervise how self-generated histories are written into K/V memory during frame-wise and chunk-wise generation.The conclusion reports evaluation across multiple initializations and 5s, 60s, and 240s horizons.

A ADDITIONAL EXPERIMENTAL DETAILS · A.1 FRAME-WISE CONFIGURATION · A.2 CHUNK-WISE CONFIGURATION

The appendix defines matched frame-wise and chunk-wise evaluation protocols across 5-, 60-, and 240-second horizons, with long-horizon metrics focused on visual persistence. It also specifies training and inference context geometries, checkpoint provenance, and controlled Self Forcing–SGF comparisons.

  • A ADDITIONAL EXPERIMENTAL DETAILS: Evaluations compare matched Self Forcing and SGF pairs at 5s, 60s, and 240s using VBench, VBench-Long, and MovieGen prompts.The 5-second setting reports all 16 VBench quality and semantic dimensions; long-horizon evaluations omit text-alignment metrics.
  • A ADDITIONAL EXPERIMENTAL DETAILS: Frame-wise and chunk-wise causal-ODE Self Forcing baselines use released Causal Forcing checkpoints, while chunk-wise bidirectional-ODE baselines use released Self Forcing checkpoints.Other Self Forcing checkpoints were reproduced under the same training settings as corresponding SGF checkpoints.
  • A.1 FRAME-WISE CONFIGURATION: Frame-wise training uses the full 5-second window without sliding-window eviction, so Pass 2 applies a teacher-forcing-style causal mask over the recorded frame sequence.This mask matches the full-context causal relation used by the frame-wise Pass-1 rollout.
  • A.1 FRAME-WISE CONFIGURATION: Frame-wise inference uses streaming context with sink 4, FIFO 16, and current chunk 1, totaling 21 latent frames for both Self Forcing and SGF.Within matched pairs, prompts, seeds, sampling, initialization, and inference context policy are held fixed; the intended difference is the sampled exit-loss gradient boundary.
  • A.2 CHUNK-WISE CONFIGURATION: Chunk-wise training uses a sliding-window context with sink 3, FIFO 6, current chunk 3, and chunk size 3.The Pass-2 reconstruction mask matches the Pass-1 sliding-window cache relation and attention geometry.
  • A.2 CHUNK-WISE CONFIGURATION: Chunk-wise inference and long-horizon evaluation use the same context policy, with released bidirectional-ODE and causal-ODE Self Forcing checkpoints as reference baselines.Controlled SGF comparisons are matched causal-CD and TF pairs sharing initialization and inference geometry.
  • A.2 CHUNK-WISE CONFIGURATION: Chunk-wise experiments test context-gradient usefulness under coarser temporal memory updates, reporting 5-second results in Tab. 6, long-horizon results in Tab. 2, and human preferences in Tab. 3.The long-horizon chunk-wise results cover 60-second and 240-second settings.

B FULL METRIC TABLES · B.1 FRAME-WISE 5-SECOND VBENCH · B.2 CHUNK-WISE 5-SECOND VBENCH

The appendix provides complete 5-second VBench evaluations for frame-wise and chunk-wise settings, testing whether SGF preserves short-video quality and remains comparable to Self Forcing. It also specifies the streaming policies and reconstruction masks used to align training with evaluation-time memory updates.

  • B FULL METRIC TABLES: The appendix reports complete 5-second VBench results, with metrics as rows and model variants as columns.Boldface marks the better value within each matched Self Forcing–SGF pair.
  • B FULL METRIC TABLES: The frame-wise table isolates SGF’s gradient-boundary change under the same initialization.Evaluation uses sink 4, FIFO 16, and current chunk 1, totaling 21 latent frames.
  • B FULL METRIC TABLES: 5-second results serve as a sanity check because SGF should not degrade standard video quality.The method is designed to improve long autoregressive extrapolation, while 60-second and 240-second results appear in the main paper.
  • B.1 FRAME-WISE 5-SECOND VBENCH: The frame-wise setting tests whether context-gradient reconstruction preserves short-video generation quality.Training uses the full 5-second window without sliding-window eviction, and Pass 2 uses a standard teacher-forcing-style causal mask.
  • B.1 FRAME-WISE 5-SECOND VBENCH: Frame-wise evaluation follows the long-horizon streaming policy of sink 4, FIFO 16, and current chunk 1.This policy gives a total context window of 21 latent frames.
  • B.2 CHUNK-WISE 5-SECOND VBENCH: The chunk-wise setting checks whether SGF remains comparable to Self Forcing with coarser-grained memory updates.Chunk-wise Pass 1 already uses a sliding-window context with sink 3, FIFO 6, current chunk 3, and chunk size 3.
  • B.2 CHUNK-WISE 5-SECOND VBENCH: Chunk-wise Pass-2 reconstruction matches the sliding-window cache relation used during rollout.This alignment makes the recovered context-gradient path follow the sampled rollout’s attention geometry.

C DIRECT CACHE-GRADIENT FEASIBILITY

This section compares gradient exposure through autoregressive history and explains why SGF uses bounded parallel reconstruction. Direct differentiable-cache training creates a serial history-formation graph, full rollout BPTT is more demanding, and SGF recovers context-side gradients without retaining either serial graph.

  • Gradient-exposure alternatives: Frozen-cache Self Forcing trains future denoising computations but detaches the historical cache formation, preventing losses from supervising earlier key-value writing.Historical K/V entries are produced by a no-gradient context-timestep forward, so gradients cannot reach that computation.
  • Gradient-exposure alternatives: Direct differentiable-cache training keeps generated latents detached but makes serial cache formation differentiable across earlier historical entries, creating a recurrent autograd graph.The K/V entry for each block depends on earlier differentiable cache state, opening a serial history-formation graph.
  • Gradient-exposure alternatives: Full rollout BPTT additionally differentiates through the denoising trajectories that produce generated latents, so memory grows with both rollout length and denoising depth.This makes full rollout BPTT strictly more demanding than direct differentiable-cache training.
  • SGF bounded reconstruction: SGF performs a no-gradient autoregressive rollout, records context latents and noisy exit states, then reconstructs them in one bounded parallel pass.The reconstruction window has fixed length N and recomputes context-side K/V projections plus future-to-context attention.
  • SGF bounded reconstruction: SGF’s recovered gradient reaches context encoding and future-to-context attention, while excluding the recorded latents’ denoising trajectory and Pass 1’s persistent-cache updates.The scaling order is MSGF < Mdirect < Mfull-BPTT, and direct differentiable-cache training runs out of memory in the reported setting.

D TWO-PASS RECOVERY FIDELITY

The appendix evaluates whether Pass 2 faithfully reconstructs Pass 1’s sampled exit computation. Across 96 prompt/exit comparisons, reconstruction closely matches the serial rollout within expected mixed-precision error, supporting its use as a bounded local surrogate rather than an exact full-rollout gradient.

  • Evaluation protocol: 96 prompt/exit comparisons evaluate Pass 2 against serial no-gradient Pass 1 rollouts across 24 prompts and four exit steps.Pass 2 reconstructs the recorded noisy states and detached self-generated context under a matched teacher-forcing attention mask.
  • Recovery results: 1.41% overall relative ℓ2 error and 0.999886 average cosine similarity show close Pass-2 recovery of Pass-1 exit predictions.The results hold across the denoising schedule.
  • Numerical scale: 1.80 times ϵbf16 is the overall relative ℓ2 error normalized by bf16 precision, decreasing from 2.73 at the noisiest exit step to 1.04 at exit step 250.The reported scale reflects accumulated roundoff from many bf16 transformer operations executed in different serial and parallel orders.
  • Interpretation and limitation: The diagnostic supports Pass 2 as a bounded local surrogate for sampled exit computation, not as recovery of the exact full serial-rollout gradient.The verification concerns forward local-attention recovery up to expected mixed-precision numerical error.

E VAE BOUNDARY AND SINK CHOICE · F ADDITIONAL RELATED-WORK DISCUSSION

The appendix shows that four sink latents cover the VAE’s short boundary-transition region while preserving recent context, and clarifies SGF’s relationship to Checkpointed Self Forcing. SGF similarly combines rollout and parallel recomputation but targets supervision of historical context K/V formation without backpropagating through the serial rollout.

  • E VAE BOUNDARY AND SINK CHOICE: Frame-wise streaming inference uses a shared sink-plus-FIFO context policy in Self Forcing and SGF, rather than an independent SGF contribution.The appendix justifies the sink size used in frame-wise experiments.
  • E VAE BOUNDARY AND SINK CHOICE: The Wan VAE begins with a 1-frame boundary group followed by 4-frame groups, making freshly encoded early latents potentially unlike steady in-stream latents.The diagnostic uses 10 videos with 81 pixel-space frames corresponding to 21 VAE latent frames.
  • E VAE BOUNDARY AND SINK CHOICE: A fresh-stream target group has 0.607 relative L2 error, while adding an anchor and W = 1, 2, 3, 4 previous groups reduces error to 0.124, 0.084, 0.067, and 0.060.The anchor alone gives 0.271 at W = 0, and improvements plateau beyond the first several positions.
  • E VAE BOUNDARY AND SINK CHOICE: The diagnostic indicates that early VAE latents form a short boundary-transition region rather than a single isolated sink token.The mismatch falls sharply after adding previous latent groups and changes little beyond the tested range.
  • E VAE BOUNDARY AND SINK CHOICE: Sink 1 is weaker on aesthetic quality and flickering, whereas sink 4 reaches the diagnostic’s stable range and sink 8 adds only marginal gains while consuming more context budget.The total streaming context budget is fixed, so larger sinks leave fewer slots for recent FIFO context.
  • E VAE BOUNDARY AND SINK CHOICE: Four sink latents are therefore used in frame-wise long-horizon experiments as the smallest sink covering the observed boundary-transition prefix.Qualitative ablations associate multi-latent sinks with better subject and scene consistency during long extrapolation.
  • F ADDITIONAL RELATED-WORK DISCUSSION: Checkpointed Self Forcing in Solaris performs no-gradient autoregressive rollout, caches clean estimates and noisy transition states, and replays a sampled denoising step in parallel.The cited implementation targets multiplayer Minecraft world modeling and uses a teacher-forcing attention mask.
  • F ADDITIONAL RELATED-WORK DISCUSSION: SGF similarly performs rollout-then-parallel recomputation, but uses stop-gradient generated latents to address the historical context-gradient gap without backpropagating through the serial rollout.In frozen-cache self-rollout training, future losses supervise reading cached history but not writing earlier self-generated latents into future-readable K/V memory.

G LIMITATIONS

SGF recovers only a bounded surrogate for the missing history-formation gradient, not the exact gradient of the full serial rollout. Its effectiveness depends on faithful Pass-2 alignment with inference, and it complements rather than replaces other long-video techniques.

  • Bounded surrogate: SGF supervises how recorded self-generated latents are written into future-readable memory, but does not backpropagate future losses into those latents or optimize denoising decisions.The method therefore does not recover the exact gradient of the full serial autoregressive rollout.
  • Alignment assumption: SGF assumes that parallel Pass-2 reconstruction faithfully reproduces the serial context relation at the sampled exit step.Mismatches in the teacher-forcing mask, sink positions, FIFO window, RoPE handling, context timestep, or chunk alignment can train a different memory writer.
  • Alignment assumption: Alignment is especially important at the clean context timestep because the cache-writing call is shared with inference.Incorrect reconstruction can recover gradients for the wrong attention relation.
  • Complementary scope: SGF is not a replacement for streaming long-rollout tuning, retrieval-augmented memory, sparse attention, stronger causal initialization, or long-context teachers.These techniques target complementary parts of the long-video generation system; SGF instead addresses an unused history-formation gradient in self-rollout training.

H ADDITIONAL QUALITATIVE RESULTS · H.1 FRAME-WISE COMPARISONS · H.2 CHUNK-WISE COMPARISONS

Matched qualitative comparisons show that SGF reduces the memory-related drift that accumulates in long-horizon frame-wise and chunk-wise generation. Across settings, SGF more consistently preserves subject identity, composition, viewpoint, background, and temporal layout than Self Forcing.

  • H ADDITIONAL QUALITATIVE RESULTS: The appendix compares Self Forcing and SGF under matched prompts, seeds, initializations, horizons, sampling configurations, and inference-context geometries, emphasizing memory-related drift.The comparisons illustrate temporal failure modes behind aggregate scores, including view changes, crop drift, and related memory errors.
  • H.1 FRAME-WISE COMPARISONS: Frame-wise Self Forcing often begins locally plausible but progressively changes subject identity, camera distance, object layout, or background, especially under weaker initializations.Because each generated frame is written into historical context, errors in historical K/V formation can accumulate at fine temporal granularity.
  • H.1 FRAME-WISE COMPARISONS: Under causal CD initialization, SGF limits 60-second crop and scale drift, preserving animal scale, background, and camera distance more stably than Self Forcing.The sea-turtle and elephant comparisons show improvement in non-catastrophic composition drift as well as outright failure.
  • H.1 FRAME-WISE COMPARISONS: Across 240-second causal CD and causal ODE comparisons, SGF more consistently preserves subjects, poses, camera distance, and scene layouts while Self Forcing develops crops, artifacts, fragments, or identity loss.Examples include the otter, anime reading, library-reading, clock-and-vase, candle/creature, and disco-kangaroo prompts.
  • H.1 FRAME-WISE COMPARISONS: In 60-second chunk-wise TF comparisons, SGF better preserves hands, actions, people, gestures, and environments than Self Forcing, which changes identities, viewpoints, or subject counts.The apple-peeling and eyebrow-makeup prompts show these differences directly.
  • H.2 CHUNK-WISE COMPARISONS: Chunk-wise generation tests whether recovered context gradients remain useful when generated multi-frame blocks are written and consumed as later context.ODE-initialized rows are reference baselines, while controlled comparisons use matched Self Forcing and SGF rows under the same initialization.
  • H.2 CHUNK-WISE COMPARISONS: Over 240 seconds under TF initialization, SGF more consistently preserves translucent cell populations and lizard subjects, whereas Self Forcing produces unrelated textures, particles, or foliage.These comparisons test whether a generated block remains useful several chunks later.
  • H.2 CHUNK-WISE COMPARISONS: Under causal CD initialization, SGF maintains more coherent people, object arrangements, viewpoints, identities, and environments than matched Self Forcing and ODE-initialized references in chunk-wise comparisons.The umbrella-couple, corgi, and running-man prompts show more stable layouts and front-facing subjects with SGF.
Loading 2607.20368v2…