PaperlayerView exact revision on arXiv

Source-linked AI summary

SGF+: Decoupling Gradient Flows for Autoregressive Video Generation

Zihan Su, Junhao Zhuang, Yaowei Li, Siwen Lu, Haoran Li, Lingen Li, Haoyu Wu, Weiyang Jin, Songchun Zhang, Haoyang Huang, Chun Yuan, Zeyue Xue, Nan Duan

arXiv:2610.10429v1cs.CV

TL;DR

Autoregressive video generation must both denoise current frames and encode them as context for future predictions, yet the roles typically share parameters. SGF+ separates those parameters while preserving interaction through causal attention, and improves visual quality and long-horizon consistency over the evaluated baselines, including generation up to 24 hours from 5s rollouts.

  • Problem

    Self Forcing leaves historical key-value caches frozen, so future-generation losses cannot supervise how earlier generated latents are encoded for later predictions; SGF then restores that path but retains shared parameters for context writing and denoising.

    Source: §1 · p. 2

  • Method

    SGF+ assigns separate parameters to context writing and denoising, while jointly optimizing both through the original generation objective and differentiable context states.

    Source: §1 · p. 2

  • Results

    With 5s training rollouts, SGF+ supports continuous generation for up to 24 hours and improves visual quality and long-horizon consistency over SF and SGF in framewise and chunkwise generation.

    Source: §1 · pp. 2–3, §5 · p. 9

  • Takeaways & Limitations

    Role-specific parameterization is an effective design principle for high-quality autoregressive video generation and native long-horizon extrapolation.

    Source: §1 · p. 3

    The paper adopts parameter separation as its approach and leaves coordinating roles through model merging for future work.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Autoregressive video generation requires denoising the current frames while writing their key-value representations as context for future predictions. However, these two roles typically share parameters, and we find that their gradients exhibit distinct patterns and systematic negative alignment, hindering the joint optimization of visual quality and temporal consistency. We introduce Self Gradient Forcing Plus (SGF+), which assigns separate parameters to context writing and denoising while preserving their interaction through causal attention. Both roles are jointly optimized using the original generation objective without auxiliary losses, with context writing supervised through its contribution to future predictions. This simple change improves visual quality and long-horizon consistency over the evaluated baselines in both framewise and chunkwise generation, without additional video training data or a longer training horizon. Trained on only 5s rollouts, SGF+ supports continuous generation for up to 24 hours without long-video fine-tuning. These results highlight role-specific parameterization as an effective design principle for high-quality autoregressive video generation and native long-horizon extrapolation.

1. Introduction

Autoregressive video generation must denoise current frames and write their representations as context for future predictions, but shared parameters may not accommodate both roles. SGF+ separates their parameters and improves visual quality and long-horizon consistency in the reported evaluations.

1. Introduction

  • SF leaves historical key-value representations frozen during subsequent predictions, so future-generation losses cannot supervise how earlier latents are encoded for future use.SGF restores this historical context-gradient path, but still shares parameters between context writing and denoising.Source: §1 · p. 2
  • Mean role-gradient directions differ by 104.2° in Attention and 106.3° in FFN; all 512 paired cosine similarities are negative in both module families.The analysis measures context-writing and denoising gradients under the same generation objective.Source: §1 · p. 2
  • SGF+ assigns independent parameters to context writing and denoising while preserving their interaction through causal attention and training both roles with the original generation objective.It retains self-generated rollouts and differentiable replay without auxiliary losses, additional video training data, or long-horizon fine-tuning.Source: §1 · p. 2
  • With 5s training rollouts, SGF+ supports continuous generation for up to 24 hours and improves visual quality and long-horizon consistency over SF and SGF in framewise and chunkwise generation.The comparisons cover VBench horizons of 5s, 60s, and 240s; no long-video fine-tuning is used.Source: §1 · pp. 2–3

2. Related Work

Related work addresses autoregressive video generation through history construction, supervision, and context management. SGF+ complements these directions by changing how context writing and denoising are parameterized.

2.1. Autoregressive Video Diffusion and World Models

  • Autoregressive video models reuse historical key-value representations, and SGF+ complements world-model approaches that use differentiable context reconstruction by separating context-writing and denoising parameters.The cited applications include visual world simulation, interactive editing, and audio-video generation.Source: §2.1 · p. 3

2.2. Forcing Strategies for Autoregressive Video Generation

  • Forcing methods vary in history construction, supervision, and gradient propagation; SGF+ retains SGF’s differentiable context supervision while changing the shared-parameter design.Other approaches vary rollout histories, diffusion objectives, distillation, adversarial supervision, rewards, or auxiliary prediction modules.Source: §2.2 · p. 3, Figure 4 · p. 4

2.3. Long-Horizon Video Generation and Context Management

  • Long-video methods extend temporal coverage or manage inference-time context, whereas SGF+ targets joint learning within a limited training window and is complementary to those mechanisms.The described alternatives include longer-horizon training, context routing, spectral correction, and positional adaptation.Source: §2.3 · p. 4

3. Method

The method restores future-loss supervision of context writing through differentiable replay, then separates its parameters from those used for denoising. This addresses measured gradient conflict while preserving the roles’ forward interaction.

3.1. Self Gradient Forcing Preliminaries

  • SGF uses two passes: a no-gradient autoregressive rollout records sampled latents, then a parallel reconstruction makes context key-value states differentiable for future-generation losses.The sampled latents and cache trajectory remain detached, avoiding backpropagation through the full rollout.Source: §3.1 · p. 5

3.2. Observations of Gradient Conflict

  • Context-writing and denoising gradients are measured separately within the same Pass-2 computation, with conflict defined by negative cosine similarity or an angle above 90°.All 512 measured paired cosine similarities are negative in both Attention and FFN.Source: §3.2 · p. 5
  • The two roles have separated gradient distributions and mean directions, and their conflicting contributions partially cancel when accumulated on shared parameters.Mean directions exceed 90° in both module families; the measured conflict spans the sampled prompts and denoising timesteps.Source: §3.2 · p. 5

3.3. Self Gradient Forcing Plus

  • SGF+ uses separate context-writer and denoiser parameters, jointly trained through the original objective and differentiable KV states without auxiliary context losses or gradient projection.Both roles remain forward-coupled; the paper leaves model-merging approaches to future work.Source: §3.3 · p. 5

4. Experiments

Experiments compare SGF+ with SF and SGF across framewise and chunkwise generation, finding stronger long-horizon quality and consistency with 5s training rollouts. Ablations and efficiency measurements examine the scope and costs of role-specific parameterization, while gradient analyses find role differences already at teacher-forcing initialization.

4.1. Experimental Setup

  • Experiments compare SF, SGF, and SGF+ at 5s, 60s, and 240s using framewise and chunkwise generation; long-horizon settings use VBench-Long and MovieGen-128 prompts.Models share a causal video diffusion student, frozen teacher, and separately trained fake-score network; SGF+ uses two full parameter sets.Source: §4.1 · pp. 5–7

4.2. Native Long-Horizon Extrapolation

  • SGF+ outperforms SF and SGF on most quality and consistency metrics across framewise and chunkwise generation at 60s and 240s, with Dynamic Degree as the main exception.Table 1 covers VBench-Long prompts at 60s and MovieGen-128 prompts at 240s; SF or SGF sometimes scores higher on Dynamic Degree.Source: Table 1 · p. 7, §4.2 · p. 7
  • In 240s qualitative comparisons, SGF+ better preserves scene details than SF and SGF in both chunkwise and framewise generation.Examples include buildings and miners, an explorer and desert setting, and a warehouse flower scene.Source: Figure 5 · p. 6, §4.2 · p. 7
  • Trained on 5s video windows, SGF+ supports continuous generation for up to 24 hours without long-video fine-tuning.The 60s and 240s evaluations also test extrapolation beyond the training horizon.Source: §4.1 · p. 5, §4.2 · p. 7

4.3. Ablation Study

  • Both Attention-only and FFN-only parameter separation underperform full separation on quality and consistency metrics.The results indicate that separating parameters in FFN alone does not fully address conflicting gradients also found in Attention.Source: Table 2 · p. 7, §4.3 · p. 8

4.4. Discussion

  • For the evaluated 81-frame workload, SGF+ inference takes 4.969 seconds versus 4.962 seconds for SGF, while training time per step rises by approximately 8.2%.Generator parameters increase from 1.4B to 2.8B; inference memory increases from 24.85 to 27.96 GB.Source: §4.4 · p. 8, Table 3 · p. 8
  • Role-specific gradient differences are visible at teacher-forcing initialization, including a sharp directional change from clean-context writing to noisy denoising between timestep indices 0 and 1.Mean role-gradient angles exceed 90° in both Attention and FFN, with the vast majority of paired cosine similarities negative.Source: §4.4 · p. 8, Figure 6 · p. 8
  • Video diffusion alternates clean-context encoding and noisy-latent denoising, unlike autoregressive language models that write KVs and predict subsequent tokens in one pass.The authors suggest these distinct input conditions and computational roles may help explain the observed gradient differences.Source: §4.4 · p. 8

5. Conclusion

The conclusion identifies conflicting optimization demands between context writing and denoising when they share parameters. SGF+ separates the roles and improves long-horizon generation, including generation for up to 24 hours from 5s training rollouts.

  • Shared-parameter Self Gradient Forcing has an optimization conflict between context writing and denoising.Source: §5 · p. 9
  • SGF+ uses role-specific parameter separation and improves visual quality and long-horizon consistency over SF and SGF in framewise and chunkwise generation.Source: §5 · p. 9
  • SGF+ supports generation for up to 24 hours from 5s training rollouts, and the role distinction is evident at teacher-forcing initialization.Source: §5 · p. 9

Supplementary Material

The supplementary material lists additional short-video results, implementation details, gradient analyses, long-generation results, qualitative comparisons, optimization analysis, and future work.

  • Supplementary Section A provides additional short-video results, and Section B provides implementation details.Source: Supplementary Material · p. 12
  • Sections C, D, E, and F cover gradient role separation at teacher-forcing initialization, 24-hour generation, layer-wise gradient separation, and additional qualitative comparisons.Source: Supplementary Material · p. 12
  • Sections G and H provide local optimization analysis of role separation and future work, respectively.Source: Supplementary Material · p. 12

A. Additional Short-Video Results

The appendix table reports standard VBench results at the 5s training horizon for framewise and chunkwise generation.

  • Table 4 covers framewise and chunkwise VBench metrics at 5s.Source: Table 4 · p. 12
  • The 5s evaluations compare SF, SGF, and SGF+ within the training horizon.Source: Table 4 · p. 12
  • Boldface marks the best value within each generation mode.Source: Table 4 · p. 12

B. Additional Implementation Details

The implementation uses a shared sink-plus-FIFO context policy within each generation mode and specifies optimizer settings for generator and critic.

C. Gradient Role Separation at TF Initialization

At teacher-forcing initialization, context-writing and denoising gradients are already distinct and negatively aligned. Their mean directions are separated by more than 90° in both Attention and FFN.

  • Figure 7 compares role-specific gradient distributions, mean directions and lengths, and paired cosine similarities across module families.Its t-SNE uses 6,400 gradient samples per role and module family; color indicates denoising timestep.Source: Figure 7 · p. 13
  • At TF initialization, mean gradient directions are 95.1° in Attention and 105.1° in FFN, both exceeding 90°.Source: C. Gradient Role Separation at TF… · p. 13
  • Most paired context-writing and denoising gradient cosine similarities are negative at TF initialization.The visualization samples 128 real videos and 50 denoising timesteps, with context fixed at t = 0.Source: C. Gradient Role Separation at TF… · p. 13, Figure 7 · p. 13

D. 24-Hour Generation Results

SGF+ examples illustrate continuous 24-hour generation from 5-second training windows. The accompanying layer-wise analyses describe attention and FFN gradient distributions, including a block with no context-writing gradients.

D. 24-Hour Generation Results

E. Layer-wise Gradient Role Separation

  • The attention analysis covers SGF gradients over 128 prompts and 4 denoising timesteps, using concatenated Q/K/V/O gradients and angular-distance t-SNE sketches.Silhouette scores summarize role separation before projection.Source: Figure 9 · p. 14
  • In FFN block 29, context-writing gradients are zero for all 512 prompt–timestep pairs, so angular distances are undefined and no t-SNE is shown.Source: Figure 10 · p. 15

F. Additional Qualitative Comparisons

The additional comparisons cover matched prompts across chunkwise and framewise generation, while the local analysis examines how separating role parameters changes descent under specified comparisons and assumptions.

F. Additional Qualitative Comparisons

G. Local Optimization Analysis of Role Separation

G.1. Fixed-replay objective and comparison geometry

  • The fixed-replay surrogate holds rollout latents, sampled denoising exits, supervision, and forward randomness fixed; its differentiable KV path carries the writer derivative.The stated product-space budget matches parameter displacement, not parameter count or computational cost.Source: §G.1 · p. 22

G.2. Descent lost under parameter sharing

  • Under the product-space budget, tied updates combine the role gradients, whereas independent updates can use their separate directions.The comparison concerns first-order decrease at the same point.Source: §G.1 · p. 22, §G.2 · p. 22

G.3. Gradient cancellation and stationary points

  • The sharing restriction excludes the role-dependent gradient component; complete cancellation makes the tied objective stationary while the split objective is not.Negative alignment alone does not imply this stationary-point separation.Source: §G.3 · pp. 22–23

G.4. Finite-step descent under a common update budget

  • For normalized updates under a common product-space budget, the split update strictly outperforms the shared update and strictly decreases the surrogate under the stated smoothness and step-size conditions.The comparison is limited to the normalized updates specified in the analysis.Source: §G.4 · p. 23

G.5. Negative alignment under a common learning rate

Under negative alignment, a role-split update can outperform a shared update under a common learning rate, while partial separation retains restrictions from tied blocks. Future work proposes role-specific parameterization for world action models and lightweight context writers.

G.5. Negative alignment under a common learning rate

  • With negative alignment and sufficient smoothness, the split update outperforms the shared update and decreases the surrogate for sufficiently small η.The comparison uses a common learning rate.Source: §G.5 · pp. 23–24
  • The common-learning-rate result requires negative alignment of aggregated minibatch gradients; individual-sample or selected-module alignment is insufficient.The shared and split updates generally use different displacement budgets under this protocol.Source: §G.5 · pp. 23–24

G.6. Partial separation across network modules

  • Partial separation retains a restriction: tied blocks average role gradients, and separating only FFN leaves the Attention contribution.The stated comparison assumes tying matching parameters reproduces the same forward computation at the common point.Source: §G.6 · p. 24

H. Future Work

  • World action models are proposed as a setting to explore role-specific parameterization from the teacher-forcing stage.Separating context writing from visual and action prediction may enable more effective learning of historical representations for subsequent predictions.Source: H. Future Work · p. 25
  • A lightweight context writer could learn compressed KV representations from redundant histories under a constrained cache budget.The proposed goal is to reduce parameter storage, cache memory, and attention cost while preserving generative capabilities and long-horizon consistency.Source: H. Future Work · p. 25
Loading 2610.10429v1…