PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Gradient Forcing for Robust Few-Step Autoregressive Video Generation

Chenxi Xia, Yixuan Xu, Shan Liu, Jiapeng Wang, Maohua Li, Hanlin Tang, Kan Liu, Tao Lan, Weiyi Lu, Xinggao Liu

arXiv:2610.14553v1cs.CV

TL;DR

Autoregressive video models can accumulate errors because they remain sensitive to errors in their generated histories. AGF uses future-block distribution-matching gradients to select adversarial history perturbations and trains on both original and perturbed histories. With five-second training rollouts, it achieves the highest 60-second VBench Total score and best four-minute visual quality among the evaluated baselines.

  • Problem

    Autoregressive video models remain sensitive to errors in their generated histories, allowing small deviations to propagate and degrade long-horizon generation.

    Source: §1 · pp. 1–2, §3.1 · p. 4

  • Method

    AGF uses future-block DMD gradients to select adversarial perturbations of generated histories and jointly trains on original and perturbed histories.

    Source: §1 · p. 2

  • Results

    With five-second rollouts for training, AGF achieves the highest 60-second VBench Total score and best four-minute visual quality among five evaluated baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    AGF sustains stable visual quality throughout four-minute generation despite using only five-second training rollouts.

    Source: §5 · p. 10

Abstract

from arXiv · show

Autoregressive video generation models generate videos sequentially, with each new block conditioned on previously generated blocks. By training on the model's own rollouts, Self Forcing and its follow-up methods substantially mitigate exposure bias caused by the mismatch between training- and inference-time history distributions. However, these models still suffer from error accumulation during long-horizon generation. We identify a key underlying limitation: the model remains highly sensitive to errors in its generated history, allowing small deviations to be amplified through subsequent generations. We refer to this underexplored problem as the local history robustness gap. To address this gap, we introduce Adversarial Gradient Forcing (AGF), a framework designed to improve the robustness of few-step autoregressive video models to perturbations and accumulated errors in their own generated histories. AGF uses distribution matching gradients from future block predictions to select adversarial perturbations of self-generated histories. It then jointly trains the generator on the original and perturbed histories to improve the robustness of future generation. Extensive experiments show that AGF achieves the highest VBench Total score on 60-second video generation and the best visual quality on four-minute video generation. Notably, AGF achieves such long-horizon robustness while requiring only 5-second rollouts for training.

1 Introduction

AGF targets long-horizon error accumulation by training few-step autoregressive video models to withstand errors in their self-generated histories. The paper reports improved long-video quality, including the highest 60-second VBench Total score and best four-minute visual quality among five baselines.

1 Introduction

  • Self Forcing trains on its own rollouts to narrow the training–inference gap, while follow-up methods add longer trajectories or supervision.Few-step denoising methods aim to reduce inference latency and support streaming and interactive video generation.Source: §1 · p. 1
  • Autoregressive models remain sensitive to errors in generated histories, allowing earlier-block errors to propagate and degrade later video quality.The paper calls this vulnerability the local history robustness gap.Source: §1 · pp. 1–2
  • AGF backpropagates future-block DMD gradients to select bounded adversarial perturbations of generated history blocks, then trains on original and perturbed histories.The original-history branch preserves performance under standard generation conditions, while the perturbed-history branch targets robustness to history errors.Source: §1 · p. 2
  • A prediction consistency regularizer is added during early training to keep perturbations from driving trajectories too far from the local data manifold.The regularizer improves consistency in viewpoint and scene structure.Source: §1 · p. 2
  • Trained on five-second rollouts, AGF achieves the highest 60-second VBench Total score and best four-minute visual quality among five baselines.AGF sustains stable quality throughout four-minute generation, while the baselines progressively degrade; ablations validate each component.Source: §1 · p. 2

2 Related Work

Prior work developed few-step autoregressive video models by changing how histories are constructed and by extending self-generated-rollout training. AGF focuses on selecting imperfect histories according to their downstream impact and using them in on-policy distribution matching.

2.1 Training Few-Step Autoregressive Video Models

  • Self Forcing trains on its own autoregressive rollouts to reduce the training–inference history mismatch, while later methods add longer trajectories, adjacent-block denoising, or gradient propagation.The subsection also traces approaches from Teacher Forcing and Diffusion Forcing through CausVid and AAPT.Source: §2.1 · p. 3

2.2 Training with Imperfect Histories

  • Prior imperfect-history methods use noise, corruptions, resampling, reused prediction errors, drifting rollouts, cache perturbations, or real-video teacher context.The cited methods do not explicitly assess each perturbation’s downstream impact to select informative training histories.Source: §2.2 · p. 3
  • AGF adds adversarial history selection to on-policy distribution matching, using perturbed histories to train robustness.Figure 2 contrasts unperturbed and once-perturbed Self-Forcing rollouts under the same model and sampling randomness; artifacts and subject distortion intensify earlier in the perturbed rollout.Source: §2.2 · p. 3, Figure 2 · p. 3

3 Method

Section 3 identifies the local history robustness gap and presents AGF, which selects harmful history perturbations for training on clean and perturbed histories. It also introduces an early-training consistency regularizer and visualizes the perturbations’ effects on later generation.

3.1 The Local History Robustness Gap

  • Small perturbations to a generated history block can cause artifacts to appear earlier and worsen in later generation; this vulnerability is the local history robustness gap.The section relates this perturbation mechanism to errors accumulated during autoregressive generation.Source: §3.1 · p. 4

3.2 Adversarial Gradient Forcing

  • AGF uses future-block DMD gradients to select a history perturbation that increases future-block DMD loss, rather than merely changing predictions.The perturbation is selected by backpropagating the later-block distribution-matching gradient through the future predictions to the history block.Source: §3.2 · pp. 4–5
  • AGF fixes the selected perturbation and jointly trains on clean and perturbed histories, with the adversarial loss focused on blocks after the perturbed block.The clean branch averages its loss over all rollout blocks, while the adversarial branch averages over subsequent blocks.Source: Figure 3 · p. 5, §3.2 · p. 6

3.3 Prediction Consistency Regularizer

  • The prediction consistency regularizer aligns clean- and adversarial-branch velocity predictions at the first denoising step, encouraging shared global scene structure while later steps refine local details.It is used during early training and then removed while AGF training continues.Source: §3.3 · p. 6

3.4 Visualizing the Effects of History Perturbations in AGF

  • AGF-selected perturbations target vulnerable fine textures, subtle color variations, and complex structures whose corruption can impair subsequent generation.The paper reports that training on these histories directs attention to preserving these features over long rollouts.Source: §3.4 · p. 6
  • Compared with Gaussian noise, AGF perturbations selectively affect scene details and produce more pronounced deviations in later generated frames.Figure 4 compares history and later-frame pixel differences for perturbations applied to the same history block.Source: §3.4 · p. 6, Figure 4 · p. 6

4 Experiments

Across experiments, AGF preserves quality better at long horizons while remaining comparable at five seconds. The evaluations also examine rollout stability, human preferences, training cost, and design choices in the perturbation and regularization strategy.

4.1 Experimental Setup

  • The evaluation covers native five-second generation through 240-second rollouts, using VBench and five-dimension Q5 for visual quality.Training uses five-second rollouts; evaluations include 944 prompts at five seconds, 240 at 60 seconds, and 120 at 240 seconds, with comparisons against five baselines.Source: §4.1 · p. 7

4.2 Main Long-Horizon Benchmark

  • AGF matches Self-Forcing comparably at five seconds, leads all methods in 60-second Quality and Total, and reaches 84.82 Q5 at 240 seconds.It ranks first in four of five quality dimensions at 240 seconds; Figure 5 shows it preserving the church and coastline through the final frame. Higher Dynamic Degree alone may reflect camera drift or visual corruption.Source: §4.2 · pp. 7–8, Table 1 · p. 7

4.3 Stability over Rollout Time

  • AGF's Q5 changes by +0.03 pp from the first to last 20-second interval, while all five baselines decline by 0.96–2.61 pp.Figure 6 reports changes relative to the first interval, averaged over prompts, with pointwise 95% bootstrap intervals.Source: §4.3 · p. 8, Figure 6 · p. 9

4.4 Human Evaluation

  • In 240 human-ranking tasks, AGF earns the best mean rank at 2.23, ahead of LongLive at 2.85.AGF receives 35.8% of first-place rankings and ranks in the top three in 84.2% of evaluations.Source: §4.4 · p. 8, Figure 7 · p. 9

4.5 Training Cost

  • AGF training takes 364.4 GPU-hours, less than Self-Forcing's 431.6 and LongLive's 2774.4 GPU-hours.AGF uses 400 outer iterations and five-second rollouts; OPSD-V requires 944.4 GPU-hours and approximately 45-second rollouts.Source: §4.5 · pp. 8–9

4.6 Ablation Studies

  • Gradient-selected perturbations improve 240-second Q5 by 9.71 pp over equal-RMS Gaussian perturbations, while regularization preserves clean-history scene consistency.Removing regularization lowers 240-second Q5 from 84.82 to 82.75 and 60-second Total from 81.47 to 77.96; regularizing more denoising steps or using either frozen-reference alternative underperforms the proposed choice. AGF also improves 240-second Q5 by 2.81 pp from a Causal Forcing Stage 2 ODE checkpoint.Source: §4.6 · pp. 9–10, Table 2 · p. 9

5 Conclusion

AGF targets history-error sensitivity in few-step autoregressive video generation and sustains stable visual quality through four-minute generation despite training on five-second rollouts.

5 Conclusion

  • AGF identifies sensitivity to errors in self-generated history as a key source of cumulative quality decay.It constructs history perturbations that are particularly detrimental to subsequent generations and trains the model to remain robust to them.Source: §5 · p. 10
  • AGF explicitly trains the model to remain robust to history perturbations that are especially detrimental to subsequent generations.The method builds on the identified sensitivity to errors in self-generated history.Source: §5 · p. 10
  • Despite training only on five-second rollouts, AGF sustains stable visual quality throughout four-minute generation.Source: §5 · p. 10

A Local History Robustness and Adversarial Selection

This appendix analyzes how history errors affect later outputs and derives AGF’s constrained perturbation selection using future-block distribution-matching gradients.

A.1 Causal Error Propagation

  • The state-difference recursion separates propagation of an existing history error from errors introduced at later steps.Transition products determine each contribution to the final state difference, which is projected onto later video blocks.Source: §A.1 · p. 14

A.2 Why Video Differences Need a Quality Criterion

  • Large output sensitivity alone does not establish distribution-matching error, so AGF selects perturbations by their effect on future-block DMD loss.The appendix gives an example where conditional distributions match although shared-noise sensitivity can grow arbitrarily large.Source: §A.2 · p. 15

A.3 Backpropagating the Future DMD Gradient

  • Backpropagating later-block DMD gradients through differentiable history key/value representations yields a gradient on the perturbed history block.A single vector–Jacobian product computes the sum without forming the Jacobians; these derivatives use recorded histories.Source: §A.3 · p. 15

A.4 Selecting a Perturbation under Sharing and RMS Constraints

  • The selected perturbation aligns with the projected history gradient while obeying frame-sharing and RMS constraints.If the shared direction is zero, AGF selects a zero perturbation; optimality is for the first-order criterion with the DMD gradient fixed.Source: §A.4 · p. 16

B.1 Applying DMD Gradients to the Generator

  • DMD compares generated and target video distributions after Gaussian noising, with its practical gradient using estimated scores and normalization.The normalization can change sample weighting relative to the unscaled KL gradient.Source: §B.1 · pp. 16–17
  • For exact scores, the score difference supplies an output gradient that is backpropagated through the generator.The gradient expectation includes prompts, generator sampling randomness, diffusion time, and noise.Source: §B.1 · p. 17

B Paired Training and Gradient Computation

  • Generator training uses a surrogate loss whose gradient applies a fixed DMD gradient without differentiating through either score model.The mean-reduced loss gradient includes a factor of 1/N; its numerical value is not an estimate of KL divergence.Source: §B.1 · p. 17

B.2 Parallel Forward Passes for Selection and Training

The training update selects a perturbation from recorded rollouts, trains on original and perturbed histories, and separately updates the fake-score model; early training adds a consistency regularizer.

B.2 Parallel Forward Passes for Selection and Training

  • Parallel forward passes recompute predictions from recorded histories, allowing all blocks to be predicted without inserting newly computed predictions into the history.The generator’s history representations are recomputed with trainable parameters during training, so gradients update both history encoding and denoising.Source: §B.2 · p. 18

B.3 Training the Fake-Score Model

  • One fake-score model is trained on a weighted mixture of original- and perturbed-history videos, including detached replay-buffer videos.The default branch weights are both 0.5; fake-score updates do not update the video generator.Source: §B.3 · pp. 18–19

B.4 Prediction Consistency Regularization and Weighting

  • The early-training regularizer compares first-step velocity predictions under the two histories and is weighted using estimated output-gradient RMS values.Its target gradient ratio rises to 0.05, remains there through iteration 200, then returns to zero for iterations 201–400.Source: §B.4 · p. 19

B.5 Complete Training Update

  • Each generator update records an autoregressive rollout, selects a history perturbation, recomputes both branches, and optimizes their DMD losses with scheduled regularization.The complete procedure stores detached branch videos for fake-score training; five fake-score updates occur per generator update.Source: §B.5 · pp. 19–20

C Visualizing Changes in History and Continuations

  • The visualizations compare decoded pixel differences in the perturbed history and a later frame against a matched-RMS Gaussian perturbation.Maps use consistent display scales; the latent-space RMS budget is matched before decoding.Source: C Visualizing Changes in History and… · p. 20

D.1 Training Configuration

The appendix documents training-cost accounting, long-horizon evaluation settings, and metric definitions and uncertainty for the reported comparisons.

D.1 Training Configuration

  • Training-cost estimates count only listed stages, exclude upstream initialization and data preparation, and use timing procedures that do not measure throughput under a common workload.AGF’s recorded time sums its regularized and pure-AGF runs; extrapolated estimates assume linear workload scaling with batch.Source: §D.1 · pp. 20–21, Table 5 · p. 24

D.2 Long-Horizon Inference Protocol

  • The evaluation uses 944 prompts at five seconds, a random 240-prompt subset at 60 seconds, and 120 prompts at 240 seconds, with shared inference settings across methods.The 60-second Temporal Flickering dimension retains fewer than 20 prompts per method after its static-video filter.Source: §D.2 · p. 22, Table 6 · p. 24

D.3 Metrics and Statistical Protocol

  • At 240 seconds, Q5 averages five visual-quality dimensions and Q6 adds Dynamic Degree; neither composite is an official VBench total.The reported bootstrap bands quantify prompt-level uncertainty, not variation across seeds, checkpoints, or evaluator versions.Source: §D.3 · pp. 23–24

E Complete Multi-Length Results

The multi-length appendix provides complete benchmark dimensions at 5, 60, and 240 seconds, alongside six-time-point visualizations of 240-second rollouts.

E Complete Multi-Length Results

  • Tables 8, 9, and 10 provide complete dimension scores for 5-second, 60-second, and 240-second evaluations, respectively.The 5-second table also reports valid prompt counts and method order; the 60-second table follows the same conventions.Source: Table 8 · p. 25, Table 9 · p. 25, Table 10 · p. 25

F Additional Long-Horizon Examples

  • The 240-second examples show six time points—0, 48, 96, 144, 192, and 240 seconds—with whole-video Dynamic Degree and Q5 scores.Q5 is the unweighted mean of subject consistency, background consistency, motion smoothness, aesthetic quality, and imaging quality.Source: F Additional Long-Horizon Examples · p. 25

F.1 Camera Drift and Loss of Scene Structure

Long-horizon examples contrast AGF’s preservation of scenes and dynamic interactions with drift, duplication, and structural corruption in other methods; the unperturbed rollout results show AGF maintaining quality over four minutes.

F.1 Camera Drift and Loss of Scene Structure

  • In camera-drift and detail examples, AGF keeps subjects with their settings and retains fine structures that competing methods lose or distort.Higher Dynamic Degree can accompany camera drift or structural degradation, so it does not necessarily indicate better visual quality.Source: §F.1 · pp. 25–27, Figure 10 · p. 26, Figure 11 · p. 26

F.2 Coherent Dynamic Scenes

  • AGF preserves coherent object manipulation and character movement, keeping the clay figure, salad ingredients, and dancing characters distinct within recognizable scenes.The examples cover sculpting, salad tossing, and character movement.Source: §F.2 · pp. 27–28, Figure 12 · p. 27

F.3 Subject Duplication and Background Corruption

  • AGF retains one main subject and recognizable backgrounds in examples where other methods introduce copies, patterned scenery, or artifacts.The examples include a gift scene, felt monsters, and a papercraft reef.Source: §F.3 · pp. 28–29, Figure 13 · p. 28

F.4 Complete Six-Method Comparisons

  • Across two further scenes, AGF combines the highest Q5 with clearer preservation of the main subjects and their spatial relationships.The bookshelf mushrooms remain recognizable on the shelf, while the dragon and torch-bearing people remain visible within a recognizable cavern.Source: §F.4 · p. 29

G.1 Quality over Unperturbed Rollouts

  • AGF stays near its initial local Q5 throughout four-minute unperturbed rollouts, while all five baselines finish below their initial scores.Table 11 averages each consecutive 20-second interval over the same 120 prompts.Source: §G.1 · p. 29, Table 11 · p. 29

G.2 Stability after Temporary History Perturbations

Under two temporary inference-time history perturbations, all methods lose quality during intervention and recover afterward; AGF returns closest to its initial level.

G.2 Stability after Temporary History Perturbations

  • Both perturbations are applied to newly generated latent blocks from 30 to 60 seconds, after which generation continues to 240 seconds without further error injection.Saved outputs are not edited; perturbations enter the history cache before encoding.Source: §G.2 · p. 29
  • The blur intervention progressively mixes each history block with a 5×5 spatial mean-filtered version, while the additive-error intervention uses a fixed gradient-derived error field.Blur strength rises from 0.0625 to 0.25; the additive perturbation is shared across the block’s three latent frames.Source: §G.2 · p. 31
  • −0.17 pp after blur and −0.13 pp after additive error are AGF’s mean Q5 changes over 90–240 seconds, versus baseline ranges of −2.10 to −1.03 pp and −2.31 to −1.01 pp, respectively.The trajectories include natural quality drift as well as intervention effects.Source: §G.2 · p. 31, Figure 15 · p. 31

H Additional Ablations

The appendix reports additional ablations of regularization choices and an alternative ODE initialization, using controlled settings for comparisons.

H Additional Ablations

  • The ablations use five-second rollouts and evaluate EMA weights at iteration 400 across the same 944/240/120 prompts at 5/60/240 seconds.Variants use one training run and one video per prompt, with ordinary, unperturbed inference.Source: H Additional Ablations · p. 31

H.1 Regularization Variants

  • Without regularization, the variant obtains 81.38/77.96 Total at 5/60 seconds and 82.75 Q5 at 240 seconds.The passage identifies this as the “None” variant in Table 2.Source: §H.1 · p. 31
  • The regularization variants compare per-step MSEs against detached clean predictions and use distinct reference targets, including a frozen generator or a frozen 14B teacher.The teacher generates a full video from shared initial noise without conditioning on the student’s generated history; the regularizer applies to continuation under perturbed history.Source: §H.1 · pp. 31–32

H.2 Alternative ODE Initialization

  • With alternative ODE initialization, all five Q5 dimensions increase over the released Stage 3 Causal Forcing model, while lower Dynamic Degree makes Q6 fall.The result reinforces reporting visual quality separately from extent of motion.Source: §H.2 · p. 32

I Human Evaluation Details

In a blinded human evaluation of four-minute videos, AGF receives the strongest preference rankings, consistent with the four-minute quality evaluation in Table 1.

I Human Evaluation Details

  • The study uses 240 ranking tasks from 12 participants, who rank six identity-hidden four-minute videos generated from the same prompt in each task.The tasks cover 120 prompts, and each participant completes 20 tasks.Source: I Human Evaluation Details · p. 32
  • Mean rank divides each method’s summed ranks by 240; Figure 16 shows the percentage assigned each rank, with darker cells indicating larger shares.The distribution complements mean rank by showing how often methods appear near the top or bottom; rows follow Figure 7’s mean-rank ordering.Source: I Human Evaluation Details · p. 32, Figure 16 · p. 33
  • AGF achieves the best mean rank (2.23), the largest first-place share (35.8%), and a top-three placement in 84.2% of ranking tasks.This preference trend is broadly consistent with the four-minute quality evaluation in Table 1.Source: I Human Evaluation Details · p. 32

Related papers

Loading 2610.14553v1…