Source-linked AI summary
End-to-End Training for Autoregressive Video Diffusion via Self-Resampling
Yuwei Guo, Ceyuan Yang, Hao He, Yang Zhao, Meng Wei, Zhenheng Yang, Weilin Huang, Dahua Lin
TL;DR
Autoregressive video diffusion is limited by exposure bias and the resulting error accumulation, while existing post-training solutions often require bidirectional teachers or discriminators. The paper introduces Resampling Forcing, a teacher-free framework that self-resamples degraded histories, trains with causal frame-level diffusion loss, and routes sparse history context. It achieves quality comparable to distilled baselines and stronger long-video temporal consistency through native-length training.
Problem
Autoregressive video diffusion models face exposure bias because ground-truth histories during training differ from imperfect model histories during inference, causing error accumulation in long rollouts.
Method
Resampling Forcing trains autoregressive video diffusion models end to end by autoregressively resampling degraded history frames, applying causal parallel frame-level diffusion loss, and routing top-k relevant history frames.
Results
Resampling Forcing achieves generation quality comparable to state-of-the-art distilled models, outperforms extrapolated baselines on longer videos, and provides superior temporal quality across evaluated video lengths.
Takeaways & Limitations
Teacher-free native long-video training offers a practical route to scalable autoregressive video generation with stable long-horizon quality and efficient sparse context.
Abstract
from arXiv · showhide
Autoregressive video diffusion models hold promise for world simulation but are vulnerable to exposure bias arising from the train-test mismatch. While recent works address this via post-training, they typically rely on a bidirectional teacher model or discriminator. To achieve an end-to-end solution, we introduce Resampling Forcing, a teacher-free framework that enables training autoregressive video models from scratch and at scale. Central to our approach is a self-resampling scheme that simulates inference-time model errors on history frames during training. Conditioned on these degraded histories, a sparse causal mask enforces temporal causality while enabling parallel training with frame-level diffusion loss. To facilitate efficient long-horizon generation, we further introduce history routing, a parameter-free mechanism that dynamically retrieves the top-k most relevant history frames for each query. Experiments demonstrate that our approach achieves performance comparable to distillation-based baselines while exhibiting superior temporal consistency on longer videos owing to native-length training.
1 Introduction
Autoregressive video generation supports causal world simulation but suffers from exposure bias, error accumulation, and growing attention costs. Resampling Forcing addresses these challenges through teacher-free self-resampling and efficient history routing, achieving comparable quality to distilled models and stronger long-horizon behavior.
- Motivation: Exposure bias arises because training uses ground-truth histories while inference conditions on imperfect model outputs, causing errors to accumulate and potentially collapse long videos.The expanding historical context also increases attention complexity during long-horizon training and inference.
- Prior approaches: Existing approaches reduce train–test mismatch through distillation or adversarial post-training, but depend on bidirectional teachers or online discriminators.These auxiliary models impede scalable training from scratch.
- Resampling Forcing: Resampling Forcing directly trains the target autoregressive model by degrading history frames through online autoregressive resampling and predicting frames with causal, parallel diffusion loss.The resampling process is detached from backpropagation to avoid shortcut learning.
- History routing: History routing retrieves the top-k relevant history frames with a parameter-free router, maintaining near-constant attention complexity during long-horizon rollout.This preserves access to relevant long-term context without dense attention over every prior frame.
- Results: Resampling Forcing achieves generation quality comparable to state-of-the-art distilled models, while native long-video training improves longer-video generation and temporal consistency.The method also offers stable quality on long videos, as illustrated against teacher forcing and short-teacher distillation.
2 Related Works
Prior work on autoregressive video generation addresses causal modeling, exposure bias, and attention costs through several complementary strategies. However, distillation and adversarial alignment remain difficult to scale, motivating teacher-free inference simulation and sparse context mechanisms.
- Autoregressive and bidirectional generation: Autoregressive video generation sequentially predicts each frame from historical context, supporting potential applications in world and game simulation.This contrasts with bidirectional models, which synthesize frames jointly using past and future context.
- Exposure-bias mitigation: Teacher forcing suffers from increasing quality degradation as video length grows because training histories differ from inference-time model predictions.Prior responses include noise injection, independent diffusion noise levels, rolling denoising, and plan-and-interpolate strategies.
- Post-training alignment: Self Forcing-style post-training simulates inference by rolling out full videos, but distillation requires pretrained bidirectional teachers and adversarial methods require online discriminators.These dependencies limit scalability and training autoregressive models from scratch.
- Prediction conditioning: Conditioning on model predictions is established in autoregressive systems through scheduled sampling, diffusion self-conditioning, and error recycling for long-video inference.These methods replace or augment clean histories with model-derived information.
- Efficient attention: Efficient video attention research explores linear-complexity designs and sparse masks that prune less-activated tokens to reduce quadratic attention costs.These approaches target the computational bottleneck created by high-dimensional spatiotemporal video tokens.
3 Method
Resampling Forcing trains autoregressive video diffusion models on degraded histories while preserving clean prediction targets, using causal masking for parallel frame-level training. Self-resampling simulates inference-time errors, and history routing limits long-horizon attention to relevant frames.
- 3.1 Background: The model factorizes video generation across frames while using diffusion to synthesize each frame, with clean history features cached for subsequent autoregressive steps.A causal mask restricts each frame to its clean history, enabling parallel training with per-frame diffusion loss.
- 3.1 Background: Teacher forcing conditions training on ground-truth histories, but inference uses imperfect predictions, causing errors to accumulate and degrade long-horizon rollouts.The expanding historical context also increases attention cost for long videos.
- 3.2 Enhancing Error Robustness: Resampling Forcing corrupts clean history frames to a sampled timestep, then uses online model weights to autoregressively complete denoising and create degraded histories.The process resamples frames conditioned on previously degraded histories, with detached gradients and KV caching for efficiency.
- 3.2 Enhancing Error Robustness: The sampling timestep balances history faithfulness against error-correction flexibility: low values risk accumulation, whereas high values increase content drift.The method samples intermediate timesteps with a logit-normal distribution and can shift them according to model strength.
- 3.3 Routing History Context: Dynamic history routing retrieves the top-k relevant history frames instead of attending densely to the full context, maintaining stable attention complexity during long rollouts.Local sliding windows reduce cost but can sacrifice long-term dependency and global consistency.
4 Experiments
Experiments evaluate Resampling Forcing against autoregressive baselines, ablate self-resampling and timestep shifting, and study sparse history routing. The method maintains long-video quality and causality while reducing attention through dynamic history selection.
- Comparisons: The method shows stronger long-term visual robustness than strict autoregressive baselines, which progressively lose color, texture, and sharpness through error accumulation.The qualitative comparison covers generated 15-second videos and contrasts the method with Pyramid Flow, CausVid, and Self Forcing.
- Comparisons: Compared with LongLive, Resampling Forcing preserves strict causality: the liquid level rises monotonically during continuous pouring instead of rising and then falling.The authors attribute LongLive’s non-causal behavior to future information leakage from its bidirectional teacher and local sub-clip distillation.
- Comparisons: Resampling Forcing maintains comparable visual quality and superior temporal quality across video lengths, matching LongLive on longer videos while avoiding a pretrained 14B-parameter bidirectional teacher.The evaluation uses VBench on separate 0–5 s, 5–10 s, and 10–15 s segments of generated 15-second videos.
- Analytical Studies: Autoregressive self-resampling achieves the highest quality among error-simulation strategies, outperforming parallel resampling and additive noise augmentation.The comparison attributes this result to better matching inference-time error accumulation across time.
- Analytical Studies: A moderate timestep-shifting scale balances error accumulation against content drift, whereas weak resampling risks accumulation and strong resampling reduces semantic consistency with history.The ablation reports robustness to the shifting factor but uses extreme values to expose these trade-offs.
5 Discussions
The discussion presents Resampling Forcing as a teacher-free approach for stable long-horizon autoregressive video generation. It combines history self-resampling with history routing to mitigate error accumulation and control attention complexity.
- 5 Discussions: Resampling Forcing uses history self-resampling to mitigate error accumulation and history routing to maintain near-constant attention complexity as context grows.The framework is described as end-to-end and teacher-free, with experiments supporting visual quality and robustness under high attention sparsity.