Source-linked AI summary
LIVE: Long-horizon Interactive Video World Modeling
Junchao Huang, Ziyang Ye, Xinting Hu, Tianyu He, Guiyu Zhang, Shaoshuai Shi, Jiang Bian, Li Jiang
TL;DR
Long-horizon autoregressive video world models suffer from compounding prediction errors. LIVE uses cycle consistency to bound error accumulation without teacher distillation and achieves stable, high-quality generation beyond training rollout lengths.
Problem
Autoregressive video world models face compounding errors from exposure bias, limiting reliable generation over long horizons.
Method
LIVE forward-rolls out from ground-truth frames, reverse-generates to reconstruct the initial state, and applies diffusion loss to enforce cycle consistency.
Results
LIVE maintains stable FID around 10 from 32 to 200 frames, while baselines degrade beyond 64 frames.
Takeaways & Limitations
LIVE robustly extends effective rollout horizons beyond the training window while preserving generation quality on long-horizon benchmarks.
Takeaways & Limitations
Teacher-based sequence-level approaches incur substantial computational overhead and can constrain output diversity through teacher-dependent distillation.
Abstract
from arXiv · showhide
Autoregressive video world models predict future visual observations conditioned on actions. While effective over short horizons, these models often struggle with long-horizon generation, as small prediction errors accumulate over time. Prior methods alleviate this by introducing pre-trained teacher models and sequence-level distribution matching, which incur additional computational cost and fail to prevent error propagation beyond the training horizon. In this work, we propose LIVE, a Long-horizon Interactive Video world modEl that enforces bounded error accumulation via a novel cycle-consistency objective, thereby eliminating the need for teacher-based distillation. Specifically, LIVE first performs a forward rollout from ground-truth frames and then applies a reverse generation process to reconstruct the initial state. The diffusion loss is subsequently computed on the reconstructed terminal state, providing an explicit constraint on long-horizon error propagation. Moreover, we provide an unified view that encompasses different approaches and introduce progressive training curriculum to stabilize training. Experiments demonstrate that LIVE achieves state-of-the-art performance on long-horizon benchmarks, generating stable, high-quality videos far beyond training rollout lengths.
1. Introduction
Autoregressive video world models suffer compounding errors over long horizons because training on ground-truth frames creates exposure bias and distributional shift. LIVE addresses this with teacher-free cycle consistency, a unified formulation, and progressive training to control error accumulation and stabilize long-horizon generation.
- Problem: Autoregressive world models face temporal error accumulation because exposure bias forces inference to condition on predictions rather than ground-truth frames.This causes compounding distributional shift over long horizons.
- Problem: Diffusion Forcing improves robustness on short sequences by adding stochastic noise to conditioning contexts but remains ineffective for long-horizon rollouts.Noised ground-truth inputs remain substantially different from genuine model rollouts with accumulated errors.
- Limitations: Self-Forcing trains on model-generated rollouts and distills a pre-trained teacher, but teacher dependence incurs substantial computational overhead.The overhead is particularly significant in domain-specific settings.
- LIVE: LIVE enforces bounded error accumulation through teacher-free cycle consistency, forwarding from ground-truth frames and reversing generation to reconstruct the initial state.The diffusion loss is computed on the reconstructed terminal state rather than matching full sequence distributions.
- LIVE: LIVE maintains distributional alignment by training fixed-length windows while explicitly modeling accumulated errors, enabling stable long-horizon generalization.The design aims to keep generation within a controlled error bound at inference time.
- Unified view and curriculum: LIVE unifies Teacher Forcing, Diffusion Forcing, and itself, while its progressive curriculum controls error tolerance through the ground-truth-to-rollout ratio.The curriculum facilitates stable optimization within each training window.
2. Related Work
Prior video diffusion models evolved from UNet-based temporal extensions to Diffusion Transformers that better capture global spatiotemporal dependencies. Causal sliding-window methods support real-time generation but face long-horizon error accumulation, while existing exposure-bias remedies only approximate rollout conditions.
- Video Diffusion Models: Video diffusion models progressed from UNet-based temporal extensions to Diffusion Transformers for modeling global spatiotemporal dependencies.Large-scale examples include Sora, Seaweed, HunyuanVideo, and Wan.
- Video Diffusion Models: Causal-attention methods with sliding windows enable real-time generation but face error accumulation during long-horizon inference.
- Mitigating Exposure Bias: Teacher Forcing creates exposure bias by training on ground-truth context while inference uses imperfect model rollouts.
- Mitigating Exposure Bias: Diffusion Forcing injects noise into ground-truth context to approximate rollout distributions, but noised ground truth remains fundamentally different from actual rollouts.
3. Preliminaries
Video world modeling learns p(x1:T | c1:T) over frame sequences conditioned on information such as camera poses and actions. Video diffusion models iteratively denoise noise, while autoregressive world models generate frames from prior context, commonly using sliding windows for interactive inference.
- Video world modeling learns the conditional distribution p(x1:T | c1:T), where x1:T is a T-frame sequence and c1:T contains per-frame conditions such as camera poses or actions.
- Video diffusion models gradually add Gaussian noise and learn an iterative reverse denoising process to predict that noise.The diffusion timestep t is sampled from [t1, . . . , tN], with Gaussian noise ϵ ∼N(0, I).
- Autoregressive world models generate each frame conditioned on previous frames and corresponding conditions, using only the most recent K frames in a sliding context window for real-time inference.Each frame xk is conditioned on x<k and c≤k, with the sliding-window variant retaining the latest K frames.
- Teacher Forcing conditions on ground-truth frames, whereas Diffusion Forcing noises the context; both address training, but inference still involves imperfect model rollouts.Diffusion Forcing independently samples context-frame noise, yet its noised ground-truth distribution differs from genuine rollouts with accumulated errors.
- Self-Forcing uses knowledge distillation from a teacher, but sequence-level distribution matching does not bound error accumulation and degrades beyond training rollout lengths.
4. Method
LIVE bounds long-horizon error accumulation through cycle consistency: it forward-rolls from ground-truth prompts, reverse-generates to recover them, and computes diffusion supervision on the reconstruction. A unified GT-ratio formulation and progressive curriculum adapt training from ground-truth contexts to increasingly imperfect rollouts.
- Cycle-Consistency Objective: LIVE forward-rolls from ground-truth frames, reverses the rollout and conditioning, and computes diffusion loss while reconstructing the initial prompt frames.This trains the model to map its imperfect rollouts back to the ground-truth manifold, creating a valid signal despite diverse future trajectories.
- Frame-Level Supervision: LIVE extends prompt-frame supervision to all T positions with independently noised repeated ground-truth frames, enabling parallel frame-level noise-prediction training.The reversed rollout context is used for conditioning at each position while the repeated prompt frames provide the supervision targets.
- Implicit Error Bounding: The recovery objective implicitly keeps forward distortion within a bounded range, preventing the monotonic quality degradation associated with autoregressive diffusion inference.The model must recover ground-truth frames from imperfect rollout context, directly reducing recovery distortion through gradient optimization.
- Unified Training Objective: Controlling the GT ratio p unifies Teacher Forcing, Diffusion Forcing, and LIVE, with p = T for pre-training and progressively smaller p during post-training.LIVE uses p < T with imperfect rollout context, whereas Teacher Forcing and Diffusion Forcing use p = T with perfect or noisy ground-truth context.
- Progressive Training Curriculum: Progressively decreasing p introduces more generated frames into context, strengthens error tolerance through harder recovery tasks, and enables robust long-horizon generation.The curriculum adapts the model to increasing error levels while maintaining recoverability through the cycle-consistency objective.
5. Experiments
Experiments show that LIVE maintains stable long-horizon generation quality and outperforms competing methods across real-world, game-engine, and interactive gameplay settings. Ablations support cycle consistency, random timestep context noise, and progressive rollout training as key design choices.
- Error Accumulation Analysis: LIVE maintains FID around 10 from 32 to 200 frames, while Teacher Forcing, Diffusion Forcing, DFoT, and Geometry Forcing degrade sharply beyond 64 frames.The models use the same architecture for TF, DF, and LIVE in this analysis.
- Quantitative Results: LIVE achieves state-of-the-art RealEstate10K performance, with especially large gains at longer rollout lengths.These results are summarized in Table 1 across the full test set and different rollout lengths.
- Qualitative Results: LIVE improves generation quality on UE Engine and maintains visual quality across extended indoor and outdoor RealEstate10K rollouts, unlike competing methods.The UE Engine comparison uses identical architectures trained with Teacher Forcing, Diffusion Forcing, and LIVE.
- Ablation Studies: Removing reverse generation substantially degrades performance, supporting cycle consistency as a valid training signal for imperfect rollouts with distributional diversity.The objective constrains recovery toward the original ground-truth state rather than directly supervising semantically divergent forward rollouts.
- Ablation Studies: Random timestep sampling for rollout-context noise achieves the best results, whereas no noise degrades at long horizons and fixed-scale noise offers only marginal improvement.The comparison evaluates no noise, fixed-scale noise, and the LIVE strategy.
- Ablation Studies: Progressively decreasing p from T to pmin outperforms setting p = 1 throughout post-training by gradually increasing rollout difficulty and error exposure.Starting with mostly ground-truth context enables easier recovery before progressively extending rollouts.
6. Conclusion
LIVE addresses error accumulation in autoregressive long-horizon interactive video generation through a cycle-consistency objective enforced by diffusion loss. The work also unifies TF, DF, and LIVE and introduces a progressive training curriculum to stabilize optimization.
- 6. Conclusion: LIVE is a long-horizon interactive video world model designed to address error accumulation in autoregressive generation.The paper identifies error accumulation as a fundamental challenge in long-horizon generation.
- 6. Conclusion: A cycle-consistency objective enforced through diffusion loss explicitly bounds long-horizon error propagation without teacher-based distillation.This objective is presented as the mechanism for avoiding reliance on teacher-based distillation.
- 6. Conclusion: The paper presents a unified perspective connecting TF, DF, and LIVE.This perspective is described as part of the paper’s broader contribution.
- 6. Conclusion: LIVE includes a progressive training curriculum that stabilizes optimization.The curriculum is introduced alongside the unified perspective.
7. Appendix
The appendix details LIVE’s training and evaluation configurations across RealEstate10K, UE Engine Videos, and Minecraft. Additional qualitative examples show that LIVE recovers from its own generated errors, improving long-rollout stability over distinct baseline failure patterns.
- RealEstate10K configuration: RealEstate10K experiments use a 774M-parameter DiT model with 18-step ODE sampling, 256×256 resolution, frame skip 2, and a 32-frame context window.Training uses approximately 50–60k videos, and metrics are reported on the complete test set containing over 7k videos.
- RealEstate10K configuration: LIVE is initialized from the converged NFD-DF checkpoint and trained for an additional 20k iterations, while NFD-TF and NFD-DF train from scratch for over 200k iterations.Experiments use 32 NVIDIA H100 GPUs, batch size 64, Adam, and learning rate 4 × 10−5.
- UE Engine Videos configuration: UE Engine Videos fine-tunes RealEstate10K weights at 352×640 resolution with frame skip 2, using 88 videos for training and 12 videos for testing across 12 scenes.The dataset contains 100 videos totaling 7,601 frames, and evaluation uniformly samples 50 starting frames from each test video.
- Minecraft configuration: Minecraft experiments use 224×384 resolution with frame skip 1 and evaluate on 300 action trajectories collected from MineDojo because WorldMem lacks an official test set.WorldMem contains approximately 10k interactive gameplay videos of 1,500 frames each, with 25-dimensional action vectors.
- Qualitative analysis: Qualitative rollouts show TF models developing color distortion and semantic inconsistency, DF models exhibiting exposure problems, and LIVE maintaining stable quality by recovering from self-generated errors.The examples span different datasets and long rollouts.