Source-linked AI summary
Long-Context Autoregressive Video Modeling with Next-Frame Prediction
Yuchao Gu, Weijia Mao, Mike Zheng Shou
TL;DR
Long-context video models need to preserve dependencies over extended videos, but short-clip training limits long-range consistency and direct long-video training is computationally expensive. The paper introduces FAR and asymmetric long short-term context modeling, reporting state-of-the-art results in short- and long-video modeling while reducing redundant long-context tokens and training costs.
Problem
Short-clip training limits long-range video consistency, while directly training on long videos is computationally expensive because vision tokens grow rapidly.
Method
FAR models causal dependencies between continuous frames, and asymmetric long short-term context modeling compresses distant context while preserving fine-grained nearby-frame information.
Results
FAR achieves state-of-the-art performance in both short- and long-video modeling.
Takeaways & Limitations
The proposed approach provides an effective baseline for efficient long-context autoregressive video modeling.
Takeaways & Limitations
Experiments cover only up to 300 frames, about 20 seconds, and do not include large-scale text-to-video training.
Abstract
from arXiv · showhide
Long-context video modeling is essential for enabling generative models to function as world simulators, as they must maintain temporal coherence over extended time spans. However, most existing models are trained on short clips, limiting their ability to capture long-range dependencies, even with test-time extrapolation. While training directly on long videos is a natural solution, the rapid growth of vision tokens makes it computationally prohibitive. To support exploring efficient long-context video modeling, we first establish a strong autoregressive baseline called Frame AutoRegressive (FAR). FAR models temporal dependencies between continuous frames, converges faster than video diffusion transformers, and outperforms token-level autoregressive models. Based on this baseline, we observe context redundancy in video autoregression. Nearby frames are critical for maintaining temporal consistency, whereas distant frames primarily serve as context memory. To eliminate this redundancy, we propose the long short-term context modeling using asymmetric patchify kernels, which apply large kernels to distant frames to reduce redundant tokens, and standard kernels to local frames to preserve fine-grained detail. This significantly reduces the training cost of long videos. Our method achieves state-of-the-art results on both short and long video generation, providing an effective baseline for long-context autoregressive video modeling.
1 INTRODUCTION
Long-context video modeling is needed for world simulation, but short-clip training and the cost of long-video tokens limit long-range consistency. The paper introduces FAR and asymmetric long short-term context modeling to address these constraints.
- Existing models trained on approximately 5-second clips capture short-term motion but fail to maintain long-term consistency, such as remembering observed environments.
- Direct long-video training can capture long-range dependencies but is computationally expensive because video processing produces many vision tokens.
- FAR models causal dependencies between continuous frames while retaining full attention within each frame.
- FAR with stochastic clean context performs better than video diffusion transformers and Token-AR, establishing a strong autoregressive video-generation baseline.
- Long short-term context modeling uses standard patchification for nearby frames and larger kernels for distant frames to preserve detail while compressing redundant context tokens.
- The paper reports state-of-the-art performance for FAR in both short- and long-video modeling.
2 RELATED WORK
Prior video-generation work includes diffusion transformers, token-level and hybrid autoregressive models, and long-context approaches based on extrapolation or direct training. This paper positions FAR as a framework for using causal dependencies between continuous frames and efficient long-context modeling.
- Video diffusion transformers provide strong visual quality, but sliding-window autoregressive generation limits the visual context available for long videos.
- FAR is presented as learning causal dependencies from continuous frames and outperforming Token AR in both short- and long-video modeling.
- Hybrid AR-Diffusion models combine diffusion objectives for image-level modeling with autoregressive contexts across visual and language generation.
- Long-context language modeling commonly uses test-time extrapolation or direct fine-tuning on long sequences, with efficient fine-tuning methods reducing computational cost.
- Existing video diffusion transformers lack effective mechanisms for long-range context, while earlier long-video prediction methods were limited in visual quality and long-range consistency.
3 PRELIMINARY
Flow Matching trains a deterministic continuous vector field between data and noise distributions, while autoregressive models factor sequence generation into conditionals on preceding elements.
- 3.1 Flow Matching: Flow Matching learns a continuous vector field that deterministically connects data and noise distributions.The method uses a simpler alternative objective to stochastic differential-equation-based diffusion training.
- 3.1 Flow Matching: A linear interpolation between a data sample and a noise sample defines the continuous trajectory used for Flow Matching.The data sample is drawn from pdata(x), while the noise sample follows N(0, I).
- 3.1 Flow Matching: The trajectory has constant velocity, providing the target velocity for learning the transport between distributions.
- 3.2 Autoregressive Models: Autoregressive models generate each sequence element conditioned on all preceding elements.This conditional structure captures sequential dependencies in the data.
- 3.2 Autoregressive Models: FAR achieves better convergence than a video diffusion transformer in unconditional video generation on UCF101.
4 FAR
FAR is a frame-level autoregressive diffusion framework designed to model video efficiently, while stochastic clean context addresses its training–inference context gap. Long short-term context modeling, asymmetric patchification, and multi-level KV caching reduce long-video token and inference costs while preserving local detail.
- 4.1 FAR: FAR applies causal attention across frames while retaining full attention within each frame.It is built on a diffusion transformer and jointly supports image generation and image-conditioned video generation through its causal mask.
- 4.2 Short-Video Modeling: During training, later frames access noised previous latents, whereas inference uses clean context frames, creating a distribution shift.This mismatch is the training–inference gap in observed context.
- 4.2 Short-Video Modeling: Stochastic clean context is introduced to bridge the observed-context gap without the doubled training cost of maintaining a clean sequence copy.Adding mild inference noise can mitigate the gap but causes low-level flickering.
- 4.3 Long-Context Video Modeling: The proposed context scheme makes token context length grow more slowly with vision context length, lowering long-video training cost and memory usage.
- 4.4 Inference-Time KV Cache: FAR uses KV caching by encoding each decoded clean frame for reuse during autoregressive generation.
- 4.2 Short-Video Modeling: Test-time extrapolation trains FAR on short videos and evaluates it on long videos, but produces significantly lower quality than the baseline sliding-window approach.
- 4.3 Long-Context Video Modeling: Long short-term context modeling keeps nearby frames at high resolution and compresses distant frames with large patchify kernels.The design preserves fine-grained local consistency while reducing redundant long-range context tokens.
- 4.4 Inference-Time KV Cache: In long-context generation, distant frames use an L2 cache with 4 tokens per frame, while recent frames use an L1 cache.When a frame exits the short-term window, it moves to L2 and the L1 cache is re-encoded.
5 EXPERIMENT
Experiments evaluate FAR on unconditional, conditional, short-video, and long-context prediction tasks. FAR achieves strong performance while asymmetric context compression and KV caching reduce long-video training and inference costs.
- Main Results: FAR achieves state-of-the-art unconditional and conditional video generation on UCF101, outperforming Latte and OmniTokenizer without additional training cost.Latte uses a video diffusion transformer, while OmniTokenizer uses Token AR; recent frame-autoregressive models require twice the training cost.
- Short-Video Prediction: FAR achieves superior short-video prediction on UCF-101 and BAIR without the multi-scale fusion strategies or optical flow used by MCVD and ExtDM.The evaluation samples 256 videos with 100 trajectories per context and selects the best trajectory for pixel-wise metrics.
- Long-Video Prediction: FAR obtains the lowest LPIPS prediction error on Minecraft and DMLab long-context prediction without prohibitive computation cost.The setup uses 144 observed frames to predict 156 future frames, with additional FVD evaluation from 36 context frames over 264 generated frames.
- Ablation Study: A [4,4] patchify kernel for distant context significantly reduces training cost without sacrificing prediction accuracy compared with a larger kernel.The selection condition is c × c × d ≤ D; with d=32 and D=768, a 4 × 4 kernel yields 512 < 768.
- Ablation Study: 104 seconds is the sampling time with long short-term context and multi-level KV caching, versus 171 seconds with KV caching alone and approximately 1341 seconds for baseline FAR.The combined design provides the largest reported speedup for generating a 256-frame video.
6 CONCLUSION
The paper presents FAR and long short-term context modeling as an efficient approach to long-context video generation. It reports strong short- and long-video performance but identifies limited scale and duration as remaining boundaries.
- Conclusion: Direct test-time extrapolation is insufficient for effective long-context video modeling, motivating efficient training on long videos.The conclusion identifies context redundancy as a key bottleneck in video autoregression.
- Conclusion: Long short-term context modeling with asymmetric patchify kernels eliminates redundant context and significantly reduces long-video training cost.The method retains nearby-frame detail while compressing distant context.
- Conclusion: FAR achieves state-of-the-art performance in both short- and long-video modeling.The conclusion presents FAR as a promising direction for long-context video generation and world modeling.
- Limitations and Future Work: The experiments do not include large-scale text-to-video training or minute-level videos, extending only to 300 frames, about 20 seconds.Future work proposes scaling FAR on text-to-video datasets and evaluating minute-level video modeling.
7 APPENDIX
Appendix materials document experimental configurations and provide additional qualitative long-video comparisons on DMLab and Minecraft. These visualizations show more consistent later predictions from FAR.
- Experimental Configurations: Ablation studies use half the training iterations while keeping other experimental settings unchanged.The appendix refers to Table 8 for detailed training and evaluation configurations.
- Qualitative Comparison: Additional DMLab and Minecraft visualizations show FAR better exploits provided context and produces more consistent later predictions than previous works.The appendix identifies these comparisons as Figures 12 and 13.
- Experimental Configurations: Table 8 records FAR’s configurations following evaluation settings from Latte, MCVD, and TECO.These settings cover the paper’s short- and long-video experiments.