PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoAR: Autoregressive Video Generation via Next-Frame & Scale Prediction

Longbin Ji, Xiaoxiong Liu, Junyuan Shang, Shuohuan Wang, Yu Sun, Hua Wu, Haifeng Wang

arXiv:2601.05966v2cs.CVcs.AI

TL;DR

Video generation models based on diffusion and flow matching can be computationally expensive and difficult to scale, while autoregressive video methods face spatial-temporal modeling and error-propagation challenges. VideoAR combines intra-frame VAR modeling, causal next-frame prediction, and a 3D tokenizer, with temporal modeling and training strategies for consistency. It reports state-of-the-art autoregressive results and faster inference, with VBench performance comparable to leading diffusion-based models.

  • Problem

    Diffusion and flow-matching video models are computationally expensive and difficult to scale, while autoregressive video methods face spatial-temporal modeling mismatches and error propagation.

    Source: §1 · p. 2, §4.2 · p. 5

  • Method

    VideoAR combines intra-frame VAR modeling with causal next-frame prediction, a 3D tokenizer, Multi-scale Temporal RoPE, Cross-Frame Error Correction, and Random Frame Mask.

    Source: §1 · p. 2, §4.1 · p. 3, §4.2 · p. 5

  • Results

    88.6 gFVD and 81.7 VBench scores are reported, alongside inference 13× faster than existing autoregressive baselines.

    Source: §7 · p. 9

  • Takeaways & Limitations

    VideoAR narrows the gap between autoregressive and diffusion-based models and establishes a foundation for future large-scale video-generation research.

    Source: §1 · p. 2

    VideoAR was trained with relatively modest sequence lengths, so long-horizon temporal coherence has not yet been exhaustively explored.

    Source: §6.1 · p. 9

Abstract

from arXiv · show

Recent advances in video generation have been dominated by diffusion and flow-matching models, which produce high-quality results but remain computationally intensive and difficult to scale. In this work, we introduce VideoAR, the first large-scale Visual Autoregressive (VAR) framework for video generation that combines multi-scale next-frame prediction with autoregressive modeling. VideoAR disentangles spatial and temporal dependencies by integrating intra-frame VAR modeling with causal next-frame prediction, supported by a 3D multi-scale tokenizer that efficiently encodes spatio-temporal dynamics. To improve long-term consistency, we propose Multi-scale Temporal RoPE, Cross-Frame Error Correction, and Random Frame Mask, which collectively mitigate error propagation and stabilize temporal coherence. Our multi-stage pretraining pipeline progressively aligns spatial and temporal learning across increasing resolutions and durations. Empirically, VideoAR achieves new state-of-the-art results among autoregressive models, improving FVD on UCF-101 from 99.5 to 88.6 while reducing inference steps by over 10x, and reaching a VBench score of 81.74-competitive with diffusion-based models an order of magnitude larger. These results demonstrate that VideoAR narrows the performance gap between autoregressive and diffusion paradigms, offering a scalable, efficient, and temporally consistent foundation for future video generation research.

1. Introduction

VideoAR adapts visual autoregression to video by combining intra-frame multi-scale synthesis with causal next-frame prediction, addressing modeling and error-propagation challenges. The introduction reports improved UCF-101 gFVD among autoregressive models and VBench performance comparable to leading diffusion methods.

1. Introduction

  • Autoregressive video generation faces mismatched spatial and temporal modeling, error propagation over long sequences, and limited control of video dynamics and duration.The introduction contrasts causal temporal progression with 2D spatial synthesis.Source: §1 · p. 2
  • VideoAR pairs VAR’s intra-frame modeling with causal next-frame prediction and extends the 2D tokenizer into a 3D architecture initialized with pretrained 2D-VAR weights.The Transformer predicts frames conditioned on text prompts and preceding frames.Source: §1 · p. 2
  • Cross-Frame Error Correction progressively increases the flip ratio across frames, while Random Frame Mask reduces reliance on preceding frames and over-memorization.The introduction also describes temporally adjustable classifier guidance for controlling dynamics and frame reencoding for extending duration.Source: §1 · p. 2
  • 90.3 gFVD on UCF-101 gives VideoAR-L a lower score than PAR-4×’s 99.5, setting a new autoregressive-model state of the art.The introduction reports this as a substantial improvement over the previous best autoregressive model.Source: §1 · p. 2
  • VideoAR-L performs comparably to CogVideo and Step-Video on VBench, generating 4-second videos at 384×672 with high fidelity and temporal consistency.The authors also report consistent quality improvements as the Transformer backbone grows.Source: §1 · p. 2
  • The authors present VideoAR as a framework combining VAR with next-frame prediction and report improved inference speed and computational efficiency over prior methods.They describe its tokenizer and Transformer backbone as achieving state-of-the-art results on standard generation benchmarks.Source: §1 · p. 2

2. Related Work

The related work contrasts diffusion-based video generation with autoregressive approaches. It traces AR visual generation from token prediction toward methods that combine temporal causality with multi-scale spatial generation.

2.1. Diffusion-based Video Generation

  • Diffusion and flow-based methods produce realistic video, while their iterative denoising remains computationally expensive and offers limited flexibility in generation length or fine-grained temporal coherence.The passage also describes AR-Diffusion approaches combining autoregressive modeling with diffusion processes.Source: §2.1 · pp. 2–3

2.2. Autoregressive Visual Generation

  • LlamaGen and MAGVIT-v2 use next-token prediction over VQ-VAE-style visual tokens, while PAR parallelizes prediction and Loong introduces temporal-balanced losses and multi-stage training.These approaches are described as improving inference efficiency or supporting video generation.Source: §2.2 · p. 3
  • Token-based AR video methods face excessive token lengths, low spatial resolution, error accumulation, and weak spatial correlations; VideoAR combines temporal causality with inter-frame multi-scale generation and cross-frame self-correction.The passage characterizes AR generation as sequential prediction of visual elements.Source: §2.2 · p. 3

3. Preliminary

The preliminary section describes VAR as a Transformer-based next-scale prediction process over multi-scale visual residuals. It also outlines how the model forms inputs from prior residuals and uses corruption during training to improve robustness.

3. Preliminary

  • VAR tokenizes an image into K multi-scale residual maps, then predicts each next-scale residual conditioned on earlier scales and the text prompt.The conditional distribution is written as p(R_k|R_1:k−1, Ψ).Source: §3 · p. 3
  • To predict R_k, the model aggregates earlier residuals, upsamples them to base resolution, and downsamples the result to the target scale.The up- and down-sampling operations are bilinear.Source: §3 · p. 3
  • During training, the Transformer receives a partially corrupted feature representation, with labels randomly flipped at probability p_flip to improve robustness to prediction errors.The stated aim is to mitigate train–test discrepancies and better generalize to autoregressive inference.Source: §3 · p. 3

4. Method

VideoAR combines a causal 3D tokenizer with multi-scale next-frame prediction, separating spatial synthesis within frames from temporal prediction across frames. Temporal positioning, error-correction training, random frame masking, and progressive training support long-form generation.

4.1. Visual Tokenizer

  • The causal 3D tokenizer compresses video into compact spatio-temporal latents, leveraging adjacent-frame redundancy while preserving fidelity.Removing non-causal temporal operations lets inference proceed chunk by chunk on extremely long videos without performance loss versus full-sequence inference.Source: §4.1 · p. 3

4. Method

  • VideoAR’s pipeline compresses video into spatio-temporal tokens, predicts multi-scale frame residuals autoregressively, and decodes them back into frames.The Transformer inputs combine text, accumulated video features, and scale embeddings; Multi-Scale Temporal RoPE encodes temporal, spatial, and scale-aware position information.Source: §4 · p. 3, Figure 2 · p. 4

4.2. Autoregressive Video Modeling

  • The Transformer predicts each frame’s residuals from previously generated frames, the current frame’s coarser scales, and the text prompt.Multi-Scale Temporal RoPE factorizes positional information across time, height, and width, supporting temporal awareness and spatial consistency across frames and scales.Source: §4.2 · pp. 4–5
  • Cross-Frame Error Correction progressively increases token corruption and carries prior-frame errors into the next frame’s first scale for correction.The model is trained on corrupted history and self-corrected targets; Random Frame Mask uses a causal sliding window to reduce over-reliance on distant frames while preserving temporal context.Source: §4.2 · p. 5
  • Three training stages progress from joint image and low-resolution video pretraining to higher-resolution training and long-form video fine-tuning.The final stage uses only high-resolution video data to capture extended motion dynamics and long-range temporal dependencies.Source: §4.2 · p. 5
  • Temporal-spatial adaptive classifier-free guidance adjusts text alignment and temporal dynamics during causal decoding.Larger guidance coefficients are associated with improved visual quality and stronger frame dynamics, while smaller coefficients yield more stable transitions and greater sampling diversity.Source: §4.2 · p. 6

5. Experiments

Experiments evaluate VideoAR on UCF-101 and real-world video-generation tasks, including tokenizer reconstruction, generation quality, efficiency, and qualitative continuation. Results report improved autoregressive benchmarks, competitive VBench performance, and gains from temporal-consistency components.

5.1. Experimental Setup

  • UCF-101 short-video generation uses 8K clips across 101 action categories, while evaluations cover tokenizer reconstruction, generation quality, and VBench perceptual and temporal metrics.Videos are resized to pixel sizes in (128, 256, 512) and sampled to 17–65 frames depending on the dataset.Source: §5.1 · p. 6

5.2. Experimental results

  • FVD 88.6 makes VideoAR-XL the best reported UCF-101 generator, while VideoAR-L uses 30 decoding steps and runs over 13× faster than PAR-4x.VideoAR-L generates a video in 0.86 seconds; its 30 decoding steps represent a more than 10× reduction.Source: §5.2 · p. 8, Table 3 · p. 8
  • The tokenizer compresses spatially by 16×, reducing sequence length 4× versus MAGVIT and OmniTokenizer while reaching rFVD 61, on par with MAGVIT’s 58.It encodes clips into a 5 × 8 × 8 latent token grid.Source: §5.2 · p. 8, Table 2 · p. 8
  • VBench score 81.74 is comparable to or exceeds results from substantially larger models, and VideoAR leads on Semantic Score at 77.15.The table caption also reports state-of-the-art results for Aesthetic Quality, Object Class, and Multiple Objects; semantic performance is associated with precise text-to-video alignment.Source: §5.2 · p. 8, Table 1 · p. 8
  • VideoAR-4B follows prompts aligned with input images and produces natural, consistent multi-iteration continuations exceeding 20 seconds.The examples include object-motion control, camera-trajectory adjustments, and single- or multishot I2V and V2V generation.Source: Figure 5 · p. 7, §5.2 · p. 9

5.3. Ablation Studies

  • Multi-scale Temporal RoPE lowers FVD from 96.04 to 94.95; Cross-Frame Error Correction reaches 92.50, and Random Frame Mask raises VBench from 76.22 to 77.00.Time-dependent Corruption lowers FVD to 93.57 before Error Inheritance Initialization yields the full-model result; Random Frame Mask is evaluated during 256px stage-I training.Source: §5.3 · p. 9, Table 5 · p. 9, Table 4 · p. 9

6. Discussion

VideoAR uses next-frame prediction with multi-scale modeling within each frame, unlike InfinityStar’s temporal-window 3D-VAR formulation. Its relatively modest training sequence lengths leave long-horizon coherence incompletely explored.

6.1. Comparsion to Concurrent work: InfinityStar

  • VideoAR combines next-frame prediction with multi-scale modeling within each frame, whereas InfinityStar generates over temporal windows using 3D-VAR.The authors describe VideoAR as trained from scratch on joint low-resolution image–video data, in contrast to InfinityStar’s fine-tuning from an 8B image-generation model.Source: §6.1 · p. 9
  • Relatively modest training sequence lengths mean long-horizon temporal coherence has not been exhaustively explored, although the framework has no inherent sequence-length limitation.The authors expect longer-context training and increased scale to improve long-term consistency.Source: §6.1 · p. 9

7. Conclusion

VideoAR extends VAR to video with a causal 3D tokenizer and Transformer generator, combining spatial and temporal modeling. Experiments report strong gFVD and VBench scores alongside faster inference than autoregressive baselines.

7. Conclusion

  • VideoAR reports gFVD 88.6 and VBench 81.7, with inference 13× faster than existing autoregressive baselines.The conclusion presents the findings as evidence for autoregressive video generation as a practical alternative to diffusion-based approaches.Source: §7 · p. 9

A. Implementation Details.

VideoAR uses a 3D video tokenizer and autoregressive Transformer, then trains the Transformer with temporal error-correction and frame-masking strategies. Inference uses different classifier-guidance schedules for UCF-101 and real-world text-to-video generation.

A. Implementation Details.

  • The tokenizer maps video clips into a discrete latent space, while an autoregressive Transformer generates from those latents.The tokenizer uses latent dimension 48 and temporal subsampling ratio 4; the Transformer follows LLaMA-family architectural principles.Source: A. Implementation Details · p. 12
  • Transformer training uses AdamW with β1 = 0.9, β2 = 0.95, a cosine learning-rate schedule of 1×10−4, and weight decay 0.05.Training also uses Cross-Frame Error Correction and Random Frame Mask; large-scale internal-data training extends the duration with the same hyperparameters.Source: A. Implementation Details · p. 12
  • Inference gradually raises CFG from γ = 1 →5 and then to γ = 10 within each UCF-101 frame, but reduces CFG from γ = 5 →3 for real-world generation.The real-world schedule reduces CFG along temporal and spatial dimensions together to maintain stronger spatial consistency.Source: A. Implementation Details · p. 12

B. Visualization of Generated Videos

VideoAR-4B generates videos across varied domains, with reported object consistency in high-dynamic scenes and high-aesthetic natural scenery. The examples also include multi-object semantic guidance and stylized or imaginative content.

B. Visualization of Generated Videos

C. Limitations and Future Work

VideoAR-4B is limited to 384×672 resolution and 8 FPS, below the paper’s stated commercial specifications. The authors also report motion drift in high-dynamic scenes and propose future changes to training and inference.

C. Limitations and Future Work

  • VideoAR-4B generates at 384×672 and 8 FPS, below the stated commercial specifications of 720P and 24 FPS.The authors attribute the constraint to limited training compute restricting sequence length and to the high overhead of full autoregressive VAR attention.Source: C. Limitations and Future Work · p. 12
  • VideoAR-4B can produce drifted motion in high-dynamic scenes, including complex human movements, which the authors attribute to autoregressive error propagation.They propose iterative inference-time rollouts and reinforcement learning as future research directions.Source: C. Limitations and Future Work · p. 12
Loading 2601.05966v2…