PaperlayerView exact revision on arXiv

Source-linked AI summary

Autoregressive Video Generation without Vector Quantization

Haoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo, Yufeng Cui, Huchuan Lu, Shiguang Shan, Yonggang Qi, Xinlong Wang

arXiv:2412.14169v2cs.CV

TL;DR

Existing visual-generation approaches face trade-offs between compact high-fidelity representations, flexible video lengths, and autoregressive in-context capabilities. NOVA addresses these issues with non-quantized temporal frame-by-frame and spatial set-by-set autoregression, achieving strong image and video results with a 0.6B-parameter model and lower training overhead. The paper also reports generalization across longer videos and diverse zero-shot applications.

  • Problem

    Vector-quantized autoregressive models require more tokens for higher quality, while diffusion models typically target fixed-length videos and lack autoregressive in-context capabilities.

  • Method

    NOVA uses non-quantized autoregression with causal frame-by-frame temporal prediction and spatial set-by-set prediction within each frame.

  • Results

    NOVA surpasses prior autoregressive video models in data efficiency, inference speed, visual fidelity, and video fluency, while achieving state-of-the-art text-to-image performance with lower training cost.

  • Takeaways & Limitations

    NOVA supports unified text-to-image and text-to-video generation, extended video durations, and diverse zero-shot applications within one autoregressive model.

  • Takeaways & Limitations

    The authors identify inconsistent video fluency as frame counts increase and note robustness and stability concerns from ground-truth contextual frames during training.

Abstract

from arXiv · show

This paper presents a novel approach that enables autoregressive video generation with high efficiency. We propose to reformulate the video generation problem as a non-quantized autoregressive modeling of temporal frame-by-frame prediction and spatial set-by-set prediction. Unlike raster-scan prediction in prior autoregressive models or joint distribution modeling of fixed-length tokens in diffusion models, our approach maintains the causal property of GPT-style models for flexible in-context capabilities, while leveraging bidirectional modeling within individual frames for efficiency. With the proposed approach, we train a novel video autoregressive model without vector quantization, termed NOVA. Our results demonstrate that NOVA surpasses prior autoregressive video models in data efficiency, inference speed, visual fidelity, and video fluency, even with a much smaller model capacity, i.e., 0.6B parameters. NOVA also outperforms state-of-the-art image diffusion models in text-to-image generation tasks, with a significantly lower training cost. Additionally, NOVA generalizes well across extended video durations and enables diverse zero-shot applications in one unified model. Code and models are publicly available at https://github.com/baaivision/NOVA.

1 INTRODUCTION

NOVA addresses efficiency and flexibility limits in autoregressive and diffusion-based video generation by using non-quantized temporal and spatial autoregressive modeling. It combines compact visual representations with in-context generation and reports strong video and image-generation results.

  • Motivation: Vector-quantized visual tokenizers struggle to achieve high fidelity and high compression simultaneously, increasing costs for high-resolution images and long videos.Higher-quality representations require more tokens, substantially increasing computation as resolution or sequence length grows.
  • Motivation: Video diffusion models use compact continuous latents but generally model fixed-length frame distributions and lack autoregressive in-context capabilities.This limits flexible video-length generation and unified context-dependent tasks compared with GPT-style autoregressive models.
  • Approach: NOVA reformulates video generation as non-quantized autoregressive prediction across temporal frames and spatial token sets.It predicts frames sequentially while using set-by-set prediction within each frame, extending non-quantized autoregression to video.
  • Approach: NOVA predicts frames in causal temporal order and token sets in random spatial order, supporting text-to-image, image-to-video, and text-and-image-to-video tasks.The framework treats text-to-video generation as a common task that can encompass these related generation settings.
  • Results: NOVA surpasses autoregressive video counterparts in data efficiency, inference speed, and video fluency while matching similarly scaled diffusion models.It achieves a VBench score of 80.1 at 2.75 FPS after 342 GPU days of A100-40G training.

2 RELATED WORKS

Prior visual-generation systems include diffusion models over fixed-length pixel or latent tokens and autoregressive models that raster-scan token sequences. Masked autoregressive methods improve parallelism through bidirectional prediction but can face context scalability and long-video coherence problems.

  • Diffusion Models: Image and video diffusion models typically learn joint distributions over fixed-length tokens in pixel or latent spaces.Video diffusion systems build on diffusion-based visual generation while operating over compact representations in many settings.
  • Evaluation Context: NOVA’s reported 2.75 FPS measurement uses a single NVIDIA A100-40G GPU with batch size 24.This specifies the hardware and batching condition for the cited throughput figure.
  • Diffusion Models: Diffusion-based approaches offer context scalability and zero-shot generalizability across diverse applications, including extending video-generation duration.These properties are highlighted as part of their broader application flexibility.
  • Raster-scan Autoregression: Raster-scan autoregressive models generate discrete pixel or latent tokens sequentially in a fixed spatial order.This formulation parallels language-model token generation but applies it to visual sequences.
  • Masked Autoregression: Masked autoregressive models predict randomly masked tokens with bidirectional attention, improving on inefficient line-by-line sequential modeling.For video, synchronous set-by-set modeling can still produce context-scalability and long-duration coherence problems, motivating NOVA’s temporal-spatial decoupling.

3 METHODOLOGY

NOVA separates video autoregression into causal frame-by-frame prediction across time and efficient set-by-set prediction within each frame, followed by continuous-space diffusion denoising. Its design combines temporal causality with bidirectional spatial modeling and addresses inference instability through motion-aware feature transformation.

  • 3.1 RETHINKING AUTOREGRESSIVE MODELS FOR VIDEO GENERATION: NOVA models video autoregressively by predicting frames in causal temporal order while generating token sets within each frame in a random spatial order.This decouples per-frame spatial generation from prediction across the video sequence.
  • 3.2 TEMPORAL AUTOREGRESSIVE MODELING VIA FRAME-BY-FRAME PREDICTION: Block-wise causal attention lets each frame attend to prompts, video flow, and preceding frames, while all tokens in the current frame attend bidirectionally.The formulation treats each frame as a causal meta-unit and preserves spatial relationships within the frame.
  • 3.3 SPATIAL AUTOREGRESSIVE MODELING VIA SET-BY-SET PREDICTION: Randomly ordered token sets act as meta causal tokens, enabling parallel masked decoding with bidirectional modeling inside each frame.NOVA adds spatial positional embeddings and uses unmasked token features to predict masked visual tokens set by set.
  • 3.3 SPATIAL AUTOREGRESSIVE MODELING VIA SET-BY-SET PREDICTION: An attempted temporal-indicator design caused image-structure collapse and declining video fluency as frame count increased.The authors attribute this to similar adjacent-frame indicators and weak robustness to cumulative inference errors from ground-truth contextual frames.
  • 3.4 DIFFUSION PROCEDURE DENOISING FOR PER-TOKEN PREDICTION: During training, a diffusion denoising loss estimates per-token probabilities in continuous-valued space, while inference starts from Gaussian noise and denoises step by step.The noisy token is conditioned on its diffusion timestep and NOVA output, with the noise schedule indexed by t.

4 EXPERIMENT

NOVA is evaluated across text-to-image, text-to-video, qualitative generation, extrapolation, multitasking, and architectural ablations. Results show strong benchmark performance, visual quality, temporal modeling, and zero-shot generalization, while the appendix studies training stability and efficiency.

  • Text-to-image results: NOVA achieves state-of-the-art text-to-image performance on GenEval, with leading T2I-CompBench and DPG-Bench results using only 16% of PixArt-α’s training overhead.The comparison covers both small-model and data-scale settings.
  • Text-to-video results: NOVA matches Emu3’s text-to-video performance at 80.12 versus 80.96 while using 0.6B rather than 8B parameters.It also outperforms CogVideo across several text-to-video metrics despite being smaller.
  • Qualitative results: NOVA produces high-fidelity images and videos with multi-view perspectives, smooth object motion, and stable scene transitions.These properties are illustrated in the qualitative text-to-image and text-to-video visualizations.
  • Zero-shot generalization: NOVA extrapolates videos beyond training length, reaching up to twice the original length by pre-filling generated frames and shifting text and BOV embeddings.The model also generates image-conditioned videos with realistic motion without text prompts.
  • Ablations: Temporal autoregressive modeling improves subject movement and reduces artifacts and temporal inconsistencies in contextual generalization and video extrapolation.Spatial-only autoregression also does not support key-value-cache acceleration during inference.
  • Ablations: The Scaling and Shift layer reduces drift between text-to-image and image-to-video losses, while rank 24 provides more accurate motion predictions.Extremely low ranks limit representation capacity and make motion modeling difficult; post-normalization before residual addition is the default design.

5 CONCLUSION

NOVA is presented as an autoregressive model for text-to-image and text-to-video generation. The conclusion emphasizes efficient, high-quality generation and identifies larger-scale experiments as future work.

  • Conclusion: NOVA combines temporal frame-by-frame prediction, spatial set-by-set generation, and continuous-space autoregression across contexts.The model is designed for both text-to-image and text-to-video generation.
  • Conclusion: NOVA delivers exceptional image quality and video fluency while minimizing training and inference overhead.The conclusion describes image generation as near-commercial quality and video generation as promising in fidelity and fluency.
  • Future directions: The paper identifies real-time and infinite video generation as possibilities for future development beyond Sora-like video diffusion models.These possibilities are presented as directions enabled by NOVA’s results, not as completed capabilities.
  • Future directions: Future work will scale model size and training data to explore NOVA’s limits.The authors describe this as a continuation of the current experiments.

APPENDIX

The appendix organizes implementation details, inference analyses, ablations, evaluations, and additional visualizations that support the main experiments.

  • Implementation details: The appendix provides additional Scaling and Shift architecture details and normalization configurations.These materials are organized in Sections A and B.
  • Efficiency and extrapolation: The appendix includes video extrapolation evaluations and inference time analysis.These are listed as Sections C and D.
  • Additional experiments: The appendix reports ablations on temporal autoregressive modeling and comprehensive DPG-Bench evaluation results.These appear in Sections E and F.
  • Qualitative results: Additional text-to-image and text-to-video visualizations are provided in Sections G and H.These sections extend the qualitative evidence accompanying the main results.

A ARCHITECTURE DETAILS OF SCALING AND SHIFT LAYER

The Scaling and Shift layer models cross-frame motion through relative distribution changes in a shared BOV-token space. NOVA uses post-normalization before residual addition, and experiments examine rank capacity and long-horizon extrapolation.

  • Scaling and Shift layer: The Scaling and Shift layer learns relative distribution variations within a unified BOV-token space instead of directly modeling each current-frame distribution.This design targets cross-frame motion changes.
  • Normalization: NOVA uses post-normalization before residual addition as its standard normalization design.The configuration is intended to control Transformer-block embedding ranges while preserving residual identity transformations.
  • Video extrapolation: After 50 autoregressive steps, extrapolation metrics are tracked against the point corresponding to three times the training length.Each step generates four frames because the VAE downsamples the temporal dimension by 4×.
  • Video extrapolation: Visualizations indicate that generated frames can remain close to the original video in content and overall image quality despite metric decline.The appendix therefore reports extrapolation up to 3× training length as a visual capability.

C VIDEO EXTRAPOLATION EVALUATIONS

NOVA is evaluated for video extrapolation by generating subsequent frames from an initial frame and text prompt. Generated frames remain visually similar to the original video, although accumulated dynamics errors cause quality metrics to worsen over time.

  • The evaluation uses 200 videos, generating subsequent frames from each initial frame and a textual prompt.LPIPS and PSNR measure extrapolation quality.
  • Generated frames begin deviating from ground truth after a few iterations because video-dynamics errors accumulate.
  • As extrapolation proceeds, per-frame PSNR decreases while LPIPS increases.The metric trends are reported in Figure 13.
  • Despite metric degradation, generated frames retain high similarity to the original video in content and image quality.This observation is reported in Figure 14.

D INFERENCE TIME ANALYSIS

NOVA’s temporal layers are substantially faster than its spatial layers during inference. The analysis identifies the spatial layers as the main opportunity for further acceleration.

  • 0.03 seconds are required by temporal layers, compared with 11.97 seconds for spatial layers in each video.Measurements use one NVIDIA A100 GPU with 40GB memory and batch size 24.
  • Temporal-layer efficiency highlights the spatial layers as the main target for further inference acceleration.

E ABLATIONS ON THE IMPACT OF TEMPORAL AUTOREGRESSIVE MODELING

The ablation shows that temporal autoregressive modeling improves motion-related VBench performance and makes video inference faster, especially for longer videos. Removing it slightly lowers the total score and substantially reduces dynamic-degree performance.

  • 75.84 versus 75.38 is the VBench total score with versus without temporal autoregressive modeling.The comparison uses the same experimental settings.
  • 23.27 versus 11.38 is the dynamic-degree score with versus without temporal autoregressive modeling.The dynamic-degree decline is more pronounced than the total-score change.
  • Temporal autoregressive modeling enables faster inference through key-value caching and frame-by-frame autoregressive processing.The speed advantage increases for longer videos.
  • The authors hypothesize that bidirectional attention requires more data and longer training to capture subtle motion changes than causal models.

F COMPREHENSIVE DPG-BENCH EVALUATION RESULTS

On DPG-Bench, NOVA outperforms most models of comparable size and matches the overall score of state-of-the-art models. Scaling brings only marginal gains and does not improve text rendering, motivating higher-quality text-to-image data.

  • NOVA outperforms most models of comparable size and matches the overall score of state-of-the-art models on DPG-Bench.The comparison is reported in Table 6.
  • Increasing model scale yields marginal improvements and does not boost text-rendering performance.
  • The authors attribute the text-rendering limitation to reliance on extensive web datasets such as LAION and DataComp.
  • Future work will focus on improving the quality of text-to-image data.

G MORE TEXT-TO-IMAGE VISUALIZATIONS

NOVA’s text-to-image visualizations show high-quality, detailed outputs across realistic, imaginative, and abstract subjects, reaching 1024×1024 resolution.

  • NOVA generates high-quality text-to-image outputs that accurately reflect textual descriptions across realistic, imaginative, and abstract subjects.Examples include landscapes, portraits, characters, and abstract concepts while maintaining detail and aesthetic quality.
  • 1024×1024 is NOVA’s maximum reported resolution for text-to-image generation.
  • The visualization prompts span portraits, fantasy characters, children, surreal scenes, and unpopulated landscapes.
Loading 2412.14169v2…