PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Next-Token: Next-X Prediction for Autoregressive Visual Generation

Sucheng Ren, Qihang Yu, Ju He, Xiaohui Shen, Alan Yuille, Liang-Chieh Chen

arXiv:2502.20388v2cs.CV

TL;DR

Visual autoregressive generation lacks a universally suitable prediction unit, and teacher forcing creates exposure bias that can accumulate inference errors. xAR generalizes tokens to flexible entities, regresses them continuously with flow matching, and trains on noisy contexts. On ImageNet-256, xAR-B outperforms larger baselines with 20× faster inference, while xAR-H reaches FID 1.24 and runs 2.2× faster than the previous best model.

  • Problem

    Visual generation lacks a universally agreed-upon token definition, while teacher forcing exposes autoregressive models to exposure bias and cascading inference errors.

  • Method

    xAR generalizes next-token prediction to flexible entities and uses flow-matching continuous regression with Noisy Context Learning on noisy preceding entities.

  • Results

    On ImageNet-256, xAR-B (172M) outperforms DiT-XL and SiT-XL (675M) with 20× faster inference, while xAR-H achieves a state-of-the-art FID of 1.24 and runs 2.2× faster than the previous best model.

  • Takeaways & Limitations

    Next-cell prediction provides the best balance between local structure and global coherence, while noisy-context training improves robustness and reduces cascading errors.

  • Takeaways & Limitations

    The paper uses a simple square cell design; dynamically shaped prediction entities remain a direction for future research.

Abstract

from arXiv · show

Autoregressive (AR) modeling, known for its next-token prediction paradigm, underpins state-of-the-art language and visual generative models. Traditionally, a ``token'' is treated as the smallest prediction unit, often a discrete symbol in language or a quantized patch in vision. However, the optimal token definition for 2D image structures remains an open question. Moreover, AR models suffer from exposure bias, where teacher forcing during training leads to error accumulation at inference. In this paper, we propose xAR, a generalized AR framework that extends the notion of a token to an entity X, which can represent an individual patch token, a cell (a $k\times k$ grouping of neighboring patches), a subsample (a non-local grouping of distant patches), a scale (coarse-to-fine resolution), or even a whole image. Additionally, we reformulate discrete token classification as continuous entity regression, leveraging flow-matching methods at each AR step. This approach conditions training on noisy entities instead of ground truth tokens, leading to Noisy Context Learning, which effectively alleviates exposure bias. As a result, xAR offers two key advantages: (1) it enables flexible prediction units that capture different contextual granularity and spatial structures, and (2) it mitigates exposure bias by avoiding reliance on teacher forcing. On ImageNet-256 generation benchmark, our base model, xAR-B (172M), outperforms DiT-XL/SiT-XL (675M) while achieving 20$\times$ faster inference. Meanwhile, xAR-H sets a new state-of-the-art with an FID of 1.24, running 2.2$\times$ faster than the previous best-performing model without relying on vision foundation modules (e.g., DINOv2) or advanced guidance interval sampling.

1. Introduction

The paper generalizes visual autoregression from next-token prediction to next-X prediction and addresses exposure bias through noisy-context training. On ImageNet-256, xAR achieves strong quality and speed results across model scales.

  • Motivation: Visual AR models lack a universally agreed-upon token definition, motivating prediction units beyond individual image patches.The paper asks how next-token prediction should generalize to next-X prediction for image generation.
  • Motivation: Traditional teacher forcing exposes models to ground-truth tokens during training, causing exposure bias and cascading errors during inference.At inference, models condition on their own predictions, which can produce context drift.
  • Method: xAR reformulates discrete token classification as continuous entity regression over flexible entities, including tokens, cells, subsamples, scales, and whole images.Flow matching guides regression at each autoregressive step.
  • Method: Noisy Context Learning trains xAR on noisy entities rather than pristine ground-truth inputs, improving robustness and mitigating exposure bias.The model learns from imperfect, corrupted, or partially inaccurate conditions instead of relying exclusively on teacher forcing.
  • Results: 20× faster inference is achieved by xAR-B (172M), which outperforms DiT-XL and SiT-XL (675M) on ImageNet-256 generation.The comparison is reported for the base model against substantially larger baselines.
  • Results: FID 1.24 establishes xAR-H as a new state-of-the-art on ImageNet-256.The largest model achieves this result while running 2.2× faster than the previous best-performing model.

2. Related Work

Prior work established autoregressive modeling in language and extended it to visual generation through discrete or learned image representations. More recent approaches model visual tokens in continuous spaces while retaining autoregressive structure.

  • AR Modeling in NLP: Autoregressive language models predict each next token from preceding context and have demonstrated scalability and zero-shot generalization.This paradigm drove progress toward general-purpose AI and influenced applications across modalities.
  • AR Modeling in Vision: Visual autoregressive methods adapt language-style sequential prediction by converting images into token sequences for high-fidelity generation.Image patches or learned latent codes provide the representations modeled by autoregressive architectures.
  • AR Modeling in Vision: VQ-VAE introduced discrete latent image codes, while Parti and LlamaGen combined learned tokens with Transformers for scalable high-fidelity image generation.These methods use compressed image representations that can be modeled autoregressively.
  • AR Modeling in Vision: MAR replaced categorical cross-entropy with a diffusion loss to model per-token probability distributions in continuous space.This represents a shift toward more abstract and continuous token modeling within autoregressive generation.

3. Method

xAR generalizes autoregressive visual generation from next-token prediction to flexible entity-level prediction and replaces teacher-forced classification with noisy-context continuous regression. Its entities can represent different spatial or resolution structures, while flow matching and Noisy Context Learning support robust autoregressive inference.

  • Next-X Prediction: xAR defines X as a flexible prediction entity that may be a patch token, cell, subsample, scale, or entire image.An entire-image entity reduces xAR to a flow-matching method, while scale entities progressively refine predictions from coarse to fine.
  • Entity Design: 8×8 cells achieve the best performance among the evaluated entity designs and become xAR’s default choice.Cell prediction aggregates neighboring tokens to capture local structures and regional contexts at a higher abstraction level.
  • Next-X Prediction: xAR replaces discrete token classification conditioned on ground-truth entities with continuous entity regression conditioned on previous noisy entities.The regression objective is guided by flow matching at each autoregressive step.
  • Noisy Context Learning: Noisy Context Learning trains xAR on independently noised preceding entities, reducing reliance on clean inputs and mitigating exposure bias.Noise levels are sampled without ordering constraints, exposing the model to varying degrees of corruption in preceding entities.
  • Inference Scheme: During inference, xAR sequentially generates each entity from a Gaussian noise sample while conditioning on previously generated clean entities.With cells as the default entity, generation proceeds cell by cell through flow matching.
  • Motivation: Traditional visual AR models face information-density and accumulated-error challenges, which xAR addresses through semantically richer entities and noisy-context training.Cells shift prediction from isolated tokens toward coherent local and regional patterns.

4. Experimental Results

Experiments on ImageNet at 256×256 and 512×512 evaluate xAR’s main results, prediction entities, cell sizes, and noisy-context schedules. xAR achieves strong generation quality and speed, while cell-based prediction and random noise perform best in the reported ablations.

  • Main Results: 1.24 FID makes xAR-H state of the art on ImageNet-256 without vision foundation models or guidance interval sampling.It improves over StyleGAN-XL, RAR, VAR, MAR, and REPA by 1.06, 0.28, 0.24, 0.73, 0.31, and 0.18 FID, respectively.
  • Main Results: 9.8 images per second lets xAR-B surpass DiT-XL by 0.55 FID while running 20× faster despite using 172M rather than 675M parameters.
  • Main Results: 1.70 FID makes xAR-L state of the art on ImageNet-512, surpassing DiT-XL/2, DiMR-XL/3R, VAR-d36, and REPA.The reported FID margins are 1.34, 1.19, 0.93, and 0.38, respectively.
  • Ablation Studies: 2.48 FID makes cell-based xAR the best prediction-entity variant, outperforming token-based xAR by 1.03 FID and scale-based xAR by 0.42 FID.The ablation compares tokens, cells, subsamples, scales, and entire images.
  • Ablation Studies: 8×8 cells achieve the best reported cell-size performance at 2.48 FID, while treating the entire image as one entity reaches 3.13 FID.The study evaluates k ∈ {1, 2, 4, 8, 16} on a 16×16 continuous latent representation.
  • Ablation Studies: Random noise performs best in Noisy Context Learning, whereas clean, increasing-noise, and decreasing-noise schedules are inferior.The clean variant is equivalent to vanilla autoregressive training with teacher forcing.

5. Conclusion

The conclusion presents xAR as a generalized next-X framework for autoregressive visual generation. It identifies next-cell prediction and Noisy Context Learning as central design choices and reports state-of-the-art ImageNet-256 and ImageNet-512 performance.

  • xAR generalizes autoregressive visual generation from next-token prediction to flexible next-X prediction with continuous entity regression.
  • Next-cell prediction provides the best balance between local structure and global coherence in the reported exploration.
  • Noisy Context Learning trains on noisy entities instead of pristine ground-truth inputs, improving robustness and reducing cascading errors.
  • xAR achieves state-of-the-art performance on ImageNet-256 and ImageNet-512.

Appendix

The supplementary material provides implementation details, speed comparisons, limitations and future directions, and additional xAR visualizations.

  • Supplementary Section A details xAR hyper-parameters, while Section B provides a comprehensive speed comparison.
  • Supplementary Section C discusses limitations and future directions, and Section D presents visualization samples generated by xAR.

A. Hyper-parameters for xAR

The appendix documents xAR’s training and inference settings through detailed hyper-parameter tables.

  • Table 6 lists the detailed training and inference hyper-parameters of xAR models.

B. Speed Comparison.

xAR achieves higher reported ImageNet-256 generation efficiency across model sizes, combining strong FID with substantially faster sampling than several diffusion-, flow-matching-, and autoregressive-based methods.

  • 20× speedup: xAR-B (172M) outperforms DiT-XL, SiT-XL, and MAR while reaching 9.8 versus 0.5 images/sec.The comparison uses diffusion-, flow-matching-, and autoregressive-based models.
  • 1.24 FID: xAR-H establishes a new ImageNet-256 state of the art while running 2.2× faster than REPA.
  • 5.3× faster: xAR-L surpasses REPA, reaching 3.2 versus 0.6 images/sec.

C. Discussion and Limitations

The experiments favor simple square 8×8 cells over rectangular alternatives, while the paper identifies dynamically shaped entities as an open direction for adapting prediction units to image content.

  • Square 8×8 cells achieve the best performance, with no noticeable difference from rectangular alternatives such as k/2×2k or 2k×k/2.The rectangular configurations introduce additional complexity without clear benefits.
  • Future work could test dynamically shaped prediction entities for regions with different semantic densities, such as objects and sky.The current work retains a simple square-cell design and reports state-of-the-art ImageNet generation results.

D. Visualization of Generated Samples

The visualizations present xAR-generated samples across diverse ImageNet categories, including animals, landscapes, and natural phenomena, and describe them as high fidelity.

  • Additional xAR-H visualization results are provided across Figures 5–13.
  • The animal examples include American eagle, macaw, golden retriever, otter, and lesser panda images.These samples are identified in Figures 5–9.
  • The landscape and natural-phenomenon examples include coral reef, geyser, valley, and volcano images described as high fidelity.These samples are identified in Figures 10–13.
Loading 2502.20388v2…