Randomized Autoregressive Visual Generation

1. Introduction

• Autoregressive models underpin language models such as GPT, Llama, and Gemini, and have also shown competitive visual-generation performance against diffusion and non-autoregressive transformers.The paper describes autoregressive modeling as a promising path toward unified models across modalities and tasks.Source: §1 · p. 1
• Visual signals are low-level and redundant, making bidirectional context more important; prior studies found causal attention on image tokens inferior to bidirectional attention.Autoregressive image models have often underperformed diffusion models and non-autoregressive transformers on visual generation tasks.Source: §1 · p. 1
• VAR and MAR restore bidirectional attention through changes to the autoregressive formulation, which can complicate integration into universal transformer architectures.VAR uses next-scale rather than next-token prediction, while MAR generalizes a masked-transformer framework.Source: §1 · p. 2
• RAR maximizes expected likelihood across factorization orders by randomly permuting the input with probability r, which decays linearly from 1 to 0 during training.This transitions training from fully random factorization toward the raster order commonly used by autoregressive image generators.Source: §1 · p. 2, Figure 2 · p. 2
• RAR retains the original autoregressive architecture and formulation while improving generation quality, preserving compatibility with language-model frameworks.The paper describes RAR as a simple, effective, and scalable training paradigm.Source: §1 · p. 2
• On ImageNet-256, RAR achieves an FID score of 1.48, substantially outperforming previous state-of-the-art autoregressive image generators.The paper presents RAR as preserving language-model compatibility while improving generation quality at no additional cost.Source: §1 · p. 2

2. Related Work

• Autoregressive visual modeling progressed from pixel sequences to discrete-valued image tokens, with later work improving tokenization and adapting language-model architectures.These developments produced notable performance improvements.Source: §2 · p. 2
• Prior autoregressive visual methods use raster order, creating unidirectional information flow; this work explores other factorization orders while retaining the autoregressive framework.The paper notes that raster order is widely adopted but not a universally correct way to arrange image tokens.Source: §2 · p. 3
• MAR also experimented with random-order autoregression, but replacing raster order with random order alone brought only marginal improvement, underscoring the role of RAR’s annealing strategy.The paper reports this observation as consistent with prior findings in.Source: §2 · p. 3

3.1. Background

• Autoregressive factorization predicts each token from preceding tokens, limiting context to one direction despite images' lack of a predefined token order.Raster order is widely adopted for images, while visual signals have bidirectional correlations and no universally correct causal sequence.Source: §3.1 · p. 3, §3.2 · p. 3

3.2. RAR: Randomized AutoRegressive Modeling

• RAR trains autoregressively across randomized factorization orders, exposing each token to varied contexts while preserving the autoregressive framework.The shared parameters learn relationships between each token and every other token across sampled permutations.Source: §3.2 · p. 3
• RAR's permutation objective improves image-generation performance, but generated tokens still lack full global context; resampling or refinement is left for future work.The authors note that such techniques may help ensure every token is generated with sufficient context, but may complicate the system.Source: §3.2 · pp. 3–4
• Target-aware positional embeddings encode which token is being predicted, preventing different permutations from producing identical logits for different target labels.The figure contrasts that failure case with target-aware positions guiding the model to predict the next token accurately.Source: §3.2 · p. 4, Figure 3 · p. 4
• The target-aware embedding is omitted for the final token and can be merged with the original positional embedding after training, adding no inference parameters or computation.The final token has no prediction target and does not participate in the loss.Source: §3.2 · pp. 4–5
• Randomness annealing starts with random permutations and linearly decays their probability to zero, returning training to raster order.The schedule uses start and end epochs; the authors describe it as balancing permutation diversity with raster order's effectiveness.Source: §3.2 · p. 5

4.1. Implementation Details

• The setup uses MaskGIT-VQGAN to encode each 256 × 256 image as 256 discrete tokens and trains ViT-based RAR variants from 133M to 1,499M parameters.Training uses ImageNet-1K; final models use ten-crop transformations, and sampling uses classifier-free guidance with a power-cosine schedule.Source: §4.1 · pp. 5–6

4.2. Ablation Studies

• The best RAR-L annealing schedule randomizes through epoch 200, decays permutation through 300, then uses raster order; FID improves from 3.08 to 2.18.Source: §4.2 · p. 6, Table 2 · p. 6
• Across six terminal scan orders, all variants score reasonably well, but row-major raster order retains an advantage and is used for final models.Source: Table 3 · p. 6, §4.2 · p. 7

4.3. Main Results

• RAR-B (261M) reaches FID 1.95 versus DiT-XL/2 2.27 and SiT-XL 2.06, using 39% of their parameters.The authors also report that RAR-B uses 91% and 81% fewer parameters than LlamaGen-3B-384 and Open-MAGVIT2-XL, respectively.Source: §4.3 · p. 7, Table 4 · p. 7
• RAR-XXL's FID 1.48 also beats VAR (1.73) and MAR (1.55), while RAR retains language-modeling compatibility.Source: §4.3 · p. 7, Table 4 · p. 7
• On ImageNet-1K 256 × 256, RAR achieves FID 1.48, outperforming LlamaGen-3B-384 (2.18), MDTv2-XL/2 (1.58), and MaskBit (1.52).Table 4 reports generation results evaluated with ADM.Source: §4.3 · p. 7, Table 4 · p. 7
• RAR-XL generates 8.3 images/sec at better FID, 11.9× MaskBit and 27.7× MAR-H; RAR-XXL is 9.1× and 21.3× faster, respectively.Throughput includes detokenization and is compared among methods with similar FID scores.Source: §4.3 · p. 8, Table 5 · p. 8
• Larger RAR models lower training loss and improve FID with or without classifier-free guidance; samples also gain fidelity and diversity, especially in dogsled.Source: §4.3 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8

• RAR improves visual generation quality while preserving the autoregressive structure, and outperforms prior autoregressive, non-autoregressive transformer, and diffusion models.Source: §5 · p. 8

• The supplementary material includes final-model hyperparameters, RAR pseudocode, scan-order visualizations, and more generated samples.Source: Appendix · p. 9

• Table 6 lists the detailed training and sampling hyperparameters for the final RAR models.Source: A. Hyper-parameters for Final RAR Models · p. 9, Table 6 · p. 9

B. Pseudo-Code for RAR

• RAR is presented with a PyTorch-style pseudo-code listing.Source: B. Pseudo-Code for RAR · p. 9

C. Visualization of Scan Orders

• The six scan orders studied in the main paper are visualized in Fig. 6.Source: C. Visualization of Scan Orders · p. 9
• The sampling probability for random permutations decreases linearly from 1 to 0 across the annealing interval, after which raster order is used.Each batch item samples a random permutation with probability r; otherwise it uses raster order.Source: C. Visualization of Scan Orders · p. 9
• The forward pass permutes tokens and next-token labels, adds target-aware positional embeddings, and computes next-token prediction loss.Source: C. Visualization of Scan Orders · p. 9

D. Visualization on Generated Samples

• Figure 6 depicts scan orders for a 16 × 16 grid of 256 tokens, with numbers marking token indices in each order.Source: Figure 6 · p. 10
• Figures 7–9 show RAR samples described as high-fidelity and diverse.The text identifies these as visualization results.Source: D. Visualization on Generated Samples · p. 9, Figure 7 · p. 10, Figure 8 · p. 11