Source-linked AI summary
SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RL
Junke Wang, Zhi Tian, Xun Wang, Xinyu Zhang, Weilin Huang, Zuxuan Wu, Yu-Gang Jiang
TL;DR
Autoregressive visual generation still trails diffusion models despite its sequential precision and multimodal compatibility. SimpleAR preserves a vanilla AR design while optimizing pretraining, SFT, GRPO-based RL, and inference. With 0.5B parameters it generates 1024×1024 images and reports competitive benchmark performance, while vLLM reduces generation time to around 14 seconds.
Problem
Autoregressive visual generation underperforms diffusion models, with discrete tokenization and long visual sequences identified as possible challenges.
Method
SimpleAR uses a vanilla autoregressive transformer with visual and text tokenizers, three-stage pretraining/SFT/RL training, and inference acceleration techniques.
Results
0.5B parameters achieve competitive text-to-image performance, including 0.59 on GenEval and 79.66 on DPG-Bench, while vLLM generates 1024×1024 images in around 14 seconds.
Takeaways & Limitations
The results highlight vanilla autoregressive visual generation as a potential alternative to diffusion models and motivate further exploration.
Takeaways & Limitations
Limited data and parameter scale constrain complex poses, objects, and text, and the model may generate content that violates physical laws.
Abstract
from arXiv · showhide
This work presents SimpleAR, a vanilla autoregressive visual generation framework without complex architecure modifications. Through careful exploration of training and inference optimization, we demonstrate that: 1) with only 0.5B parameters, our model can generate 1024x1024 resolution images with high fidelity, and achieve competitive results on challenging text-to-image benchmarks, e.g., 0.59 on GenEval and 79.66 on DPG; 2) both supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) training could lead to significant improvements on generation aesthectics and prompt alignment; and 3) when optimized with inference acceleraton techniques like vLLM, the time for SimpleAR to generate an 1024x1024 image could be reduced to around 14 seconds. By sharing these findings and open-sourcing the code, we hope to reveal the potential of autoregressive visual generation and encourage more participation in this research field. Code is available at https://github.com/wdrink/SimpleAR.
1 Introduction
Autoregressive visual generation offers precise sequential prediction and multimodal compatibility, but currently trails diffusion models. SimpleAR addresses this gap by retaining a vanilla AR framework while optimizing training and inference.
- Motivation: Autoregressive models generate visual tokens sequentially, supporting precise and coherent prediction for fine-grained control.They are also naturally compatible with modalities such as language and audio.
- Motivation: Autoregressive visual generation currently underperforms diffusion models, with discrete tokenizers and long visual sequences proposed as possible causes.Variants such as MAR and VAR improve benchmark results but compromise the intrinsic next-token-prediction pattern.
- SimpleAR: SimpleAR preserves the simplicity of vanilla autoregressive modeling while optimizing pretraining, SFT, and GRPO-based reinforcement learning.The approach targets generation quality without complex architectural modifications.
- SimpleAR: 0.5B parameters enable SimpleAR to generate 1024×1024 images with superior aesthetics and achieve 0.59 on GenEval.Scaling parameters and tokens consistently improves fidelity and structural coherence.
- Inference: Approximately 14 seconds are required to generate a 1024×1024 image when SimpleAR is deployed with vLLM.The work also investigates speculative sampling for inference acceleration.
2 Related Work
Related work develops autoregressive and unified multimodal models that represent visual content with tokens and model sequences with transformers. Other approaches combine autoregression for text with diffusion for images.
- Autoregressive Models: Autoregressive visual generation tokenizes images and videos into discrete tokens, then uses causal attention to model next-token dependencies.Representative systems include DALL-E, Parti, and LlamaGen.
- Unified Multimodal Models: Unified multimodal models integrate vision and language understanding and generation within shared architectures.Chameleon and Emu3 use unified autoregressive transformers for diverse modalities.
- Hybrid Models: Transfusion and Show-o combine next-token prediction for text with diffusion processes for images.This design handles discrete and continuous data within one model.
3 Method
SimpleAR combines compact visual tokenization, a decoder-only autoregressive transformer, three-stage training, and inference optimizations. Its pipeline models joint text-image sequences while using RL and systems techniques to improve quality and speed.
- Architecture: SimpleAR uses a pretrained visual tokenizer, a text tokenizer, and a decoder-only transformer to autoregressively model joint text and image tokens.Unlike diffusion models or prior AR models that require an additional text encoder, the model integrates text encoding and visual generation.
- Architecture: Images are converted into compact discrete visual-token sequences, flattened in raster order, and combined with text tokens before transformer decoding.The visual tokens represent codebook indices for learned visual patches.
- Training: The three-stage training paradigm uses large-scale pretraining, high-quality SFT, and RL to improve generalization, fidelity, instruction-following, and multimodal alignment.RL is also intended to alleviate exposure bias.
- Training: GRPO samples groups of outputs from an old policy and optimizes a trainable policy against group-computed advantages with a KL penalty.The method initializes from the SFT checkpoint and uses a frozen reference model; CLIP provides the reward.
- Inference: During inference, visual tokens are sampled sequentially and decoded into images, using greedy search and classifier-free guidance by default.The default topK equals the 64000-entry codebook size.
- Inference: KV caching reduces autoregressive decoding complexity from O(N^2) to O(N), while vLLM uses paged attention for efficient serving.Speculative Jacobi decoding verifies draft-model candidate sequences with the target model.
4 Experiments
Experiments show that SimpleAR’s compact autoregressive model performs competitively across text-to-image benchmarks, benefits from scaling and GRPO, and can be substantially accelerated at inference. Qualitative results also reveal strong fidelity and instruction following alongside limitations on complex content.
- Comparisons with State-of-the-Art: 0.59 on GenEval and 79.66 on DPG-Bench demonstrate competitive performance for the 0.5B-parameter SimpleAR model.The model outperforms comparable-scale methods with fewer than 1B parameters.
- Ablation Studies: Replacing 1D positional encoding with 2D does not significantly improve DPG-Bench performance in the reported pretraining comparison.The authors nevertheless consider 2D positional encoding necessary for dynamic-resolution and video generation.
- Ablation Studies: +0.6 on GenEval is achieved by the CLIP-ViT-H-14 reward in GRPO for the 0.5B model, while both tested reward modules improve performance.Qualitative results associate the CLIP reward with better text rendering and perception of quantifiers and spatial descriptions.
- Ablation Studies: Reward values gradually increase during GRPO training, while GenEval performance shows a positive correlation with reward progression.The authors interpret this as evidence that CLIP-ViT-H-14 provides feedback aligned with task performance.
- Inference Speedup: Speculative Jacobi decoding reduces autoregressive generation steps by around 2× and slightly improves DPG, but does not practically reduce testing latency.The method cannot use KV cache and must forward the entire sequence at each iteration.
- Visualizations and Failure Cases: SimpleAR generates high-fidelity, aesthetically pleasing images with strong instruction following, but limited data and parameter scale constrain complex poses, objects, and text.The model may also synthesize content that does not adhere to physical laws.
5 Conclusion and Future Work
SimpleAR preserves a vanilla autoregressive architecture while optimizing training and inference, achieving competitive text-to-image performance with 0.5B parameters and approximately 14-second 1024×1024 generation using vLLM. The paper identifies limitations in data and model scale, visual tokenization, text-to-video generation, and native multimodal capabilities.
- SimpleAR uses a vanilla autoregressive framework while optimizing its training pipeline through pretraining, supervised finetuning, and GRPO training.
- 0.5B parameters yields competitive performance on existing text-to-image benchmarks.
- 14 seconds is sufficient for SimpleAR to generate a 1024×1024 image when served with vLLM.
- Cosmos-Tokenizer has limited reconstruction of fine-grained details such as faces and text, leaving room for stronger visual tokenizers.
- Text-to-video generation remains challenging because outputs must remain contextually and temporally coherent.
- End-to-end native multimodal models integrating vision, text, and other modalities remain a promising research direction for SimpleAR.