Source-linked AI summary
NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scale
NextStep Team, Chunrui Han, Guopeng Li, Jingwei Wu, Quan Sun, Yan Cai, Yuang Peng, Zheng Ge, Deyu Zhou, Haomiao Tang, Hongyu Zhou, Kenkun Liu, Ailin Huang, Bin Wang, Changxin Miao, Deshan Sun, En Yu, Fukun Yin, Gang Yu, Hao Nie, Haoran Lv, Hanpeng Hu, Jia Wang, Jian Zhou, Jianjian Sun, Kaijun Tan, Kang An, Kangheng Lin, Liang Zhao, Mei Chen, Peng Xing, Rui Wang, Shiyu Liu, Shutao Xia, Tianhao You, Wei Ji, Xianfang Zeng, Xin Han, Xuelin Zhang, Yana Wei, Yanming Xu, Yimin Jiang, Yingming Wang, Yu Zhou, Yucheng Han, Ziyang Meng, Binxing Jiao, Daxin Jiang, Xiangyu Zhang, Yibo Zhu
TL;DR
Autoregressive text-to-image systems face trade-offs from heavy diffusion processing, discrete VQ tokenization, and persistent quality gaps. NextStep-1 addresses these issues with a 14B autoregressive model and 157M flow-matching head over unified discrete text and continuous image tokens. It achieves state-of-the-art autoregressive text-to-image performance and competitive image-editing results, while exposing stability and latency challenges.
Problem
Existing autoregressive image generators rely on heavy diffusion models or discrete VQ tokens, while continuous-latent approaches remain behind diffusion methods in image quality and consistency.
Method
NextStep-1 combines a 14-billion-parameter autoregressive Transformer with a 157-million-parameter flow-matching head and an image tokenizer for unified text and continuous-image token prediction.
Results
NextStep-1 achieves state-of-the-art autoregressive text-to-image performance, including 0.54 on WISE, and competitive editing scores of 6.58 on GEdit-Bench and 3.71 on ImgEdit-Bench.
Takeaways & Limitations
The unified continuous-token approach supports high-fidelity image synthesis and diverse instruction-based image editing across challenging scenarios.
Takeaways & Limitations
Higher-dimensional continuous latents can produce generative artifacts, while sequential decoding and multi-step flow sampling create substantial inference latency.
Abstract
from arXiv · showhide
Prevailing autoregressive (AR) models for text-to-image generation either rely on heavy, computationally-intensive diffusion models to process continuous image tokens, or employ vector quantization (VQ) to obtain discrete tokens with quantization loss. In this paper, we push the autoregressive paradigm forward with NextStep-1, a 14B autoregressive model paired with a 157M flow matching head, training on discrete text tokens and continuous image tokens with next-token prediction objectives. NextStep-1 achieves state-of-the-art performance for autoregressive models in text-to-image generation tasks, exhibiting strong capabilities in high-fidelity image synthesis. Furthermore, our method shows strong performance in image editing, highlighting the power and versatility of our unified approach. To facilitate open research, we will release our code and models to the community.
1. Introduction
NextStep-1 addresses limitations of diffusion-dependent and VQ-based autoregressive image generation by using continuous image tokens within a unified next-token framework. It achieves state-of-the-art autoregressive performance across text-to-image benchmarks and shows strong image-editing results.
- Existing autoregressive approaches rely on heavy diffusion models or discrete VQ tokens, facing exposure bias and suboptimal image tokenization.
- Continuous latent approaches remain behind state-of-the-art diffusion methods, particularly in image quality and consistency.
- 0.54 on WISE, 0.67 on GenAI-Bench advanced prompts, 85.28 on DPG-Bench, and 0.417 on OneIG-Bench demonstrate broad text-to-image capability.
- 6.58 on GEdit-Bench and 3.71 on ImgEdit-Bench show competitive instruction-based image-editing performance.
- NextStep-1 unifies discrete text and continuous image tokens under a next-token prediction paradigm.
2. Framework
The framework represents text and images in one causal sequence, using modality-specific heads to predict discrete text tokens and continuous image tokens. Its tokenizer and flow-matching design support high-dimensional continuous visual representations.
- Images are converted into continuous visual tokens and combined with discrete text tokens in a unified multimodal sequence.
- The model samples each next token conditionally, using a language-modeling head for text and a flow-matching head for continuous image tokens.
- Training combines cross-entropy for text tokens with mean-squared-error flow matching for velocities mapping noised patches to clean patches.
- The image tokenizer produces 16-channel latents with 8× spatial downsampling and applies channel-wise normalization for stability.
- A 2×2 space-to-depth transformation converts a 256×256 image into a flattened sequence of 256 compact tokens.
- A causal Transformer processes the sequence, while lightweight language-modeling and patch-wise flow-matching heads compute modality-specific losses.
3. Data
NextStep-1 is trained on a multimodal corpus spanning text-only, image-text, image-to-image, and interleaved data. These datasets are curated to support language retention, text-to-image generation, editing, perception, and contextual understanding.
- The training corpus contains text-only, image-text pair, image-to-image, and interleaved data categories serving distinct generative capabilities.
- 400B text-only tokens are included to preserve the language capabilities of the underlying large language model.
- Image-text pairs form the foundation for text-to-image generation and undergo sourcing, quality filtering, deduplication, and detailed recaptioning.
- 550M high-quality image-text pairs provide training data for aesthetic quality and broad world knowledge.
- Instruction-guided image-to-image data covers visual perception, controllable generation, restoration, and general image editing.
- Interleaved data includes video-interleaved content, tutorials, character-centric scenes, and multi-view data for sequential multimodal associations.
4. Training Recipe
Training proceeds through tokenizer optimization, a three-stage pre-training curriculum, and post-training with supervised fine-tuning followed by preference optimization. The curriculum culminates in annealing on a highly curated dataset to improve final outputs.
- 4.1. Training Image Tokenizer: The tokenizer is trained for 50K steps with AdamW, batch size 512, and learning rate 1 × 10^-5.
- 4.2. Pre-Training: Pre-training uses a three-stage curriculum whose model parameters are trained end-to-end except for the pre-trained image tokenizer.
- 4.2. Pre-Training: Stage 1 uses 20% text-only, 60% image-text, and 20% interleaved data at 256×256 resolution, consuming approximately 1.23T tokens.
- 4.2. Pre-Training: Stage 2 introduces dynamic higher-resolution training at 256×256 and 512×512 base areas with aspect-ratio buckets.
- 4.2. Pre-Training: One epoch on a curated 20M-sample subset significantly improves image structure, composition, texture, and aesthetic appeal.
- 4.3. Post-Training: Post-training aligns outputs through supervised fine-tuning followed by Direct Preference Optimization.
- 4.3. Post-Training: The SFT dataset contains 5M samples combining human-selected pairs, distilled generative-model images, and Chain-of-Thought data.
- 4.3. Post-Training: DPO preference pairs are constructed from approximately 20,000 diverse prompts using candidate images scored by ImageReward.
5. Model Performance
NextStep-1 performs strongly across text-to-image alignment, world-knowledge reasoning, and image editing benchmarks. Its results include competitive autoregressive performance and gains from Self-CoT or prompt rewriting on selected evaluations.
- Image–Text Alignment: NextStep-1 achieves robust image–text alignment across counting, grounding, spatial alignment, compositionality, and long-context multi-object generation.It scores 0.63 on GenEval, 0.88 and 0.67 on basic and advanced GenAI-Bench prompts, and 85.28 on DPG-Bench.
- Image–Text Alignment: 0.417 on OneIG-Bench exceeds Emu3 at 0.311 and Janus-Pro at 0.267 for English prompts.The benchmark covers alignment, text rendering, reasoning, and stylistic control.
- World Knowledge: 0.54 on WISE is the best result among autoregressive models and exceeds most diffusion models.The score rises to 0.67 with Self-CoT and to 0.79 under prompt rewriting, or 0.83 with Self-CoT.
- Image Editing: NextStep-1-Edit achieves 6.58 on GEdit-Bench-EN and 3.71 on ImgEdit-Bench after finetuning on 1M edit-only samples.These results indicate competitive image-editing performance against advanced diffusion-based models.
6. Discussions
The discussion examines how the autoregressive transformer, flow-matching head, and tokenizer jointly determine generation quality, while identifying stability, latency, resolution, and fine-tuning challenges.
- 6.1. What Governs Image Generation: the AR Transformer or the FM Head?: Continuous image tokens are modeled directly autoregressively with a flow-matching objective, rather than relying on a separate diffusion model for the entire image.The framework generates image patches sequentially and uses a lightweight flow-matching head for continuous image tokens.
- 6.1. What Governs Image Generation: the AR Transformer or the FM Head?: Different flow-matching head sizes produced remarkably similar results, suggesting that the transformer backbone performs the core generative modeling.The ablation reinitialized and trained small, base, and large heads for 10k steps.
- 6.2. Tokenizer is the Key to Image Generation: High classifier-free guidance can amplify token-level distributional shifts, causing later-token statistics to diverge and producing visual artifacts.At guidance scale 3.0, per-token means and variances diverged significantly, unlike at scale 1.5.
- 6.2. Tokenizer is the Key to Image Generation: Channel-wise normalization stabilizes per-token statistics and enables strong guidance without degrading image quality.The tokenizer design directly addresses the distributional instability identified under high guidance scales.
- 6.2. Tokenizer is the Key to Image Generation: Higher tokenizer noise intensity can increase generation loss while improving synthesis quality; NextStep-1 uses γ = 0.5 and produces the highest-fidelity images.The paper attributes this effect to noise regularization, which improves decoder robustness and latent dispersion, although their relative importance remains unclear.
- 6.3. Limitations and Challenges: The approach faces practical limits from high-dimensional latent artifacts, sequential inference latency, high-resolution training, and unstable fine-tuning on small datasets.The reported bottlenecks include serial LLM decoding, flow-matching sampling, longer high-resolution convergence, and overfitting or negligible gains during small-data SFT.
Contributors and Acknowledgments
This section identifies the researchers, contributors, sponsors, and acknowledgment recipients associated with NextStep-1.
- Researchers are identified as individuals involved in developing NextStep-1.
- Contributors provided support in data, systems, platforms, early version work, or part-time contributions.
- Core executors are marked with ★, while the project leader is marked with †.
- The section lists sponsors and thanks Tianhong Li and Yonglong Tian for insightful discussions.