Source-linked AI summary
NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generation
Huichao Zhang, Liao Qu, Yiheng Liu, Hang Chen, Yangyang Song, Yongsheng Dong, Shikun Sun, Xian Li, Xu Wang, Yi Jiang, Hu Ye, Bo Chen, Yiming Gao, Peng Liu, Akide Liu, Zhipeng Yang, Qili Deng, Linjie Xing, Jiyang Liu, Zhao Wang, Yang Zhou, Mingcong Liu, Yi Zhang, Qian He, Xiwei Hu, Zhongqi Qi, Jie Shao, Zhiye Fu, Shuai Wang, Fangmin Chen, Xuezhi Chai, Zhihua Wu, Yitong Wang, Zehuan Yuan, Daniel K. Du, Xinglong Wu
TL;DR
Existing multimodal systems separate language reasoning from visual generation or incur efficiency and representation gaps. NextFlow uses a unified decoder-only transformer with next-scale visual prediction, a dual-codebook tokenizer, and a large-scale training and alignment recipe. It reports strong unified-model performance, diffusion-rivaling visual quality, and 5-second 1024 × 1024 generation, while retaining limitations from discrete visual representations and shared-parameter optimization.
Problem
Existing systems separate language reasoning from visual generation, while hybrid and raster-scan approaches face representation gaps and high computational cost.
Method
NextFlow combines a unified decoder-only transformer, next-scale visual prediction, dual-codebook tokenization, and coarse-scale prefix-tuning within a 6-trillion-token training pipeline.
Results
NextFlow rivals top-tier diffusion models in visual quality, outperforms specialized image-editing models, and generates 1024 × 1024 images in 5 seconds.
Takeaways & Limitations
A single decoder-only transformer can perceive, reason, and create across modalities while supporting efficient generation and interleaved multimodal tasks.
Takeaways & Limitations
Discrete vector quantization creates an information bottleneck relative to continuous latent spaces, and balancing text and visual objectives remains difficult at lower parameter counts.
Abstract
from arXiv · showhide
We present NextFlow, a unified decoder-only autoregressive transformer trained on 6 trillion interleaved text-image discrete tokens. By leveraging a unified vision representation within a unified autoregressive architecture, NextFlow natively activates multimodal understanding and generation capabilities, unlocking abilities of image editing, interleaved content and video generation. Motivated by the distinct nature of modalities - where text is strictly sequential and images are inherently hierarchical - we retain next-token prediction for text but adopt next-scale prediction for visual generation. This departs from traditional raster-scan methods, enabling the generation of 1024x1024 images in just 5 seconds - orders of magnitude faster than comparable AR models. We address the instabilities of multi-scale generation through a robust training recipe. Furthermore, we introduce a prefix-tuning strategy for reinforcement learning. Experiments demonstrate that NextFlow achieves state-of-the-art performance among unified models and rivals specialized diffusion baselines in visual quality.
1 Introduction
NextFlow addresses the separation between multimodal understanding and visual generation with a unified decoder-only transformer. Its next-scale generation, training recipe, and unified representation target efficient multimodal creation, while experiments report strong visual quality and image-editing performance.
- Diffusion models provide visual fidelity, while LLMs provide reasoning and in-context learning, but existing systems largely separate these capabilities.
- Hybrid systems use different representations, creating re-encoding overheads and a gap between generation and understanding.
- NextFlow unifies multimodal understanding and generation in a single decoder-only transformer using next-scale prediction instead of raster-scan generation.
- 5 seconds enables 1024 × 1024 image generation, while experiments report competitive text-to-image performance, stronger image editing, and diffusion-rivaling visual quality.
- A 6-trillion-token training recipe addresses next-scale instabilities and adds coarse-scale prefix-tuning for reinforcement learning.
2 Model Architecture
NextFlow combines a dual-codebook tokenizer with multiscale positional encoding and next-scale visual generation in a unified decoder-only architecture. Its training and decoding mechanisms address resolution variation, coarse-scale structural imbalance, exposure bias, and lost high-frequency detail.
- Tokenizer: The dual-codebook tokenizer jointly preserves semantic content and pixel-level fidelity in discrete visual tokens.Semantic and pixel-level features are learned separately but aligned through a shared mapping and jointly constrained codebook lookup.
- Tokenizer: Dynamic-resolution processing combines a variable-resolution semantic encoder, a CNN pixel branch, and multi-scale VQ.This allows training at native resolutions without fixed input aspect-ratio constraints.
- Positional Encoding: Multi-Scale 3D RoPE assigns diagonal positions to text and normalized spatial coordinates plus scale information to vision tokens.Normalized coordinates support resolution-invariant training across grids such as 16 × 16 and 32 × 32, while scale-length embeddings disambiguate target resolutions.
- Training Stabilization: Scale-aware loss reweighting increases the importance of early-scale predictions, counteracting their lower token counts and protecting coarse image structure.The weighting preserves total vision loss while using spatial resolution and a reweighting hyperparameter to control scale dependence.
- Training Stabilization: Self-correction trains the model to recover from suboptimal previous-scale choices, while independently retrieved residual features reduce visual-input complexity and local artifacts.During training, top-k nearest codebook indices are sampled while the top-1 index remains the target; residual features replace accumulated inputs.
- Optional Diffusion Decoder: An optional diffusion decoder refines high-frequency detail but can alter fine structures, so it is disabled in the reported experiments.The trade-off is especially relevant for local editing and identity preservation, where strict spatial consistency matters.
3 Training Odyssey
NextFlow’s training odyssey combines multi-stage tokenizer and model training with targeted fixes for instability, representation imbalance, and reinforcement-learning variance. The resulting recipe uses scale-aware optimization, self-correction with residual features, and prefix-tuning for coarse visual scales.
- Training pipeline: NextFlow’s training pipeline progresses through tokenizer development, alignment, pre-training, and post-training while tracking Geneval across cumulative training tokens.The tokenizer uses a dual-branch design, followed by multimodal alignment and progressive pre-training at 256-, 512-, and 1024-level resolutions.
- Tokenizer training: The dual-branch tokenizer converges faster and achieves better generative performance than single-branch VQGAN despite slightly lower reconstruction PSNR.The comparison uses identical 40k-step training protocols; the single-branch baseline has marginally higher raw reconstruction fidelity.
- Self-correction: Residual-feature self-correction improves performance, whereas applying correction to accumulated features degrades it; p = 1.0 on 60% of visual tokens per scale is optimal.The reported optimum reaches 0.56 at 50k steps, while direct accumulated-feature correction falls below the non-corrected baseline.
- Text-only data: Mixing 25% text-only data does not adversely affect text-to-image generation quality.The ablation tests whether preserving text capabilities harms text-to-image performance.
- Scale instability: At 512-level pre-training, Geneval drops from 0.67 to 0.57 as higher token counts amplify the importance of coarse-scale losses.The observed degradation motivates reweighting earlier scales, which determine global image layout.
- Reinforcement learning: Prefix-tuning updates only the first m scales, such as m = 8, while freezing finer scales to concentrate high-variance reinforcement-learning signals on coarse decisions.This accelerates convergence, preserves pretrained generative quality, and adapts high-level attributes without noisy later-scale updates.
4 Infrastructure
NextFlow’s infrastructure addresses heterogeneous multimodal workloads and large-vocabulary memory demands through balanced packing, fused kernels, offline image indexing, and distributed training. These measures reduce idle time and memory requirements during large-scale pre-training.
- Distributed training: NextFlow uses 1024 GPUs with DeepSpeed ZeRO and gradient checkpointing for distributed pre-training.The infrastructure is designed to support the model’s large-scale multimodal training workload.
- Workload balance: Fixed maximum-computation-budget packing reduces inter-GPU idle time and improves overall training throughput.The strategy precomputes TFLOPS for sequence lengths and balances heterogeneous workloads accordingly.
- Memory efficiency: Fused linear projection and cross-entropy computation reduce peak memory usage by approximately 20GB per GPU.The fused kernel avoids storing the full output-logit tensor in memory.
- Data preparation: Offline image-index extraction eliminates online encoding latency and keeps encoders offloaded from GPUs during training.Precomputed indices are packed offline with the original data ordering and support self-correction inputs.
5 Data
NextFlow’s multimodal data spans understanding, generation, editing, interleaved documents, video-derived sequences, and text reasoning. The curation pipeline combines broad sources with filtering, balancing, and task-specific construction to improve diversity and quality.
- Dataset scope: The training corpus covers visual understanding, image generation, image editing, interleaved image-text generation, and text generation.The dataset is designed to support both multimodal and pure-text capabilities.
- Visual understanding: Image-understanding data combines captioning, text-rich images, documents, and world-knowledge associations to strengthen visual perception and OCR.Text-rich examples include tables, charts, plots, and other document types.
- Image generation: The text-to-image corpus uses diverse open-source, curated, and in-house images filtered by heuristics and aesthetic models.The collection is described as billion-scale and broad in image content and type.
- Image editing: The editing data combines traditional local or global transformations with subject-driven generation, supplemented by synthetic tasks to correct open-source distribution bias.The synthetic set broadens coverage beyond common additions, removals, and replacements.
- Interleaved data: Video clips are converted into interleaved image-text documents using temporal frame sequences, transition text, quality filtering, motion constraints, and semantic balancing.The resulting balanced corpus contains approximately 5.3M clips.
- Frame selection: Motion-adaptive sampling removes static or redundant frames while retaining independent object motion and large scene transitions.Training inputs are capped at 5 frames at 512px or 3 frames at 1k resolution.
6 Model Performance
NextFlow achieves competitive or state-of-the-art results across text-to-image, world-knowledge, aesthetic, and image-editing evaluations while supporting interleaved generation and in-context visual transformation. Its unified model also exposes a capacity limitation when multimodal understanding and generation share a dense 7B decoder.
- Text-to-image generation: The RL-finetuned model reaches 0.84 on GenEval and 88.32 on DPG, outperforming or matching strong diffusion baselines.These results are reported for prompt-following text-to-image evaluation.
- World knowledge: NextFlow RL scores 0.62 on WISE, matching Qwen-Image and exceeding Show-o at 0.30 and Janus-Pro-7B at 0.35.WISE evaluates semantic understanding and world knowledge for text-to-image generation.
- Aesthetic quality: NextFlow RL achieves 78.8 on PRISM-Bench, matching top-tier systems on imagination, composition, style, and text rendering.The benchmark evaluates multiple generative aspects using GPT-4.1.
- Image editing: On ImgEdit, NextFlow RL scores 4.49 overall, surpassing Qwen-Image at 4.27 and Emu3.5 at 4.41.It also reports strong Adjust and Remove sub-task scores and high subject consistency on OmniContext.
- Image editing: On EditCanvas, NextFlow RL scores 8.04 overall and 8.78 on Subject-Driven Generation across 56 fine-grained tasks.The benchmark contains over 5,000 high-quality samples spanning traditional editing and subject-driven generation.
- Unified multimodal generation: The unified model generates alternating text-image sequences and supports CoT prompting and in-context image transformation.CoT raises WISE from 0.60 to 0.70 in an exploratory study, while in-context prompts apply inferred stylistic mappings.
- Limitation: The 7B decoder’s shared capacity creates a bottleneck for simultaneously supporting multimodal understanding and image generation.Because high-quality understanding data is scarce, late-stage understanding training uses a 10% task-inversion probability.
7 Conclusion, Limitations and Future Directions
NextFlow is presented as a proof of concept for unified multimodal autoregressive modeling, combining understanding, generation, editing, and reasoning in one decoder-only transformer. The authors identify information bottlenecks, optimization difficulty, and future opportunities in data, model, tokenization, and multimodal reinforcement-learning scaling.
- Conclusion: NextFlow uses next-scale prediction, a dual-codebook tokenizer, scale-reweighting, and prefix-tuning GRPO as a unified multimodal training recipe.The conclusion frames these components as a robust recipe validated on 6 trillion tokens.
- Limitations: Vector quantization imposes an information bottleneck relative to continuous latent spaces, sometimes requiring an optional diffusion decoder for realistic refinement.The limitation concerns fine visual reconstruction rather than the unified architecture’s overall scope.
- Limitations: Balancing text-generation and visual-synthesis objectives in shared parameters remains difficult, especially at lower parameter counts.This is identified as an optimization challenge within the shared model.
- Future Directions: Future work prioritizes denser multimodal reasoning data, larger or MoE models, and tokenizers with higher compression without sacrificing reconstruction quality.The proposed directions target understanding, generation fidelity, instruction following, and sequence-length reduction.
- Future Directions: The fully autoregressive design supports multimodal reinforcement learning and intermediate visual generation for image-based reasoning.The authors contrast this with hybrid architectures that optimize discrete text and continuous visual latents separately.
A.1 Predefined Scale Schedules
The predefined schedules organize progressive image generation across aspect ratios by preserving approximate geometry and spatial area through 18 scale steps. Different target resolutions use schedule prefixes, supporting outputs from 1:4 to 4:1 while maintaining computational efficiency.
- Schedule Construction: Each aspect ratio r receives an 18-step progressive schedule from an initial patch to the final resolution.The schedule is represented as a sequence of height-width tuples.
- Schedule Construction: Each scale tuple approximately preserves aspect ratio r, with particular emphasis on higher scales where visual fidelity is critical.The design keeps the spatial area approximately constant across aspect ratios at each scale index.
- Resolution Settings: 256-scale images use the first 12 schedule scales, while 512-scale images use the first 16.These prefixes select the appropriate progression length for each target resolution.
- Resolution Settings: The schedules support photorealistic inference across aspect ratios from 1:4 to 4:1 while maintaining computational efficiency.The predefined schedules are intended to provide consistent quality across diverse output dimensions.
A.2 Inference Efficiency Analysis
The efficiency analysis compares NextFlow’s dynamic next-scale inference with MMDiT’s fixed-length denoising at matched sampling steps. NextFlow reduces computational cost by generating fewer tokens early and using KV-caching, yielding substantially lower FLOPs at the same resolution.
- Experimental Setup: Both models are evaluated for 1024 × 1024 images with approximately 64 × 64 latent tokens and 18 sampling steps.The hidden size is tested at 2048 and 4096.
- Cost Models: MMDiT uses a fixed 4096-token sequence across 18 denoising steps, whereas NextFlow dynamically generates scale-specific token counts with KV-caching.For NextFlow, sq is the number of new tokens at the current step and skv is the cumulative prior-token count.
- Comparative Results: 6× fewer FLOPs are required by NextFlow than MMDiT for generating an image at the same resolution.The comparison is made under matched sampling-step conditions.
- Comparative Results: 7.2× more FLOPs are consumed by MMDiT in FFN and linear-projection components than by NextFlow.Next-scale prediction lowers average load because early steps generate fewer tokens.
- Comparative Results: 4.7× more attention computation is required by MMDiT than by NextFlow.The analysis attributes the difference to NextFlow’s dynamic sequence lengths and autoregressive KV-caching.
B EditCanvas benchmark
EditCanvas is a hierarchical benchmark for systematic image-editing evaluation, combining traditional editing and subject-driven generation. Its fine-grained taxonomy and curated dataset are designed to reveal performance differences across diverse editing scenarios.
- Taxonomy: The benchmark uses a three-level hierarchy with 56 fine-grained tasks: 40 traditional-editing tasks and 16 subject-driven-generation tasks.Examples include object addition, removal, replacement, and repositioning.
- Dataset: EditCanvas contains 5,221 high-quality samples and evaluates subject-driven tasks for instruction following and content consistency using GPT-4.1.The dataset is presented as a large-scale, structured benchmark.
- Benchmark Distribution: The benchmark is organized into Traditional Editing at 67.8% and Subject-Driven Editing at 32.2%.The distribution is shown in the hierarchical task chart.
- Benchmark Scope: EditCanvas integrates traditional editing and subject-driven generation into one unified evaluation framework.Traditional editing modifies a reference image, while subject-driven generation extracts a subject to create a new scene.
- Taxonomy: Its finer taxonomy supports distinguishing specific capabilities such as object addition from object repositioning.This enables more precise analysis of model strengths and weaknesses than coarser benchmark categories.
- Dataset: The dataset combines semi-automated generation with multi-round human verification to provide scale and quality.It is described as larger and more diverse than GEdit-Bench’s 606 examples.