Source-linked AI summary
ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representations
Junke Wang, Xiao Wang, Jiacheng Pan, Xuefeng Hu, Feng Li, Jingxiang Sun, Chaorui Deng, Zilong Chen, Yunpeng Chen, Kaibin Tian, Matthew Gwilliam, Hao Chen, Danhui Guan, Kun Xu, Weilin Huang, Zuxuan Wu, Haoqi Fan, Yu-Gang Jiang, Zhenheng Yang
TL;DR
Existing multimodal systems often use separate visual representations for understanding and generation, creating fragmented architectures and inference overhead. ARM unifies these tasks with discrete visual tokens, a 7B autoregressive model, and preference optimization, achieving competitive performance across understanding, generation, and editing, including a G_O score of 6.68 on GEdit-Bench-EN.
Problem
Existing multimodal methods commonly use separate visual representations for understanding and generation, fragmenting the architecture and increasing inference overhead.
Method
ARM uses a semantic visual tokenizer and a single 7B autoregressive transformer over interleaved text and visual tokens, with preference optimization for generation and editing.
Results
ARM achieves competitive performance across multimodal understanding, generation, and editing, including a G_O score of 6.68 on GEdit-Bench-EN.
Takeaways & Limitations
The results support next-token prediction with unified discrete representations as a promising approach for integrating multimodal understanding, generation, and editing.
Abstract
from arXiv · showhide
This paper introduces ARM, a discrete representation-based AutoRegressive Model that unifies image understanding, generation, and editing within a next-token prediction framework. ARM is built on three efforts: first, we train a discrete semantic visual tokenizer that maps images into compact token sequences. Our tokenizer is supervised with multiple objectives that jointly promote semantic discriminability, language alignment and faithful reconstruction, thereby supporting diverse tasks in a shared latent space. With this, we train a 7B autoregressive model over large-scale text and image token sequences, seamlessly developing vision-language perception and generation capabilities. Finally, to further improve preference-aligned behavior for text-to-image generation and instruction-guided editing, ARM applies reinforcement learning (RL) to optimize task-level objectives such as visual quality, instruction adherence, and edit consistency. Surprisingly, the results show that RL not only substantially improves performance on the target tasks (e.g., raising WISE overall from 0.50 to 0.56, GEdit-Bench-EN G_O from 5.75 to 6.68), but also induces cross-task synergy between text-to-image generation and editing. Collectively, these findings highlight autoregressive modeling, when paired with strong representations and preference optimization, as a scalable foundation for multimodal intelligence. Code: https://github.com/wdrink/ARM.
1 Introduction
ARM addresses the fragmentation of multimodal understanding and generation by using unified discrete visual representations. It combines a supervised visual tokenizer with a 7B autoregressive model and achieves strong performance across understanding, generation, and editing.
- Motivation: Existing methods use separate visual encoders for understanding and generation, creating fragmented visual latent spaces and requiring extra capacity to bridge them.They also maintain redundant representations of the same visual input.
- Approach: ARM introduces a discrete visual tokenizer trained with complementary supervision to preserve text-aligned semantics and appearance details for recognition, synthesis, and editing.A 7B autoregressive model is then trained on interleaved text and visual token sequences.
- Results: 40.2 on MMMU and 87.3 on POPE demonstrate ARM’s strong multimodal understanding performance.These results substantially outperform prior methods relying on discrete visual representations.
- Results: 0.86 on GenEval and 0.56 on WISE show leading-level image-generation performance relative to diffusion baselines.The passage reports these as representative generation results for ARM.
2 Related Work
Related work distinguishes semantic and fine-grained visual tokenizers, while unified vision-language models increasingly target both image understanding and generation. Visual generation includes autoregressive models that generate from discrete image tokens and masked prediction models that generate VQ tokens in parallel.
- Unified Visual Tokenizer: Semantic encoders preserve high-level representations for visual understanding but miss fine-grained details needed for precise image generation and editing.CLIP and SigLIP represent the semantic-encoder category, whereas VQVAEs and VAEs support generation through reconstruction-based training.
- Unified Vision Language Model: Unified MLLMs combine image understanding and generation, but systems using semantic encoders with separate diffusion decoders struggle with editing because fine-grained details are lost.Next-GPT, SEED-X, and EMU2 are cited as examples of this design.
- Visual Generation: Autoregressive visual-generation models map images to discrete tokens, while masked prediction models generate VQ tokens in parallel.The passage identifies these as two of three main approaches to visual generation.
3 Methods
ARM uses a unified autoregressive transformer over interleaved discrete text and visual token sequences, with modality-specific detokenizers for language and pixels. Its pipeline combines a semantically aligned, detail-preserving visual tokenizer, large-scale next-token training, and preference-based reinforcement learning for visual token prediction.
- Overall architecture: ARM tokenizes multimodal inputs into one-dimensional discrete sequences, models them with next-token prediction, and detokenizes predicted tokens into language or pixels.This single autoregressive backbone supports image understanding, generation, and editing.
- Unified visual tokenizer: The visual tokenizer uses a frozen pretrained SigLIP2 encoder, Finite Scalar Quantization without an explicit codebook, and a symmetric projection module.The design preserves semantically strong visual features while producing discrete representations.
- Unified visual tokenizer: Four complementary objectives train the tokenizer for language alignment, pixel-level fidelity, cross-modal alignment, and feature preservation.They comprise caption, pixel reconstruction, sigmoid contrastive, and feature distillation losses.
- Unified visual tokenizer: Pixel-space diffusion reconstruction avoids lossy VAE bottleneck compression and provides more stable optimization than GAN-style decoders.A separate high-capacity latent diffusion model is used for high-quality detokenization from learned quantized embeddings.
- Autoregressive training and preference optimization: ARM flattens image-to-text, text-to-image, text-only, and interleaved image-text data into multimodal token sequences for standard next-token prediction.After this large-scale training, GRPO aligns visual token prediction with preference feedback for generation and editing, using reward-normalized advantages and KL regularization.
4 Experiments
ARM delivers competitive multimodal understanding and generation within a unified autoregressive architecture, while reinforcement learning improves both generation and editing. Its shared discrete visual latent space also enables cross-task gains and supports controllable instruction-guided editing.
- Understanding: ARM narrows the historical understanding gap between discrete and continuous unified models while retaining a fully autoregressive architecture.It achieves 87.3 on POPE, 40.2 on MMMU, 1463 on MMEPerc, and 73.1 on SeedBench.
- Understanding: 87.3 on POPE and 40.2 on MMMU place ARM on par with or above representative continuous unified models and ahead of prior discrete models.ARM also reaches 1463 on MMEPerc and 73.1 on SeedBench, indicating strength on knowledge-intensive and reasoning-heavy queries.
- Text-to-image generation: ARM achieves competitive text-to-image performance across object-attribute binding, spatial control, global coherence, relational constraints, and reasoning-based evaluation.These results are obtained within a single unified autoregressive model, without specialized prompting or auxiliary mechanisms.
- Image editing: 6.68 is ARM’s G_O on the GEdit-Bench-EN full set after reinforcement learning, up from 5.75.Preference optimization reduces incomplete edits, excessive modifications, and attribute drift, improving instruction adherence and visual quality.
- Reinforcement learning: T2I RL improves the GEdit score from 5.75 to 5.92, while Edit RL also improves the reciprocal text-to-image task.Initializing Edit RL from T2I RL weights and then applying joint RL further improves all tasks, with joint RL achieving the highest overall scores.
- Visualizations: ARM maintains structured layouts in complex scenes and follows compound editing instructions combining appearance, pose, and style changes.These visualizations indicate controllability under interleaved text and visual tokens.
5 Conclusion
ARM is an autoregressive large multimodal model built on unified discrete visual representations. Its semantic tokenizer, 7B multimodal training, and Group Relative Policy Optimization support preference-aligned generation and editing.
- ARM is an autoregressive large multimodal model built on unified discrete visual representations.
- Its semantic tokenizer discretizes images into compact token sequences preserving language-aligned semantics and visual details.
- A 7B model is trained autoregressively on large-scale multimodal tokens using this tokenizer.
- Group Relative Policy Optimization aligns model outputs with preference feedback for generation and editing.