PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification

Wujian Peng, Lingchen Meng, Yuxuan Cai, Xianwei Zhuang, Yuhuan Yang, Rongyao Fang, Chenfei Wu, Junyang Lin, Zuxuan Wu, Shuai Bai

arXiv:2606.18249v2cs.CV

TL;DR

Existing unified multimodal systems typically use separate visual tokenizers, limiting shared-context modeling across understanding and generation. UniAR uses one discrete tokenizer with parallel bitwise prediction and reports state-of-the-art image-generation performance while remaining competitive on multimodal understanding benchmarks.

  • Problem

    Existing unified multimodal systems typically use separate visual tokenizers because understanding needs semantics while generation needs low-level detail, complicating shared-context modeling.

  • Method

    UniAR uses multi-level feature fusion, lookup-free bitwise quantization, parallel bitwise prediction, and a visual decoder within one unified autoregressive framework.

  • Results

    UniAR achieves state-of-the-art image-generation performance, including 0.873 on OneIG-EN, while remaining competitive on multimodal understanding benchmarks.

  • Takeaways & Limitations

    A single discrete visual tokenizer can support unified understanding, generation, and editing within one autoregressive model.

  • Takeaways & Limitations

    Joint pre-training omits pure-text data, and reinforcement learning is limited to image generation.

Abstract

from arXiv · show

Unified Multimodal Modeling aims to integrate visual understanding and generation within a single system. However, existing approaches typically rely on two disparate visual tokenizers, which splits the representation space and hinders truly unified modeling. We propose UniAR, a unified autoregressive framework where a single discrete visual tokenizer serves as the key bridge between understanding and generation, enabling a shared context in which the model can directly interpret its own generated visual tokens without additional re-encoding. UniAR adapts a pretrained vision encoder with multi-level feature fusion and a lookup-free bitwise quantization scheme, preserving both high-level semantics and low-level details while scaling the effective visual vocabulary at minimal cost. Building on this, the unified autoregressive model adopts parallel-bitwise-prediction to jointly predict spatially grouped, multi-level visual codes, substantially reducing visual sequence length and accelerating generation. Finally, a diffusion-based visual decoder operates on discrete visual tokens to decode high-fidelity images. Through large-scale pre-training, followed by supervised fine-tuning and reinforcement learning, UniAR achieves state-of-the-art performance on image generation and image editing while remaining competitive on multimodal understanding benchmarks. The project page is available at https://sharelab-sii.github.io/uniar-web.

1 Introduction

UniAR addresses the split between visual understanding and generation by using a single discrete visual tokenizer within a unified autoregressive framework. Its multi-level, bitwise visual representation and parallel prediction support compact, efficient high-resolution generation while retaining competitive multimodal understanding.

  • Motivation: Unified multimodal models seek shared-context understanding and generation, but existing paradigms typically separate these capabilities.Shared context means directly interpreting generated visual content without re-encoding.
  • Approach: UniAR uses a single discrete visual tokenizer to unify visual understanding and generation while addressing representation design and vocabulary-scaling challenges.The framework is motivated by the need for tokens that support both tasks and a larger vocabulary at minimal cost.
  • Tokenizer: Multi-level feature fusion preserves shallow-layer details and deep-layer semantics, while lookup-free binary quantization efficiently expands the visual vocabulary without an explicit codebook.The representation is designed to support both discriminative and generative tasks.
  • Training: Training proceeds through large-scale multimodal pretraining, supervised fine-tuning, and reinforcement learning, with the tokenizer and decoder frozen throughout model training.The decoder is introduced during reinforcement learning for reward computation.
  • Results: UniAR achieves state-of-the-art image generation, text rendering, and instruction-following performance while remaining competitive on multimodal understanding benchmarks.The reported contributions emphasize broad image-generation effectiveness alongside standard multimodal understanding performance.
  • Generation: Parallel bitwise prediction and a DiT-based decoder with resolution upsampling enable efficient high-resolution generation from compact visual sequences.The model jointly predicts grouped visual codes, and the decoder reconstructs images from discrete visual tokens.

2 Related Works

Prior unified multimodal approaches use dual tokenizers, decoupled cascaded systems, or hybrid Transformers, limiting shared visual context or architectural synergy. UniAR instead uses multi-level lookup-free bitwise quantization and parallel-bitwise prediction to unify visual representation and improve efficiency.

  • Visual Tokenization: Understanding emphasizes high-level semantics, whereas generation requires high-frequency details such as texture and color, motivating dual-tokenizer designs.Dual tokenizers accommodate both objectives but inherently partition the visual context.
  • Unified Modeling: Decoupled cascaded architectures combine a frozen LMM with a diffusion model but often lack architectural synergy between understanding and generation.
  • Unified Modeling: Hybrid Transformers generate text autoregressively and images via flow-matching as an alternative unified-modeling paradigm.
  • Comparison with X-Omni: UniAR uses lookup-free bitwise quantization across multi-level features instead of an explicit codebook, expanding theoretical codebook size with lower overhead.
  • Comparison with X-Omni: UniAR’s parallel-bitwise-prediction paradigm quadruples inference speed by reducing autoregressive steps.

3 UniAR

UniAR unifies visual understanding, generation, and editing through a shared discrete visual tokenizer, a multimodal autoregressive model, and a DiT-based decoder. Its design combines multi-level quantized visual representations, grouped bitwise prediction, and token-to-image decoding for efficient unified modeling.

  • Architecture: UniAR comprises a shared visual tokenizer, a unified autoregressive model for visual and text tokens, and a DiT-based visual decoder.The tokenizer supports understanding, generation, and editing, while the decoder translates visual tokens back into pixel space.
  • Visual Tokenizer: The visual tokenizer adapts a pretrained vision encoder with BSQ, replacing reconstruction MSE with LMM cross-entropy and freezing the encoder after discrete adaptation.The 64-bit binary representation is spatially merged by aggregating 2 × 2 visual features into one token.
  • Visual Tokenizer: Multi-level fusion incorporates the final vision-encoder layer and three intermediate layers to capture multi-granular visual cues.The hierarchical features serve both generation and understanding.
  • Unified Autoregressive Model: The autoregressive model predicts grouped BSQ indices bitwise, with group size g = n_level × n_spatial, while retaining the original LLM output layer for text.Each visual prediction covers hierarchical levels and spatial units defined by the merger.
  • Visual Decoder: At each autoregressive step, predicted BSQ indices represent a 2 × 2 spatial region whose cells contain multi-level vectors, preserving spatial order for visual conditioning.The resulting conditioning signal is fused with noisy DiT hidden states, and bicubic upsampling enables high-resolution synthesis with a low autoregressive budget.

4 Experiments

UniAR is evaluated across image generation, image editing, and multimodal understanding benchmarks, achieving strong or state-of-the-art results across these tasks. Its discrete-token autoregressive design also supports reinforcement-learning improvements in text rendering.

  • Visual Generation: 0.86 overall on GenEval, surpassing GPT-4o and Flux.1-dev in instruction-following text-to-image synthesis.The result particularly reflects accurate object counting, color-attribute binding, and spatial relationships.
  • Visual Generation: 0.873 on OneIG-EN, surpassing GPT-4o in English text rendering, while also outperforming Gemini 2.5 Flash Image on LongText-Bench.The paper reports state-of-the-art text-rendering performance across the evaluated English subsets.
  • Image Editing: 3.73 overall on ImgEdit-Bench, surpassing Flux.1 Kontext, BAGEL, and OmniGen2 on image-editing tasks.Flux.1 Kontext is described as specifically designed for editing, while BAGEL and OmniGen2 are unified models.
  • Multimodal Understanding: UniAR significantly outperforms existing unified models on OCRBench, DocVQA, InfoVQA, and MVBench, while surpassing LLaVA-Onevision overall.Its performance is also comparable to state-of-the-art open-source vision-language models.
  • Reinforcement Fine-Tuning: 71.1 to 84.0 on OneIG-EN after 500 RL steps at 512-pixel resolution, with further improvement after 100 additional steps at 960 resolution.The experiment demonstrates that reinforcement learning improves text rendering as training steps increase.

5 Conclusion · Appendix

UniAR is presented as a unified autoregressive model for understanding, generation, and editing, built around shared visual representations and bitwise visual tokens. The appendix reports that increasing reinforcement-learning steps improves text-rendering capability.

  • Appendix: As reinforcement-learning steps increase, text-rendering capability improves significantly.Figure 6 reports this trend without providing a numeric value.
  • 5 Conclusion: UniAR unifies understanding, generation, and editing within a single autoregressive model.The framework is explicitly designed to combine these three capabilities.
  • 5 Conclusion: A unified visual tokenizer maps generative and understanding tasks into the same semantic space.This shared space is described as strengthening model unification.
  • 5 Conclusion: UniAR incorporates bitwise visual tokens for unified modeling.The conclusion identifies bitwise visual tokens as one of the framework’s three key designs.
  • 5 Conclusion: The bitwise visual-token design substantially enlarges representational capacity while incurring less computation.The supplied passage states both benefits, though its final computational qualification is truncated.
  • 5 Conclusion: The conclusion presents shared semantic mapping as a mechanism for strengthening model unification.This follows the tokenizer design described for generative and understanding tasks.

A Implementation Details · A.1 Pre-training and Supervised Fine-tuning

UniAR pre-training uses two sequential token-length stages with native-resolution visual encoding and a fixed 32 × 32 compression ratio. Training covers visual understanding, text-to-image generation, and image editing, while supervised fine-tuning adapts the formats to ChatML for dialogue and instruction following.

  • A.1 Pre-training and Supervised Fine-tuning: Pre-training proceeds in two sequential stages with maximum sequence lengths of 8K and 32K tokens.The procedure uses native-resolution visual encoding and flexibly processes data based on total token counts.
  • A.1 Pre-training and Supervised Fine-tuning: A constant 32 × 32 compression ratio represents each 512 × 512 image with 256 autoregressive tokens.This compression ratio is maintained for both understanding and generation tasks.
  • A.1 Pre-training and Supervised Fine-tuning: Visual understanding examples use the template {prompt; image_tokens; answers}.The template separates the prompt, visual tokens, and answer sequence.
  • A.1 Pre-training and Supervised Fine-tuning: Text-to-image generation examples use the template {prompt; image_tokens}.The format pairs a text prompt with image tokens.
  • A.1 Pre-training and Supervised Fine-tuning: Image-editing examples use the template {prompt; reference_image_tokens; image_tokens}.The format includes a prompt, reference-image tokens, and output image tokens.
  • A.1 Pre-training and Supervised Fine-tuning: Bold template tokens contribute to the autoregressive loss during training.The templates are subsequently adapted to ChatML for supervised fine-tuning.
  • A.1 Pre-training and Supervised Fine-tuning: ChatML adaptation supports multi-turn dialogue and instruction following during supervised fine-tuning.The detailed training recipe is provided in Table 6.

A.2 Reinforcement Learning

UniAR uses reinforcement learning with multiple specialized rewards, prompt data drawn from diverse sources, and two-stage GRPO training across resolutions. The procedure combines quality, text-rendering, and instruction-following objectives while controlling optimization with KL regularization.

  • Reward system: Multiple rewards target image quality, text rendering, and instruction following through HPSv2, UnifiedReward, PaddleOCR edit distance, and FlowGRPO object detection.HPSv2 and UnifiedReward improve aesthetic quality and reduce artifacts; PaddleOCR rewards accurate ground-truth text rendering.
  • Data collection: Prompts combine SFT samples, open-source multimodal datasets, GenEval and OCR training sets, and long-text prompts synthesized by a large language model.Sources include BLIP3o-60k and ShareGPT-4o-Image, alongside datasets created or used by FlowGRPO.
  • Training details: GRPO training uses a constant learning rate of 5 × 10−6, a KL-divergence coefficient of 0.01, batches of 32 prompts, and 16 generated images per prompt.These settings are part of a two-stage procedure spanning 512 × 512 and 960 × 960 resolutions.
  • Training details: Training proceeds for 500 steps at 512 × 512 resolution, followed by 100 additional steps at 960 × 960 resolution.The first stage quickly improves image quality and instruction following, while the higher-resolution stage enhances long-text capabilities.

A.3 Evaluation Configurations

Evaluation uses classifier-free guidance with benchmark-dependent scales and low sampling temperature, while inference resolution and decoder upsampling vary by benchmark.

  • Autoregressive sampling: Classifier-free guidance combines conditional and unconditional predictions with scale s= 2.5 for GenEval and s= 2.0 for other benchmarks.Sampling temperature is set to 0.1 during evaluation.
  • Inference resolutions: GenEval and ImgEdit generate 512 × 512 visual tokens, then apply ×2 decoder upsampling to produce 1024 × 1024 output images.The DiT decoder performs the upsampling.

A.4 Training and Inference Cost Analysis

UniAR improves training efficiency through offline bit-packed visual-token preprocessing and accelerates image-generation inference through a high downsampling ratio that reduces prediction steps.

  • Training efficiency: Approximately 30% higher pre-training throughput than continuous tokens is achieved at sequence length 8K using offline bit-packed visual representations.Visual inputs can be pre-tokenized and stored offline before training.
  • Inference efficiency: UniAR achieves significantly faster 1024-resolution image generation than Janus-Pro [5] and X-Omni [14] on the same A100 GPU without CFG.The comparison measures only autoregressive generation time, excluding the visual decoder.
  • Inference efficiency: Its 32× downsampling ratio quadratically reduces the number of visual prediction steps, driving the inference-speed advantage.The reported generation-time comparison excludes visual-decoder cost.

B Limitations and Future Works

UniAR’s joint pre-training excludes pure-text data because of resource constraints, and reinforcement learning currently covers only image generation. Future work will optimize the data mixture, scale pre-training, and improve post-training, especially through reinforcement learning.

  • Limitations: Resource constraints prevent including pure-text data in joint pre-training, leaving substantial room to optimize the data mixture for understanding and generation.The authors identify data-mixture optimization as an important direction for jointly supporting both capabilities.
  • Limitations: Reinforcement learning is currently limited to image generation.The paper notes that post-training, particularly through reinforcement learning, still has considerable room for improvement.
  • Future Works: Future work will scale both dataset size and model parameters through large-scale pre-training.The authors also plan to further explore improvements in post-training, especially reinforcement learning.
Loading 2606.18249v2…