PaperlayerView exact revision on arXiv

Source-linked AI summary

OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generation

Han Li, Xinyu Peng, Yaoming Wang, Zelin Peng, Xin Chen, Rongxiang Weng, Jingang Wang, Xunliang Cai, Wenrui Dai, Hongkai Xiong

arXiv:2509.03498v3cs.CV

TL;DR

OneCAT addresses the challenge of building an efficient unified multimodal model without external vision components. It uses a pure decoder-only transformer with modality-specific experts, unified autoregressive training, and multi-scale visual generation, achieving strong cross-task performance and improved inference efficiency. The paper presents this architecture as a viable foundation for unified multimodal intelligence.

  • Problem

    Existing unified multimodal systems rely on external encoders, tokenizers, or diffusion components that limit architectural simplicity and inference efficiency.

    Source: §1 · p. 1, §2.3 · p. 4

  • Method

    OneCAT combines modality-specific MoE routing, hidden-state distillation, unified autoregressive training, and multi-scale visual autoregression in a pure decoder-only model.

    Source: §1 · pp. 1, 3

  • Results

    OneCAT achieves strong performance across multimodal understanding, generation, and editing benchmarks while significantly improving inference efficiency.

    Source: §8 · p. 17

  • Takeaways & Limitations

    The results support pure decoder-only autoregressive modeling as a viable foundation and baseline for general-purpose multimodal intelligence.

    Source: §8 · p. 17

    Diffusion-based unified approaches still inherit significant inference latency and require separate encoders and tokenizers during inference.

    Source: §2.3 · p. 4

Abstract

from arXiv · show

We introduce OneCAT, a unified multimodal model that seamlessly integrates understanding, generation, and editing within a novel, pure decoder-only transformer architecture. Our framework uniquely eliminates the need for external components such as Vision Transformers (ViT) or vision tokenizer during inference, leading to significant efficiency gains, especially for high-resolution inputs. This is achieved through a modality-specific Mixture-of-Experts (MoE) structure trained with a single autoregressive (AR) objective, which also natively supports dynamic resolutions. Furthermore, we pioneer a multi-scale visual autoregressive mechanism within the Large Language Model (LLM) that drastically reduces decoding steps compared to diffusion-based methods while maintaining state-of-the-art performance. Our findings demonstrate the powerful potential of pure autoregressive modeling as a sufficient and elegant foundation for unified multimodal intelligence. As a result, OneCAT sets a new performance standard, outperforming existing open-source unified multimodal models across benchmarks for multimodal generation, editing, and understanding.

1 Introduction

OneCAT addresses the latency and fusion limits of modular multimodal systems with a unified decoder-only autoregressive model. Its modality-specific experts, unified training, and multi-scale generation support understanding, generation, and editing while improving inference efficiency.

Motivation and overall contribution

  • OneCAT unifies multimodal understanding, generation, and editing in a single decoder-only autoregressive model trained with one objective.The design avoids the complex multi-stage pipelines associated with separate task modules.Source: §1 · pp. 1, 3

Unified architecture

  • Modality-specific MoE routing sends continuous vision, discrete vision, and text tokens to specialized experts for early cross-modal fusion without external ViT or vision-tokenizer components.The model directly processes raw visual inputs as patch embeddings alongside text tokens.Source: §1 · p. 1
  • A multi-scale autoregressive mechanism predicts image tokens from low to high resolutions, while text uses next-token prediction to support faster, coarse-to-fine generation.A scale-aware adapter extracts scale-specific representations to improve image generation speed and quality.Source: §1 · p. 1

Unified training paradigm

  • A custom MLLM teacher transfers visual knowledge through hidden-state distillation before unified expert pretraining, mid-training, and supervised fine-tuning.The strategy is designed to avoid the supervisory bottleneck caused by smaller ViT teachers.Source: §1 · p. 3

Results and significance

  • OneCAT reports state-of-the-art unified-model performance and faster inference, especially for high-resolution inputs and outputs.The paper attributes these gains to encoder-free processing, earlier cross-modal fusion, and the unified autoregressive objective.Source: §1 · p. 3

2 Related Work

Related work progresses from compositional multimodal systems toward decoder-only and unified architectures, but existing approaches retain information, training, or efficiency limitations. OneCAT combines encoder-free early fusion with next-scale visual generation in a single decoder-only transformer.

2.1 Compositional MLLMs

  • Compositional MLLMs connect vision encoders to LLMs through trainable connectors, but sequential processing increases latency and late fusion can lose fine-grained visual detail.These systems remain effective but complicate end-to-end optimization.Source: §2.1 · p. 4

2.2 Decoder-only MLLMs

  • Decoder-only MLLMs remove separate vision encoders or tokenizers to improve efficiency and early fusion, yet their training efficiency and unified generation capabilities remain open challenges.Prior methods use patch embeddings, visual distillation, MoE, or causal transformers to improve visual processing.Source: §2.2 · p. 4

2.3 Unified Visual Understanding and Generation

  • Unified understanding-and-generation models often use vision tokenizers, separate encoders, adapters, or diffusion components, which can cause information loss, task conflicts, modularity, or inference latency.BAGEL and Mogao retain separate encoders and tokenizers while inheriting diffusion-model latency.Source: §2.3 · p. 4
  • OneCAT uses modality-specific experts inside a pure decoder to provide multimodal capabilities without external vision encoders or tokenizers during inference.This design targets both architectural unification and inference efficiency.Source: §2.3 · pp. 4–5

2.4 Next Scale Prediction for Visual Generation

  • Next-token prediction becomes inefficient for high-resolution images because sequence length grows quadratically, while diffusion models require slow iterative sampling.VAR instead generates hierarchical tokens from low to high resolution through next-scale prediction.Source: §2.4 · p. 5
  • OneCAT unifies next-token prediction for text and next-scale prediction for visual generation within one decoder-only transformer, augmented by a scale-aware adapter.Images are generated hierarchically from the lowest to the highest resolution scales.Source: §2.4 · p. 5

3 OneCAT

OneCAT is a unified decoder-only model that handles text, understanding, generation, and editing through modality-specific experts and hierarchical visual prediction.

3 OneCAT

  • OneCAT uses a pure decoder-only architecture with raw images converted directly into continuous visual tokens, avoiding external vision encoders or tokenizers during inference.Visual generation instead uses next-scale prediction, producing images hierarchically from coarse to fine.Source: §3 · p. 5

3.1 Modality-MoE

  • The Modality-MoE routes text, continuous visual, and discrete visual tokens to specialized Text, Und., and Gen. FFN experts.Shared QKV and attention layers promote parameter efficiency and cross-modal alignment.Source: §3.1 · p. 6
  • Hard routing assigns each token to the expert associated with its modality and task, while the model replicates Qwen2.5 FFNs to construct the three experts.OneCAT is initialized from the pretrained Qwen2.5 LLM.Source: §3.1 · p. 6

Multimodal Understanding.

OneCAT combines direct visual patch processing, hierarchical generation, and modality-specific attention patterns to support understanding, editing, and generation in one decoder.

Multimodal Understanding.

  • The patch embedding layer converts raw images into continuous visual tokens, while a multi-scale VAE supplies discrete generation targets and is unnecessary during inference.Reference images for editing use the same patch embedding pathway rather than a separate tokenizer.Source: Multimodal Understanding · p. 6
  • Image editing conditions autoregressive discrete visual-token prediction on reference-image tokens routed through the understanding expert.The shared representation supplies low-level features for pixel consistency and deeper semantic features.Source: Multimodal Understanding · p. 6

3.2 Scale-Aware Adapter for Hierarchical Generation

  • Scale-Aware Adapters assign separate low-rank modules to each VAE scale so hierarchical tokens receive scale-specific processing.Each adapter is jointly trained as a permanent Gen. FFN component with rank r=64.Source: §3.2 · p. 6

3.3 Multimodal versatile attention mechanism

  • Text tokens use causal attention, continuous visual tokens use full attention, and multi-scale discrete visual tokens use block-causal attention.Within-scale generation tokens attend freely, while cross-scale attention remains causal.Source: §3.3 · pp. 6–7, Figure 4 · p. 7

4 Model Training Pipeline

OneCAT is trained through staged teacher construction, expert pretraining, unified mid-training, and supervised fine-tuning, progressively expanding task and resolution coverage.

4 Model Training Pipeline

  • The three-stage pipeline proceeds from Stage-1 multimodal pretraining to fully unfrozen unified mid-training and final supervised fine-tuning.Stage 1 preserves pretrained language capabilities by freezing the remaining LLM parameters.Source: §4 · p. 7, §4.2 · p. 8, §4.3 · p. 9

4.1.2 Stage 1-2: Expert Pretraining

  • Stage-1 expert pretraining optimizes the Und. FFN and Patch Embedding layer for understanding and the Gen. FFN for generation.Understanding uses next-token prediction plus hidden-state distillation, while generation uses next-scale prediction at 256×256 resolution.Source: §4.1 · p. 7, §4.1.2 · p. 8

4.1.1 Stage 1-1: Teacher Training

  • A customized teacher connects InternViT [19] and Qwen2.5 [105] through a two-layer MLP, improving distillation stability through backbone parameter consistency.The ViT and LLM are frozen while the connector is trained on image-text captions.Source: §4.1.1 · pp. 7–8

4.1.2 Stage 1-2: Expert Pretraining

  • Understanding distillation matches student and teacher hidden states across transformer layers and token types, with LUnd = LNTP + λLDistill and λ = 0.02.Distillation images use a fixed 448 × 448 resolution.Source: §4.1.2 · p. 8

4.2 Stage-2: Unified Mid-Training

  • Stage-2 unified mid-training adds the Scale-Aware Adapter, native-resolution understanding, thumbnails for global context, and dynamic generation resolutions from 288 to 864 pixels.The model is trained jointly on image-to-text, text-to-image, editing, and text-only dialogue tasks.Source: §4.2 · pp. 8–9

4.3 Stage-3: Unified Supervised Fine-tuning

  • Stage-3 supervised fine-tuning continues native-resolution training and expands generated-image side lengths to 288–1776 pixels.The curated data targets instruction following and visual generation quality.Source: §4.3 · p. 9

5 Data Setup

OneCAT’s data setup combines large-scale image-text and generation corpora with diverse visual instructions, editing examples, and text-only data across three training stages.

Stage 1

  • Stage-1 uses approximately 436 million image-text pairs for understanding and 52 million filtered text-to-image samples for generation.A small 10-million-sample subset trains the custom teacher, and the understanding-to-generation token ratio is approximately 8:1.Source: §5 · p. 9

Stage 2

  • Stage-2 adds 70 million diverse visual-instruction samples, 8 million image-editing samples, and 40 million text-only instructions.Oversampling yields a text-only:understanding:generation token ratio of approximately 1:2:6.Source: §5 · p. 9

Stage 3

  • Stage-3 uses 13 million high-quality understanding and text-only samples plus 3 million visual-generation samples.Its overall token ratio across text-only, multimodal understanding, and generation is approximately 1:5:6.Source: §5 · pp. 9–10

Evaluation setup

  • Table 3 compares multimodal understanding across benchmarks using higher-is-better scores and reports activated LLM and vision-module parameters.Unified-model best and second-best results are highlighted through bold and underlining conventions.Source: Table 3 · p. 10

6 Implementation details

OneCAT’s implementation uses MoE variants, distributed-training strategies, and classifier-free guidance to balance multimodal workloads and improve generation.

Model Configurations

  • OneCAT-1.5B and OneCAT-3B are initialized from Qwen2.5-1.5B-instruct and Qwen2.5-3B-instruct, respectively.OneCAT-3B has 9B total parameters, with 3B activated per token during forward computation.Source: §6 · p. 10

Data Packing and Gradient Accumulation

  • Uneven gradient accumulation controls effective batch sizes and token ratios across text and image-generation tasks.Data packing concatenates variable-length samples into contiguous sequences to improve distributed workload balance and throughput.Source: §6 · p. 10

Unbiased Global Batch Gradients

  • Global Batch Reduced Loss uses globally aggregated token counts to produce gradients mathematically equivalent to the global batch objective.Local token counts are combined with an All-Reduce operation before loss normalization.Source: §6 · p. 11

Classifier-free guidance (CFG)

  • Classifier-free guidance randomly drops conditional text and reference-image tokens during training, then combines conditional and unconditional logits at inference.This procedure is used to enhance visual generation quality.Source: §6 · p. 11

7 Evaluation

OneCAT is evaluated across understanding, generation, editing, efficiency, and ablation settings. It achieves strong benchmark performance while improving high-resolution inference speed and validating its distillation and unified-training choices.

7.1 Multimodal understanding.

  • OneCAT-3B achieves new encoder-free-model bests on AI2D (77.8), ChartQA (81.2), InfoVQA (64.8), and DocVQA (91.2).It also scores 1630 on MME-P, 78.8 on MMBench, and 52.2 on MM-Vet.Source: §7.1 · pp. 12–13
  • OneCAT-3B outperforms unified models using external vision encoders or tokenizers despite activating fewer parameters.The model has a slight gap to Qwen2.5-VL-3B, attributed primarily to training-data scale and quality differences.Source: §7.1 · p. 13

7.2 Visual Generation.

  • 0.90 overall on GenEval establishes OneCAT-3B as a SOTA unified-model result, exceeding BAGEL-7B at 0.88 and Mogao-7B at 0.89.The comparison uses raw prompts rather than LLM-based prompt rewriting.Source: §7.2 · p. 14, Table 5 · p. 14
  • 84.53 on DPG-Bench surpasses Janus-Pro-7B at 84.19 and Mogao-7B at 84.33 among unified models.DPG-Bench measures compositional text-to-image generation.Source: §7.2 · p. 14, Table 6 · p. 15
  • 3.43 overall on ImgEdit-Bench places OneCAT-3B among top-performing unified models and above many specialized editing models.It leads in Adjust (3.70), Extract (2.42), and Background (3.79) categories.Source: §7.2 · p. 14, Table 7 · p. 15

7.4 Comparison of Inference Efficiency

  • 61% lower first-token latency and approximately 10× faster 1024 × 1024 generation demonstrate OneCAT’s high-resolution inference advantage.OneCAT requires 2.85s for text-to-image generation and 4.61s for image editing at 1024 × 1024.Source: §7.4 · pp. 15–16, Table 8 · p. 16

7.5.1 Ablation Study on Understanding Distillation (Und. Distil.) Stage

  • Full-layer hidden-state distillation improves visual learning efficiency and multimodal performance more than last-layer, logit, EvE [23], or VoRA [91] alternatives.Using Qwen2.5-VL-1.5B as teacher causes unstable training and lower distillation efficiency because of teacher–student parameter misalignment.Source: §7.5.1 · p. 16, Table 9 · p. 16, Table 10 · p. 16

7.5.2 Ablation Study on Unified Mid-Training (Stage-2)

  • The adopted 1:2:6 Stage-2 token ratio balances training cost and performance, while removing SAA causes an obvious performance drop.Increasing generation tokens improves generation performance without significantly affecting understanding.Source: §7.5.2 · pp. 16–17, Table 11 · p. 17, Figure 8 · p. 17

8 Conclusion

OneCAT unifies understanding, generation, and editing in a pure decoder-only model, using multiscale visual tokenization to support coarse-to-fine autoregressive generation.

8 Conclusion

  • OneCAT integrates understanding, generation, and editing within a single pure decoder-only architecture.The design eliminates external encoders and tokenizers, uses modality-specific MoE routing, and improves inference efficiency.Source: §8 · p. 17, Table 12 · p. 17

A.1 Multiscale Tokenization

  • Multiscale tokenization represents images at K resolutions whose spatial dimensions increase monotonically with scale index.The feature map is recursively resized and quantized, with Q implemented using Binary Spherical Quantization.Source: §A.1 · pp. 24–25
  • The multiscale representation assumes the final feature map can be approximated by summing all scales after upsampling to the final resolution.This coarse-to-fine structure supports the model’s multiscale visual autoregressive mechanism.Source: §A.1 · p. 24

A.2 Visual Auto-Regressive Training

OneCAT generates images autoregressively across multiple visual scales, using predefined schedules for varied aspect ratios and classifier-free guidance for conditional generation and editing.

A.2 Visual Auto-Regressive Training

  • The model generates a sequence of visual scales autoregressively, predicting each scale from preceding scales while conditionally factoring tokens within the current scale for parallel decoding.This reduces image synthesis to generating scale representations rather than every token sequentially, but the independence assumption can propagate early-scale errors.Source: §A.2 · p. 25
  • Early-scale mistakes can be amplified at higher resolutions because conditional independence and imperfect training expose generation to error propagation.Bitwise Self-Correction trains on corrupted conditioning scales to improve robustness, although the cited passage attributes this technique to prior work.Source: §A.2 · p. 25

A.3 Predefined Scale Schedules

  • Predefined scale schedules preserve approximate target aspect ratios and roughly constant area across scales, standardizing sequence lengths and supporting varied output ratios.The schedules use K = 13 scales for a 1024 × 1024 image, while a 512 × 512 image can truncate the schedule to K = 10.Source: §A.3 · p. 25, Table 13 · p. 26

B Details of Class-Free Guidance

  • The implementation randomly drops conditional text and reference-image tokens during training and uses λt = 20 for text-to-image generation, versus λi = 1 and λt = 3 for editing.These settings define the guidance strengths used during inference for the two task types.Source: B Details of Class-Free Guidance · pp. 25–26
  • Classifier-free guidance combines conditional and unconditional logits, with text guidance for text-to-image generation and dual text-reference guidance for image editing.For editing, the method first blends logits conditioned on text and a reference image with text-only logits, then combines the result with unconditional logits.Source: B Details of Class-Free Guidance · pp. 25–26

C Additional ablation study

The ablations examine fusion strategy, distillation targets, unified mid-training, and scale-aware visual representations, showing efficiency benefits, the importance of joint token distillation, and scale-specific feature roles.

C.1 Effect on Early and Late Fusion in MLLM.

  • Early and late fusion perform on par across training scales, while early fusion offers a distinct computational-efficiency advantage.The comparison uses a randomly initialized InternViT, a frozen pretrained Qwen2.5-1.5B-instruct LLM, and simplified SFT on LLaVA-665k.Source: §C.1 · p. 27, Table 14 · p. 27

C.2 Effect on Distilling Only Visual Tokens

  • Distilling only continuous visual tokens causes a slight overall performance drop, indicating that both visual and text tokens are important distillation targets.This result is reported in the comparison of alternative token-distillation strategies.Source: §C.2 · p. 27, Table 15 · p. 28

C.3 Effect on The Increase of Training Tokens of Unified Mid-Training (Stage-2)

  • OneCAT-1.5B is evaluated at multiple unified mid-training checkpoints on multimodal understanding and generation benchmarks after simplified unified SFT.The SFT combines LLaVA-665K and BLIP3o-60K, with oversampling to balance understanding and generation training tokens.Source: §C.3 · p. 28, Figure 11 · p. 28

D Visualization of discrete visual tokens of different scales

  • Lower-scale visual tokens primarily encode low-frequency global information, whereas higher-scale tokens capture high-frequency details, supporting the scale-aware adapter design.The visualization compares feature maps and FFT frequency-intensity maps across scales and LLM layers.Source: D Visualization of discrete visual… · pp. 28–29, Figure 12 · p. 29

E More Qualitative results

Qualitative results illustrate OneCAT’s multimodal generation, editing, and understanding capabilities across varied tasks, resolutions, and aspect ratios.

E More Qualitative results

  • OneCAT produces text-to-image outputs across aspect ratios and resolutions, performs diverse image edits, and supports mathematical reasoning, OCR, and detailed image captioning.The qualitative comparisons include style transfer, object adjustment, attribute modification, object removal, and background editing alongside multimodal understanding examples.Source: E More Qualitative results · p. 29, Figure 13 · p. 30, Figure 14 · p. 31

Related papers

Loading 2509.03498v3…