MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer

1 Introduction

• Prior unified models lag understanding-only counterparts, especially on text-rich benchmarks, while discrete and continuous token preferences create representation conflict.Dual tokenizers can force the LLM to process tokens from distinct semantic and spatial spaces; decoupled approaches can lose scaling benefits for generation.Source: §1 · p. 1
• Manzano uses one shared visual encoder with continuous and discrete adapters, then jointly trains an autoregressive LLM on text, understanding, and generation data.A diffusion decoder renders pixels from generated image tokens.Source: §1 · p. 3
• Manzano achieves state-of-the-art performance on understanding and generation, with its 3B model improving text-rich understanding while remaining competitive in generation.Ablations indicate minimal cross-task conflict under joint training.Source: §1 · p. 3
• Scaling the LLM decoder from 300M to 30B improves both understanding and generation benchmarks, while a larger diffusion decoder improves image structural integrity.The latter finding is validated by large-scale human evaluations.Source: §1 · p. 3

2.1 MLLMs for Image Understanding

• Understanding-focused MLLMs use trainable vision-language interfaces, from gated cross-attention and Q-Former connectors to lightweight MLP projections, but lack high-quality image generation.Their training recipes and scaling strategies are described as more mature than those of unified models.Source: §2.1 · p. 3

2.2 Unified Multimodal Models

• Unified models use autoregressive, decoupled LLM-diffusion, or hybrid AR-diffusion designs; Manzano follows the autoregressive paradigm with a shared semantic tokenizer and diffusion synthesis decoder.Its tokenizer produces continuous understanding features and quantized generation features.Source: §2.2 · pp. 3–4

2.3 Diffusion Models for Image Generation

• Diffusion models refine noise into images, with latent diffusion reducing compute requirements; Manzano conditions a latent-domain DiT flow-matching decoder on LLM-generated visual token embeddings.This conditioning differs from conventional text-to-image diffusion models that use pretrained text-encoder embeddings.Source: §2.3 · p. 4

3 Model

• Manzano’s hybrid tokenizer supplies continuous and discrete visual representations to an autoregressive LLM that predicts text or image tokens, which an image decoder renders into pixels.The architecture comprises a hybrid vision tokenizer, an LLM decoder, and an image decoder.Source: §3 · p. 4

3.1 Design Choices

• Continuous features serve image-to-text understanding, discrete tokens serve text-to-image generation, and both branches share an encoder and semantic space.The LLM predicts high-level semantics while the diffusion decoder renders high-fidelity pixel details; the unified decoder uses one autoregressive objective without per-task heads or auxiliary losses.Source: §3.1 · pp. 4–5

3.2 Architecture

• The tokenizer uses a ViT backbone and parallel adapters: the continuous branch projects compressed spatial features, while the discrete branch applies finite scalar quantization before projection.Spatial-to-Channel compression reduces spatial tokens by a factor of 9; the discrete adapter uses a 64K codebook in the reported experiments.Source: §3.2 · p. 5

Unified LLM Training

• During unified training, continuous features support understanding and discrete image tokens support generation; at inference, the decoder renders predicted image tokens into pixels.The image decoder is trained on a pretrained hybrid tokenizer, and the separated decoders support independent scaling.Source: §3.1 · p. 5, Figure 4 · p. 6, §3.2 · p. 6

4.1 Data

• The three-stage recipe mixes text-only, image-understanding, and generation data, with curated instruction data used during supervised fine-tuning.Pretraining uses large-scale mixed data, continued pretraining uses higher-quality understanding and generation data, and fine-tuning targets instruction following.Source: §4.1 · p. 6

4.1.1 Pre-training & Continued Pre-training

• Pretraining and continued pretraining draw on captioning and interleaved image-text data for understanding and text-image pairs for generation.The continued understanding stage includes 24M capability-oriented examples; generation pretraining uses 1B in-house text-to-image pairs.Source: §4.1.1 · pp. 6–7

4.1.2 Supervised Fine-tuning

• SFT combines image-text and text-only understanding data with real and synthetic text-image generation examples.The understanding mix includes knowledge, document and chart understanding, and reasoning data; generation training expands DreamO examples with 90K dataset pairs and 4M synthetic pairs.Source: §4.1.2 · p. 7

4.2.1 Hybrid tokenizer training

• Tokenizer pretraining randomly selects one of two adapters to align its features with a small LLM decoder, then retains the trained tokenizer for downstream use.The shared encoder and parallel branches are trained across understanding domains, aligning both branches to the same space.Source: §4.2.1 · p. 7

4.2.2 Unified LLM Training

• Unified LLM training uses continuous features for text-target understanding and discrete image-token targets for generation, with a 1:0.5 text-to-image loss weight.The vision encoder and discrete adapter are frozen to maintain a fixed image-token vocabulary; training proceeds through pretraining, continued pretraining, and SFT.Source: §4.2.2 · pp. 7–8

4.2.3 Image Decoder Training

• The image decoder is trained progressively, beginning at 256x256 for 400K steps and then fine-tuning at 512, 1024, and 2048 resolutions for 100K steps each.Each stage uses images whose short sides exceed the target resolution.Source: §4.2.3 · p. 8

• Understanding benchmarks cover general VQA, knowledge and reasoning, and text-rich document and chart tasks.The listed benchmarks include SeedBench, RealWorldQA, MMBench, AI2D, ScienceQA, MMMU, MathVista, ChartQA, TextVQA, DocVQA, InfoVQA, and OCRBench.Source: §5.1 · p. 8
• Generation evaluation combines prompt-following benchmarks GenEval and DPGBench with WISE for world-knowledge-informed generation.Source: §5.1 · p. 8
• Human evaluation rates 800 challenging prompts for structural integrity, instruction following, and aesthetic quality.Three raters independently assess each sample; scores are averaged, with entity information masked and sample order randomized.Source: §5.1 · p. 8

• The tokenizer ablation compares pure-discrete, dual-encoder, and hybrid strategies, while omitting continued pre-training in the unified-model ablations.Source: §5.2 · pp. 8–9
• The pure-discrete baseline uses discrete features for both tasks; the dual-encoder baseline uses semantic understanding features and MagViT-2 generation tokens.For dual-encoder generation, images are resized to 128x128, yielding 256 tokens per image.Source: §5.2 · p. 9
• The hybrid tokenizer shows the least task conflict and outperforms pure-discrete and dual-encoder baselines on all tasks.Pure-discrete quantization particularly harms text-rich understanding; dual-encoder still underperforms the hybrid on understanding, especially knowledge benchmarks.Source: §5.2 · p. 9
• Unified training with the hybrid tokenizer performs on par with task-only models on nearly all tasks, including at 300M scale.The ablation compares 300M and 3B decoders, with training steps adjusted to match the relevant understanding or generation token exposure.Source: §5.2 · p. 9, Figure 5 · p. 9

• Scaling experiments vary the LLM decoder or image decoder separately and skip continued pre-training in unified LLM training.Source: §5.3 · p. 9
• Compared with 300M, 3B gains +14.2 (General), +18.8 (Knowledge), +10.9 (Text-rich), +11.0 (GenEval), +1.48 (DPG), and +12.0 (WISE).Further scaling to 30B yields smaller but consistent gains over 3B; qualitative examples also improve in text rendering, instruction following, and image quality.Source: §5.3 · p. 10, Figure 6 · p. 10
• Scaling the image decoder improves structural integrity by +9.9; instruction following is unchanged, aesthetic quality decreases slightly, and WISE improves by +2.0.GenEval and DPGEval remain nearly identical across image decoder sizes.Source: §5.3 · p. 10, Figure 6 · p. 10
• GenEval and DPG performance saturate as the model grows, while substantial improvements appear on world-knowledge generation tasks.The authors note that existing benchmarks may capture only part of overall capability and can be boosted through targeted data tuning [50].Source: §5.3 · p. 10

5.4.1 Image Understanding

• Qualitative examples show improved text rendering and creativity with larger LLM decoders, and more complete integration of prompt concepts at 30B.Source: Figure 7 · p. 11
• At 3B, Manzano outperforms unified models within 7B and matches or exceeds the best 3B specialists; at 30B it ranks first on ScienceQA, MMMU, and MathVista, and third on AI2D.The 30B decoder was pre-trained on roughly half the tokens used for other sizes due to compute limits.Source: §5.4.1 · pp. 10–11, Table 2 · p. 12
• Manzano generally outperforms other unified models on general visual question answering and is competitive with state-of-the-art specialists at both scales.Source: §5.4.1 · p. 11, Table 2 · p. 12
• Manzano achieves the best results on ChartQA, TextVQA, DocVQA, and OCRBench; on InfoVQA it leads unified counterparts and specialists.Source: §5.4.1 · p. 12, Table 2 · p. 12

5.4.2 Image Generation

• Manzano achieves state-of-the-art results among unified models on GenEval and WISE; scaling from 3B to 30B especially improves WISE while maintaining strong GenEval performance.WISE additionally evaluates semantic grounding through world-knowledge-informed attributes.Source: §5.4.2 · p. 12, Table 3 · p. 14

5.4.3 Comparison with Unified Models

• At similar scale, the 3B model exceeds X-Omni and BAGEL on DocVQA, OCRBench, and SEEDBench while remaining competitive on MathVista and ChartQA.The 30B model further surpasses existing unified models across knowledge, general VQA, and text-rich domains.Source: §5.4.3 · pp. 12, 14

5.5 Capability Extension to Image Editing

• Fine-grained image editing remains challenging, and an LLM alone lacks native mechanisms for direct pixel-level control.The cited work reports this difficulty for the decoupled LLM–diffusion paradigm as well.Source: §5.5 · p. 14
• Reference-image conditioning lets the LLM handle instruction following and semantic coherence while the diffusion decoder enforces precise pixel-level control.Both components jointly condition on the reference image.Source: §5.5 · p. 14
• Manzano demonstrates instruction-guided editing, style transfer, inpainting, outpainting, and depth estimation.The paper reports pixel-level controls across these five editing tasks.Source: §5.5 · p. 14

6 Conclusion

• Manzano combines a hybrid image tokenizer and unified autoregressive backbone with a lightweight diffusion decoder that renders pixels from generated image tokens.The LLM predicts high-level semantics as text and image tokens.Source: §6 · p. 14
• The authors report state-of-the-art understanding, substantial generation gains among unified models, and minimal task interference.These findings are validated by interplay and scaling ablations.Source: §6 · p. 14
• Conditioning both the LLM and image decoder on a reference image supports instruction-following editing with pixel-level control.Figure 9 covers instruction-guided editing, style transfer, inpainting, outpainting, and depth estimation.Source: §6 · p. 14, Figure 9 · p. 15
• The authors suggest that unification need not sacrifice accuracy for creativity and identify conversational editing, reasoning, and additional modalities as future directions.They propose exploring the same hybrid-tokenizer, unified-backbone, and image-decoder recipe.Source: §6 · p. 14