VoT: Vision-of-Thought for Unified Multimodal Representation Alignment

1 Introduction

• Static text embeddings or key–value caches leave a modality gap that makes transferring an LLM’s structured knowledge to diffusion difficult.The paper describes a bottleneck between knowing what to generate and rendering it accurately.Source: §1 · p. 1
• VoT uses autoregressive discrete visual tokens as step-by-step plans, extending VLM reasoning into visual composition before rendering.Unlike query-based approaches such as MetaQuery [22] and BLIP-3o [33], VoT generates visual tokens autoregressively.Source: §1 · p. 2
• The tokenizer quantizes VLM semantic features and combines VLM alignment, feature reconstruction, and vector-quantization objectives.VLM alignment and reconstruction are complementary, and scaling the teacher VLM consistently benefits token quality and downstream alignment.Source: §1 · p. 3
• VoT’s MoT-based integration and two-stage model training substantially improve semantic alignment and reasoning-driven text-to-image generation.The approach clones VLM experts to initialize the VoT branch and freezes the VLM branch; training uses text–image pairs throughout.Source: §1 · p. 3

2.1 Unified Multimodal Understanding and Generation

• Unified multimodal models can compromise generation fidelity or understanding depth, while diffusion-based unification struggles to transfer VLM knowledge through static conditioning.VoT introduces discrete visual thoughts to separate semantic planning from pixel rendering while retaining the specialized strengths of VLMs and diffusion transformers.Source: §2.1 · p. 3

2.2 Visual Tokenization Strategies

• VoT addresses the lack of closed-loop training that preserves visual information while making discrete tokens readable by a VLM.Earlier tokenizers prioritize pixel fidelity; SEED [11] and LaVIT [14] use post-hoc alignment, while Emu [28] uses continuous representations.Source: §2.2 · pp. 3–4

2.3 Reasoning and Planning in Visual Generation

• VoT integrates discrete visual plans into generation through a unified token vocabulary, enabling visual reasoning about composition, spatial relations, and attributes before pixel values are set.Unlike methods such as LayoutGPT, which treat planning and generation as separate stages, VoT frames the process as text-to-thought-to-image generation.Source: §2.3 · p. 4

3 Overview of VoT

• The VLM first produces structured VoT tokens as explicit visual plans, which the diffusion branch renders in the unified model.The architecture inserts a discrete visual-thought layer rather than mapping text directly to pixels.Source: §3 · p. 4
• Stage 0 learns a discrete codebook in VLM semantic space using feature reconstruction, VLM alignment, and vector-quantization losses.The tokenizer pipeline encodes images with a frozen VLM ViT, adapts features, and discretizes them.Source: §3 · p. 5, Figure 3 · p. 5
• Stage 1 trains lightweight VoT experts to predict visual tokens from text prompts using targets from paired images; Stage 2 jointly fine-tunes VoT and diffusion while freezing the VLM branch.The frozen branches in Stage 1 are the pretrained VLM and diffusion branches; in Stage 2 the VLM remains frozen to preserve understanding and reduce training cost.Source: §3 · p. 5

4 VLM-Aligned VoT Tokenizer Training

• The tokenizer is designed to produce discrete visual representations compatible with VLM semantics rather than prioritize pixel-level fidelity.Its unified alignment objective aims to minimize distributional shift between visual tokens and the VLM semantic space.Source: §4 · p. 5

4.1 Semantically Aligned Tokenizer Architecture

• The tokenizer combines frozen Qwen2.5-VL ViT features with a trainable adapter and quantizes adapted semantic features into discrete codes.The adapter projects foundation features toward a quantizable manifold while retaining VLM semantic structure; a straight-through estimator passes quantized features onward.Source: §4.1 · p. 6
• Increasing decoder depth from 2 layers in Exp.D to 6 layers in Exp.E raises AutoEval from 69.32 to 70.37.The authors attribute the gain to increased capacity to invert quantization in the teacher feature space, allowing tokens to focus on language-grounded semantics while the decoder smooths quantization noise.Source: §4.1 · p. 6

4.2 Training Objectives

• The tokenizer objective combines feature reconstruction, VLM alignment, and vector-quantization losses to preserve teacher features, make tokens readable, and stabilize quantization.The reconstruction loss targets frozen teacher ViT features, while the alignment loss uses caption prediction by the frozen VLM.Source: §4.2 · p. 6

5 Unified Vision-Language Model with VoT

• The unified architecture extends Mixture-of-Transformers with understanding, visual-token prediction, and diffusion-synthesis pathways.The three branches share a transformer backbone but use modality-specific components.Source: §5 · p. 7, §5.1 · p. 7

5.1 Unified Three-Branch MoT Backbone

• Text and VoT branches use causal attention, while the diffusion branch uses bidirectional attention and branch-specific feed-forward networks.Each branch applies independent normalization and query, key, and value projections; inference generates VoT tokens autoregressively before diffusion renders an image conditioned on text and VoT key–value caches.Source: §5.1 · p. 7

5.2 Training Recipe

• Stage 1 warms up VoT experts with the pretrained backbone frozen, training them to predict discrete visual tokens without perturbing existing capabilities.The training uses targets from the frozen Stage 0 tokenizer and an autoregressive cross-entropy objective over a 65,536-way codebook, predicting 1,024 VoT tokens per image.Source: §5.2 · p. 8
• Stage 2 jointly trains VoT and diffusion pathways so they can co-adapt, with diffusion loss also updating shared attention layers.Training drops VoT tokens with probability 0.5 and text tokens with probability 0.1 when VoT is present, and randomly masks 0–20% of VoT key–value pairs.Source: §5.2 · p. 8

6.1 Implementation Details

• Stage 0 uses native-resolution pairs (188K–1M pixels); the unified model adds a 7B VoT branch to Mogao-14B, totaling 21B parameters.GenEval uses 553 prompts and generates four images per prompt.Source: §6.1 · p. 8, §6.2 · p. 8

6.2 Text-to-Image Generation

• 0.91 overall GenEval beats FLUX.1-dev (0.82), BAGEL (0.82), and Mogao (0.89); counting scores 0.86 vs. 0.83 for Mogao.The table reports VoT’s highest overall score and advantages in spatial and counting tasks.Source: §6.2 · pp. 8–9, Table 1 · p. 9
• On the internal instruction benchmark, VoT outperforms Seedream 4.0 and GPT-Image-1; selected cases show more accurate attributes and surreal scenes than GPT-Image-1.Figure 5 places VoT on the left and GPT-Image-1 on the right in each image pair.Source: §6.2 · pp. 9–10, Figure 5 · p. 10

6.3 Ablation Study

• Exp.A’s ∼5.08 final autoregressive loss coexists with 58.67 AutoEval, showing that lower token-prediction loss does not guarantee downstream utility.Figure 6 plots VLM alignment loss on the left and VoT autoregressive loss on the right.Source: Figure 6 · p. 10, §6.3 · p. 11
• AutoEval rises 53.12 → 70.37 with VoT-token training versus Mogao on the same T2I data, and SFT/RLHF further improve the score.Exp.F follows supervised fine-tuning and reinforcement learning from human feedback.Source: Table 2 · p. 11, §6.3 · p. 11
• AutoEval rises 58.67 → 60.98 → 64.64 as reconstruction and then a 1-layer adapter are added, while tokenizer LM loss falls ∼1.93 → ∼1.79 → ∼1.76.Source: §6.3 · p. 11
• With a Qwen2.5-VL 7B teacher, tokenizer LM loss falls to ∼1.48 and AutoEval rises to 69.32 from 64.64 with the Qwen2-VL 2B setup.The stronger-teacher setting also has the lowest autoregressive loss among reconstruction-based settings.Source: §6.3 · p. 11

• VoT inserts a discrete visual-planning layer between VLMs and diffusion decoders to bridge semantic understanding and visual synthesis.The conclusion describes structured VoT tokens as plans generated before rendering.Source: §7 · p. 11

• The training compute budget is measured in GPU-hours on high-performance GPUs.Source: Table 3 · p. 14

• The MoT architecture’s frozen VLM backbone and parallel experts limit transfer to unidirectional reasoning-to-generation.Future work will explore bidirectional optimization without catastrophic forgetting.Source: B Limitations and Future Work · p. 14

• Figure 7 presents additional text-to-image generation results.Source: Figure 7 · p. 14