Source-linked AI summary
UniWeTok: An Unified Binary Tokenizer with Codebook Size $\mathit{2^{128}}$ for Unified Multimodal Large Language Model
Shaobin Zhuang, Yuang Ai, Jiaming Han, Weijia Mao, Xiaohui Li, Fangyikang Wang, Xiao Wang, Yan Li, Shanchuan Lin, Kun Xu, Zhenheng Yang, Huaibo Huang, Xiangyu Yue, Hao Chen, Yali Wang
TL;DR
Unified MLLMs need visual representations that support reconstruction, semantic extraction, and generation, but existing tokenizers struggle to provide all three together. UniWeTok addresses this with a unified discrete tokenizer using a 2^128 binary codebook, specialized distillation and generative-prior losses, a hybrid architecture with SigLu, and three-stage training. It achieves competitive multimodal understanding, generation, and editing results while reaching FID: UniWeTok 1.38 vs. REPA 1.42 with Training Tokens: UniWeTok 33B vs. REPA 262B.
Problem
Existing visual tokenizers struggle to jointly support high-fidelity reconstruction, semantic extraction, and generative suitability for Unified MLLMs.
Method
UniWeTok uses a 2^128 binary codebook with Pre-Post Distillation, Generative-Aware Prior, SigLu, a convolution-attention hybrid backbone, and three-stage curriculum training.
Results
UniWeTok shows competitive multimodal understanding, generation, and editing, including FID: UniWeTok 1.38 vs. REPA 1.42 and Training Tokens: UniWeTok 33B vs. REPA 262B.
Takeaways & Limitations
The paper presents UniWeTok as a unified discrete tokenizer that supports semantic concepts, fine-grained texture details, and generative priors within one framework.
Abstract
from arXiv · showhide
Unified Multimodal Large Language Models (MLLMs) require a visual representation that simultaneously supports high-fidelity reconstruction, complex semantic extraction, and generative suitability. However, existing visual tokenizers typically struggle to satisfy these conflicting objectives within a single framework. In this paper, we introduce UniWeTok, a unified discrete tokenizer designed to bridge this gap using a massive binary codebook ($\mathit{2^{128}}$). For training framework, we introduce Pre-Post Distillation and a Generative-Aware Prior to enhance the semantic extraction and generative prior of the discrete tokens. In terms of model architecture, we propose a convolution-attention hybrid architecture with the SigLu activation function. SigLu activation not only bounds the encoder output and stabilizes the semantic distillation process but also effectively addresses the optimization conflict between token entropy loss and commitment loss. We further propose a three-stage training framework designed to enhance UniWeTok's adaptability cross various image resolutions and perception-sensitive scenarios, such as those involving human faces and textual content. On ImageNet, UniWeTok achieves state-of-the-art image generation performance (FID: UniWeTok 1.38 vs. REPA 1.42) while requiring a remarkably low training compute (Training Tokens: UniWeTok 33B vs. REPA 262B). On general-domain, UniWeTok demonstrates highly competitive capabilities across a broad range of tasks, including multimodal understanding, image generation (DPG Score: UniWeTok 86.63 vs. FLUX.1 [Dev] 83.84), and editing (GEdit Overall Score: UniWeTok 5.09 vs. OmniGen 5.06). We release code and models to facilitate community exploration of unified tokenizer and MLLM.
1 Introduction
UniWeTok targets the tension among efficient compression, high-fidelity reconstruction, semantic extraction, and generative suitability in unified multimodal modeling. It combines a massive binary codebook with new training, architectural, and curriculum strategies, achieving strong generation and downstream multimodal results.
- Motivation: Pixel-level modeling is computationally expensive, motivating visual tokenizers that compress images into compact representations for unified vision-language modeling.Continuous tokenizers can suffer error accumulation and mode collapse during autoregressive generation, motivating discrete-token modeling.
- Approach: 75% fewer visual tokens are achieved while maintaining exceptionally high reconstruction quality through 32× spatial downsampling and a 2^128 codebook.This compression is presented as making discrete visual tokens suitable for Unified MLLMs.
- Motivation: Codebooks exceeding 2^128 increase token information density but introduce new complexities for downstream generation and understanding.Text-to-image approaches still use codebooks constrained to 2^32 and do not extend to multimodal understanding.
- Approach: UniWeTok combines 32× spatial downsampling and a 2^128 codebook with Pre-Post Distillation, Generative-Aware Prior, SigLu, hybrid convolution-attention architecture, and three-stage curriculum learning.The curriculum adjusts resolutions and training distributions for variable resolutions and perceptually sensitive scenarios such as faces and text.
- Results: FID: UniWeTok 1.38 vs. REPA 1.42, while Training Tokens: UniWeTok 33B vs. REPA 262B in ImageNet class-to-image generation.The Unified MLLM also reports DPG Score: UniWeTok 86.63 vs. FLUX.1 [Dev] 83.84 and GEdit Overall Score: UniWeTok 5.09 vs. OmniGen 5.06.
2 Related Work
Prior visual tokenizers improve discrete representation quality and multimodal capabilities through codebook optimization, scaling, and unified understanding-generation designs. However, the supplied related-work passages emphasize unresolved limitations in reconstruction, autoregressive unification, and downstream suitability.
- Visual Tokenizers: VQVAE and VQGAN use vector quantization to produce discrete visual tokens but suffer low reconstruction quality from unstable codebook utilization.Subsequent work improves performance through optimization strategies, additional modules, or larger grouped codebooks.
- Visual Tokenizers: Multiple-codebook methods such as ImageFolder, DualToken, and TokenFlow target improvements in model understanding and reconstruction capabilities.These methods represent one line of work focused on scaling or structuring codebooks.
- Open Direction: Discrete-token extraction must preserve suitability for downstream generation while improving extraction performance through a binary codebook.This requirement connects tokenizer representation design with generation-oriented downstream use.
- Unified MLLMs: Unified MLLM research combines visual understanding and generation through embedding decoding, diffusion objectives, or discrete visual tokenizers.The cited approaches include Emu2, Show-o, Transfusion, Emu3, and Chameleon.
3 Method
UniWeTok combines distillation, generative-prior training, hybrid architecture, and progressive curriculum training to support reconstruction, semantic extraction, generation, and variable-resolution use.
- Training Framework: Pre-Post Distillation aligns pre- and post-quantization representations with a pretrained semantic encoder to enhance semantic extraction.The teacher encodes the input image into semantic latents, while attention-pooled encoder representations are aligned using cosine similarity losses.
- Training Framework: Generative-Aware Prior trains a lightweight BitDance model on quantized latent sequences with a next-token diffusion task.The prior exposes the tokenizer to downstream generation objectives during training.
- Architecture: UniWeTok uses a convolution-attention hybrid backbone whose residual blocks downsample and extract local features before transformer blocks capture global context.The decoder mirrors this structure, and concurrent channel expansion with downsampling improves reconstruction capability.
- Architecture: SigLu constrains encoder outputs to [−1, 1], making token entropy loss equivalent to commitment loss and stabilizing semantic distillation.This permits setting α=0 and replacing the commitment term with token entropy loss.
- Training Pipeline: A three-stage progressive pre-training strategy addresses fixed-resolution constraints and improves adaptation to variable resolutions, faces, and text.Stage 1 uses fixed-resolution 256×256 pre-training; later stages introduce multiple resolutions and text-face annealing, with stage 3 improving face and text processing.
4 Experiments
Experiments evaluate UniWeTok’s ablations, training curriculum, ImageNet generation, and unified MLLM performance across reconstruction, understanding, generation, and editing.
- Ablation Studies: Semantic distillation improves semantic extraction, while the prior loss improves downstream generation without sacrificing reconstruction or understanding.The prior loss is also reported to enhance understanding capability.
- Ablation Studies: SigLu activation enables effective post-distillation by constraining the encoder feature space and stabilizing convergence.Without SigLu, post-distillation almost fails to converge; pre-distillation stabilizes convergence but does not ensure semantics in discrete latents.
- Ablation Studies: 51.32% zero-shot Top-1 accuracy is achieved when pre- and post-distillation are combined.The combined strategy yields the best performance in the ablation.
- Ablation Studies: Doubling bottleneck channels lowers rFID from 1.58 to 1.12 and improves semantic accuracy by over 7%.The ablation identifies wider bottlenecks as beneficial for both visual compression and semantic extraction.
- Ablation Studies: The hybrid backbone achieves rFID 1.35 and Top-1 35.41%, combining convolutional texture processing with Transformer semantic modeling.CNNs achieve rFID 1.75 but lack semantics, while Transformers reach Top-1 26.09% but struggle with detail.
- Comparison with State-of-the-Art: Stage 2 supports variable-resolution reconstruction, while stage 3 improves processing of faces and text.The three-stage curriculum targets resolution variability and perceptually sensitive content.
- Comparison with State-of-the-Art: UniWeTok-H reaches FID 1.38 on ImageNet using 33B training tokens and generating 64 tokens at inference.It compresses 256 × 256 images into 64 tokens, a 75% reduction compared with mainstream tokenizers.
- Comparison with State-of-the-Art: The unified MLLM uses 25% of the visual token count, supports native variable-resolution training, and performs competitively in understanding, generation, and editing.UniWeTok-Gen surpasses various diffusion-based models, while UniWeTok-Edit surpasses a diffusion model at a similar parameter scale.
5 Conclusion
UniWeTok is presented as a unified discrete tokenizer for high-fidelity reconstruction, multimodal understanding, and generation. Its architecture and training objectives combine massive-codebook representation with semantic and generative capabilities.
- Conclusion: UniWeTok targets the conflict between high-fidelity reconstruction, multimodal understanding, and generation.The tokenizer is designed as a single framework for these capabilities.
- Conclusion: A convolution-attention hybrid backbone, SigLu activation, Pre-Post Distillation, and Generative-Aware Prior support a 2^128-codebook tokenizer.The framework is reported to extract semantic concepts, fine-grained texture details, and generative prior effectively.
A Model Architecture
Figure 5 provides a detail illustration of the UniWeTok model architecture.
- Model Architecture: Figure 5 details the UniWeTok model architecture.
B More Ablation Implementation Details
Additional ablation materials list supporting tables for the training loss, SigLu activation, Pre-Post Distillation, bottleneck channel, Generative-Aware Prior, training configuration, and hybrid architecture.
- Training loss: Additional training-loss ablations are reported in Tables 14, 15, and 16.
- SigLu activation: Additional SigLu activation ablations are reported in Tables 17, 18, and 19.
- Pre-Post Distillation: Additional Pre-Post Distillation ablations are reported in Tables 20, 21, and 22.
- Bottleneck channel: Additional bottleneck-channel ablations are reported in Tables 23 and 24.
- Generative-Aware Prior: Additional Generative-Aware Prior ablations are reported in Tables 25, 26, and 27.
- Training configurations: Additional DataComp-1B training-configuration ablations are reported in Tables 28, 29, and 30.
- Hybrid architecture: Additional hybrid-architecture ablations are reported in Tables 31, 32, and 33.
- Implementation details: BitDance-T has 8.6M parameters, making its added training overhead negligible.
C More Visualization Results
This section provides additional visualization results for multimodal understanding and image editing, alongside tables covering ablations and architectural settings.
- Multimodal Understanding: Additional multimodal-understanding visualizations are provided for UniWeTok-Chat.The section directs readers to further visualization results in a figure.
- Image Editing: Additional image-editing visualizations are provided for UniWeTok-Edit across three figures.The results are presented in Figures 7, 8, and 9.
- Training Components: Tables 14–16 compare settings with and without Pre-Post Distillation and the Generative-Aware Prior.The listed configurations progress from removing both components to including both.
- Distillation and Activation: Tables 17–22 report pre-distillation, post-distillation, and Pre-Post distillation settings, including a SigLu activation variant.These tables enumerate alternative distillation configurations.