Source-linked AI summary
UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encoding
Yang Jiao, Haibo Qiu, Zequn Jie, Shaoxiang Chen, Jingjing Chen, Lin Ma, Yu-Gang Jiang
TL;DR
Existing visual encodings struggle to support multimodal understanding and image generation together because discrete tokens lose information and task-specific encoders reduce flexibility. UniToken combines discrete and continuous visual representations, and experiments report state-of-the-art performance across diverse multimodal tasks while identifying task-interference and data-distribution principles.
Problem
Discrete-only encodings lose information and overlook high-level semantics, while decoupled encoders restrict flexibility when supporting understanding and generation together.
Method
UniToken uses a task-agnostic unified visual representation combining discrete tokens for low-level details with continuous tokens for high-level semantics.
Results
UniToken achieves state-of-the-art performance across diverse multimodal tasks and outperforms unified models while competing with specialists in understanding and generation.
Takeaways & Limitations
The experiments provide empirical guidance on task interference and task-specific data distributions for unified multimodal understanding and generation.
Abstract
from arXiv · showhide
We introduce UniToken, an auto-regressive generation model that encodes visual inputs through a combination of discrete and continuous representations, enabling seamless integration of unified visual understanding and image generation tasks. Unlike previous approaches that rely on unilateral visual representations, our unified visual encoding framework captures both high-level semantics and low-level details, delivering multidimensional information that empowers heterogeneous tasks to selectively assimilate domain-specific knowledge based on their inherent characteristics. Through in-depth experiments, we uncover key principles for developing a unified model capable of both visual understanding and image generation. Extensive evaluations across a diverse range of prominent benchmarks demonstrate that UniToken achieves state-of-the-art performance, surpassing existing approaches. These results establish UniToken as a robust foundation for future research in this domain. The code and models are available at https://github.com/SxJyJay/UniToken.
1. Introduction
UniToken addresses limitations of discrete-only and task-decoupled visual encoding by combining discrete and continuous tokens in a unified representation. Experiments identify task-interference and data-distribution considerations, while evaluations show strong performance across understanding and generation.
- Prior paradigms: Existing unified models use either discrete visual tokens or separate continuous and discrete encoders for different tasks.Discrete-only approaches include Chameleon and Emu3, while Unified-IO2 and Janus decouple visual encoding by task.
- Limitations: Discrete-only encoding loses information through quantization, while decoupled encoders reduce flexibility and add mode-switching burdens.The paper links these limitations to weaker multimodal knowledge absorption and increasing complexity as functionality scales.
- UniToken: UniToken combines discrete and continuous visual tokens in a task-agnostic representation that carries both low-level details and high-level semantics.The unified representation is designed to support visual understanding and image generation within one model.
- UniToken: Scaling visual resolution and tuning the ViT significantly boost multimodal comprehension, especially in text-reading scenarios.These techniques amplify the visual representation before multimodal training.
- Empirical insights: Experiments find that combined discrete-continuous encoding avoids task interference, whereas discrete-only encoding is prone to it.The study also reports that the understanding-to-generation data proportion should vary with training-data scale.
- Empirical insights: UniToken competes with state-of-the-art approaches in both multimodal understanding and image generation.The contribution summary presents this as a central outcome of the unified modeling approach.
2. Related Works
Related work develops autoregressive visual generation and shared multimodal models, but visual encoding choices create a gap between semantic understanding and image reconstruction. UniToken adopts unified encoding and prediction to address this gap.
- Multimodal understanding: MLLMs commonly use CLIP or SigLIP vision encoders to adapt semantically enriched visual concepts to LLMs.Adapters and sequential multimodal instruction tuning further connect visual representations to language modeling.
- Visual generation: Autoregressive visual generation quantizes images into discrete codebook IDs and models their distribution with transformers.VQ-VAE and VQ-GAN established this generation paradigm, while later work improves tokenization and scales training.
- Visual generation: Discrete image-generation tokens capture low-level details but often overlook high-level semantics, limiting multimodal comprehension.Their codebooks are designed primarily for reconstruction and generation rather than semantic understanding.
- Unified modeling: Chameleon and Lumina-mGPT jointly train VQ-GAN-based visual tokens with VQA and generation data, but lag behind conventional MLLMs in understanding.This illustrates the difficulty of using a generation-oriented tokenizer for multimodal comprehension.
- Unified modeling: UniToken adopts unified visual encoding and prediction heads regardless of task type.This design directly targets shared processing for multimodal understanding and image generation.
3. UniToken
UniToken unifies multimodal understanding and image generation through dual visual encoding and staged training. Its design combines continuous semantic features with discrete image tokens, while training and inference procedures support both task types.
- Architectural Designs: UniToken encodes visual inputs with a dual visual encoder that produces continuous and discrete tokens for the language model.SigLIP provides continuous features, while Chameleon’s VQ-Tokenizer produces discrete codebook IDs mapped to vocabulary features.
- Architectural Designs: The unified representation combines high-level semantics and low-level details, enabling task-dependent knowledge use across understanding and generation.A separator distinguishes discrete and continuous image features within the multimodal sequence.
- Architectural Designs: UniToken augments visual encoding through higher-resolution grid processing and regulated ViT learning rates.The resolution technique partitions images into multiple grids, while learning-rate control prevents ViT collapse and improves performance across benchmarks.
- Training Recipes: UniToken uses staged training that first aligns continuous features, then jointly trains understanding and generation, and finally improves instruction following.Stage I trains the SigLIP ViT and adapter on 2.5 million image captions; Stage II trains all parameters on 20 million samples; Stage III uses curated multimodal and text-to-image data.
- Training Objectives: The training objective applies cross-entropy selectively: answer text tokens for visual understanding and discrete image tokens for image generation.This task-specific loss allocation follows the autoregressive prediction setup.
- Inference: Inference uses greedy decoding for understanding and multinomial sampling with classifier-free guidance for diverse image generation.The guidance scales are 5.5 for GenEval and 5.0 for T2i-Compbench++.
4. Experiments
UniToken is evaluated across multimodal understanding, image generation, task-interference, data-proportion, and qualitative settings. The experiments compare it with specialized and unified baselines and examine how visual encoding affects joint-task performance.
- Data Proportion: Table 5 evaluates training-data proportions across aggregated understanding, text-reading, and generation metrics under varying data scales.The table reports “General,” “TextRead,” GenEval, and T2I-Compbench++ scores using averages over their specified benchmark groups.
- Multimodal Understanding: UniToken outperforms leading understanding-only and unified models across most multimodal benchmarks, including gains of +5.6 on SEEDBench, +7.0 on MathVista, and +102 on OCRBench over LLaVA-v1.6(HD).It also outperforms leading approaches such as Emu3 (chat version) and Janus despite inheriting a checkpoint with limited visual understanding capabilities.
- Image Generation: UniToken achieves competitive or superior image-generation performance against diffusion and autoregressive hybrid approaches, while remaining comparable to other autoregressive unified models.It performs better on “Two Obj.” and “Counting” but worse on “Positions” and “Color Attri.”, possibly because the text-to-image dataset lacks relevant prompts.
- Task Interference: Joint training severely degrades Chameleon’s image generation, whereas UniToken remains robust for both visual understanding and image generation.The comparison attributes the difference to lower susceptibility to task interference from unified discrete-continuous encoding.
- Qualitative Results: Qualitative results show support for English and Chinese question answering across diverse formats, although some hallucinations remain.For image generation, UniToken produces finer textures and more intricate visual details than Janus-Pro-7B.
5. Conclusion
UniToken integrates discrete and continuous visual tokens to bridge visual understanding and image generation. Experiments report state-of-the-art performance across diverse multimodal tasks and insights into task interference and data distribution.
- Conclusion: UniToken combines discrete and continuous tokens into a unified visual representation for visual understanding and image generation.The paper presents this representation as a foundation for unified multimodal modeling.
- Conclusion: Experiments report state-of-the-art performance across diverse multimodal tasks while providing insights into task interference and data distribution challenges.The conclusion positions UniToken as a foundation for future research in unified multimodal modeling.