Source-linked AI summary
TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation
Haokun Lin, Teng Wang, Yixiao Ge, Yuying Ge, Zhichao Lu, Ying Wei, Qingfu Zhang, Zhenan Sun, Ying Shan
TL;DR
Unified multimodal autoregressive models face costly training and limited comprehension because low-level VQ tokens lack high-level semantics. TokLIP semanticizes those tokens with CLIP-level representations while disentangling comprehension and generation objectives, and it reports data-efficient comprehension, strong representation quality, and improved generation. Its main scope limitation is information loss from frozen VQ encoders, especially for text-rich figures.
Problem
Unified multimodal autoregressive models face high training overhead and limited comprehension because VQ tokens lack high-level semantic representations.
Method
TokLIP combines an off-the-shelf low-level VQ tokenizer with a causal ViT-based token encoder that captures CLIP-level continuous semantics for multimodal autoregression.
Results
TokLIP delivers data-efficient image representation and multimodal comprehension while retaining VQ reconstruction and enhancing autoregressive image generation.
Takeaways & Limitations
Disentangling semantic comprehension from low-level generation allows TokLIP to use standard state-of-the-art VQ tokenizers in unified autoregressive models.
Takeaways & Limitations
Frozen VQ encoders can reduce performance on text-rich figures because quantization loses fine-grained information.
Abstract
from arXiv · showhide
Pioneering token-based works such as Chameleon and Emu3 have established a foundation for multimodal unification but face challenges of high training computational overhead and limited comprehension performance due to a lack of high-level semantics. In this paper, we introduce TokLIP, a visual tokenizer that enhances comprehension by semanticizing vector-quantized (VQ) tokens and incorporating CLIP-level semantics while enabling end-to-end multimodal autoregressive training with standard VQ tokens. TokLIP integrates a low-level discrete VQ tokenizer with a ViT-based token encoder to capture high-level continuous semantics. Unlike previous approaches (e.g., VILA-U) that discretize high-level features, TokLIP disentangles training objectives for comprehension and generation, allowing the direct application of advanced VQ tokenizers without the need for tailored quantization operations. Our empirical results demonstrate that TokLIP achieves exceptional data efficiency, empowering visual tokens with high-level semantic understanding while enhancing low-level generative capacity, making it well-suited for autoregressive Transformers in both comprehension and generation tasks. The code and models are available at https://github.com/TencentARC/TokLIP.
1 Introduction
TokLIP addresses weak comprehension and costly training in unified multimodal autoregressive models by semanticizing low-level VQ tokens with CLIP-level knowledge while preserving standard discrete-token training. It separates comprehension and generation objectives and reports data-efficient comprehension, representation, and generation benefits.
- Unified autoregressive models using VQ and word tokens remain computationally expensive to train and show weak comprehension performance.
- Prior approaches discretize high-level semantic features, but entangle reconstruction and text-alignment objectives while introducing information loss through quantization.
- TokLIP semanticizes low-level VQ tokens with a causal ViT-based encoder, incorporating CLIP-level semantics into end-to-end multimodal discrete-token prediction.
- TokLIP disentangles comprehension and generation objectives, preserving plain VQ-token autoregression while enabling direct use of state-of-the-art VQ tokenizers.
- Using less than 20% of VILA-U’s pretraining data, TokLIP significantly outperforms VILA-U and QLIP in zero-shot ImageNet classification while retaining low-level VQ reconstruction.
2 Related Works
The related work reviews vector-quantized tokenizers and unified multimodal models, highlighting their strengths for generation and integration alongside persistent semantic and training challenges. TokLIP addresses these tensions by semanticizing low-level discrete visual tokens rather than discretizing high-level continuous features.
- Vector-quantized tokenizers: Vector quantization compresses high-dimensional data into discrete token sequences, supporting efficient learning across modalities.VQ-VAE introduced assignment to a learnable codebook, while VQGAN improved generation quality with adversarial and perceptual losses.
- TokLIP: TokLIP semanticizes low-level discrete visual tokens using a VQ encoder and a CLIP-initialized causal token encoder trained with contrastive and distillation losses.The overview describes quantizing images with an off-the-shelf VQGAN encoder before semanticizing the resulting codes.
- Vector-quantized tokenizers: Recent VQ tokenizers demonstrate strong potential for autoregressive image generation but struggle to provide necessary semantic information.
- Unified comprehension and generation: Unified multimodal modeling seeks to combine comprehension and generation, but these tasks have conflicting objectives within large language models.The surveyed approaches include external diffusion, while unified tokenizers aim to integrate text and image tokens in one autoregressive model.
- Unified comprehension and generation: State-of-the-art unified-tokenizer methods face reconstruction-versus-alignment conflicts and information loss from quantization, making stable training difficult.
3 Method
TokLIP enriches low-level VQ tokens with high-level continuous semantics while preserving end-to-end autoregressive multimodal modeling. Its design separates generation-oriented discrete representations from comprehension-oriented semantic features and aligns learned features closely with SigLIP.
- TokLIP Tokenizer: TokLIP combines an off-the-shelf VQ tokenizer with a causal ViT-based token encoder to produce semantic visual features.VQ code embeddings are mapped through an MLP before entering the token encoder.
- TokLIP Tokenizer: The tokenizer uses causal attention so its outputs support end-to-end next-token prediction in a multimodal autoregressive Transformer.Causal attention restricts each token to preceding tokens, matching the autoregressive training paradigm.
- Training Objectives: TokLIP trains its MLP and token encoder with contrastive and distillation objectives while freezing the VQGAN encoder to preserve reconstruction capability.The combined objective is L = L_contra + L_distill, and freezing VQGAN decouples reconstruction from TokLIP training.
- Design Rationale: The framework disentangles comprehension and generation by using high-level continuous features for understanding and low-level discrete VQ tokens for generation.This design avoids the objective conflict and quantization-specific training strategies used by approaches that discretize high-level semantic features.
- Feature Analysis: TokLIP features achieve 0.83 cosine similarity with SigLIP, exceeding VILA-U’s 0.75, and its pooled image representations nearly match SigLIP’s distribution.The comparisons use token features for one image and pooled representations from 1,500 MSCOCO test images.
4 Experiments
TokLIP achieves strong semantic understanding and generation across classification, retrieval, multimodal comprehension, and image-generation evaluations, while retaining causal autoregressive behavior. Ablations show that pretrained vision initialization, [CLS]-token distillation, and causal attention are important design choices, though frozen VQ encoders remain limiting for text-rich figures.
- Comprehension capacity: TokLIP achieves 76.4% ImageNet accuracy at 256 resolution and 68.00% TR@1 and 52.87% IR@1 at 384 resolution.The retrieval scores exceed QLIP by 7.14% and 9.87%, respectively.
- Multimodal comprehension: TokLIP-L outperforms Chameleon and Show-o across evaluated benchmarks and surpasses Emu3-chat on most benchmarks despite less data and lower resolution.TokLIP-L also outperforms LLaVA-v1.5 on SEED Bench, MMB, MMMU, and AI2D while performing on par on MME and POPE.
- Data efficiency: TokLIP uses less than 20% of VILA-U’s pretraining data while outperforming established tokenizers in zero-shot ImageNet classification and retaining low-level reconstruction capability.The results support TokLIP’s data efficiency and preservation of generation-oriented VQ behavior.
- Ablation studies: Pretrained vision initialization and [CLS]-token distillation improve semantic performance, whereas distilling all tokens does not provide gains.MLP projection also outperforms direct codebook mapping, whose 16,532-entry codebook complicates learning meaningful representations.
- Visual generation: TokLIP outperforms discrete VQGAN in class-conditional generation, indicating that incorporating high-level semantics can improve visual generation.Qualitative examples show high-quality images that capture visual concepts; the experiments use the LlamaGen pipeline.
- Limitations: Frozen VQ encoders can reduce performance on text-rich figures because quantization loses fine-grained text information.The paper identifies higher-resolution training, text-rich data, tokenizer unlocking, and reinforcement learning as possible future improvements.
- Attention ablations: Bidirectional attention improves classification and retrieval, but causal attention is necessary for generation: FID 15.05 versus 214.36 for bidirectional TokLIP and 17.3 for VQGAN.Bidirectional attention introduces non-causal information flow that breaks next-token autoregression, so the paper selects causal attention for the unified architecture.
5 Conclusion
TokLIP enables end-to-end autoregressive training with multimodal discrete tokens while combining semantic understanding with image generation. Empirical validation reports stronger image representation, lower training cost, and improved image generation capability.
- TokLIP enables end-to-end autoregressive training of a single Transformer on multimodal discrete-token sequences.
- TokLIP disentangles comprehension and generation objectives while enhancing comprehension performance.
- TokLIP efficiently uses state-of-the-art VQ tokenizers within the unified multimodal framework.
- TokLIP achieves exceptional image representation capabilities, reduces training cost, and enhances image generation capability.
A Additional Implementation Details
The implementation uses detailed training hyperparameters and maps VQGAN features into the CLIP feature space before token encoding. A codebook alternative directly maps each code to that space.
- TokLIP training uses a detailed hyperparameter configuration documented in Table A1.
- The VQGAN-to-token-encoder projection maps 8-dimensional features to 4×dimensional CLIP-feature hidden states, applies GeLU, then projects to CLIP feature dimension.
- The codebook approach directly maps each code into the CLIP feature space through a large embedding layer.
B.1 Further Discussion
For bidirectional TokLIP, distilling features from all tokens significantly improves downstream performance. The result is attributed to closer architectural similarity with the teacher model and greater inherited semantic knowledge.
- All-token feature distillation significantly improves downstream performance for bidirectional TokLIP on CC3M.
- The improvement indicates that bidirectional TokLIP more closely resembles the teacher architecture and inherits more semantic knowledge.
B.2 Effects of TokLIP with small LLMs.
With Qwen-2.5-1.5B and the LLaVA-v1.5 dataset, TokLIP achieves competitive results against MAR despite using substantially less training data.
- TokLIP achieves competitive results compared with MAR using 1.2M rather than 29.8M training examples.Both models use the same smaller language model, Qwen-2.5-1.5B.