Source-linked AI summary
ViQ: Text-Aligned Visual Quantized Representations at Any Resolution
Xumin Yu, Zuyan Liu, Zhenyu Yang, Yuhao Dong, Shengsheng Qian, Jiwen Lu, Han Hu, Yongming Rao
TL;DR
Discrete visual representations must reconcile language-compatible semantics with preservation of image detail, but existing approaches often sacrifice one for the other. ViQ uses text-aligned pre-training and progressive, position-aware quantization to support native-resolution inputs, achieving competitive multimodal performance, low-level reconstruction, and 20%-70% training acceleration.
Problem
Discrete visual representations struggle to balance high-level semantics with low-level detail while remaining compatible with token-based language modeling and computationally efficient.
Method
ViQ uses two-stage text-aligned pre-training and progressive feature discretization with proximal representations and position-aware quantization for native-resolution inputs.
Results
Across multimodal tasks, ViQ is competitive with continuous and discrete encoders, accelerates training by 20%-70%, and preserves details with PSNR 22.73 and rFID 0.62.
Takeaways & Limitations
ViQ offers a viable unified discrete framework for vision and language that combines native-resolution representation quality with improved multimodal training efficiency.
Takeaways & Limitations
ViQ’s integration and synergistic effects with foundation models of 70B parameters and beyond remain empirically unexplored.
Abstract
from arXiv · showhide
A unified representation for text and vision is a natural pursuit, as it enables simpler multimodal modeling and more efficient training. However, representing images as discrete signals in the same way as text inevitably introduces severe information loss. Existing work struggles to balance low-level details and high-level semantics in discrete representations: reconstruction-oriented representations often lack semantic information, whereas semantically stronger features typically suffer from severe loss of detail. We present ViQ, a Visual Quantized Representations framework, which is designed to balance semantics and details in discrete representations while supporting inputs at native resolutions, thereby enabling it to serve as a unified and general discrete representation for arbitrary visual inputs. Our approach structures quantization learning into two stages: text-aligned pre-training and feature discretization. With text-aligned pre-training, we enhance the visual encoder semantic-rich supervision from the pretrained language model and enable it to process native-resolution visual inputs. During discretization, we propose a proximal representation learning strategy to progressively compact the feature space, along with a position-aware head-wise quantization mechanism that enables flexible processing of arbitrary resolutions. Extensive experiments on multimodal tasks demonstrate that ViQ achieves competitive performance compared to state-of-the-art multimodal vision encoders with continuous and high-dimensional visual features, while maintaining high precision in low-level reconstruction. We also show that multimodal training with visual quantized representations largely improves efficiency, yielding up to 20\%-70\% acceleration with different base LLMs and training recipes.
1 Introduction
ViQ addresses the mismatch and computational burden of continuous visual features by learning native-resolution discrete representations that balance semantic understanding with low-level detail. It combines text-aligned pre-training with visual quantization and achieves competitive multimodal performance alongside substantial training speedups.
- Motivation: Continuous visual features are mismatched with text’s discrete token modeling and impose substantial hardware demands during multimodal training.These limitations motivate unified discrete visual representations.
- Motivation: Existing discrete visual representations quantize images into finite token sets but struggle to preserve both high-level semantics and low-level visual details.The challenge is balancing semantic information with faithful visual representation.
- Method: ViQ uses two phases—text-aligned pre-training and visual quantization learning—to produce semantically rich, native-resolution discrete visual representations.The framework is designed to raise discrete representations to performance comparable with widely used continuous features while supporting native-resolution inputs.
- Results: 57.2 and 63.9 are ViQ’s average scores with Qwen2.5-1.5B and Qwen2.5-7B, surpassing previous state-of-the-art scores of 57.0 and 63.8, respectively.The aggregated score covers nine benchmarks in visual question answering, world knowledge, and document and chart recognition, under 6B visual encoder parameters.
- Results: 20% to 70% speedups are achieved with the ViQ visual encoder across multimodal training recipes with varying sequence lengths.The comparison is against conventional training strategies.
2 Related Works
Related work spans conventional visual encoders for multimodal language models and quantized visual encoders for compact image representations. CLIP-style encoders provide a common foundation but often require fixed input resolutions, while VQ-based tokenizers discretize continuous features using learned codebooks.
- Visual Representations for MLLMs: CLIP-style models commonly serve as visual encoders in multimodal language models, but their fixed input resolutions constrain flexibility.Examples include CLIP, SigLIP, SigLIP2, AIM, and DFN.
- Quantized Visual Encoders: VQ-based visual tokenizers use discrete latent spaces to connect raw pixels with compact representations for visual generation.VQ-VAE introduced a learnable codebook that discretizes continuous features through nearest-neighbor assignment.
3 ViQ: Visual Quantized Representations
ViQ learns visual quantized representations in two stages: text-aligned pre-training creates a multimodal visual encoder, and progressive quantization compresses continuous features into discrete codes. The method supports native-resolution inputs while preserving visual detail through proximal representations, multi-head quantization, spatial position encoding, and low-level supervision.
- Text-Aligned Pre-training: ViQ first performs text-aligned pre-training on continuous features to align vision and language embeddings for multimodal learning.This stage uses language supervision and self-distillation to retain pretrained knowledge while adapting the encoder for multimodal data.
- Any Resolution Adaptation: The encoder transitions from fixed-resolution to native-resolution inputs by progressively increasing image pixels during training and packing multimodal data for efficiency.Training begins with low-resolution images at native aspect ratios and gradually reaches native resolution.
- Feature Discretization: ViQ progressively compresses high-dimensional visual features through a proximal representation before applying finite scalar quantization in a compact latent space.Bottleneck and inverted-bottleneck layers reduce feature complexity, with the final dimension satisfying d << D << C.
- Multi-Head Quantization: Multi-head quantization expands each visual patch into 2 × 2 visual codes, then recombines the four sub-tokens to preserve the original one-token-per-patch downsampling rate.Expanded patches are processed independently, keeping quantized representations independent for representation learning and downstream tasks.
- Position Encoding: A 2D rotary position embedding encodes height and width before quantization, preserving relative spatial relationships across varying resolutions.The embedding explicitly incorporates spatial resolution information into token features.
- Low-Level Supervision: Multi-stage training retains text and self-distillation objectives while adding reconstruction supervision on compressed features against pretrained VAE latents.Under fixed unit-variance Gaussian likelihood, the reconstruction loss reduces to mean-squared error rather than pixel-level reconstruction.
4 Experiments
Experiments evaluate ViQ’s implementation, multimodal understanding, efficiency, reconstruction quality, and design choices. ViQ combines competitive multimodal performance with strong reconstruction and substantial training speedups across model sizes.
- Implementation: ViQ reduces SigLIP2-g features from C = 1536 to D = 128 and then d = 6, using FSQ with a codebook size of 64,000.Training uses Qwen2.5-VL-0.5B for text supervision and a fixed pretrained Qwen-Image encoder for low-level visual supervision.
- Multimodal Understanding: ViQ achieves competitive multimodal performance and matches or surpasses representative general-purpose encoders on the aggregated score despite quantization.The results attribute this to native-resolution perception and quantization-aware training that preserve visual perception during quantization.
- Multimodal Understanding: ViQ’s gains are most pronounced on OCR, document, and infographic understanding tasks, which depend more directly on precise low-level visual details.The paper contrasts these detail-intensive tasks with general benchmarks that rely more heavily on the backbone language model’s knowledge and reasoning.
- Efficiency: 70% and 78%: ViQ accelerates forward time for the 0.5B Qwen2.5 model under 4k and 16k training settings, respectively.The comparison includes SigLIP2-g, ViQ, and ViQ offline code extraction time across Qwen2.5 models from 0.5B to 7B.
- Reconstruction Quality: ViQ attains an SSIM of 0.66 and an rFID of 0.62, second only to UniTok among discrete tokenizers, while its PSNR is comparable to QLIP-B.ViQ’s PSNR is 22.73 versus 23.16 for QLIP-B on the 256×256 ImageNet-1K validation set.
- Ablation Studies: Intermediate proximal representations substantially improve final performance, while FSQ outperforms SimVQ and combining text, self-distillation, and reconstruction losses improves results on detail-intensive tasks.The reconstruction-loss gains are strongest for OCR and Chart, whereas captioning and VQA improve relatively little.
5 Conclusion
ViQ is a quantized multimodal encoder that unifies visual and language representations at native resolution. Its two-stage training pipeline and progressive techniques reduce information loss while achieving competitive multimodal-task performance.
- ViQ unifies visual and language representations at native resolution through a quantized multimodal encoder.
- Its two-stage training pipeline combines text-aligned pretraining with feature discretization to reduce information loss in discrete visual representations.
- Carefully designed architectures and progressive training techniques minimize information loss while supporting competitive performance in multimodal tasks.
Appendix · A More Details · A.1 Training Details
The training details describe a two-stage process that moves from fixed-size visual processing toward native-resolution inputs, then discretizes features through a bottleneck and reconstruction branch. Training uses large-scale vision-language data, staged learning rates, and a VAE-oriented prediction head.
- A.1 Training Details: 3B vision–language tokens were used to train a ViT with a lightweight LLM on captioning, chart understanding, OCR, and diagram comprehension.This first stage constrained image size within 3842 while preserving native aspect ratios.
- A.1 Training Details: A factor of 16 downsampled vision tokens before they were fed into the LLM for training efficiency.
- A.1 Training Details: 1B vision-language tokens were used in Stage 1 at a resolution of 768 pixels, with the bottleneck optimized during this phase.
- A.1 Training Details: The reconstruction branch passed final vision features through a head designed for VAE latent feature prediction.
- A.1 Training Details: A 3-layer MHSA followed by a convolutional upsampling layer formed the reconstruction head.
B Size-Matched Comparison
A size-matched ViQ-0.4B variant achieves 56.8 average accuracy, outperforming sub-1B continuous baselines and indicating gains beyond model scale.
- B Size-Matched Comparison: 56.8 overall average accuracy is achieved by ViQ-0.4B, outperforming continuous baselines with fewer than 1B parameters.ViQ-0.4B is based on SigLIP2-SO400M and is evaluated under the same protocol as Table 1.
C Additional Analysis on Quantization Methods
Additional quantization comparisons show that LFQ and vanilla VQ remain below FSQ, while parameter-free quantizers optimize more stably than codebook-learning methods. Matching the regularization norm to the quantization geometry is important, as an L∞ norm reduces SimVQ’s score relative to L2.
- Quantization-method comparison: LFQ scores 66.8 and vanilla VQ 65.7, compared with 68.7 for FSQ and 66.6 for SimVQ.LFQ exceeds SimVQ but remains below FSQ, while vanilla VQ is slightly below SimVQ.
- Quantization-method comparison: Parameter-free quantizers optimize more stably within ViQ’s framework than codebook-learning methods.The comparison includes LFQ and vanilla VQ under the same proximal-representation setting.
- Regularization-norm analysis: Using L∞ instead of L2 for SimVQ lowers its score from 66.6 to 66.1, indicating that the regularization norm should match the quantization geometry.The evaluated pairings use L1 for LFQ and L2 for vanilla VQ.
D Encoder Memory and Throughput
ViQ’s encoder closely matches SigLIP2-g in peak GPU memory and scales near-linearly with image token length. Its throughput is about 15% lower on average, primarily due to any-resolution processing, 2D RoPE computation, and the quantization bottleneck.
- Encoder Memory: ViQ closely matches SigLIP2-g in peak GPU memory across image token lengths.The encoder’s memory footprint scales near-linearly with image token length, indicating negligible overhead from the proximal bottleneck and quantization.
- Encoder Throughput: ViQ is on average about 15% slower than SigLIP2-g in throughput.The modest throughput cost mainly comes from any-resolution processing, 2D RoPE computation, and the quantization bottleneck.
E More Visualizations
The section visualizes ViQ’s reconstruction effects on detailed images spanning different resolutions and themes.
- E More Visualizations: ViQ reconstruction is showcased on detailed images with varied resolutions and themes.The selected images are intended to demonstrate reconstruction effects across diverse visual inputs.