Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?

1 Introduction

• Discrete visual tokens provide a shared interface that lets multimodal models inherit mature LLM architectures, training paradigms, and scaling recipes.The paper also describes their use as visual chains of thought or intermediate semantic representations.Source: §1 · p. 1
• The paper challenges the assumption that quantization error entails semantic loss, using multimodal understanding as a proxy for semantic fidelity.A discrete representation with an order-of-magnitude larger quantization error can still achieve substantially better understanding.Source: §1 · pp. 1–2
• Effective bit budget is proposed as a key determinant of semantic capacity, with larger budgets improving understanding when added codes remain informative.The authors note that digital visual signals are themselves finite-bit encodings.Source: §1 · pp. 2–3
• FVQ quantizes complementary, fixed-dimensional feature views with symmetric parallel codebooks, continues improving at larger bit budgets, and ultimately matches continuous representations in understanding.This approach targets the saturation of existing multi-codebook quantizers.Source: §1 · p. 2
• 22.8% over UniAR and 59.2% over Emu3-Chat on a nine-metric average; the industrial-scale model leads discrete-representation models and competes with Qwen3-VL.The authors describe it as the first discrete-representation model competitive with flagship continuous-representation models.Source: §1 · p. 3

2 What Determines the Semantics of Discrete Visual Representations?

• Section 2 frames the inquiry around two analyses: reconstruction error versus semantic fidelity, and bit-budget scaling across quantizers.Figure 2 summarizes these analyses.Source: §2 · p. 3, Figure 2 · p. 3

2.1 Rethinking Quantization Error and Semantic Fidelity

• Representations with similar reconstruction errors can have substantially different understanding performance, while markedly different errors can retain comparable semantic information.An odd-group RVQ variant omits the coarse component from group 0 and has much greater error than VQ, yet markedly outperforms it and performs comparably to the even-group variant.Source: §2.1 · pp. 3–4

2.2 Bit-Budget Scaling in Discrete Visual Representations

• Increasing discrete bit budgets improves semantic fidelity and downstream understanding, ultimately allowing discrete representations to match continuous ones.The analysis evaluates downstream multimodal understanding as the bit budget is scaled.Source: §2.2 · p. 4
• A larger nominal budget is insufficient if the additional codes do not remain semantically informative.The paper motivates scaling strategies that preserve useful information for each added codebook.Source: §2.2 · pp. 4–5
• RVQ and PQ saturate at high budgets: doubling codebooks from 8 to 16 yields only marginal understanding gains as later codes contribute less semantic information.RVQ receives weaker residuals, while PQ divides features into increasingly narrow channel groups.Source: §2.2 · pp. 4–5

3.1 Overview

• FVQ uses dense tight-frame projections, parallel independent codebooks, and information-preserving synthesis to scale nominal bit budget without structurally reducing the information available to added codebooks.Each codebook receives a fixed-dimensional view derived from the full feature, rather than a progressively narrower or weaker input.Source: §3.1 · p. 5, §3.2 · p. 6

3.2 Factorized Vector Quantization

• The tight-frame condition makes the aggregate transform isotropic, preserving feature geometry before quantization and ensuring no nonzero direction is discarded by all views.The analysis–synthesis transform exactly recovers the feature before quantization; distortion arises from replacing projected views with nearest codewords.Source: §3.2 · p. 6
• Each patch is represented by the tuple of independently selected codebook IDs, with nominal bit budget B = L log2 C as the number of codebooks L grows.All codebooks have symmetric roles, and each receives a fixed-dimensional dense view of the full feature.Source: §3.2 · pp. 5–6

4.1 Implementation Details

• The main model pairs Qwen3-8B with a 16-codebook FVQ tokenizer applied to Qwen3 ViT and is trained on industrial-scale multimodal data.The main training configuration uses 100B, 10B, 60B, and 30B tokens for quantizer training, alignment, pre-training, and supervised fine-tuning, respectively.Source: §4.1 · p. 7

4.2 Main Results

• FVQ continues to improve as the bit budget grows and ultimately matches continuous QwenViT average understanding performance, while RVQ and PQ gains diminish at high budgets.The comparison uses identical VLM training and evaluation settings; Table 2 reports the benchmark results underlying the scaling trend.Source: §4.2 · p. 7, Table 2 · p. 8
• The model achieves state-of-the-art performance among discrete-representation models, matches Qwen3-VL-8B on MathVista, and achieves the best RealWorldQA result.On OCR and document tasks, it performs comparably to Qwen3-VL-8B on OmniDocBench and DocVQA and surpasses continuous-representation models on both CharXiv tasks.Source: §4.2 · p. 7, Table 3 · p. 8, Table 4 · p. 9

4.3 Visualization of Fine-Grained Visual Understanding

• The model succeeds on three fine-grained cases: a 21,253 × 3,912-pixel long table, irregular handwritten selections, and transcription of a dense page exceeding 5,000 output characters.The cases probe understanding beyond standard OCR.Source: §4.3 · p. 8, Figure 4 · p. 9

4.4 Ablation Studies

• Eight-codebook FVQ yields substantial average gains over single-codebook VQ on both MoonViT-v2 and SigLIP2 features, supporting bit budget as a determinant of semantic capacity across encoders.Table 5 compares VQ and FVQ at per-patch bit budgets of 14 and 112, respectively.Source: §4.4 · p. 9, Table 5 · p. 9
• Under LLaVA-OneVision SFT, FVQ achieves the strongest overall performance among the compared methods when industrial-scale pre-training and SFT data are omitted.This comparison retains the alignment stage and then fine-tunes on LLaVA-OneVision SFT data.Source: §4.4 · p. 10, Table 6 · p. 10

5 Conclusion

• FVQ sustains gains at larger bit budgets, and the industrial-scale model achieves state-of-the-art performance among discrete models while competing with continuous models, including on fine-grained OCR and document understanding.The conclusion identifies effective bit budget as a key determinant of semantic capacity and reports that existing multi-codebook quantizers saturate as added codebooks contribute diminishing semantic information.Source: §5 · p. 10

A Mathematical Formulation of Visual Quantization

• The mathematical formulation defines feature-space vector quantization and derives the tight-frame analysis and synthesis operators used by FVQ.Source: A Mathematical Formulation of Visual… · p. 15

A.1 Quantizer Training

• The tokenizer quantizes pretrained VFM features and is trained to reconstruct frozen features without pixel-level supervision; discrete IDs feed the VLM while quantized vectors support tokenizer reconstruction.Source: §A.1 · p. 15
• Tokenizer training combines feature-reconstruction, codebook, and commitment losses, with reconstruction targeting frozen VFM features rather than pixels.Source: §A.1 · p. 15

A.2 Factorized Vector Quantization: Design and Derivation

• FVQ combines dense fixed-dimensional projections, tight-frame synthesis, and symmetric parallel codebooks, allowing nominal bit budget to grow with L without structurally reducing information available to new codebooks.Source: §A.2 · pp. 15, 17
• The tight-frame constraint makes the stacked transform full-rank and uniformly energy-preserving, so factorization itself introduces no reconstruction error before nearest-neighbor quantization.It supplies a collective constraint on the stacked projections; dimensionality Ld ≥ D alone does not guarantee uniform or stable coverage.Source: §A.2 · pp. 16–17
• FVQ uses QR decomposition to construct fixed projections, retaining the orthonormal-column basis Q after orthonormalization.Source: §A.2 · p. 16

B.1 Native and Unified Multimodal Modeling

• Native multimodal work changes when multimodal learning begins or where visual processing occurs, while discrete-token approaches unify vision and language under shared objectives or interfaces.The cited work also explores visual representations as intermediate computational states; this paper studies the semantic fidelity and scalability of discrete visual tokens.Source: §B.1 · pp. 17–18

B.2 Semantic Visual Representations and Tokenization

• Pretrained vision-language encoders supply semantic feature spaces for tokenization, but discrete representations have continued to lag continuous ones in multimodal understanding.The paper isolates discretization in a frozen VFM feature space and evaluates retained semantic information through downstream understanding.Source: §B.2 · p. 18

• Structured scalar or binary codes and methods improving explicit codebook learning or utilization offer other approaches to vector quantization.Source: §B.3 · p. 18
• Residual vector quantization encodes successive residuals, while product quantization partitions features into independent channel groups; gains can saturate as codebook count grows.Source: §B.3 · p. 18
• FVQ addresses multi-codebook scaling through fixed-dimensional full-feature views and symmetric parallel codebooks.Source: §B.3 · p. 19

• Training proceeds through tokenizer training, alignment, pre-training, and supervised fine-tuning, using 100B, 10B, 60B, and 30B tokens, respectively.Source: §C.1 · p. 19
• Alignment freezes the tokenizer and LLM while training visual token embeddings and the merger; pre-training and fine-tuning then keep the tokenizer frozen.Source: §C.1 · p. 19
• All four stages use AdamW with β1 = 0.9 and β2 = 0.96, and Table 7 summarizes remaining hyperparameters.Source: §C.1 · p. 19, Table 7 · p. 19

C.2 Additional Quantitative Results

• The 16× configuration performs slightly better overall than the 32× configuration, suggesting Pre-Buffer is not essential.The 16× setup retains the QwenViT merger; the 32× setup uses Pre-Buffer.Source: §C.2 · p. 19, Table 8 · p. 19
• Table 8 compares quantization position and post-embedding adaptation at spatial downsampling factors of 16× and 32×.Source: Table 8 · p. 19

C.3 Additional Qualitative Visualizations

• The selected examples span dense OCR, long-document retrieval, fine-grained chart comparison, and visual reasoning, and are not statistical evidence.The figures include extreme-resolution document retrieval, dense plot comparison, geometry reasoning, and spatial reasoning.Source: §C.3 · p. 19, Figure 7 · p. 22, Figure 8 · p. 22
• The Chinese and English full-page OCR examples receive OCRBenchV2 similarity scores of 0.9690 and 0.9828, respectively.Source: Figure 5 · p. 20, Figure 6 · p. 21
• The worked setup models the three cuboids by combining identical bricks along length, width, or height, using dimensions l, w, and h.The supplied algebra derives w = 3l and h(w − l) = 12.Source: §C.3 · p. 23
• The brick problem asks for one brick’s surface area from three two-brick cuboids with surface areas 72, 96, and 102 cm2.Source: §C.3 · p. 23