UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm

1. Introduction

• Existing MLLMs rely largely on text-centric supervision, while post-training visual signals arrive after representations are shaped and usually serve as auxiliary constraints.The paper identifies pretraining-stage vision supervision as an unexplored opportunity to shape visual features from the outset.Source: §1 · pp. 1–2
• Direct pixel-level VQ supervision can lose input information, create image-text gradient conflicts across feature spaces, and depend on external visual decoders.The paper links low-dimensional reconstruction targets and high-dimensional language targets to conflicting optimization paths.Source: §1 · p. 2
• UVU bypasses VQ with lossless continuous visual encoding and an iterative hierarchical clustering algorithm that builds a pixel-level codebook for joint image-token and text-token generation.The codebook extends the vocabulary for unified autoregressive supervision.Source: §1 · p. 2
• UVU reports significantly improved multimodal understanding across multiple tasks under unified visual supervision during pretraining.The paper presents end-to-end vision-language supervision as a primary driver for improving multimodal comprehension.Source: §1 · p. 2

2.1. Multimodal Understanding

• MLLMs use pretrained visual encoders to produce continuous image features, project them toward language embeddings, and train the LLM for text generation under predominantly text-dominant supervision.The paper characterizes this as yielding sparse learning of visual knowledge.Source: §2.1 · pp. 2–3

2.2. Visual Supervision in MLLMs

• Prior visual-supervision approaches use vision-text alignment or image-token prediction and reconstruction, including DiT-guided feature prediction, intermediate-layer alignment, and masked image modeling.The cited examples include,,, and.Source: §2.2 · p. 3
• Most existing approaches introduce visual supervision after pretraining, when representations are already shaped, so it acts mainly as an external constraint rather than a driver of fine-grained feature formation.The potential of vision-grounded supervision during initial pretraining remains largely unexplored.Source: §2.2 · p. 3
• Figure 2 compares image-text loss-gradient angles over training: AR+VQ stays above 90°, while UVU remains below 90°.The caption interprets the AR+VQ angles as orthogonal or opposing and UVU's as enabling synergistic optimization.Source: Figure 2 · p. 3

3. Why Does Visual Supervision Impair Multimodal Understanding?

• The analysis uses a cascaded VQVAE autoregressive framework and focuses on dimensional misalignment, treating input discretization and external-decoder reliance as structural design problems.Gradient orthogonality is quantified as the angle between image-loss and text-loss gradients; ∇i and ∇t denote those gradients, and θ their angle.Source: §3 · p. 3
• Image- and text-loss gradient angles consistently reach or exceed 90°, indicating orthogonal or opposing optimization directions associated with degraded multimodal understanding.Image loss prioritizes low-level reconstruction, while text loss promotes linguistic coherence without bridging cross-modal representation misalignment.Source: §3 · p. 3

4.1. Pixel-Level Visual Codebook

• The codebook construction uses initial K-means, duplicate removal, uniform sampling, incremental supersampling, and iterative refinement with caching.The procedure targets high coverage and precision.Source: Figure 3 · p. 4
• The clustering visualization labels each patch with its image token ID and distance to the cluster center.In each patch’s right-side values, the token ID appears above the distance.Source: Figure 4 · p. 4
• The codebook uses 3072-dimensional vectors formed by flattening 32×32×3 pixel patches.It contains 200K codewords and targets precise perception for images at resolutions of 512×512 or higher.Source: §4.1 · p. 4
• Pixel Space Coverage measures the minimum of the clusters’ maximum sample-to-center distances, divided by the number of active clusters.An active cluster has at least one assigned sample.Source: §4.1 · p. 4
• Pixel Representation Precision measures reconstruction accuracy on the sample set using the complement of normalized mean squared error.Each patch is assigned to its nearest codeword.Source: §4.1 · p. 5
• The joint objective maximizes the codebook quality metrics to promote broad coverage of natural images and precise intra-cluster representations.The stated max is the global maximum variance across the test set for normalization.Source: §4.1 · p. 5

C (PSC + PRP) . (4)

• The codebook pipeline clusters 2 billion 3072-dimensional patches from 1 million images, using positional encoding, duplicate filtering, and iterative hierarchical sampling and reclustering.Each iteration samples 20 million patches, filters long-tail clusters, and updates centers from cached samples until convergence.Source: C (PSC + PRP) . (4) · p. 5

4.2. Vision-Language Unified Autoregressive

• UVU feeds projected continuous SigLIP-2 features and text embeddings into one LM decoder for autoregressive prediction of image and text tokens indexed by the pixel-level codebook.The framework integrates the modalities without discretizing visual input features or using an external visual decoder.Source: §4.2 · pp. 5–6, Figure 5 · p. 6
• Image-token sampling is inverse-frequency weighted and dynamically balanced against text-token counts during pretraining; image-token loss is discontinued during SFT.The inverse-frequency probabilities are computed from 5 billion image-token samples.Source: §4.2 · p. 6

5.1. Experimental Setup

• Experiments use Qwen2.5-3B-Instruct, SigLIP2, and 1.04 trillion tokens, and evaluate 12 vision-centric multimodal benchmarks.The 200K-entry codebook is derived from 2 billion 32×32×3 patches; training uses a 32K sequence length.Source: §5.1 · pp. 6–7, §5.4 · p. 8

5.2. Ablation Study

• UVU improves visual understanding, whereas AR+VQ substantially degrades it, under identical training data and experimental settings.The comparison includes models without visual supervision, with AR+VQ supervision, and with UVU.Source: §5.2 · p. 7, Table 1 · p. 7
• 200K balances reconstruction quality and inference efficiency: PRP and understanding improve with codebook size, PSC stays stable, and 500K offers no substantial gains.The authors select a 200K-entry codebook.Source: Table 2 · p. 7, §5.2 · p. 7
• Positional encoding in the feature dimension and iterative hierarchical clustering each significantly contribute to codebook quality and downstream multimodal understanding.Source: Table 3 · p. 7, §5.2 · p. 7

5.3. Qualitative Results

• UVU reconstructs visual details during training without external visual decoders, demonstrating learned fine-grained perceptual and spatial knowledge.Source: §5.3 · p. 7, Figure 6 · p. 8

5.4. Quantitative Results

• UVU substantially improves over leading text-only models, including Qwen2.5-VL, with particularly strong results on RefCOCO, LISA, CVBench, and BLINK.The authors anticipate further gains from adding image-generation data in future iterations.Source: Table 4 · p. 8, §5.4 · p. 8

5.3. Qualitative Results

• UVU’s attention concentrates more precisely on semantically relevant regions than the no-supervision and AR+VQ baselines.Figure 7 compares heatmaps for counting giraffes or dogs.Source: §5.3 · p. 7, Figure 7 · p. 8

• Prior unified autoregressive approaches to visual supervision can compromise multimodal understanding through visual-feature discretization and image/text gradient orthogonality.Source: §6 · p. 8
• UVU uses continuous visual encoding and iterative hierarchical clustering to build a pixel-level codebook, enabling autoregressive generation of image and text tokens.Source: §6 · p. 8
• Evaluations across diverse tasks show UVU surpassing state-of-the-art multimodal large language models in multimodal understanding.Source: §6 · p. 8