PaperlayerView exact revision on arXiv

Source-linked AI summary

VIVAS: Vitalizing Visual Perception in VLM Pre-training via Vision-language Unified Autoregressive Supervision

Zhehan Kan, Yubo Zhu, Xinghua Jiang, Zhixiang Wei, Shifeng Liu, Wei Tong, Sheng Zhong, Qingmin Liao, Wenming Yang, Xin Li, Yinsong Liu, Deqiang Jiang, Xing Sun

arXiv:2609.27948v1cs.CV

TL;DR

Text-dominant optimization in VLM pretraining is associated with insufficient fine-grained visual perception and limited multimodal understanding. VIVAS adds a dense, modality-aligned visual vocabulary to a unified vision-language token space and autoregressively supervises visual and linguistic content; it achieves state-of-the-art performance on most multimodal-understanding tasks across 39 benchmarks with 3.8% additional inference latency.

  • Problem

    Text-dominant optimization in standard VLM training treats visual signals as passive conditioning inputs, limiting fine-grained visual perception and multimodal understanding.

    Source: §1 · p. 2

  • Method

    VIVAS adds a modality-aligned visual vocabulary to a unified token space and autoregressively supervises visual details and linguistic content during pretraining.

    Source: §1 · p. 3

  • Results

    Across 39 benchmarks, VIVAS achieves state-of-the-art performance on most multimodal-understanding tasks with 3.8% additional inference latency.

    Source: §4.2 · p. 11

  • Takeaways & Limitations

    The authors report that strengthening visual perception improves multimodal understanding.

    Source: §1 · p. 3

Abstract

from arXiv · show

While Vision-Language Models (VLMs) demonstrate strong capabilities, they continue to suffer from a critical limitation: insufficient fine-grained visual perception, which fundamentally limits their multimodal understanding. We attribute this bottleneck to text-dominant optimization biases during pre-training, which encourage the model to overlook fine-grained visual details, thereby limiting the capability of multimodal understanding. We investigate that overcoming this bottleneck requires two key elements: (1) a unified token space paradigm that ensures stable training dynamics, and (2) a modality-aligned dense visual supervision signal enriched with both structural granularity and semantic information to capture critical visual representations. Based on these insights, we propose VIVAS, a framework built upon the unified token space paradigm, which introduces a dense-structural-semantic vision tokenizer, which expands the textual vocabulary into a unified vision-language vocabulary by incorporating a visual vocabulary. During pretraining, VIVAS performs vision-language unified autoregressive supervision over both visual details and linguistic content, thereby enhancing visual perception to improve multimodal understanding. Trained end-to-end on 12.4T tokens, VIVAS achieves state-of-the-art performance across 7 tasks and 39 multimodal benchmarks.

1 Introduction

VIVAS targets the fine-grained perception deficit linked to text-dominant training with a unified vision-language token space and a dense, structurally and semantically informed visual vocabulary. Its unified autoregressive supervision improves multimodal understanding across seven tasks and 39 benchmarks with 3.8% additional inference latency.

1 Introduction

  • Existing VLMs’ text-dominant optimization treats visual signals as passive inputs, discarding fine-grained details and limiting multimodal understanding.The paper identifies this as a visual information bottleneck that impedes dense perception.Source: §1 · p. 2
  • A unified token space integrates visual tokens into a shared vision-language predictive stream, enabling stable and rapid convergence unlike architectural separation.Architectural separation with alignment modules for visual reconstruction leads to training instability and convergence difficulties.Source: §1 · pp. 2–3
  • VIVAS’s tokenizer fuses DINOv3’s dense structural features with SigLIP2’s semantic features through cross-attention to build a modality-aligned visual vocabulary.The paper finds text-decoupled vocabularies can degrade multimodal understanding, while sparse semantic features provide limited complementary information for fine-grained perception.Source: §1 · pp. 2–3
  • 7 tasks and 39 benchmarks: VIVAS significantly improves multimodal understanding with 3.8% additional inference latency.The framework applies unified autoregressive supervision to both visual details and linguistic content during pretraining.Source: §1 · p. 3
  • VIVAS expands the textual vocabulary with visual tokens and autoregressively supervises visual details and linguistic content together during pretraining.The method is built on the unified token space paradigm and the proposed visual vocabulary.Source: §1 · pp. 2–3

2 Related Work

Existing VLMs project continuous visual features into LLM space, where text-centric optimization bottlenecks retention of dense visual details. Earlier visual supervision is introduced mainly during post-training, while VIVAS applies dense semantic visual supervision during pretraining through unified autoregressive prediction.

2.1 Multimodal Understanding

  • Pretrained vision encoders provide continuous visual features that are projected into LLM space, but text-centric optimization bottlenecks dense-detail retention.The encoders are optimized via contrastive learning on extensive image-text pairs.Source: §2.1 · p. 4

2.2 Visual Supervision incorporated in VLMs

  • Prior VLM visual supervision is introduced during post-training and yields marginal improvements; VIVAS instead introduces dense semantic visual signals during pretraining.The paper describes pretraining as the critical window for fundamentally optimizing representational capacity.Source: §2.2 · p. 4

3 Methodology

The methodology targets the loss of fine visual detail under text-centered VLM training by combining stable unified-token supervision with a dense, modality-aligned visual vocabulary. It trains visual and linguistic predictions together before post-training alignment.

3.1 Preliminary

  • Text-only generation treats visual input as passive conditioning, risking discarded fine detail; VIVAS end-to-end optimizes the vision encoder, projector, and LLM decoder.Source: §3.1 · pp. 4–5

3.2 What Visual Supervision Are Needed for VLM Pre-training?

3.3 Dense-Structural-Semantic Visual Supervision

  • The tokenizer uses DINOv3 structural features as spatial queries over SigLIP2 semantic keys and values, then combines fused and structural features before codebook quantization.Source: §3.3 · pp. 7–8
  • The tokenizer objective combines perceptual and adversarial fidelity with VQ and entropy terms, omits pixel-wise ℓ1 loss, and reaches 97.7% codebook utilization.Source: §3.3 · pp. 8–9

3.4 Vision-Language Unified Autoregressive Pre-training

  • During multimodal pretraining, VIVAS interleaves visual and text tokens and optimizes next-token losses for both modalities, with λ=0.5 weighting visual prediction.Source: Unified Training Objective · p. 9
  • Training progresses from text-only language-backbone pretraining to end-to-end multimodal training, then SFT and RL post-training with visual supervision deactivated.Source: Progressive Training Curriculum · pp. 9–10

4 Experiment

Across 39 multimodal benchmarks, VIVAS achieves state-of-the-art performance on most tasks, with gains on fine-grained perception and only 3.8% inference-time overhead. Ablations and scaling analyses examine where visual supervision helps and how performance changes with training data.

4.1 Implementation Details

  • VIVAS uses a Youtu-LLM backbone with SigLIP2-so400m-patch16-naflex and a four-stage recipe spanning language pretraining, multimodal pretraining, SFT, and RL.The recipe uses approximately 10T tokens of text and 2.4T tokens for multimodal pretraining; SFT and RL use approximately 4B and 0.2B tokens, respectively.Source: Training Recipe · p. 10, Model Components · p. 11

4.2 Quantitative Results

  • VIVAS achieves state-of-the-art performance on most of 39 multimodal benchmarks, with pronounced gains on fine-grained tasks and 3.8% inference-time overhead.The comparison includes leading VLMs and a baseline without pretraining visual supervision.Source: §4.2 · p. 11, Table 3 · p. 12

4.3 Out-of-Distribution and Fine-Grained Generalization

  • On GEOBench-VLM, VLMsAreBiased, and V*, VIVAS outperforms the baseline without visual supervision, including on satellite imagery underrepresented in standard training corpora.The GEOBench-VLM result suggests the supervision does not merely fit the original data distribution.Source: §4.3 · pp. 11, 13, Table 6 · p. 14

4.4 Ablation Studies

  • Pretraining-only visual supervision yields substantial gains, whereas post-training-only supervision yields marginal gains and applying it in both phases yields modest gains.The authors attribute the combined-phase result primarily to interference with instruction following during post-training.Source: Table 2 · p. 11, §4.4 · p. 13
  • A SigLIP2–DINOv3 encoder-side baseline without visual-token supervision underperforms full VIVAS and fails to outperform the no-supervision baseline, while adding inference cost.VIVAS keeps SigLIP2-based continuous input unchanged and adds visual supervision during pretraining.Source: §4.4 · pp. 13–14, Table 8 · p. 15
  • A 150,000-entry, 768-dimensional codebook reaches 97.7% utilization, while omitting L1 loss improves visual understanding by avoiding texture bias.The authors say L1 can allocate representational capacity to high-frequency noise at the expense of high-level semantic abstraction.Source: Table 4 · p. 13, Table 5 · p. 13, §4.4 · p. 14

4.5 Analysis of Data Scaling

  • Average performance across 27 benchmarks rises from 0.43 to >0.74 over 2.4T pretraining tokens without early saturation; error follows power-law scaling.The estimated scaling exponents are α ≈ 0.102 in Stage 2 and α ≈ 0.079 in Stage 3.Source: §4.5 · p. 14, Figure 5 · p. 15

5 Conlcusion

VIVAS addresses fine-grained visual perception limitations by introducing visual supervision during pretraining. Its unified token-space framework combines DINOv3 structural features with SigLIP2 semantics, and experiments show improved multimodal understanding with minimal inference overhead.

5 Conclusion

  • VIVAS introduces visual supervision during pretraining to address fine-grained visual perception limitations attributed to text-dominant optimization bias.The conclusion states that the unified token-space paradigm enables stable and effective training for vision-language autoregressive models.Source: §5 · p. 15
  • Its dense-structural-semantic tokenizer combines DINOv3 structural features and SigLIP2 semantic representations into a modality-aligned visual vocabulary.VIVAS applies unified supervision over visual and textual tokens.Source: §5 · p. 15
  • Experiments show consistent multimodal-understanding improvements with minimal inference overhead.Source: §5 · p. 15

Related papers

Loading 2609.27948v1…