PaperlayerView exact revision on arXiv

Source-linked AI summary

Let ViT Speak: Generative Language-Image Pre-training

Yan Fang, Mengcheng Lan, Zilong Huang, Weixian Lei, Yunqing Zhao, Yujie Zhong, Yingchen Yu, Qi She, Yao Zhao, Yunchao Wei

arXiv:2605.00809v2cs.CV

TL;DR

MLLM vision encoders need pretraining objectives that better match the LLMs’ generative operation. GenLIP directly trains visual tokens to predict language tokens with one transformer and one language-modeling objective, achieving competitive or superior benchmark performance with substantially less training data.

  • Problem

    Contrastive vision-language pretraining emphasizes discriminative alignment, creating an objective mismatch with the next-token prediction used by MLLMs.

  • Method

    GenLIP jointly models visual and textual tokens with a single transformer, directly training a ViT for autoregressive language prediction without contrastive losses or additional text modules.

  • Results

    GenLIP achieves competitive or superior performance across multimodal benchmarks, matching or outperforming strong baselines with substantially less training data.

  • Takeaways & Limitations

    GenLIP provides a direct, efficient, and scalable generative pretraining formulation for vision encoders used in MLLMs.

  • Takeaways & Limitations

    Validation is limited to the academic-scale LLaVA-NeXT setting, so generalizability to cutting-edge MLLMs remains unverified.

Abstract

from arXiv · show

In this paper, we present \textbf{Gen}erative \textbf{L}anguage-\textbf{I}mage \textbf{P}re-training (GenLIP), a minimalist generative pretraining framework for Vision Transformers (ViTs) designed for multimodal large language models (MLLMs). To better align vision encoders with the autoregressive nature of LLMs, GenLIP trains a ViT to predict language tokens directly from visual tokens using a standard language modeling objective, without contrastive batch construction or an additional text decoder. This design offers three key advantages: (1) \textbf{Simplicity}: a single transformer jointly models visual and textual tokens; (2) \textbf{Scalability}: it scales effectively with both data and model size; and (3) \textbf{Performance}: it achieves competitive or superior results across diverse multimodal benchmarks. Trained on 8B samples from Recap-DataComp-1B, GenLIP matches or surpasses strong baselines despite using substantially less pretraining data. After continued pretraining on multi-resolution images at native aspect ratios, GenLIP further improves on detail-sensitive tasks such as OCR and chart understanding, making it a strong foundation for vision encoders in MLLMs.

1 Introduction

MLLM vision encoders are often pretrained with objectives that mismatch the LLM’s generative next-token objective. GenLIP addresses this with a simplified framework that directly trains visual tokens for language prediction and achieves strong performance with efficient scaling.

  • Motivation: Contrastive pretraining favors discriminative alignment, whereas MLLMs are optimized for next-token prediction.Generatively trained vision features yield lower perplexity when attached to an LLM, indicating better compatibility with its objective.
  • GenLIP: GenLIP trains a Vision Transformer to predict language tokens directly using a standard autoregressive language-modeling objective.The framework removes contrastive batch construction and additional text modules.
  • Advantages: GenLIP uses a single vision backbone and standard autoregressive objective without contrastive losses or additional text modules.Its unified design jointly models visual and textual inputs and supports early cross-modal alignment.
  • Advantages: GenLIP scales effectively with data and model size, yielding consistent gains in the reported experiments.The paper identifies scalability as a core advantage of its simplified generative formulation.
  • Results: GenLIP matches or outperforms strong baselines pretrained on much larger corpora while using only 8B pretraining samples.Native-aspect-ratio adaptation further improves downstream performance, particularly on OCR-sensitive tasks.
  • Conclusion: GenLIP provides a direct and efficient formulation of generative vision-language pretraining.The authors present it as a strong foundation for future multimodal language models.

2 Related Work

Vision-language pretraining spans contrastive dual encoders, encoder-decoder generative systems, and simplified single-transformer designs. GenLIP differs by using one transformer and one language-modeling objective to pretrain a scalable vision encoder for modular MLLMs.

  • Dual-Encoder Contrastive Pretraining: Dual-encoder contrastive methods align image and text representations in a shared embedding space using InfoNCE or similar objectives.They are effective for discriminative tasks but primarily emphasize global alignment rather than deep cross-modal interaction.
  • Encoder-Decoder Generative Pretraining: Encoder-decoder generative methods cascade a vision encoder with a text decoder and train using autoregressive generation objectives.Examples include systems that generate image patches and text tokens or use captioning loss.
  • Discussion: Existing methods often use multiple towers or optimization objectives, increasing complexity and limiting efficiency.Alignment may also occur at later stages rather than within the image encoder itself, constraining early cross-modal interactions.
  • Single-Transformer Pretraining: Single-transformer approaches simplify architecture but may still rely on multiple objectives.SuperClass uses token-level classification targets, while VL-BEiT and OneR unify representation learning with multiple objectives.
  • Discussion: GenLIP uses one transformer and one language-modeling objective to pretrain a scalable vision encoder from scratch for modular MLLMs.Unlike SAIL, which builds a native MLLM around pretrained LLMs, GenLIP targets a reusable vision encoder.

3 Approach

GenLIP uses a unified Transformer to directly predict text from image tokens, with prefix-LM attention and gated attention supporting cross-modal modeling. Its two-stage pretraining progresses from fixed low-resolution images to diverse native-resolution data.

  • Model Architecture: GenLIP processes concatenated image and text tokens with a unified Transformer, letting the ViT model visual-language sequences directly.The architecture uses modality-specific embeddings, a unified Transformer, Layer Normalization, and a language-modeling head.
  • Model Architecture: Image tokens attend bidirectionally while text tokens attend causally, with MRoPE encoding positions in the concatenated multimodal sequence.GenLIP replaces absolute image-patch positions and full attention with MRoPE and prefix-LM attention.
  • Training Objective: The autoregressive objective predicts each text token from preceding visual and textual tokens, directly modeling the conditional distribution P(T|I).The negative log-likelihood is applied exclusively to the textual portion of each image-text sequence.
  • Gated Attention: Gated attention regulates information flow per token and head, reducing attention concentration on a first-token sink and encouraging spatially distributed visual features.The gate modulates attention outputs element-wise, alleviating loss spikes, accelerating convergence, and stabilizing scaling behavior.
  • Pretraining Details: GenLIP uses two-stage pretraining: 8B samples at fixed 224×224 resolution, followed by one epoch of higher-resolution native-aspect-ratio adaptation.The second stage uses 37M image-text samples and keeps visual tokens within [16, 1024]; layer scale and drop path stabilize deeper networks.
  • Discussion: GenLIP pursues a simple vision-encoder pretraining paradigm without novel architectural components, supporting integration into MLLMs.Table 1 summarizes model configurations and the two-stage pretraining setup.

4 Experiments

GenLIP is evaluated against contrastive and generative baselines across frozen multimodal benchmarks and model scales. It delivers consistent gains, especially on Doc&OCR tasks, while scaling with data and model size.

  • Experimental Setup: The evaluation covers frozen visual representations across Doc&OCR, General VQA, and Caption benchmarks, with results summarized using an overall average.The protocol replaces the vision encoder in LLaVA-NeXT, freezes it, and fine-tunes the language model.
  • Main Results: GenLIP improves ALL AVG over SigLIP2 by 2.5, 2.0, and 3.7 points at the L/16, So/16, and g/16 scales, respectively.On seven Doc&OCR tasks, GenLIP exceeds SigLIP2 by 4.3, 3.3, and 5.9 points at those scales.
  • Main Results: With Qwen2.5-7B, GenLIP outperforms SigLIP2 by 2.4 and 4.7 average-score points at the So/16 and g/16 scales.The strongest and most consistent gains remain on Doc&OCR benchmarks.
  • Baseline Comparison: GenLIP surpasses CLIP, SigLIP, AIMv2, and OpenVision2 across both frozen settings despite the latter generative baselines using additional text decoders and, for OpenVision2, stronger training resources.The results suggest improved data efficiency from GenLIP’s minimalist architecture and objective.
  • Overall Findings: Across frozen and standard evaluations, GenLIP performs strongly on diverse tasks and consistently excels on Doc&OCR benchmarks requiring fine-grained visual-text alignment.The extended results also report performance advantages across all three model sizes.
  • Scaling: Increasing training samples from 1.0B to 8.0B improves multimodal performance, with steeper gains through 4.0B and flatter gains afterward.VQA and caption improvements from 4.0B to 8.0B are minor.

4.4 Ablations

The ablations examine data efficiency, architectural choices, gated attention, resolution adaptation, transferability, and qualitative generation. They show that gated attention and native-aspect-ratio adaptation improve the learned representations, while GenLIP remains competitive under matched data budgets.

  • Gated Attention: Gated attention improves data efficiency, convergence, and final performance, with especially clear benefits in the low-data regime.The comparison evaluates GenLIP variants with and without gated attention across data scales.
  • Gated Attention: Against register-token alternatives, gated attention achieves the best overall average and strongest performance on most Doc&OCR benchmarks.The ablation uses GenLIP-So/16 models trained on 1B samples.
  • Resolution Adaptation: Native-aspect-ratio adaptation is evaluated by comparing frozen representations after two pretraining stages across multiple input resolutions.The figure reports average scores for OCR, VQA, and Caption tasks as resolution changes.
  • Transferability: Frozen GenLIP features transfer to ImageNet-1K classification and ADE20K segmentation, with GenLIP-g/16 reaching 85.2 top-1 accuracy and 44.5 mIoU.The discriminative ability scales with model size, and gated attention alleviates degradation from attention sinks.
  • Qualitative Analysis: Direct captioning becomes longer and more detailed after stage 2 and improves with model scale, while patch readouts align selected local regions with meaningful language concepts.These experiments support joint visual-linguistic modeling and local visual-text alignment.

5 Conclusions

GenLIP uses a unified transformer and standard language modeling objective to align visual and textual modalities directly. The paper reports strong data efficiency and scalability, while noting limits in model generality, dataset scale, and caption acquisition cost.

  • Conclusion: GenLIP jointly models visual and textual inputs with one transformer and directly optimizes the vision backbone for generative language prediction.The approach aligns the modalities through early fusion.
  • Conclusion: The method demonstrates strong data efficiency and scalability for vision-language representation learning.This conclusion follows the reported evaluation results summarized by the authors.
  • Limitations: Validation is limited to the academic-scale LLaVA-NeXT setting, so generalization to cutting-edge MLLMs remains unverified.The authors identify this as a limitation of the current validation scope.
  • Limitations: The pretraining dataset is limited to 1.0B scale, larger-volume scaling remains unexplored, and high-quality captions impose significant acquisition costs.These limitations constrain conclusions about larger data regimes and practical data requirements.

A Cambrian-1 style Evaluation

The Cambrian-1-style evaluation extends frozen representation testing with broader reasoning and perception benchmarks. GenLIP retains its advantage across model sizes and remains particularly strong on OCR- and chart-oriented tasks under a larger language model.

  • Evaluation Setup: The extended suite adds MMMU, MathVista, MMVP, and RealWorldQA to the frozen evaluation and reports results across all three model sizes.The main results use LLaVA-NeXT-Qwen2.5-1.5B, with Qwen2.5-7B results provided separately.
  • Results: Under Qwen2.5-7B, GenLIP improves over SigLIP2 by 1.6 and 3.0 average-score points at the So/16 and g/16 scales, respectively.At L/16, GenLIP achieves the best overall average among the compared L-scale encoders.
  • Results: The clearest gains under the larger LLM occur on ChartQA, OCRBench, TextVQA, and DocVQA.These results match the paper’s broader observation that generative pretraining is especially effective for fine-grained visual-text alignment.

B Supplementary Qualitative Results

Supplementary examples show that GenLIP develops meaningful OCR and patch-level semantic readout, with larger models handling fine-grained content better while errors remain on difficult details.

  • GenLIP models show non-trivial OCR ability on receipts, geometric layouts, tiny characters, and numbers, although clear errors remain.The examples test document understanding, shape counting and placement, and recognition of small text.
  • GenLIP-g16-S2 recovers invoice table structure more accurately than smaller variants, missing only one number and “Opener”.
  • GenLIP-g16-S2 is substantially more accurate on geometric-shape counting and placement, with a remaining acute-versus-right-triangle error.
  • GenLIP-g16-S2 identifies a tiny number correctly but still makes a spatial error, whereas smaller variants miss or misread the number.
  • Larger models recognize and describe subtle details more accurately, while long number strings, precise spatial layouts, and tiny text remain challenging.
  • Additional cases show stronger patch-semantic alignment for the stage-2 model and provide further examples of direct semantic readout from image patch embeddings.

C Additional Implementation Details

The implementation uses fixed-resolution foundational pretraining, higher-resolution native-aspect-ratio adaptation, and normalized aggregation across frozen-vision benchmarks.

  • The frozen visual representation evaluation replaces the default LLaMA3-8B, 780K SFT set, and anyres preprocessing with Qwen2.5 models, 3M SFT data, and resize-crop preprocessing.
  • ALL AVG averages 14 benchmarks equally after mapping MME-P from 0–2000 into [0, 100].
  • Stage 1 trains on fixed 224 × 224 inputs for 8B samples to learn foundational visual representations.
  • Stage 2 adapts GenLIP with higher-resolution caption data and native aspect ratios while keeping visual tokens between 16 and 1024.

D Discussion: Attention Sink and Gated Attention

GenLIP’s Prefix-LM setup creates attention-sink behavior that can weaken visual features; gated attention is introduced to reduce this concentration and improve representation quality.

  • Attention sinks arise when softmax attention distributes fixed unit mass, causing a few persistent register-like tokens to absorb disproportionate attention.
  • Vision and language transformers exhibit different sink patterns: low-semantic visual tokens attract attention in ViTs, while early tokens dominate in autoregressive language models.
  • GenLIP’s Prefix-LM uses bidirectional visual-prefix attention and causal text-suffix attention, encouraging visual information useful for generation to concentrate in preceding visual tokens.
  • The w/o GA variant has degraded linear-probing results, motivating gated attention because it alleviates concentrated sink behavior and improves learned visual features.

E Discussion: GenLIP Meets Language Priors

Controlled comparisons indicate that strong language-model initialization can preserve language-side shortcuts rather than encourage distributed visual grounding, whereas training from scratch shifts attention toward visual evidence.

  • Pretrained Qwen3-0.6B initialization does not produce stronger frozen visual representations in this setting.
  • Overall average performance rises from 53.6 to 54.8 with gated attention in Qwen-initialized SAIL, reaches 56.0 when trained from scratch, and is 56.3 for GenLIP-So/16.
  • Strong LLM initialization may let captioning rely on a language prior with weak visual conditioning, producing less distributed and grounded representations for standalone-encoder use.
  • Qwen-initialized variants allocate more generated-text attention to language-side anchors and less to visual tokens, while scratch training shifts attention toward visual tokens.
  • Removing language initialization increases pressure to ground language prediction in the visual prefix and better aligns captioning with grounded visual-representation learning.
Loading 2605.00809v2…