PaperlayerView exact revision on arXiv

Source-linked AI summary

From Pixels to Words -- Towards Native Vision-Language Primitives at Scale

Haiwen Diao, Mingxuan Li, Silei Wu, Linjun Dai, Xiaohua Wang, Hanming Deng, Lewei Lu, Dahua Lin, Ziwei Liu

arXiv:2510.14979v2cs.CVcs.AI

TL;DR

Native VLMs face unresolved challenges in unifying vision and language while addressing the costs and constraints of modular designs. This paper introduces NEO, built from shared multimodal primitives and trained on image-text data. It reports competitive performance with modular counterparts and gains over native VLM competitors on visual-centric benchmarks, alongside weaker results on knowledge- and OCR-heavy tasks.

  • Problem

    Modular designs fragment training and leave intermodal balance unresolved, while shared native modules can neglect differences in modality encoding and interaction.

    Source: §1 · p. 2

  • Method

    NEO uses unified native primitives for pixel-word encoding, alignment, and reasoning, with a pre-Buffer and post-LLM training scheme that becomes end-to-end.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    NEO narrows the gap with leading modular VLMs and gains on visual-centric benchmarks over native competitors, but lags on knowledge- and OCR-heavy tasks.

    Source: §4.2 · pp. 7–8

  • Takeaways & Limitations

    NEO’s unified design and reusable components offer a reported path toward scalable native multimodal models.

    Source: §5 · p. 10

    NEO lags on knowledge- and OCR-heavy tasks, and the 9B model does not surpass the 2B model on DocVQA and InfoVQA, which the authors link to limitations in the training corpus.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

The edifice of native Vision-Language Models (VLMs) has emerged as a rising contender to typical modular VLMs, shaped by evolving model architectures and training paradigms. Yet, two lingering clouds cast shadows over its widespread exploration and promotion: (-) What fundamental constraints set native VLMs apart from modular ones, and to what extent can these barriers be overcome? (-) How to make research in native VLMs more accessible and democratized, thereby accelerating progress in the field. In this paper, we clarify these challenges and outline guiding principles for constructing native VLMs. Specifically, one native VLM primitive should: (i) effectively align pixel and word representations within a shared semantic space; (ii) seamlessly integrate the strengths of formerly separate vision and language modules; (iii) inherently embody various cross-modal properties that support unified vision-language encoding, aligning, and reasoning. Hence, we launch NEO, a novel family of native VLMs built from first principles, greatly narrowing the gap with top-tier modular counterparts across diverse real-world scenarios. With 390M image-text examples, NEO efficiently develops visual perception from scratch while mitigating vision-language conflicts inside a dense and monolithic model crafted from our elaborate primitives. We position NEO as a cornerstone for scalable and powerful native VLM development, paired with a rich set of reusable components that foster a cost-effective and extensible ecosystem. Our code and models are publicly available at: https://github.com/EvolvingLMMs-Lab/NEO.

1 INTRODUCTION

The introduction asks how native VLMs can overcome limits of modular architectures while unifying vision and language. It presents NEO as a first-principles model with shared primitives, staged training, and reusable components.

1 INTRODUCTION

  • Modular VLMs combine pretrained vision encoders, projectors, and LLMs, but contend with strong visual-semantic biases, complex infrastructure, and component-scaling challenges.Native VLM efforts also face efficiency, optimization, and linguistic-knowledge challenges when learning visual representations inside pretrained LLMs.Source: §1 · p. 1
  • Native VLMs need unified primitives that jointly encode, align, and reason across modalities while preserving modality-specific interaction patterns.The proposed design combines flexible position encoding, multi-head native attention, and Native-RoPE compatible with pretrained LLMs.Source: §1 · p. 2
  • NEO uses pre-Buffer and post-LLM layers during pretraining, then merges them into an end-to-end model during later training stages.The pre-Buffer remains a reusable pretrained asset.Source: §1 · p. 2
  • NEO is trained on 390M image-text samples and approaches leading modular VLMs of comparable scale across diverse benchmarks.The paper also presents reusable components as a way to simplify subsequent development and reduce barriers to native VLM research.Source: §1 · p. 2

2 RELATED WORKS

Related work contrasts modular VLMs, which combine separate vision and language components, with native VLMs that integrate modalities earlier. The discussion motivates NEO’s unified, first-principles approach.

2.1 MODULAR VISION-LANGUAGE MODELS

  • Modular VLMs pair pretrained vision encoders with LLMs through adapters, achieving strong benchmark performance while requiring multi-stage training and interventions to address encoder biases and integration trade-offs.These biases can limit resolution flexibility, fine-grained detail, and sensitivity across scales.Source: §2.1 · p. 3

2.2 NATIVE VISION-LANGUAGE MODELS

  • Native VLMs use early fusion, with approaches including projected image patches, discrete visual tokenizers, MoE or divide-and-conquer strategies, and visual encoder supervision.Prior evidence reports that native models can approach modular counterparts with sufficient data and progressive training, while shared-style modules may still miss modality-specific properties.Source: §2.2 · p. 3
  • NEO is framed as a unibody system whose primitives are compatible with vision- and language-modeling patterns, using modality-agnostic pre-Buffer and end-to-end training.Figure 3 depicts native attention with bidirectional image dependencies, causal word and frame interactions, and modality-specific rotary position embeddings.Source: §2.2 · p. 3, Figure 3 · p. 4

3 METHODOLOGY

NEO combines lightweight image and text embeddings with native multimodal primitives in a monolithic backbone. Its training proceeds from visual pretraining to alignment-focused mid-training and instruction fine-tuning.

3.1 MODEL ARCHITECTURE

  • Image patches are embedded with two convolutional layers and text with the original LLM tokenizer, then visual and textual tokens are merged in one backbone.Each visual token corresponds to a 32 × 32 image patch, with positional and patch embeddings mapped into a unified space.Source: §3.1 · p. 4
  • Native-RoPE adds separate height and width dimensions alongside the temporal dimension, with distinct base frequencies and spatial versus temporal dependency roles.The design decouples channel and frequency allocation across H, W, and T.Source: §3.1 · pp. 4–5
  • Native multi-modal attention gives image tokens full bidirectional attention while text tokens retain causal attention, supporting visual dependencies and vision-language correspondences.FlexAttention is used to reduce memory overhead and increase throughput.Source: §3.1 · p. 5
  • During pretraining, a modality-shared pre-Buffer translates pixel-word inputs while the post-LLM inherits selected parameters from a pretrained LLM; the components are later merged.The pre-Buffer is randomly initialized, and this separation exists only during pretraining.Source: §3.1 · p. 5

3.2 TRAINING PROCEDURE

  • Pretraining uses 345M web-scale and synthetic image-caption pairs, with a 3:7 language-to-multimodal data ratio and only selected visual and post-LLM parameters optimized.The training objective is next-token prediction; the new QK heads are described as counteracting language bias and safeguarding capabilities against low-quality data.Source: §3.2 · pp. 5–6
  • Mid-training updates the full architecture on 40M captioning, conversation, detection, and OCR samples, and supervised fine-tuning uses about 4M bilingual instructions.The stages target vision-language alignment and varied visual scenarios, followed by complex instruction following and dialogue patterns.Source: §3.2 · p. 6, Figure 4 · p. 6

4 EXPERIMENTS

NEO is evaluated against modular and native VLMs across diverse vision-language benchmarks, then analyzed through ablations of its training stages and native primitives. The results report competitive modular comparisons, gains over native alternatives, and specific limits on knowledge- and OCR-heavy tasks.

4.2 MAIN RESULTS

  • Evaluation spans chart, diagram, document, visual-perception, reasoning, and hallucination benchmarks, while the encoder-based baseline uses Qwen3 with InternViT and staged projector and encoder training.The baseline trains its projector on 10M samples, then unfreezes the vision encoder for another 30M.Source: §4.2 · pp. 6–7
  • NEO is highly competitive with encoder-based models and narrows the gap to leading modular VLMs, while outperforming native VLM competitors on visual-centric benchmarks.The comparisons cover general vision-language and visual question-answering benchmarks.Source: §4.2 · p. 7, Table 1 · p. 7, Table 2 · p. 8
  • NEO lags on knowledge- and OCR-heavy tasks, and the 9B model does not surpass the 2B model on DocVQA and InfoVQA.The authors attribute the latter pattern to limitations in the current training corpus.Source: §4.2 · p. 8

4.3 ABLATION STUDIES

  • Average accuracy improves as pre-Buffer depth increases, with gains beginning to saturate beyond eight layers; NEO uses 12 layers at 2.2B and six at 9B.These layer counts balance performance and efficiency.Source: §4.3 · p. 8, Figure 5 · p. 8
  • Mixed attention consistently beats causal attention, and Native-RoPE gains at least 0.8% over the compared RoPE variants; setting height and width base frequencies to 1M severely impairs local-semantic perception.The authors link Native-RoPE’s advantage to disentangling height, width, and temporal components.Source: §4.3 · p. 9, Table 3 · p. 9
  • After two-stage retraining, PB3 trails CLIP, InternViT, and SigLIP by 1.7%, 2.4%, and 3.7% in average benchmark performance, respectively, and is 2.5% below full NEO using 22M samples.The passage presents PB3 as reducing the training cost of developing native VLMs.Source: §4.3 · p. 9, Figure 6 · p. 9
  • With Qwen3-1.7B held constant, NEO scores 44.0% versus 33.4%, 40.2%, and 41.5% for EVEv1.0, EVEv1.5, and EVEv2.0, respectively.The comparison is designed to isolate architectural effects from backbone and data differences.Source: §4.3 · p. 9, Figure 7 · p. 9
  • Across 2.2B and 9B model sizes, performance improves consistently as training data scales increase across the reported stages.The passage describes strong multimodal performance across diverse real-world tasks following progressive training.Source: §4.3 · p. 9, Figure 8 · p. 9

5 CONCLUSION

NEO unifies vision and language in a native VLM, using hybrid attention, modality-aware rotary embeddings, and a pre-Buffer/post-LLM training paradigm. The authors report advances in multimodal understanding and reasoning and present reusable components as a foundation for scalable architectures.

5 CONCLUSION

  • NEO integrates vision and language in one framework without separate visual encoders or ad-hoc alignment modules.Source: §5 · p. 10
  • Hybrid attention and modality-aware rotary position embeddings capture fine-grained interactions between pixels and words.Source: §5 · p. 10
  • Pre-Buffer and post-LLM training is presented as ensuring efficient convergence and robust alignment while maintaining end-to-end learning.Source: §5 · p. 10
  • Experiments report advances in multimodal understanding and reasoning, while the native primitives provide a foundation for reusable, scalable components.Source: §5 · p. 10

ETHICS STATEMENT

The work uses open-access datasets with defined usage policies and reports no new ethical or safety concerns beyond those associated with existing models. Dataset bias and misuse remain possible risks, for which the authors emphasize careful curation, responsible deployment, and transparent reporting.

ETHICS STATEMENT

  • All resources come from open-access datasets with explicitly defined usage policies.Source: ETHICS STATEMENT · p. 10
  • The authors say the work introduces no ethical or safety concerns beyond those already associated with existing models.Source: ETHICS STATEMENT · p. 10
  • Dataset bias and potential misuse cannot be entirely ruled out, motivating careful curation, responsible deployment, and transparent reporting.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The authors emphasize reproducibility through detailed reporting and a commitment to release code, model weights, and documentation. They identify dataset selection, training strategies, and evaluation settings as documented in the paper.

REPRODUCIBILITY STATEMENT

USAGE OF LARGE LANGUAGE MODELS

During manuscript preparation, large language models were used only as writing assistants. The authors state that methodological, experimental, and conclusion-related content was developed by them, and that only human-verified LLM edits were incorporated.

USAGE OF LARGE LANGUAGE MODELS

A.1 SUPERVISED FINE-TUNING DATASETS

This section identifies the datasets used during supervised fine-tuning.

A.1 SUPERVISED FINE-TUNING DATASETS

  • The section summarizes datasets used during supervised fine-tuning.The supplied caption gives no dataset names or quantities.Source: Table 4 · p. 20

A.2 IMPLEMENTATION DETAILS

The implementation-details section covers pre-training, mid-training, and supervised fine-tuning; the discussion addresses NEO’s constraints, design context, scale, and future applications.

A.2 IMPLEMENTATION DETAILS

  • The section covers implementation details for pre-training, mid-training, and supervised fine-tuning.Source: Table 5 · p. 20

A.3 LIMITATION AND DISCUSSION

  • Scarce training data and limited compute leave NEO’s potential largely untapped, especially in knowledge-intensive and OCR-focused domains.The authors say NEO nevertheless rivals state-of-the-art VLMs despite these constraints.Source: §A.3 · p. 21
  • Qwen3VL’s dense visual-language linking and relative positions echo NEO’s choices, while DeepStack underscores early pixel-word association.Source: §A.3 · p. 21
  • Further pre-training investment is needed to unlock NEO’s potential; selectively open-sourcing intermediate components can reduce follow-up training costs.The models are also presented as a baseline for reinforcement-learning research.Source: §A.3 · p. 21
  • NEO-2.2B closely matches equivalent-capacity modular VLMs; designs from 0.6 to 8 billion parameters are described as matured and suited to edge deployment.The passage notes edge use for settings with limited compute or strict real-time requirements.Source: §A.3 · p. 21
  • NEO currently focuses on dense image-text models, while sparse architectures are in development and future targets include visual-language generation, long-video understanding, and embodied AI.The architecture incorporates attention and rotary positional encodings for video-generation-related tasks, although the current focus is text and images.Source: §A.3 · p. 21
Loading 2510.14979v2…