Source-linked AI summary
From Pixels to Words -- Towards Native One-Vision Models at Scale
Haiwen Diao, Jiahao Wang, Penghao Wu, Yuhao Dong, Yuwei Niu, Yue Zhu, Zhongang Cai, Weichen Fan, Linjun Dai, Silei Wu, Xuanyu Zheng, Mingxuan Li, Yuanhan Zhang, Bo Li, Hanming Deng, Huchuan Lu, Quan Wang, Lei Yang, Lewei Lu, Dahua Lin, Ziwei Liu
TL;DR
Modular VLMs struggle to preserve native spatial signals and unified reasoning across images and video, while native models remain less explored beyond single-image understanding. NEO-ov uses an encoder-free monolithic backbone with end-to-end pixel-word and spatiotemporal modeling, achieving competitive performance against modular systems and stronger fine-grained perception and spatial reasoning.
Problem
Modular VLMs fragment native visual signals, limiting fine-grained spatial correspondence and unified reasoning across heterogeneous image, multi-image, and video inputs.
Method
NEO-ov uses a monolithic, encoder-free backbone that jointly learns visual perception, temporal dynamics, and cross-modal correspondence directly from raw inputs.
Results
NEO-ov establishes a performance frontier among native VLMs, substantially improves multi-image and video understanding, and remains competitive with leading modular systems across diverse benchmarks.
Takeaways & Limitations
Unified native architectures provide a promising path toward scalable, general-purpose one-vision models with strong fine-grained perception and spatial reasoning.
Takeaways & Limitations
A performance gap remains between NEO-ov and top-tier modular systems on certain single-image and video understanding benchmarks.
Abstract
from arXiv · showhide
Current vision-language models (VLMs) typically stitch together separate image encoders and language decoders via multi-stage alignment, a modular framework that inevitably fragments pixel-level signals across frames and scatters early pixel-word interactions. In parallel, native VLMs, despite impressive performance on single images, remain largely unexplored in multi-image, video understanding, and spatial intelligence. Hence, we introduce NEO-ov, a native foundation model that learns cross-frame and pixel-word correspondence end-to-end, without any external encoders, auxiliary adapters, or post-hoc fusion. By eliminating module boundaries entirely, NEO-ov enables fine-grained and unified spatiotemporal modeling to emerge natively inside the model. Notably, NEO-ov largely narrows the gap to modular counterparts while excelling at fine-grained visual perception, validating that native "one-vision" architectures are not only feasible but competitive at scale. Beyond empirical performance, we unveil systematic architectural analyses and detailed training recipes to facilitate subsequent native multimodal modeling. Our code and models are publicly available at: https://github.com/EvolvingLMMs-Lab/NEO.
1 Introduction
Existing modular VLMs struggle to unify heterogeneous visual inputs and early pixel–word interactions, while native VLMs have not yet simultaneously unified image, video, and spatial understanding. NEO-ov addresses this gap with an encoder-free, monolithic foundation model that jointly learns visual, temporal, and cross-modal representations end-to-end.
- Limitations of Existing VLMs: Modular VLMs use pretrained image or video encoders followed by projectors, creating separate processing pathways for visual inputs.These models have evolved toward multi-image, video, and spatial understanding while retaining an encoder–decoder design.
- Limitations of Existing VLMs: Image encoders favor static frame-level representations, whereas video encoders emphasize temporal dynamics and generalize poorly to single-image or interleaved inputs.Both designs also struggle with early pixel–word interactions.
- Native VLMs: Native VLMs avoid explicit vision encoders, but unifying single-image, multiple-image, video, and spatial understanding remains an open frontier.Native video approaches can model fine-grained video–language interactions and temporal dependencies, yet may inherit biases from distilled static visual encoders.
- NEO-ov: NEO-ov eliminates pretrained encoders and unifies spatial and temporal modeling within a single monolithic backbone trained end-to-end from raw inputs.It jointly learns visual perception, temporal dynamics, and cross-modal alignment, surpassing existing native VLMs and approaching encoder-based competitors using the same LLMs.
2 Related Work
Related work contrasts modular VLMs, which compress visual inputs before language reasoning, with native multimodal models that integrate pixels and words directly. Existing native efforts establish feasibility and extend to video, while remaining limited in broader unified perception settings.
- Modular VLMs: Most VLMs connect external visual encoders to LLMs through lightweight adapters, following the Encoder-MLP-LLM pipeline popularized by LLaVA.This modular design is also adopted by models including the InternVL and Qwen-VL series.
- Modular VLMs: The encode-then-project paradigm compresses visual signals before reasoning, constraining how language models access native visual information.This limitation arises from the modular separation between visual encoding, projection, and language reasoning.
- Modular VLMs: Image–text encoders such as CLIP and SigLIP prioritize high-level semantics while discarding texture, local geometry, and fine spatial structure.As a result, modular VLMs face limitations in fine-grained perception and precise geometric reasoning, especially for spatial intelligence.
- Native Multimodal Models: Native multimodal models integrate image patches directly into decoder-only Transformers, demonstrating the feasibility of unified pixel-and-word modeling without separate visual encoders.Fuyu and EVE are cited as early examples of this fully native approach.
- Native Multimodal Models: Recent native VLMs extend end-to-end modeling to fine-grained video–language interactions and temporal dynamics but focus primarily on video understanding.They do not address the broader combination of single-image understanding, multi-image reasoning, spatial intelligence, and other unified perception tasks.
3 NEO-ov: Native One-Vision Modeling
NEO-ov extends native unified autoregressive vision-language modeling to multi-image, video, and spatial understanding by serializing visual units and text in one sequence. Its THW-aware attention and three-stage end-to-end training jointly support spatial, temporal, and cross-modal reasoning.
- Unified Native Modeling: NEO-ov represents images, frames, regions, and text in a unified sequence, enabling cross-image reasoning, temporal understanding, and spatial localization.It extends unified autoregressive modeling beyond single-image understanding to multi-image, video, and spatial intelligence.
- Unified Native Modeling: Each image is independently encoded at arbitrary resolution, allowing token counts and visual detail to adapt to spatial size for fine-grained comparison.Multiple images appear as distinct visual units following their textual order.
- Unified Native Modeling: Video is serialized as temporally ordered sampled frames with timestamps and global duration cues, allowing temporal localization and cross-frame reasoning within the same framework.The representation samples f frames and preserves explicit timestamps and frame order.
- Spatial-Temporal Attention: Within each image or frame, tokens attend bidirectionally, while interactions across visual units remain causal, jointly modeling dense intra-unit structure and cross-image or cross-frame dependencies.This mixed-attention design extends native mixed attention to multiple images and video frames within one end-to-end autoregressive backbone.
- Training Recipe: Training proceeds through approximately 20M image-text pairs, nearly 60M multimodal samples, and approximately 6M instruction-tuning samples across pretraining, mid-training, and supervised fine-tuning.The stages progressively develop visual perception, spatial-temporal reasoning, and multimodal instruction-following capabilities.
4 Experiment
NEO-ov is evaluated across image, video, multi-image, and spatial-intelligence tasks, where it establishes strong native-VLM performance and remains competitive with modular and specialist systems. Architectural analyses further associate its gains with native early interactions and progressive training.
- Evaluation Scope: NEO-ov is evaluated across three domains: image understanding, video understanding, and spatial intelligence.
- Image Understanding: At both 2B and 8B scales, NEO-ov consistently surpasses prior native architectures, especially on reasoning-intensive and hallucination-sensitive benchmarks.The highlighted benchmarks include MMMU, HallB, and InfoVQA.
- Image Understanding: Without pretrained visual encoders, NEO-ov matches or surpasses modular VLMs on several reasoning and perception benchmarks, though OCR-intensive tasks remain challenging.
- Multi-Image and Video Understanding: At both 2B and 8B scales, NEO-ov substantially improves over prior native VLMs on VideoMME, MVBench, and MLVU, demonstrating temporal and long-context visual understanding.It also remains highly competitive with several modular VLMs across additional multi-image and video benchmarks.
- Spatial Intelligence: As a general-purpose native VLM, NEO-ov achieves comparable or better spatial-intelligence performance than spatial-specialist models at both 2B and 8B scales.It shows clear advantages over other general VLMs across benchmarks including VSI-Bench, MMSI, Mindcube-tiny, ViewSpatial, SITE, 3DSR, EmbSpatial, and SPAR.
- Architectural and Training Analysis: Native attention and deeper interactions yield larger spatial-intelligence gains, while performance improves consistently from Stage 1 to Stage 2 across diverse benchmark groups.The analyses attribute these gains to early pixel-pixel and pixel-word interactions in the unified backbone and to progressive training.
5 Conclusion
NEO-ov is presented as a fully native vision–language foundation model unifying image, multi-image, video, and spatial understanding in one monolithic backbone. It learns visual perception, temporal dynamics, and cross-modal correspondence end-to-end from raw inputs without external visual encoders.
- NEO-ov unifies single-image understanding, multi-image reasoning, video comprehension, and spatial intelligence within one monolithic backbone.
- Unlike modular VLMs, NEO-ov learns visual perception, temporal dynamics, and cross-modal correspondence directly from raw inputs through end-to-end training.
- NEO-ov performs this unified learning without relying on external visual encoders.
6 Limitations
NEO-ov still trails top-tier modular systems on some single-image and video benchmarks, while OCR-intensive and document-centric capabilities remain underexplored. Further improvements in multimodal data, model capacity, and long-context training may strengthen native multimodal reasoning and perception.
- Performance gap: NEO-ov remains behind top-tier modular systems such as Qwen3-VL on certain single-image and video understanding benchmarks.The paper attributes this gap largely to the scale and quality of current multimodal training data.
- OCR and documents: OCR-intensive and document-centric tasks remain underexplored because NEO-ov lacks diverse, high-quality supervision for documents, charts, and dense text perception.The paper expects improved OCR-related data scale and quality to strengthen native architectures.
- Future exploration: Further scaling in model capacity, multimodal data diversity, and long-context training may unlock stronger multimodal reasoning and perception.NEO-ov already shows promising capabilities in multi-image reasoning, video understanding, and spatial intelligence, but native multimodal modeling remains far from fully explored.
7 Ethical Considerations
The study uses open-access resources under defined usage policies and reports no additional ethical or safety concerns beyond existing models, while acknowledging residual risks and disclosing limited LLM assistance in manuscript preparation.
- Risks and Mitigation: The resources come from open-access datasets with explicitly defined usage policies, but dataset bias and potential misuse remain possible risks.The authors emphasize careful data curation, responsible deployment, and transparent reporting to mitigate these challenges.
- LLM Use Disclosure: LLMs were used only for manuscript writing assistance, while authors developed and critically reviewed all methodological, experimental, and concluding content.Only human-verified edits were incorporated into the final text.