PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training

Junlin Han, Shengbang Tong, David Fan, Yufan Ren, Koustuv Sinha, Philip Torr, Filippos Kokkinos

arXiv:2509.26625v1cs.LGcs.AIcs.CVcs.MM

TL;DR

The paper investigates how text-only language pre-training gives LLMs visual priors and how those priors can be cultivated. Controlled analyses distinguish reasoning and perception priors, and the authors propose a data-mixing strategy for vision-aware pre-training.

  • Problem

    The paper asks how visual priors arise during language pre-training, whether they comprise distinct abilities, and how they can be used to build more capable multimodal LLMs.

    Source: §1 · p. 3

  • Method

    The authors use controlled ablations of model scale, data scale and composition, vision encoders, and visual instruction data, and propose balancing reasoning-centric with visually descriptive text.

    Source: §1 · p. 3

  • Results

    Visual reasoning is predominantly cultivated by reasoning-centric data, while perception emerges more diffusely from broad, diverse data.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The proposed data-mixing strategy balances reasoning-centric and visually descriptive text to cultivate visual priors for vision-aware LLMs.

    Source: §1 · p. 3

    The study is confined to static images, leaving visual priors for dynamic modalities such as video as an open question.

    Source: §7 · p. 18

Abstract

from arXiv · show

Large Language Models (LLMs), despite being trained on text alone, surprisingly develop rich visual priors. These priors allow latent visual capabilities to be unlocked for vision tasks with a relatively small amount of multimodal data, and in some cases, to perform visual tasks without ever having seen an image. Through systematic analysis, we reveal that visual priors-the implicit, emergent knowledge about the visual world acquired during language pre-training-are composed of separable perception and reasoning priors with unique scaling trends and origins. We show that an LLM's latent visual reasoning ability is predominantly developed by pre-training on reasoning-centric data (e.g., code, math, academia) and scales progressively. This reasoning prior acquired from language pre-training is transferable and universally applicable to visual reasoning. In contrast, a perception prior emerges more diffusely from broad corpora, and perception ability is more sensitive to the vision encoder and visual instruction tuning data. In parallel, text describing the visual world proves crucial, though its performance impact saturates rapidly. Leveraging these insights, we propose a data-centric recipe for pre-training vision-aware LLMs and verify it in 1T token scale pre-training. Our findings are grounded in over 100 controlled experiments consuming 500,000 GPU-hours, spanning the full MLLM construction pipeline-from LLM pre-training to visual alignment and supervised multimodal fine-tuning-across five model scales, a wide range of data categories and mixtures, and multiple adaptation setups. Along with our main findings, we propose and investigate several hypotheses, and introduce the Multi-Level Existence Bench (MLE-Bench). Together, this work provides a new way of deliberately cultivating visual priors from language pre-training, paving the way for the next generation of multimodal LLMs.

1 Introduction

Visual priors can emerge from text-only language pre-training and support visual capabilities. The paper systematically investigates their structure and origins, and proposes a data-mixing strategy to cultivate them.

1 Introduction

  • Prior work reports visual capabilities in text-trained LLMs, including generating code for complex scenes, adapting to vision with limited image-text data, and supporting pure-vision tasks.These phenomena motivate investigating what visual knowledge language pre-training encodes.Source: §1 · pp. 2–3
  • The paper uses controlled ablations of model scale, data scale and categories, mixture ratios, vision encoders, and visual instruction data to investigate visual-prior origins and structure.It frames visual priors as implicit knowledge or prior vision capabilities, rather than hand-crafted biases or Bayesian prior distributions.Source: §1 · p. 3
  • Visual priors comprise separable perceptual and reasoning components.The paper presents this as a systematic investigation of their nature and origins.Source: §1 · p. 3
  • Visual reasoning is predominantly cultivated by reasoning-centric data, while perception emerges more diffusely from broad, diverse data.The paper reports these as distinct origins for the two components.Source: §1 · p. 3
  • The proposed pre-training strategy balances reasoning-centric and visually descriptive text to cultivate visual priors for vision-aware LLMs.The stated aim is stronger multimodal performance.Source: §1 · p. 3
  • The paper introduces MLE-Bench for fine-grained perception evaluation and blind visual instruction tuning as an adaptation tool and probe of language-based task solving.The latter is described as revealing how models can “hack” visual tasks with language.Source: §1 · p. 3
  • The authors connect their findings to the Platonic Representation Hypothesis and describe them as empirical support for it.They relate the work to understanding how modality-specific capabilities may be encoded in language and to models’ internal world models.Source: §1 · p. 3

2 Problem Formulation

This section sets out the default training and evaluation protocols for the LLM and multimodal experiments, including how language–vision representation alignment is measured.

2.1 Training protocol.

  • The LLM suite follows the Llama-3 architecture across five model sizes, from 340M to 13B parameters, and training volumes from 0B to 1T tokens.The default setting fixes model size at 3B parameters and training data at 30B tokens.Source: §2.1 · p. 4
  • LLM pre-training draws on 16 data sources, including academic, code, mathematics, literature, and web-crawl data.Each source contains at least 50B tokens.Source: §2.1 · p. 4
  • Multimodal adaptation has visual alignment followed by supervised fine-tuning, using a frozen vision encoder and language model during projector training.The default encoder is MetaCLIP-B/16; fine-tuning mixes vision-language and language-only instruction data.Source: §2.1 · p. 4
  • Adaptation uses 1M image-caption pairs for alignment and a 3.5M-example instruction-tuning subset, including about 1.5M language-only and 2M vision-language instructions.The curated subset has a higher proportion of language-only instructions than the original data curation.Source: §2.1 · p. 4

2.2 Evaluation protocol.

  • LLM evaluation reports average perplexity on Wikitext and LAMBADA and unweighted average accuracy across eight zero-shot reasoning and question-answering benchmarks.The benchmark suite includes PIQA, HellaSwag, WinoGrande, ARC, Copa, SciQA, OpenbookQA, and BoolQA.Source: §2.2 · p. 4
  • Multimodal evaluation covers 16 benchmarks grouped into general, knowledge, OCR and chart VQA, and vision-centric categories.The categories probe perception, visual knowledge and reasoning, fine-grained reading, and spatial or other abstract visual reasoning.Source: §2.2 · pp. 4–5
  • A parsing strategy extracts intended answers from free-form responses to assess core vision ability independently of instruction-following format.The authors state that this supports reliable evaluation of models with and without language pre-training.Source: §2.2 · p. 5
  • Language–vision alignment is assessed by comparing feature-space kernels, then averaging mutual nearest-neighbor overlap scores against three vision backbones.The overlap uses k = 20 nearest neighbors; the backbones are ViT-Large, DINOv2-Giant, and CLIP-Huge.Source: §2.2 · p. 5

3 Demystifying LLM Visual Priors: Studies and Findings

The section traces how model scale and pre-training data shape visual abilities, derives a data mixture for vision-aware LLMs, and analyzes the structure and sources of visual priors. Its findings distinguish reasoning abilities shaped by language pre-training from perception abilities more sensitive to vision encoders and visual instruction tuning.

3.1 Impact of model and data sizes.

  • VQA performance generally improves with model and data size, but category scaling differs: OCR & Chart is more model-size-sensitive, while smaller models plateau earlier on Vision-Centric VQA.General and Knowledge VQA improve consistently with both model and data size.Source: §3.1 · p. 6, Figure 1 · p. 6

3.2 Impact of pre-training data sources.

  • Pre-training source matters: reasoning-centric and visual-world-description corpora correlate with strong Vision-Centric VQA, and the top-performing source-trained models score above 42%.The comparison fixes the LLM at 3B parameters and trains each model on 30B tokens from one source.Source: §3.2 · pp. 6–7, Figure 2 · p. 7

3.3 Impact of reasoning and visual data categories and proportions.

  • Increasing reasoning-centric data in the pre-training mix progressively enhances visual abilities, with gains observed up to a 75% ratio; visual-world data is crucial in small amounts, but its contribution saturates quickly.Source: §3.3 · p. 7

3.4 Deriving a data mixture for more vision-aware LLMs.

  • Mix6 ranks highest overall, trading generally higher vision accuracy for slight language declines; the vision-favorable target is approximately 60% reasoning and 15% visual content.The target ratio comes from a search over the 300B-token pool; mix6 is the balanced mixture in the practical 50B-token experiments.Source: Table 2 · p. 9, §3.4 · pp. 9–10, Table 1 · p. 9

3.5 The structure and origin of learned visual priors.

  • General and OCR performance correlate moderately, as do Knowledge and Vision-Centric performance, while weak or slightly negative cross-group correlations suggest loosely coupled perception and reasoning priors.The reported within-group correlations are 0.37 for General/OCR and 0.33 for Knowledge/Vision-Centric.Source: Figure 4 · p. 10, §3.5 · p. 10
  • Reasoning priors are predictably enhanced by reasoning-centric data, while perception's source remains diffuse: web-crawl performs best on General and OCR, but no specific category consistently boosts perception.The authors characterize perception as potentially a general byproduct of large-scale language modeling and introduce MLE-Bench for more direct assessment.Source: §3.5 · pp. 10–11

3.6 Deconstructing multimodal abilities: vision or language.

  • Visual reasoning performance improves with reasoning-data share across three vision encoders, whereas perception varies by encoder and depends more on visual instruction tuning.Removing perception-targeted tuning produces the largest drops on OCR & Chart and General VQA; reasoning-tuning removal produces smaller perception-task drops.Source: §3.6 · pp. 11–12, Figure 5 · p. 11, Figure 6 · p. 12

4 Discussion and Hypotheses

The discussion examines whether perception priors vary with object scale, whether text-trained reasoning transfers to visual tasks, and whether structured text relates to vision alignment.

4 Discussion and Hypotheses

  • The section presents its three hypotheses as interpretive frameworks, not definitive conclusions.They concern perception-prior structure, cross-modal reasoning, and the relationship between language-data structure and vision alignment.Source: §4 · p. 12

4.1 Is the perception prior multi-level? An evaluation using the MLE-Bench

  • Web-crawl pretraining led the 3B model overall on MLE-Bench, with its clearest advantage for small-to-medium objects and a much smaller gap for large objects.Figure 7 compares 16 single-source models; MLE-Bench groups 1,861 image questions by the target object's relative pixel size.Source: §4.1 · p. 13, Figure 7 · p. 13

4.2 Is reasoning a universal, cross-modal skill already acquired during pre-training?

  • Increasing code reasoning from 0% to 100% raised Logical Soundness from 4.52% to 9.52% and Reasoning Depth from 8.31 to 53.25.The authors suggest the depth increase may partly reflect a stylistic transfer from code's long, structured sequences.Source: §4.2 · pp. 14–15, Figure 9 · p. 14
  • The authors report that reasoning learned during pre-training transfers to visual tasks, including when training uses code alone.Their qualitative example contrasts simplistic reasoning at 0% code data with a detailed application of the prompt's rule at 100%.Source: Figure 8 · p. 14, §4.2 · p. 15

4.3 Does language data structure drive representational alignment with vision?

  • LLM-vision alignment generally improved as structured reasoning data increased, peaked at 75%, then declined at 100%.The authors suggest the all-reasoning mixture may lack vocabulary needed to map abstract structure to diverse visual concepts.Source: §4.3 · p. 15, Figure 10 · p. 15

5 Scaling Up and Training a Vision-Aware LLM

The authors scaled their data-mixture approach to two 7B models trained on 1T tokens each. Their balanced model outperformed the language-favorable model on visual benchmarks while retaining competitive language proficiency.

5.1 Settings and models

  • The scale-up pre-trained two 7B LLMs for 1T tokens each, using a language-favorable mix0 recipe and a balanced mix6 recipe designed to cultivate visual priors without compromising language proficiency.Both models were subsequently adapted into MLLMs using the complete Cambrian data suites.Source: §5.1 · p. 16

5.2 Results

  • The Balanced model had lower average perplexity than the Language-favorable model, 7.49 vs. 8.72, and slightly higher average accuracy, 0.655 vs. 0.647.It began surpassing the language-favorable model in language performance after approximately 600B tokens.Source: §5.2 · p. 16, Table 3 · p. 16
  • The Balanced model achieved a higher overall VQA average than the Language-Favorable model, 38.64 vs. 37.32, and outperformed it on all visual-task categories.The table covers General, Knowledge, OCR & Chart QA, and Vision-Centric tasks.Source: §5.2 · p. 16, Table 3 · p. 16

6 Related Work

Prior work connects vision encoders and LLMs through multimodal architectures and visual instruction tuning, while data-mixture research has largely optimized language performance. This paper extends that data-composition question to visual priors and multimodal potential.

6.1 From LLMs to MLLMs.

  • MLLM architectures developed from Flamingo- and BLIP-2-style cross-attention connectors to LLaVA-style projection layers with visual instruction tuning (Alayrac et al., 2022; Li et al., 2023; Liu et al., 2023a).The paper focuses on adapter-style architectures, which it says permit clean analysis of visual priors from language pre-training.Source: §6.1 · p. 17

6.2 The role of data in shaping foundation model capabilities.

7 Limitations and Future Research Directions

The analysis is centered on adapter-style MLLMs and static images, and it does not assess the safety and ethical implications of the learned visual priors.

7 Limitations and Future Research Directions

  • The findings primarily concern adapter-style MLLMs and may not generalize to discrete visual-tokenization or end-to-end joint-training approaches.In co-trained models, separating priors originating solely from language is difficult, and prior formation and use could differ.Source: §7 · p. 18
  • The study examines capability, not whether text-based priors encode biased visual associations that could produce harmful downstream behavior.The authors identify fairness and safety auditing as a next step.Source: §7 · p. 18
  • The study is confined to static images, leaving visual priors for dynamic modalities such as video unexamined.The authors point to temporal reasoning, action recognition, and causality in video as future research directions.Source: §7 · p. 18

8 Conclusion

The paper deconstructs visual priors acquired through text-only pre-training and argues that their separable components offer a basis for deliberate cultivation. It also identifies harmful visual stereotypes as a risk that capability-focused work has not investigated.

8 Conclusion

  • Controlled experiments manipulating data composition deconstruct visual priors acquired through text-only pre-training and examine their fundamental drivers.The conclusion frames this as a shift from merely observing visual priors to interrogating their drivers.Source: §8 · p. 18
  • The authors connect transferable, modality-agnostic reasoning abilities to the possibility of learning a unified representation of the world from a single modality.They describe this as further empirical support for that idea.Source: §8 · p. 18

A Broader Impact

  • The study focuses on capability and does not investigate whether learned priors reinforce harmful visual stereotypes in downstream multimodal systems.The authors identify this as a significant risk because pre-training text contains societal biases.Source: A Broader Impact · p. 19
  • Visual priors comprise separable perception and reasoning components, each cultivated by different types of text.The paper presents this understanding as an actionable path toward more efficient cultivation of these abilities.Source: A Broader Impact · p. 19
  • The authors propose strategically curating text-only pre-training data to strengthen a foundation for vision tasks before multimodal training.They state that this targeted approach improves final multimodal performance and reduces computational resources required.Source: A Broader Impact · p. 19

B Visual World and Reasoning-centric Language Data Classification

The appendices describe how language data and visual instruction-tuning data are classified into predefined content categories. They also specify that Table 4 reports the percentage of text segments assigned each category for each corpus.

B Visual World and Reasoning-centric Language Data Classification

  • A 32B dense LLM classifies 1024-token segments from each language data source into one or more predefined categories.The classification prompt asks the model to identify significant concepts using only the listed categories.Source: B Visual World and Reasoning-centric… · p. 19
  • The language-data categories distinguish visual concepts, attributes, and relationships from code, math, and science reasoning.The prompt defines these as labels for visual entities, properties, and relations, and for reasoning-focused content.Source: B Visual World and Reasoning-centric… · p. 19
  • Table 4 reports the percentage of text segments from each data corpus classified into each conceptual category.The percentages represent the proportion of segments in a source assigned a given label.Source: B Visual World and Reasoning-centric… · p. 19, Table 4 · p. 20

C Visual Instruction Tuning Data Classification

  • A 7B VLM classifies Cambrian-7M visual instruction-tuning data as perception-oriented, reasoning-oriented, or neither.The prompt distinguishes perception questions from complex analytical reasoning tasks and standard questions.Source: C Visual Instruction Tuning Data… · p. 20

D Robust Parsing for VQA Evaluations

The VQA evaluation procedure addresses free-form model answers with a prioritized parser rather than exact-letter matching alone. It checks answer letters first, then option text, and finally option-text substrings.

D Robust Parsing for VQA Evaluations

  • Free-form or conversational VQA answers can be correct despite not matching a ground-truth option letter exactly.The authors motivate parsing that does not unfairly penalize differently formatted correct responses.Source: D Robust Parsing for VQA Evaluations · p. 20
  • The parser first extracts explicit option letters using patterns such as “The correct answer is (B)” or outputs starting or ending with “(B)”.It proceeds through prioritized steps and stops when it finds a valid answer.Source: D Robust Parsing for VQA Evaluations · p. 20
  • If letter extraction fails, the parser checks for an exact, case-insensitive match to an option’s text.For example, “A car” is mapped to option B when it is the associated option text.Source: D Robust Parsing for VQA Evaluations · p. 21
  • As a final fallback, the parser matches option text within verbose answers and returns the longest matching option to prevent ambiguity.The authors state that this multi-tiered strategy aims to measure visual capability rather than formatting adherence.Source: D Robust Parsing for VQA Evaluations · p. 21

E Multi-Level Existence Benchmark Construction

The benchmark tests object existence across size-defined image splits, using segmentation masks to determine object area and plausible but incorrect distractors.

E Multi-Level Existence Benchmark Construction

  • Images are split by object-area proportion into small (0–30), medium (30–60), and large, dominant (60–100) categories.The benchmark uses ground-truth segmentation masks from SA-1B and ADE20K to calculate object area.Source: E Multi-Level Existence Benchmark… · p. 21
  • Each image receives a multiple-choice object-existence query with distractors drawn from the dataset vocabulary and filtered against ground truth.An LLM filters out distractor names that refer to objects present under different names.Source: E Multi-Level Existence Benchmark… · p. 21
  • Table 5 reports model performance by the three object-size splits and weighted overall accuracy.It evaluates identifying objects of varying sizes.Source: Table 5 · p. 22

F Multi-Level Existence Benchmark Results

On MLE-Bench, models have distinct size-dependent performance profiles and robust recognition across all object scales remains challenging. Blind visual instruction tuning improves performance in most tasks, with gains concentrated in knowledge categories and a risk of hallucination when images are absent.

F Multi-Level Existence Benchmark Results

  • MLE-Bench reveals distinct model performance profiles, while robust perception across all object scales remains a challenge.The benchmark is presented as a tool for diagnosing weaknesses and driving more comprehensive visual understanding.Source: F Multi-Level Existence Benchmark… · p. 22
  • gpt-5 achieves the highest overall performance, with weighted accuracy of 82.97%; it scores 90.69% on medium-sized objects, the largest scene category, and 73.6% on small objects.Its small-object performance is lower than its medium-object performance.Source: F Multi-Level Existence Benchmark… · p. 22

G Blind visual instruction tuning

  • Blind tuning first uses text-only instruction tuning to teach task format, then uses images in a later phase focused on vision capabilities.The authors report broad performance improvements in most tasks.Source: G Blind visual instruction tuning · pp. 22–23
  • The authors caution against making blind tuning standard practice because models should identify absent images rather than hallucinate.They report that this behavior may persist even in frontier models and leave the trick optional for future investigation.Source: G Blind visual instruction tuning · p. 23
  • Blind tuning gains are most pronounced in Knowledge categories, while OCR & Chart VQA tasks suffer a performance drop.The authors attribute the pattern to stronger use of internal knowledge and reasoning priors for tasks inferable from textual context.Source: G Blind visual instruction tuning · p. 23

H Hallucinations in Blind VQA

In blind VQA, stronger models do not necessarily hallucinate less when images are withheld. Additional qualitative examples contrast the effects of reasoning-centric and visual-data mixtures on visual reasoning and perception.

H Hallucinations in Blind VQA

  • Tests of randomly selected MLE-Bench questions without their images show that stronger models do not necessarily hallucinate less.Responses in the examples range from guessing despite lacking image access to explicitly asking for the image.Source: H Hallucinations in Blind VQA · pp. 23, 25

I Additional Qualitative Examples

  • For depth perception, the 100% reasoning-data model gives a detailed explanation applying foreground and background concepts, unlike the blunt 0% model.Figure 12 compares answers from five models trained with reasoning-data mixtures ranging from 0% to 100%.Source: Figure 12 · p. 26
  • For color constancy, the model with 25% visual data answers correctly, while models trained on more visual data give incorrect answers and flawed explanations.Figure 13 compares five models trained with visual-data mixtures ranging from 0% to 100%.Source: Figure 13 · p. 27

Related papers

Loading 2509.26625v1…