PaperlayerView exact revision on arXiv

Source-linked AI summary

Reading Between the Layers: Cross-Modal Alignment and Representational Hierarchy in Vision Transformers

Sugam Karki, Raúl Vázquez, Yova Kementchedjhieva

arXiv:2610.32463v1cs.CL

TL;DR

How vision encoders organize written text and its meaning across layers remains unclear, despite their text-recognition abilities. The paper probes rendered text and visual concepts with layer-wise and cross-modal analyses, finding that strong alignment appears only in the deepest layers alongside a surface-to-semantic hierarchy.

  • Problem

    How vision encoders develop cross-modal semantic alignment and organize language-relevant representations across layers remains unclear.

    Source: §1 · p. 1, §2.1 · p. 2

  • Method

    The study combines layer-wise probing of rendered text with cross-modal alignment and retrieval analyses across vision encoders.

    Source: §1 · p. 2

  • Results

    Strong cross-modal transfer appears only in the deepest layers after unimodal recognition; CLIP shows a surface-to-semantic hierarchy, while DINO and MAE have weaker cross-modal structure.

    Source: §4.2 · p. 5, §5.2 · p. 7

  • Takeaways & Limitations

    These findings support interpreting the measured behaviors as language-relevant representational organization and cross-modal alignment, not full human-like reading comprehension.

    Source: Limitations · p. 9

    The alignment proxy does not cover broader reading comprehension, and mostly isolated-word probes leave complex sentence- and discourse-level evaluation outside the study’s scope.

    Source: Limitations · pp. 9–10

Abstract

from arXiv · show

While large vision-language models excel at optical character recognition, it remains unclear how their underlying vision encoders process text and its meaning, and whether they support anything resembling reading comprehension. We systematically study transformer-based vision encoders, using cross-modal alignment between rendered text and visual concepts together with probing and retrieval-based analyses as a controlled window into these broader capabilities. Our layer-wise analyses span orthographic, grammatical, and semantic organization, and reveal that strong cross-modal mapping emerges only in the deepest layers, after unimodal object and text recognition have already developed, coinciding with a progression from surface-level to semantic representations. Although this hierarchy is strongest in contrastively trained models, larger self-supervised models and text-rich training data partially support similar behaviors without explicit language supervision. Together, our results suggest that increasing model and training-data scale, as well as text-rich training data, can partially compensate for the absence of language supervision.

1 Introduction

The introduction frames vision-encoder text processing as a question of moving beyond symbol recognition toward semantic alignment. It proposes layer-wise probing and cross-modal analyses, reporting that strong alignment emerges only in the deepest layers alongside a progression from surface to semantic representations.

1 Introduction

  • Prior work suggests vision encoders can recognize in-image text, but the scope and representational dynamics of their cross-modal semantic alignment remain systematically understudied.The introduction distinguishes recognizing written symbols from connecting them to semantic meaning.Source: §1 · p. 1
  • The study uses rendered-text/visual-concept alignment, probing, and retrieval to examine vision encoders as a controlled window into broader reading-comprehension capabilities.It asks when alignment emerges, what representational structure underlies it, and how training objective, scale, and data composition shape it.Source: §1 · pp. 1–2
  • Strong cross-modal semantic alignment emerges only in the deepest layers, after unimodal object and text recognition, alongside a hierarchy from orthography through grammar to semantics.The hierarchy is clearest in CLIP; self-supervised models show weaker structure, with clearer behavior in a larger model and modest gains from text-rich data.Source: §1 · p. 2

2 Background

Prior studies examine text recognition and cross-modal matching in vision encoders, but leave open how written words become semantically grounded and organized across layers. This section situates that gap within research on hierarchical representations and on the effects of supervision, data, and scale.

2.1 From Decoding to Cross-Modal Semantic Alignment in Vision Encoders

  • Prior work finds that CLIP can match rendered word images to their depicted concepts, while debate remains over whether late-layer differentiation reflects semantic grounding beyond orthographic disambiguation.Earlier compositional evaluations assess final cross-modal embeddings and do not isolate how image encoders process written language as pixels.Source: §2.1 · p. 2
  • Open questions include whether image encoders develop language-relevant representations, whether these follow a surface-to-semantic hierarchy, and how they relate to visual capabilities.The authors address these gaps with controlled layer-wise probing and cross-modal alignment experiments.Source: §2.1 · pp. 2–3

2.2 Hierarchical Organization Across Layers

  • Prior work reports surface-to-semantic layer hierarchies in language models and PIXEL, but not in ViT-MAE, and finds weaker syntactic organization in VLM text encoders than in unimodal language models.The paper extends this inquiry to vision encoders processing rendered text through their image encoder, including models never trained on language.Source: §2.2 · p. 3

2.3 Role of Training Paradigm, Data, and Scale

  • How supervision, data composition, and scale shape cross-modal semantic alignment remains an open question, despite prior evidence that these factors affect OCR in vision encoders.The cited work attributes much of CLIP’s OCR advantage over DINO to text-rich data, with language supervision adding benefit and increased scale further offsetting its surface-reading advantage.Source: §2.3 · p. 3

3 Methodology

The methodology renders text as images to control visual form and connect text with natural-image concepts. It analyzes frozen representations across model depth and compares vision encoders trained under different paradigms and at different scales.

3 Methodology

  • Rendered text varies in font, size, and layout while preserving linguistic content, reducing reliance on fixed visual cues and enabling cross-modal pairing with natural images of corresponding concepts.This strategy addresses uncontrolled layout, context, and background in real text-bearing images.Source: §3 · pp. 3–4
  • At every transformer block, the study extracts frozen post-LayerNorm residual-stream representations using either the CLS token or mean-pooled patch tokens.MAE inference uses a zero mask ratio so all patch tokens are present.Source: §3 · p. 4
  • The comparison covers approximately 1B-parameter CLIP, DINO, and ViT-MAE encoders trained on the same data, plus DINO models from 300M to 7B parameters.Encoder weights remain frozen during probe training so classifiers measure pretrained representations without task adaptation.Source: §3 · p. 4

4 RQ1: Cross-Modal Semantic Alignment

The section tests cross-modal semantic alignment through layer-wise classification, retrieval, and orthographic controls. CLIP’s advantage over DINO and MAE concentrates in the deepest layers.

4.1 Tasks

  • Cross-modal classification trains on frozen representations from natural images or rendered class labels and tests category prediction across modalities, averaging the two transfer directions.Within-modality classification provides a comparison with image and text recognition.Source: §4.1 · p. 4
  • Zero-shot retrieval uses cosine similarity without added classifiers, while relational retrieval tests image-text matching for cardinality, direction, and relative size.Relational tasks use controlled scenes of fruit objects and report average Recall@K in both retrieval directions.Source: §4.1 · p. 4
  • Orthographic controls pair class names with semantically unrelated, edit-distance-one distractors and compare their cosine similarities to the image embedding.A positive cosine gap favors the semantically correct match over the orthographic distractor.Source: §4.1 · pp. 4–5

4.2 Results and Discussion

  • Image recognition develops before text recognition, and cross-modal transfer, when present, emerges only after both in the deepest layers.ImageNet-1K and zero-shot retrieval show the same late-layer pattern.Source: §4.2 · p. 5, Figure 3 · p. 5
  • CLIP’s late-layer alignment extends to direction, size, and cardinality, with cardinality strongest, size next, and direction substantially weaker.Absolute Recall@1 remains low across tasks; DINO and MAE remain near chance across depth.Source: §4.2 · p. 5, Figure 4 · p. 5
  • Across 1B encoders, CLIP progresses from surface information early to intermediate grammatical signals and deepest-layer semantic organization, unlike DINO and MAE’s weak grammatical and near-absent semantic signals.Source: Figure 5 · p. 6
  • Early and middle layers do not distinguish correct text from orthographic distractors, but deepest-layer CLIP representations increasingly favor the semantically matching text.DINO and MAE show no comparable separation.Source: §4.2 · pp. 5–6
  • Across cross-modal classification, relational retrieval, and orthographic control, CLIP outperforms DINO and MAE, with its advantage concentrated in the deepest layers.Source: Figure 3 · p. 5, Figure 4 · p. 5, §4.2 · p. 6

5 RQ2: Representational Structure

The section probes orthographic, grammatical, and semantic information in frozen vision-encoder representations, finding that CLIP develops a clearer surface-to-semantic hierarchy than DINO and MAE. In CLIP, semantic organization consolidates in the deepest layers, where strong cross-modal alignment also emerges.

5.1 Tasks

  • Surface probes test character presence and word length, using blank rectangle controls to subtract performance attributable to visual width cues.The adjusted score subtracts rectangle-control performance from probe performance.Source: §5.1 · p. 6
  • Grammatical probes use isolated-word and contextual part-of-speech prediction, with a five-word window intended to reduce reliance on lexical priors.The contextual image circles the target word.Source: §5.1 · pp. 6–7
  • Semantic probes test WordNet-derived supersenses and concreteness, while the evaluation uses frozen representations, cross-validation, and label controls near chance.Rectangle controls are also applied to concreteness prediction because word length correlates with abstractness.Source: §5.1 · p. 7

5.2 Results and Discussion

  • CLIP shows a surface-to-semantic hierarchy: orthographic signals emerge early, grammatical signals peak at intermediate depth, and semantic organization consolidates near depth 0.9.The semantic-probe emergence coincides with strong cross-modal transfer and retrieval alignment.Source: §5.2 · p. 7
  • DINO and MAE show strong surface signals but substantially weaker grammatical and semantic structure, consistent with language supervision shaping deeper organization at this scale.At this scale, the absence of language supervision limits the deeper structures that support cross-modal alignment in CLIP.Source: §5.2 · p. 7

6 RQ3: The Role of Scale and Data Composition

The section tests whether model scale or text-rich training data can strengthen self-supervised DINO representations and cross-modal transfer. Larger-scale models show the clearest gains, while text-rich filtering improves some outcomes but does not match the 7B model.

6.1 Tasks

  • The experiments compare DINO variants by model size and by text-heavy data composition using the tasks and evaluation protocols established earlier.The scale variants are 300M, 2B, and 7B; the data variants are DINO-2B models trained on filtered MC subsets.Source: §6.1 · p. 8

6.2 Results and Discussion

  • Scale improves cross-modal transfer and representational organization, with the largest gains at 7B; the 300M-to-2B comparison mainly improves surface representations, while 7B also uses more training data.The 300M and 2B models share a 2B-image dataset, whereas 7B is trained on 8B images, so its gains combine increased capacity and dataset scale.Source: §6.2 · pp. 8–9, §6.1 · p. 8
  • Text-rich filtering stabilizes deeper text recognition and modestly improves late-layer cross-modal transfer and semantic probing, but does not match DINOv2 7B or CLIP.The data-composition variants use OCR-based filtering of the MC dataset, retaining 50.3% of images in the light variant and 1.3% in the heavy variant.Source: §6.1 · p. 8, §6.2 · p. 9

7 Conclusion

The conclusion reports that strong cross-modal alignment between rendered text and natural images emerges in the deepest layers, after unimodal recognition has developed. It connects this late alignment with hierarchical linguistic representations and reports stronger gains from model and training-data scale than from text-rich data alone.

7 Conclusion

  • Strong alignment between rendered text and natural images appears only in the deepest layers, after unimodal object and text recognition have developed.Zero-shot and relational retrieval corroborate the pattern; spatial direction remains the most difficult relational attribute to recover.Source: §7 · p. 9
  • The transition to alignment coincides with a hierarchy from orthographic information through grammatical regularities to semantic organization.Late-layer CLIP representations support object-label correspondence and attribute-level structure such as cardinality and size.Source: §7 · p. 9
  • Among the studied models, model size and training-data scale yield larger gains than text-rich data, which stabilizes late text recognition and modestly improves semantic alignment.Text-rich data does not match the stronger transfer and semantic organization observed in the 7B model.Source: §7 · p. 9

Limitations

The study measures language-relevant organization mainly through cross-modal alignment, which is narrower than human-like reading comprehension. Its probing and isolated-word focus, and its evaluation of standalone encoders, limit what the findings establish.

Limitations

  • Cross-modal semantic alignment is an operational proxy, not evidence of full human-like reading comprehension involving reasoning, discourse, world knowledge, dependencies, and pragmatics.Interpret the findings as evidence of language-relevant organization and alignment, not comprehensive reading comprehension.Source: Limitations · p. 9
  • Probing measures information accessible in representations, but the analyses cannot fully establish how models internally compute or use linguistic representations.Cross-modal transfer, retrieval, and orthographic controls supplement probing but do not resolve that limitation.Source: Limitations · pp. 9–10
  • Many tasks use isolated words, leaving more complex sentence-level and discourse-level evaluations outside the study’s scope.Contextual POS experiments partially address the limitation.Source: Limitations · p. 10
  • The study evaluates standalone vision encoders, so how its findings transfer to instruction-tuned or generative multimodal systems remains unclear.Downstream language models may alter or amplify language-relevant representations in the visual backbone.Source: Limitations · p. 10

Disclaimers

The study relies on preexisting models and datasets under their respective licenses, and the authors used AI tools for code development and debugging while retaining responsibility for the scientific work.

Disclaimers

  • The study uses preexisting models and datasets, consistent with their respective licenses.Source: Disclaimers · p. 10
  • AI tools supported code development and debugging; the authors state that experimental design, analyses, and scientific conclusions are solely their work.The authors take full responsibility for the paper’s contents.Source: Disclaimers · p. 10

A Dataset Construction Details

The character-presence and letter-count datasets use filtered SUBTLEX-US words and rendered images, with balancing procedures tailored to each classification task.

A.1 Character Presence

  • Character-presence data pairs 100 frequent words containing each letter with 100 not containing it, yielding 200 samples per letter and 5,200 rendered images.Words are selected deterministically from the frequency-sorted corpus after filtering and deduplication.Source: §A.1 · p. 12

A.2 Letter Count

  • Letter-count data samples 100 frequent words per exact length across four bins, producing 300 words per bin and 1,200 total samples.Each word has a rendered-text image and a rectangle control preserving visual extent while removing orthographic content.Source: §A.2 · p. 12

A.3 POS Tagging

The appendix details datasets for linguistic and semantic probes, cross-modal classification and controls, relational retrieval, and the vision encoders used in the study.

A.3 POS Tagging

  • The isolated-word dataset retains the top 200 lemmas in each of NOUN, VERB, ADJ, and ADV after requiring at least 10 occurrences and 98% dominant-tag purity; contextual data targets ambiguity.Contextual lemmas must occur under at least two target POS classes, each at least three times, with a minimum total frequency of 10 occurrences.Source: §A.3 · p. 12

A.4 Supersense Classification

  • Supersense classification uses 50 concrete nouns in each of 10 manually curated semantic categories.Source: §A.4 · p. 13

A.5 Concreteness

  • Concreteness classification separates ratings at or below 2.0 from those at or above 3.0, excludes intermediate words, and length-matches the 500-word classes.The matching reduces the possibility of exploiting visual word extent as a shortcut.Source: §A.5 · p. 13

A.6 Cross-Modal Classification

  • Cross-modal classification pairs Caltech-256 and ImageNet-1K images with rendered class labels and tests transfer in both image-to-text and text-to-image directions.The Caltech-256 set retains 143 classes; ImageNet-1K uses all 1,000.Source: §A.6 · p. 13

A.7 Orthographic Control

  • The orthographic control compares Caltech-256 images and correct labels against rendered, edit-distance-1 distractor words for 90 classes.The correct and distractor renders share canvas dimensions to control image size and layout.Source: §A.7 · p. 13

A.8 Relational Retrieval

  • Relational retrieval uses synthetic fruit image-text pairs for cardinality, relative size, and spatial direction.The datasets include 80 compound labels each for cardinality and size, and 1,520 relational triples for direction.Source: §A.8 · pp. 13–14

B Models

  • Table 1 identifies the vision encoders used in the study.Source: Table 1 · p. 14

C Further Results

Further results show that image recognition, text recognition, and cross-modal alignment emerge in sequence in CLIP, while retrieval and grammatical signals are strongest in deeper layers. Relational alignment varies by model and attribute, with spatial direction the most difficult.

C.1 Cross-modal Classification for ImageNet1k

  • Image recognition develops before text recognition and cross-modal alignment in CLIP, while cross-modal transfer is absent in DINO and MAE.Source: Figure 8 · p. 14

C.2 Zero-shot Retrieval

  • CLIP and DINOv2 7B show sharp late-layer retrieval increases, while DINOv2 1B and MAE remain near chance across depth.This pattern appears for both Recall@1 and Recall@10.Source: Figure 9 · p. 14, Figure 10 · p. 15

C.3 Relational Retrieval and Orthographic Control

  • CLIP shows strong late-layer relational alignment, DINOv2 7B weaker measurable alignment, and smaller self-supervised models near chance; spatial direction is hardest.Source: Figure 11 · p. 15

C.4 Contextual POS

  • Contextual POS signals strengthen in deeper layers of CLIP and DINOv2 7B, while DINOv2 1B and MAE remain substantially weaker.Source: Figure 12 · p. 15
Loading 2610.32463v1…