PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Transformers Learn Word Meaning from Images Alone

Anders Gjølbye, Sebastian Ray Mason, Stella Frank, Lars Kai Hansen

arXiv:2610.13252v1cs.CVcs.CL

TL;DR

The paper asks whether vision models pretrained on images alone can learn semantic relations among written words, not merely recognize word forms. It derives word directions from rendered words and matched controls, then evaluates their semantic structure. The largest DINOv3 models show word-level semantic structure, while synonym evidence is weaker and sentence understanding is not established.

  • Problem

    It is unclear whether image-only vision models learn semantic relations among written words, rather than merely distinguishing their appearance.

    Source: §1 · p. 1

  • Method

    The authors compare vocabulary-centered embedding differences between rendered words and matched controls, testing semantic neighbors under spelling controls and alignment with visual classes.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 5

  • Results

    DINOv3-H and DINOv3-7B show category membership and country-capital associations under spelling controls, weaker synonym evidence, and alignment with named visual categories.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    Image-only pretraining can yield word-level semantic structure, but these findings do not establish sentence understanding or instruction following.

    Source: §8 · p. 9

    The findings do not establish taxonomic hierarchies or sentence-level composition, and consistent offsets do not improve analogy retrieval over the country-only baseline.

    Source: §7 · p. 9

Abstract

from arXiv · show

Can a vision model learn what words mean from pixels alone? We show that the two largest DINOv3 vision transformers organize words rendered in images by word meaning, despite being pretrained using self-supervision on images only, without paired captions, class labels, or other language supervision. We derive word representations from embedding differences between word images and matched controls, then test whether semantically related words are neighbors within each model's embedding space, while controlling for spelling similarity. We find that brand names in the same industry group together, as do nouns from the same semantic domain. Country-capital pairs exhibit consistent vector offsets, similarly to word embeddings trained on text. These word representations also align with the representations of the visual object categories they name. DINOv3-H and DINOv3-7B are, to our knowledge, the first self-supervised vision models to show substantial semantic structure for rendered words. Comparisons with text and speech encoders reveal shared structure among representations of the same words. Contrastive alignment with either encoder further strengthened DINOv3-H's word-level semantic structure, while leaving the other finetuned vision models largely unchanged. Image-only training can therefore teach vision models not just what words look like, but also how their meanings relate.

1 INTRODUCTION

The paper asks whether image-only self-supervision can give vision models knowledge of semantic relations among written words, beyond recognizing objects or distinguishing word forms. It reports such structure in the largest DINOv3 models and tests it through controlled word representations and comparisons with visual and language encoders.

1 INTRODUCTION

  • The study asks whether image-only training can produce semantic relations among words, which text embeddings encode through patterns of co-occurrence.Distinguishing words by appearance alone does not establish knowledge of how their meanings relate.Source: §1 · p. 1
  • Earlier findings on DINOv2 did not establish semantic understanding of overlaid words, while DINOv3 was found to represent rotated and mirrored text differently.That result concerned visual structure, not word meaning.Source: §1 · p. 1
  • DINOv3-H and DINOv3-7B capture category membership and country-capital associations under spelling controls, with weaker evidence for synonymy.Their word directions also align with representations of named visual categories and share structure with text and speech encoders.Source: §1 · p. 2
  • The authors derive word directions from rendered words and matched controls, average across backgrounds, and control for spelling similarity and rendering conditions.Figure 1 illustrates nearest rendered words and country-to-capital offsets; its six displayed pairs were selected from 60 and do not quantify consistency.Source: §1 · p. 2, Figure 1 · p. 2

2 RELATED WORK

Prior work establishes semantic organization in vision models and investigates written text in images, but much of that work involves language-supervised training or visual rather than lexical structure. The paper situates its question alongside research on distributional word geometry and cross-modal representational convergence.

2 RELATED WORK

  • Self-distillation organizes vision representations by semantic category, while masked reconstruction emphasizes local visual features and texture.At scale, self-supervised image encoders can match language-supervised encoders in visual question answering; controlled smaller-scale comparisons find caption supervision strengthens fine-grained categories and text.Source: §2 · p. 2
  • CLIP and related models represent written text in images, but their training uses image-caption or image-alt-text pairs.Earlier DINOv2 comparisons found grouping by visual content rather than written-word meaning; this paper instead characterizes semantic relations learned without language supervision.Source: §2 · pp. 2–3
  • The Platonic Representation Hypothesis proposes convergence across modalities, while later studies report weaker neighborhood agreement at larger evaluation scales and little evidence of global convergence under spectral measures.Correcting for model-size biases, Gröger et al. (2026) find significant local-neighborhood agreement but little global convergence.Source: §2 · p. 3
  • Distributional semantic models use contextual co-occurrence to represent word meanings, with country-capital relations as an example of similar vector offsets.The study tests comparable geometry for rendered words while controlling spelling similarity, and separately tests alignment between words and named visual categories.Source: §2 · p. 3

3 METHODS

The methods compare word representations from rendered text with matched controls against visual classes and other encoders. They evaluate semantic neighbors, word-to-class alignment, and changes after contrastive finetuning.

3 METHODS

  • The study tests semantic relations among rendered words, compares word directions with named visual categories, then evaluates cross-encoder structure and contrastive finetuning.Finetuning uses written or spoken captions.Source: §3 · p. 3

3.1 MODELS

  • Fourteen vision models include image-only self-supervised models, language-supervised CLIP and SigLIP2, and an untrained ViT-B baseline.The encoder uses its final-layer class token, or SigLIP2’s mean patch tokens, without a projection head or per-image normalization.Source: §3.1 · p. 3

3.2 WORDS AND IMAGES

  • The lexical vocabulary supports relation and category tests, while 200 ImageNet class names test whether rendered names align with their visual classes.Relation lists include synonyms competing with close-spelling words, antonyms, negation pairs, singular-plural pairs, and 60 country-capital pairs; spelling clusters and length-matched shuffles control form effects.Source: §3.2 · pp. 3–4

3.3 WORD AND VISUAL DIRECTIONS

  • Rendered words and matched controls share backgrounds and rendering conditions; subtracting their representations isolates the response to intact lettering.Differences are averaged across backgrounds and centered over the vocabulary to form word directions.Source: §3.2 · p. 5, §3.3 · p. 5
  • Visual directions are class-image means centered over all 1,000 classes, keeping the reference fixed when comparing the 200 named classes.The word and visual directions are compared directly in the same embedding space, without a fitted mapping.Source: §3.3 · p. 5, §3.4 · p. 5

3.4 MEASURES

  • Cosine-neighbor measures assess category purity and partner retrieval, while word-class tests use matched cosine similarity and top-10 retrieval.The methods also measure shared 10-nearest-neighbor structure and linear CKA across vision, text, and speech encoders.Source: §3.4 · p. 5, §3.5 · p. 6

3.5 ALIGNMENT AND FINETUNING

  • Contrastive finetuning matches COCO images to written captions or synthesized speech while excluding images with detected text.After finetuning, the projection is discarded and the same word tests are repeated, so only vision-model weights have changed.Source: §3.5 · pp. 5–6

4 LEXICAL STRUCTURE

DINOv3-H and DINOv3-7B organize rendered words by meaning, while spelling-based retrieval alone does not establish semantic structure. The results distinguish semantic relations from visual similarity in word form.

Semantic relations

  • DINOv3-H and DINOv3-7B group rendered words by meaning: 60% of countries retrieve their capital among the ten nearest neighbors, and nouns and brands cluster by category about as often as in CLIP-B.DINOv2 retrieves capitals for 0% of countries; synonym top-10 accuracy is 10% in DINOv3-7B versus 69–74% in Qwen3 text encoders.Source: §4 · pp. 6–7

Spelling

  • All models retrieve one-edit words, but the untrained ViT-B reaches 33% top-10 accuracy, consistent with pixel and patch similarity rather than general spelling recognition.DINOv2 and MAE retrieve one-edit words readily but retrieve plural and base partners far less often.Source: §4 · p. 7
  • Strong spelling-related retrieval does not predict semantic retrieval: SigLIP2-B reaches 100% for plural or base partners but 18% for capitals, while DINOv3-H reaches 60% for capitals.Qwen3 text encoders show the reverse pattern, with 69–74% synonym accuracy and 21–26% one-edit accuracy; semantic structure is therefore interpreted relative to the untrained ViT-B baseline.Source: §4 · p. 7

5 WORD-CLASS ALIGNMENT

DINOv3-H and DINOv3-7B word directions align with the visual categories named by those words. DINOv3-H’s alignment is substantial despite no training-time alignment signal, and model size alone does not explain the pattern.

Cross-encoder alignment

  • DINOv3-H and DINOv3-7B are most aligned with text encoders in Figure 3, measured by shared nearest neighbors and linear CKA.The figure compares column models with row encoders on the lexical vocabulary; every cell exceeds a null that relabels equal-length words.Source: Figure 3 · p. 8

Named classes

  • For over half of 200 class names, DINOv3-H ranks the named class first among 1,000 visual classes (median rank = 1), and this alignment persists when visual directions use only images with no detected text.Matched cosine similarity is 0.230 with no-text images versus 0.234 with all images; DINOv3-H is on par with CLIP-B despite no alignment signal during training.Source: §5 · pp. 7–8, Table 2 · p. 7

Model size

  • DINOv3-H exceeds DINOv3-7B on matched cosine similarity and word-class retrieval by a factor of 1.7 to 2.0, although DINOv3-7B is no weaker in Table 1.Web-DINO-1B and DINOv2-G show no word-class alignment, so size alone does not determine model behavior.Source: §5 · p. 8

6 ALIGNMENT AND FINETUNING

The paper compares vision-model word representations with text and speech encoders, then tests contrastive finetuning. DINOv3-H gains semantic structure from either supervision, with especially strong retrieval from fewer rendered examples.

Text encoders

  • DINOv3-H and DINOv3-7B share 12–14% of their ten nearest neighbors with text encoders, the highest agreement among self-supervised vision models; linear CKA shows the same pattern.Vision models without detectable semantic structure share at most 5% of their neighbors with text encoders, while Qwen3 text encoders retrieve 21–26% of one-edit partners.Source: §6 · p. 8

Shared word form

  • Agreement with image and speech encoders largely reflects word form: every vision-only model agrees more with CLIP-B and SigLIP2-B than with any text encoder.MAE-H reaches linear CKA of 0.54 with SigLIP2-B, above DINOv3-7B’s 0.48, despite MAE showing no semantic structure.Source: §6 · p. 8

Finetuning

  • Contrastive finetuning on images with no detected text improves synonym retrieval and noun grouping in DINOv3-H, while the other nine finetuned models change minimally.The reported result concerns finetuning; it does not describe equivalent gains across the other models.Source: §6 · p. 8
  • With four rendered images per word, synonym top-10 accuracy rises from about 2% to 11% with text supervision and 8% with speech supervision.After text finetuning, synonym retrieval and noun purity both exceed the pretrained model’s scores using all 96 images.Source: §6 · p. 9

7 DISCUSSION

The discussion bounds the findings to lexical relations and leaves their source unresolved. It also distinguishes representational evidence from downstream behavior, especially when considering finetuning and safety.

7 DISCUSSION

  • Category membership and country-capital associations are clearer than synonymy; the findings establish neither taxonomic hierarchies nor sentence-level composition.Consistent offsets also do not improve analogy retrieval over the country-only baseline.Source: §7 · p. 9
  • The source of semantic structure remains unresolved: surrounding visual content and text embedded in images could both contribute.The images come mainly from public Instagram posts, where embedded text could provide linguistic co-occurrence signals.Source: §7 · p. 9
  • Text finetuning lets four rendered images per word exceed all 96 pre-finetuning images on synonym retrieval and noun purity, but cannot distinguish eliciting existing relations from learning new ones.The evidence concerns representations rather than downstream behavior; cross-encoder convergence is also limited by a null that controls word length but not all shared word-form properties.Source: §7 · p. 9
  • Image-only pretraining can produce lexical and factual associations, and later language finetuning can strengthen some, but these experiments establish neither instruction following nor harmful behavior.The findings motivate testing whether rendered instructions influence downstream systems.Source: §7 · p. 9

8 CONCLUSION

The conclusion reports word-level semantic structure in the largest DINOv3 models despite the absence of explicit language supervision. The structure connects written words with one another and with visual referents, but does not establish sentence understanding or instruction following.

8 CONCLUSION

  • DINOv3-H and DINOv3-7B encode category membership and country-capital associations under spelling controls, with weaker evidence for synonymy.Their word directions also align with representations of the depicted object categories they name and share neighborhood structure with text encoders.Source: §8 · p. 9
  • Contrastive finetuning strengthens synonym retrieval and noun category grouping in DINOv3-H, lowers its capital retrieval, and changes the other finetuned models little.The conclusion reports this pattern for finetuning toward text or speech encoders.Source: §8 · p. 9
  • The findings establish word-level semantic structure, not sentence understanding or instruction following.Image-only pretraining can teach relationships among written words and their visual referents, beyond the appearance of text.Source: §8 · p. 9

A DATA AND RENDERING

The methods construct matched word-image controls and balanced vocabularies, then define retrieval and similarity measures for testing lexical structure and alignment. The section also specifies how rendering, spelling, and candidate relations are handled.

A DATA AND RENDERING

  • Frequency estimates do not distinguish word senses, and some items fall below the one-per-million target; the negation and plural lists also include relation-label exceptions.The negation list is not a uniform test of negation, and some plural pairs have distinct lexicalized senses.Source: A DATA AND RENDERING · pp. 13–14
  • The eight vocabulary lists comprise triples, pairs, and individual words or pseudowords; each item appears in one list, and triples define two pairs sharing an anchor.The lists were mainly sampled from WordNet, with screening for repeated words, rendering fit, morphology, letter overlap, and balanced category counts within spelling clusters.Source: Table 3 · p. 13, A DATA AND RENDERING · p. 13
  • Noun and brand categories are balanced across four spelling clusters, but this does not remove associations between categories and individual spelling features.Meaning pairs are screened against inflectional and derivational relatedness, and their average letter overlap is close to that of length-matched random pairs.Source: A DATA AND RENDERING · pp. 13–14
  • Words are rendered on text-free backgrounds under varied typography, with each word image paired against a phase-scrambled control sharing its background and layout.The 96 backgrounds each use one of 24 settings; the control replaces the letter opacity mask while preserving the background, bounding box, position, color, and panel.Source: A DATA AND RENDERING · p. 14, Figure 4 · p. 15

B.1 MEASURES

  • Neighbor purity measures the share of a word’s ten nearest same-set candidates that share its category or industry, while partner retrieval ranks a designated partner in the full vocabulary.Partner retrieval reports top-10 accuracy and median rank; the main lexical results query from the first word of each pair.Source: §B.1 · p. 16, B MEASURES AND STATISTICAL TESTS · p. 16
  • Word–class comparisons use word-to-class retrieval, class-to-word retrieval, and matched cosine similarity, with chance levels of 0.010, 0.050, and zero, respectively.Matched cosine similarity compares each class name’s word direction with the visual direction of the class it names.Source: §B.1 · p. 16
  • Mutual 10-NN alignment compares shared nearest-neighbor sets, whereas linear CKA compares centered similarity matrices and weights similarities toward principal components of largest variance.The mutual-neighbor score has expected value 0.006 for unrelated neighborhoods; linear CKA’s chance varies across Figure 3 cells.Source: §B.1 · p. 16

B.2 STATISTICAL TESTS

The statistical tests shuffle relevant labels or assignments, estimate uncertainty across items or resamples, and compare trained models with an untrained baseline. The section also outlines tests of retrieval direction, spelling effects, held-out category grouping, and model depth.

B.2 STATISTICAL TESTS

  • Permutation tests shuffle labels or assignments while holding vectors fixed; the methods use one-sided p-values and apply Holm correction to specified comparison families.Matched cosine uses 2,000 class-to-word reassignments, while the two-clue analysis evaluates all sign-flip combinations across twelve grids.Source: §B.2 · pp. 16–17
  • Alignment across encoders is tested by shuffling equal-length word labels; all observed comparisons exceed their permutation scores and remain significant after Benjamini–Hochberg correction.The tests use 1,000 permutations with the same permutations across Figure 3 comparisons.Source: §B.2 · p. 17
  • Uncertainty estimates vary across words, classes, pairs, or items, using binomial errors, item-level errors, or resampling-based errors as appropriate.The two-clue analysis reports 95% percentile bootstrap intervals from 20,000 resamples of twelve clue grids.Source: §B.2 · p. 17

C LEXICAL STRUCTURE

  • The trained-versus-untrained underline rule flags differences exceeding twice the estimated standard error, but it ignores model pairing and multiple comparisons.The section’s lexical analyses also test query direction and spelling, category grouping on held-out words, and structure across model depth.Source: §B.2 · p. 17, C LEXICAL STRUCTURE · p. 18

C.1 PAIRS

This block examines pair retrieval, category grouping, and how rendered-word directions relate to visual classes across model depth.

C.1 PAIRS

  • Reverse queries lower country-capital retrieval in DINOv3-H and DINOv3-7B but raise it in CLIP-B; synonym retrieval improves when partners are queried in both large DINOv3 models.Within-item comparisons find no vision model favors synonyms over antonyms after correction; one-edit matches may reflect letter-position overlap rather than spelling sensitivity beyond it.Source: §C.1 · p. 18

C.2 GROUPING BY CATEGORY

  • DINOv3-H, DINOv3-7B, CLIP-B, and SigLIP2-B lead vision models in noun and brand neighbor purity and held-out category accuracy, exceeding all length-controlled permutation scores.The untrained ViT-B also shows significant noun neighbor purity, while restricting noun neighbors to the query’s spelling cluster leaves the two largest DINOv3 models above that baseline.Source: §C.2 · pp. 20–21, Table 6 · p. 21

C.3 DEPTH

  • Noun purity rises from middle layers onward in DINOv3-H, DINOv3-7B, CLIP-B, and SigLIP2-B; country-capital retrieval improves later, with capitals ranking first at those depths.DINOv3-L separates from the untrained baseline mainly at the final layer, while other vision models remain close to it throughout.Source: §C.3 · p. 21, Table 7 · p. 22

D.1 THE CLASS NAMES

  • Class names are restricted to unambiguous, frequent, lowercase WordNet names denoting one ImageNet-1k class; 200 names are retained from 224 eligible candidates.Selection excludes words whose frequency derives from proper names and classes with substantial detected writing; judgments were checked with a language model, not independently validated by human raters.Source: §D.1 · p. 23

D.2 THE CLASS IMAGES

  • Class images come from ImageNet-1k validation data, with at least 49 images retained per class; DINOv3’s curation does not report the near-duplicate removal reported for DINOv2.The 96 backgrounds are excluded from the class-image pool.Source: §D.2 · p. 23

D.3 ALIGNMENT AND RETRIEVAL IN FULL

  • In DINOv3-H, within-class image cosine similarity has a median of 0.36; similar competing visual directions, such as corn and ear, can rank above a word’s own class.The diagnostics include image consistency within a class and similarity between each visual direction and its nearest other direction.Source: §D.3 · p. 24

D.4 ALIGNMENT BY DEPTH

  • 0.055 is the maximum pre-final word-to-class top-10 accuracy for every model except SigLIP2-B, which reaches most of its final score one depth earlier.Through relative depth 0.77, matched cosine similarity stays within 0.032 of zero in every model.Source: §D.4 · p. 25, Table 10 · p. 25

D.5 TWO-CLUE ANALYSIS

The analysis tests whether two rendered clues jointly identify a target, then specifies how model and text or speech representations are constructed for comparison.

D.5 TWO-CLUE ANALYSIS

  • Clue gains compare target-versus-competitor margins for both clues intact against each single-clue condition, requiring evidence for both contributions after correction.The analysis uses 96 items from twelve grids; clue renderings are compared with phase-randomized controls on the same backgrounds, and targets are compared only within their grid.Source: §D.5 · p. 26
  • CLIP-B is the only model with corrected significant gains from both clues; DINOv3-H and DINOv3-7B show a narrow-clue contribution, but not a broad-clue contribution distinguishing targets sharing the narrow clue.SigLIP2-B’s intervals exceed zero for both gains but do not pass the corrected test; the reported tests do not establish DINOv3’s gains exceed the untrained ViT-B’s.Source: §D.5 · pp. 26–27
  • The two-clue comparisons concern alignment with rendered target words, not the named objects; author-selected items and few independent grids limit generalization.Word co-occurrence during training and visual differences between pair and single-clue images may also account for gains.Source: §D.5 · p. 27

E.1 MODELS AND REPRESENTATIONS

  • Most models use the final transformer block’s hidden state before final normalization; SigLIP2-B instead uses mean patch tokens, without its attention-pooling head.CLIP-B’s visual projection is also not applied.Source: §E.1 · p. 28

E.2 TEXT AND SPEECH VECTORS

  • Text and speech vectors average each word across shared carriers; speech vectors concatenate four relative-position bins to preserve where sounds fall within a word.The same carrier set is used across the vocabulary to avoid per-word corpus co-occurrence entering the measurement.Source: §E.2 · pp. 28–29

E.3 CONTRASTIVE FINETUNING

The authors contrastively finetune self-supervised vision models against frozen text or speech encoders and assess word-level readouts before and after training.

E.3 CONTRASTIVE FINETUNING

  • Finetuning uses symmetric InfoNCE between projected image representations and frozen caption-encoder vectors, with written captions embedded by Qwen3 and spoken captions by w2v-BERT 2.0.The batch contains 1,056 image-caption pairs; both sides are unit-normalized.Source: §E.3 · p. 29
  • Finetuning raises DINOv3-H’s noun neighbor purity and synonym retrieval in the reported readouts.The figure compares written-caption and spoken-caption runs and also reports split-half agreement between word directions.Source: Figure 6 · p. 30
  • 461,195 captions remain after excluding images with detected text, and validation holds out images together with all their captions.The written and spoken runs use the same images and captions; training lasts up to three epochs, retaining weights with the best validation loss.Source: §E.3 · pp. 29–30
  • 0.10 is the standard deviation of DINOv3-H’s capital top-10 accuracy across readout draws before finetuning, larger than its change after the written run.Each readout uses four word images, with five randomly sampled typography settings and the same draws before and after finetuning.Source: §E.3 · pp. 30–31

Related papers

Loading 2610.13252v1…