Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale

A. Sophia Koepke; Daniil Zverev; Shiry Ginosar; Alexei A. Efros

Source-linked AI summary

Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale

A. Sophia Koepke, Daniil Zverev, Shiry Ginosar, Alexei A. Efros

arXiv:2604.18572v2Apr 20, 2026cs.CVcs.AIcs.LG

TL;DR

The paper asks whether models trained on different modalities converge toward the same representation of reality, an issue made important by claims that language may make other modalities unnecessary. It reevaluates mutual nearest-neighbor alignment across larger galleries and less restrictive correspondence settings, finding that measured convergence is fragile and often reflects only partial semantic overlap. The results suggest that different modalities can produce equally rich but differently organized representations.

Problem
The paper examines whether models trained on different modalities converge to a shared representation of reality, a question relevant to whether modality choice matters.
Method
The paper reevaluates mutual kNN alignment across large, dense galleries, many-to-many correspondences, and multiple modality pairs.
Results
Alignment falls sharply at million-sample scale, decreases further under many-to-many correspondence, reflects coarse rather than fine-grained agreement, and does not consistently increase with newer language models.
Takeaways & Limitations
Cross-modal convergence evidence is conditional, and low mutual kNN agreement can reflect different organization of rich representations rather than weak representations.
Takeaways & Limitations
Mutual kNN does not extend cleanly to many-to-many settings, so reduced scores may reflect metric limitations as well as weaker alignment.

Abstract

from arXiv · show

The Platonic Representation Hypothesis suggests that neural networks trained on different modalities (e.g., text and images) align and eventually converge toward the same representation of reality. If true, this has significant implications for whether modality choice matters at all. We show that the experimental evidence for this hypothesis is fragile and depends critically on the evaluation regime. Alignment is measured using mutual nearest neighbors on small datasets ($\approx$1K samples) and degrades substantially as the dataset is scaled to millions of samples. The same behavior is observed beyond text-image, for text-audio and text-video alignment. The alignment that remains between model representations reflects coarse semantic overlap rather than consistent fine-grained structure. Moreover, the evaluations in Huh et al. are done in a one-to-one image-caption setting, a constraint that breaks down in realistic many-to-many settings and further reduces measured alignment. We also find that the reported trend of stronger language models increasingly aligning with vision does not appear to hold for newer models. Overall, our findings suggest that the current evidence for cross-modal representational convergence is considerably weaker than subsequent works have taken it to be. Models trained on different modalities may learn equally rich representations of the world, just not the same one.

1 Introduction

The paper reexamines whether models trained on different modalities converge to a shared representation, finding that the evidence depends strongly on gallery size, correspondence assumptions, and model generation.

1 Introduction: Alignment drops sharply as the gallery grows from 1024 to millions of samples.The authors argue that denser galleries require finer-grained structural agreement between modalities.
1 Introduction: Vision and language models agree on correct classes but rarely retrieve the same instance in controlled settings.This indicates that remaining alignment is coarse and semantic rather than consistently fine-grained.
1 Introduction: Relaxing one-to-one image-caption pairing to realistic many-to-many correspondence further reduces measured agreement.Multiple valid descriptions or images make mutual nearest-neighbor evaluation less reliable.
1 Introduction: The reported relationship between stronger language models and greater vision alignment weakens for newer models.The earlier trend does not consistently appear in recent models.
1 Introduction: Similar scale-dependent behavior appears for text-audio and text-video alignment, while low agreement does not imply weak representations.Different modalities can learn rich, meaningful structures while organizing information differently.

2 Related Work

Related work studies representational similarity, multimodal correspondence, and the limits of alignment metrics. These studies motivate the paper’s focus on whether apparent convergence survives broader data and evaluation settings.

2 Related Work: The Umwelt perspective challenges a single shared representation by emphasizing modality-specific perceptual worlds.Von Uexküll, Gibson, and Wittgenstein each frame perception or meaning as dependent on an organism’s capacities and world.
2 Related Work: Representational similarity research includes model stitching, universal latent-space proposals, and feature-correlation metrics such as CKA.These approaches examine equivalence, shared structure, or correlations across independently trained networks.
2 Related Work: Multimodal alignment research progressed from paired annotations and shared embedding spaces to transformations between independently trained encoders.Examples include CLIP, linear mappings, anchor-set comparisons, category-level alignment, and unsupervised translation.
2 Related Work: Reported alignment varies with modality similarity, shared versus unique information, encoder design, temporal richness, and text availability.These findings show that alignment is sensitive to both the data and architecture used.
2 Related Work: Global similarity measures can lose convergence evidence after null calibration, whereas local neighborhood similarity may remain more stable under restricted evaluations.Functional agreement can also persist despite differences in internal representations.

3 Experimental setup

The paper measures cross-model alignment with mutual k-nearest neighbors over shared, L2-normalized representations, comparing independently retrieved neighborhoods. The setup uses image and language encoders and reports the mean overlap-based score across samples.

Mutual kNN retrieves each sample’s k nearest neighbors independently in the two representation spaces after L2 normalization.The shared gallery contains feature vectors from two models, such as image and text encoders.
Figure 2 compares ten within-modality nearest neighbors for image and text embeddings in sparse WIT-1024 and dense WIT-1M galleries.Nearest-neighbor structure becomes more semantically refined as gallery density increases.
The per-sample score is the overlap between the two k-nearest-neighbor sets, normalized by k.
The overall mutual-kNN score averages the per-sample overlap, with 1 indicating identical neighborhoods and 0 indicating no overlap.For independent random retrieval, expected overlap approaches chance level as gallery size grows.
The experiments use DINOv2 as the primary vision encoder and OpenLlama3b as the primary language model, extracting representations from all encoder layers.

4 How much do representations align?

Mutual kNN evidence for cross-modal representational convergence weakens sharply under larger, denser galleries and realistic many-to-many correspondences. Remaining agreement reflects coarse semantic structure, while models organize fine-grained information differently across text-image, text-audio, and text-video settings.

4.1 Alignment across dataset scales: 0.135 and 0.058 on 1024 samples fall to 0.008 and 0.001 on LAION-15M for k = 10 and k = 1, respectively, as gallery density demands finer agreement.Nearest neighbors become closer and more semantically similar, increasing the demand for agreement on subtle distinctions.
4.1 Alignment across dataset scales: Alignment remains relatively stable for k=n/100, indicating coarse categorical overlap rather than evidence that representations are organized identically.The stable broad correspondence is consistent with models trained on overlapping web data, but lacks fine-grained convergence signal.
4.1 Alignment across dataset scales: At WIT-1M, the stronger-language-model alignment trend largely vanishes for fixed small k, weakening the claim that language capability predicts visual alignment.The gap between language models narrows substantially at the larger scale.
4.2 What is captured by cross-modal mutual kNN alignment?: On ImageNet, correct-class retrieval improves with gallery density while strict same-instance alignment stays near 11%, showing different within-class organization.At 49 images per class, DINOv2 reaches 46.1% correct-class retrieval and OpenLlama 58.0%, whereas strict alignment remains flat.
4.2 What is captured by cross-modal mutual kNN alignment?: Low mutual kNN alignment does not indicate poor unimodal representations: both models learn structured, high-quality features but arrange fine-grained information differently.At ipc=1, strict alignment is 23.1% while both models retrieve a correct-class neighbor only 11.7%, allowing semantically plausible but technically incorrect matches.
4.3 What happens when the data is not bijective?: Relaxing one-to-one image-caption matching still lowers mutual kNN scores, but the metric cannot distinguish genuine misalignment from multiple valid correspondences.This limitation matters because real multimodal data is inherently many-to-many, unlike the bijective evaluation underpinning earlier evidence.
Cross-modal scope: The same scale-dependent pattern appears for text-audio and text-video: fine-grained alignment drops, coarse alignment remains stable, and stronger-language-model trends are weak.The capability trend is slightly positive for text-video and essentially flat for text-audio.

5 Discussion and future work

The paper argues that cross-modal convergence evidence is conditional: alignment drops at million-scale galleries, weakens under many-to-many correspondences, and does not consistently increase with newer language models. It concludes that modalities may retain rich but distinct representational organizations rather than a single shared representation.

5 Discussion and future work: Alignment drops sharply from small galleries to million-scale datasets, and degrades further when evaluation relaxes one-to-one pairing assumptions.The evidence is highly sensitive to gallery size and correspondence structure.
5 Discussion and future work: The reported relationship between stronger language models and higher vision alignment does not consistently hold for recent models.This weakens the claim that scaling language models necessarily drives increasing cross-modal convergence.
5 Discussion and future work: Low mutual kNN agreement does not imply weak representations: models can learn rich, semantically meaningful structure while organizing fine-grained information differently.The paper characterizes the remaining cross-modal agreement as local and partial rather than universally absent.
5 Discussion and future work: The paper proposes that modalities inhabit distinct but coherent representational “caves,” with alignment that is partial rather than a single Platonic representation.This interpretation preserves the possibility of shared structure while rejecting broad convergence as established.
5 Discussion and future work: Real-world multimodal data are many-to-many, so convergence claims based on bijective pairings may overstate alignment.The paper calls for evaluation methods that better represent non-bijective correspondence structure.

Supplementary Material Back into Plato’s Cave: Examining Cross-modal Representational

The supplementary material introduces additional experiments designed to test whether the paper’s main findings arise from a confounding variable.

Additional experiments test whether the paper’s main findings are caused by a confounding variable.

A.1 Sanity check: does mutual kNN inherently drop at scale even within modalities?

Sanity checks show that mutual kNN does not inherently collapse with gallery size: unimodal alignment remains stable, while the cross-modal drop persists across evaluation choices, captions, and stronger or different model families.

A.1 Sanity check: does mutual kNN inherently drop at scale even within modalities?: Unimodal mutual kNN remains stable across gallery sizes, showing that scale alone does not inherently collapse the metric.OpenLlama alignment stays at 0.59–0.62 and DINOv2 alignment at 0.35–0.45 across scales.
A.1 Sanity check: does mutual kNN inherently drop at scale even within modalities?: The scaling trend persists when alignment is evaluated at k=n/100, indicating stable coarse semantic overlap rather than fine-grained convergence.The WIT-1024 k=10 trend largely matches the WIT-1M k=n/100 trend.
A.1 Sanity check: does mutual kNN inherently drop at scale even within modalities?: Re-selecting layer pairs at every gallery scale leaves the alignment trend unchanged, so fixed layers are not responsible for the observed drop.This remains true with per-scale and per-k layer searches, including null-calibrated scores.
A.1 Sanity check: does mutual kNN inherently drop at scale even within modalities?: Recaptioning images with detailed descriptions raises overall alignment scores but does not prevent their decline as the gallery grows.The recaptioned gallery uses descriptions of around 500 words per image.
A.1 Sanity check: does mutual kNN inherently drop at scale even within modalities?: The alignment degradation persists across stronger models, alternative vision objectives, and multiple language-model families.The pattern is therefore not specific to DINOv2-base, OpenLlama-3b, or one particular training objective or model family.

B Additional ImageNet experiments

Additional ImageNet experiments place the evaluation in a dense retrieval regime and show that individual modality retrieval improves with gallery density while cross-modal mutual kNN alignment remains flat, including for stronger models and k=10.

B Additional ImageNet experiments: The scaling pattern is also observed across different vision objectives and language-model families.The supplementary figures pair OpenLlama with Pixio-style SSL and CLIP encoders, and pair DINOv2 with Gemma and Mistral.
B Additional ImageNet experiments: ImageNet provides a dense retrieval setting comparable to WIT-1M, making it a meaningful test bed for the large-gallery analysis.Even one image per class yields neighbor distances comparable to WIT-1M.
B Additional ImageNet experiments: Individual retrieval improves with gallery density, but cross-modal agreement remains flat even for substantially stronger vision-language model pairs.This preserves the gap between unimodal retrieval accuracy and cross-modal alignment.
B Additional ImageNet experiments: The same pattern holds for mutual kNN with k=10: individual retrieval accuracy rises as the gallery densifies, while cross-modal alignment does not.
B Additional ImageNet experiments: The supplementary analysis uses natural WIT correspondences to complement synthetic many-to-many experiments, including one-to-many datasets of 24,875 image-text pairs.WIT contains naturally occurring repeated captions and images, followed by within-group deduplication and controlled sampling.

C.2 Mutual kNN also decreases on non-synthetic data when the bijective assumption is relaxed

Mutual kNN alignment decreases when realistic many-to-many image-caption correspondences relax the bijective assumption, confirming the effect beyond synthetic data.

Alignment decreases in both retrieval directions as images per caption and captions per image increase from 1 to 5.The evaluation uses DINOv2-base and OpenLlama-3b on WIT-based T2I and I2T datasets, measuring k=1 and k=10 alignment.
With multiple valid correspondences, each modality can retrieve a good match while being less likely to select the same nearest neighbor.
The same alignment decline appears on non-synthetic WIT data, confirming that the pattern is not an artifact of synthetic correspondences.

D.0.2 Does the alignment vs performance trend hold?

The reported alignment–language-performance relationship extends partly to recent models on language-modeling benchmarks but fails to generalize to specialized reasoning benchmarks.

The analysis fits alignment from benchmark performance on 19 base models, then tests extrapolation to 36 heterogeneous recent models across four DINOv2 variants.The new-model population includes pretrained, instruction-tuned, and reasoning-distilled models.
For HellaSwag and Wikitext, R2_avg(new) remains positive at 0.297 and 0.489, respectively.These benchmarks primarily measure next-token prediction quality and commonsense language understanding.
For ARC, GSM8K, MMLU, and LogiQA2, the base-model alignment trend does not hold for recent language models.These benchmarks probe science question answering, arithmetic, general knowledge, and logical reasoning.
R2_avg(new) is negative for the four reasoning benchmarks, ranging from −0.575 on ARC to −1.753 on GSM8K.The fitted relationship predicts recent-model alignment worse than simply predicting the mean.
Recent models stronger than the best base model do not show higher mutual kNN alignment with DINOv2 features; alignment appears to saturate or decrease.

E Experimental setup

The experiments use deduplicated million-scale image-caption galleries, multiple DINOv2 variants and language models, and generated captions whose length affects measured alignment.

The language-model trend analysis evaluates six benchmarks, including Wikitext, HellaSwag, GSM8K, ARC, LogiQA2, and MMLU.
The experimental setup includes WIT-1M and LAION-15M galleries after image and caption deduplication.WIT-1M contains 1 million sampled pairs, while LAION-15M contains 15 million sampled pairs from deduplicated pools.
Image duplicates are removed with perceptual hashing, while identical captions are removed by exact string matching.The pipeline also removes gallery samples matching WIT-1024 query captions to avoid trivial text-based retrieval.
The captioning pipeline uses Gemini-generated descriptions for ImageNet and WIT-1M, with WIT-1M-recap using the 500w variant for efficiency.The WIT-1M-recap set contains captions for 999,971 images averaging 478 words.
Longer generated ImageNet captions yield higher mutual kNN alignment, with approximately 500-word captions near the maximum and the longest captions performing best.The experiments therefore use extreme_long captions averaging approximately 981 words for ImageNet.
The vision evaluation uses four DINOv2 variants spanning ViT-S/14 through ViT-G/14, while the language evaluation covers 55 models across 13 families.

F Additional qualitative results

Qualitative retrieval examples show occasional cross-modal agreement at scale, alongside cases where agreement in a small gallery disappears as the gallery becomes denser.

Additional WIT-1M and LAION-15M examples visualize mutual nearest-neighbor matches between DINOv2 and OpenLlama-3b across gallery scales.Green borders mark cross-modal mutual kNN matches.
Some examples agree in the WIT-1024 gallery but lose agreement as the gallery densifies.