PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Vision and Language Encoders Represent the World Similarly?

Mayug Maniparambil, Raiymbek Akshulakov, Yasser Abdelaziz Dahou Djilali, Sanath Narayan, Mohamed El Amine Seddik, Karttikeya Mangalam, Noel E. O'Connor

arXiv:2401.05224v2cs.CVcs.AIcs.CLcs.LG

TL;DR

The paper asks whether unaligned vision and language encoders represent the world similarly and whether they can be matched without training. It uses CKA-based seeded graph matching and local CKA retrieval, reporting semantic similarity and stronger cross-domain and cross-lingual retrieval than relative representations.

  • Problem

    It is unclear whether unaligned vision and language encoders share semantic structure that can support matching without the statistical similarity provided by aligned encoders.

    Source: §1 · p. 1

  • Method

    The paper uses CKA to formulate seeded graph matching as permutation optimization and proposes local CKA retrieval for unaligned encoders.

    Source: §1 · p. 2, §4 · p. 3

  • Results

    Some unaligned encoder pairs have CKA scores comparable to aligned encoders, and local CKA retrieval outperforms relative representations on cross-domain and cross-lingual caption tasks.

    Source: §1 · p. 1, §8 · p. 8

  • Takeaways & Limitations

    The findings support training-free communication between unaligned encoder spaces across cross-domain and cross-lingual retrieval settings.

    Source: §8 · p. 8

    The practical QAP approach assumes access to a base set for seeded matching.

    Source: §4.1 · p. 3

Abstract

from arXiv · show

Aligned text-image encoders such as CLIP have become the de facto model for vision-language tasks. Furthermore, modality-specific encoders achieve impressive performances in their respective domains. This raises a central question: does an alignment exist between uni-modal vision and language encoders since they fundamentally represent the same physical world? Analyzing the latent spaces structure of vision and language models on image-caption benchmarks using the Centered Kernel Alignment (CKA), we find that the representation spaces of unaligned and aligned encoders are semantically similar. In the absence of statistical similarity in aligned encoders like CLIP, we show that a possible matching of unaligned encoders exists without any training. We frame this as a seeded graph-matching problem exploiting the semantic similarity between graphs and propose two methods - a Fast Quadratic Assignment Problem optimization, and a novel localized CKA metric-based matching/retrieval. We demonstrate the effectiveness of this on several downstream tasks including cross-lingual, cross-domain caption matching and image classification. Code available at github.com/mayug/0-shot-llm-vision.

1. Introduction

The paper asks whether independently trained vision and language encoders share semantic structure and whether that structure can support training-free cross-modal matching. It reports comparable CKA scores for some unaligned encoder pairs and proposes matching and retrieval methods that exploit this similarity.

1. Introduction

  • Aligned encoders use contrastive training to increase paired image-text similarity and push negative captions away, enabling cross-modal retrieval by cosine similarity.The passage notes that unaligned encoders lack this statistical similarity, so extra transformations are needed to bridge the gap.Source: §1 · p. 1
  • Prior work mapped vision-encoder outputs to language-model inputs with a trained linear layer, using three million CC-3M image-caption pairs.The cited work reports image captioning and VQA performance.Source: §1 · p. 1
  • Some unaligned vision–language encoder pairs achieve CKA scores on COCO comparable to aligned encoders such as CLIP, indicating semantic similarity.The comparison uses image-caption pairs from COCO.Source: §1 · p. 1
  • The paper proposes seeded matching that maximizes CKA and local CKA retrieval to connect unaligned encoders without training.The methods are evaluated on cross-domain and cross-lingual caption tasks and ImageNet-100 classification.Source: §1 · p. 2

2. Related Work

Prior representation-similarity research largely analyzed unimodal models, while several approaches to connecting unaligned encoders relied on training or anchor-sensitive comparisons. This work instead explores CKA as an optimization objective for training-free cross-modal tasks.

2. Related Work

  • Earlier CKA studies compared representations within vision or language models, while this work seeks to use CKA as an optimization objective.The passage describes CKA as introduced for comparing inner representations across networks [22].Source: §2 · p. 2
  • Relative representations match unaligned embeddings through cosine similarities to anchors but are sensitive to anchor selection and embedding noise [34] [35].Model stitching and LiMBeR use trainable stitching layers to evaluate representation similarity [4] [14] [31].Source: §2 · p. 2
  • The paper proposes an implicit similarity measure that compares similarity patterns, using CKA and parallel image-text anchors for training-free cross-modal tasks.The authors argue this design can make alignment more robust to anchor selection and noise in the original embeddings.Source: §2 · p. 2

3. Preliminaries

The preliminaries define CKA as a comparison between kernels computed from two sets of representations. The measure uses HSIC and a centering matrix.

3. Preliminaries

  • CKA compares two representation sets, X and Y, through their respective N-by-N kernels, which may use linear or RBF kernel functions.The columns of X and Y are the inputs to the kernel functions.Source: §3 · p. 2
  • HSIC is the criterion used to define CKA from the two kernels.The passage introduces HSIC as the Hilbert-Schmidt Independence Criterion [30].Source: §3 · p. 2
  • The centering matrix is C = I − (1/N)11⊤.The text points readers to [22] for broader properties of CKA.Source: §3 · p. 2

4. Proposed Method

The method treats caption-image alignment as finding the image permutation that maximizes CKA, then offers global and localized matching paths using aligned base pairs.

4. Proposed Method

  • CKA is highest for ground-truth image-caption ordering and decreases monotonically as pairs are randomly shuffled, motivating permutation-based matching.Source: Table 1 · p. 3, §4 · p. 3

4.1. QAP Matching

  • QAP casts CKA maximization as a graph-matching relaxation, but its global optimum is NP-hard; the seeded fast solver matches unmatched queries against aligned base pairs.Source: §4.1 · p. 3

4.2. Local CKA based Retrieval and Matching

  • Local CKA scores a candidate query pair by adding it to an aligned base set, supporting single-source retrieval and linear-sum assignment matching.Source: §4.2 · pp. 3–4

4.3. Stretching and Clustering

  • Stretching rescales each feature by its inverse empirical standard deviation before kernel computation, while k-means on image embeddings selects base samples.Source: §4.3 · p. 4

5. Experiments

The experiments compare unaligned vision–language matching methods with baselines across caption matching and retrieval, cross-lingual transfer, and ImageNet-100 classification. They report competitive training-free results, with performance varying by task and encoder.

5.1. Vision and Language Encoders

  • All-Roberta-large-v1 has the best CKA and matching score across the evaluated vision models and serves as the primary language encoder in subsequent tasks.Source: §5.1 · p. 4

5.2. Baselines

  • Baselines include linear regression trained on aligned examples, relative representations based on aligned anchors, and CLIP cosine similarity.Caption matching uses cosine-similarity cost matrices and linear sum assignment.Source: §5.2 · p. 4

5.3. Downstream Tasks

  • The analysis evaluates vision encoders on COCO and NoCaps caption matching and retrieval, multilingual caption matching and retrieval, and ImageNet-100 classification.For ImageNet-100, class text representations are derived from captions based on WordNet lemmas, definitions, and hypernyms, then compared with image representations using local CKA.Source: §5.3 · pp. 4–5

5.4. Results

  • The experiments use COCO base samples for both in-domain COCO and cross-domain NoCaps queries, and compare matching and retrieval performance.Results are averaged over three query-sampling seeds.Source: §5.4 · p. 5, Table 2 · p. 5
  • DINOv2 with Local Kernel CKA achieves top-5 retrieval scores of 70.5% on COCO and 61.8% on NoCaps; ImageNet-100 shows a narrower gap from CLIP.The COCO validation set supplies base samples for NoCaps queries in the cross-domain setting.Source: §5.4 · p. 6, Table 3 · p. 6
  • For cross-lingual caption matching, QAP surpasses CLIP by 12% on average, while local CKA exceeds CLIP by over 17% for retrieval at 5.CLIP retrieval at 5 drops 30 points for Latin-alphabet languages versus English and collapses to zero for non-Latin languages; QAP and local CKA are consistent across languages.Source: §5.4 · p. 6

5.5. Matching complexity

  • QAP has worst-case O(n^3) complexity, while local CKA retrieval is O(n^4); an additive implementation could reduce local CKA to O(n^3).The reported setup uses 320 base and 500 query samples; QAP runs on CPU, with runtimes potentially reducible from 40 seconds.Source: §5.5 · p. 6

6. Analysis

The analysis relates vision–language semantic alignment, measured with CKA and QAP matching accuracy, to vision-encoder training data and supervision. It finds that both dataset scale and training paradigm are associated with alignment.

Data scale and quality

  • More training data generally corresponds to semantically richer visual embeddings, while curated data can outperform uncurated data of the same size, especially for smaller models.Source: §6 · p. 8
  • DINOv2’s CKA and QAP matching accuracy are comparable to CLIP despite no language supervision and less training data: LVD-142 has 142M examples versus CLIP’s 400M.Source: §6 · p. 8

Training supervision

  • With comparable dataset sizes, language-supervised vision encoders show greater semantic alignment than self-supervised encoders; ImageNet class-label supervision also outperforms self-supervision in the reported comparisons.Source: §6 · p. 8

7. Ablations

The ablations show that CKA, stretching, and clustering each improve matching, with the strongest performance when the components are combined.

7. Ablations

  • The ablation compares QAP and local CKA matching/retrieval with main components included or omitted, using COCO validation base and query sets of 320 and 500 samples.When CKA is omitted, normalized correlation matrices represent each graph.Source: §7 · p. 8
  • QAP accuracy rises from 10.1% to 48.8% with CKA, from 48.8% to 57.3% with stretching, and from 57.3% to 65.5% with clustering.The table caption reports the best matching and retrieval performance when clustering and stretching are both employed.Source: §7 · p. 8, Table 6 · p. 8

8. Conclusion

The conclusion reports high semantic similarity between well-trained vision and language encoders, regardless of training paradigm, and uses that similarity to match unaligned encoders. The proposed methods also outperform relative representations on cross-domain and cross-lingual caption matching and retrieval.

8. Conclusion

  • Well-trained vision encoders trained on sufficiently large datasets show semantic similarity to language encoders comparable to aligned encoders, regardless of training paradigm.Source: §8 · p. 8
  • The paper uses CKA and a caption-matching task to study whether vision and language encoders represent the world similarly.Source: §8 · p. 8
  • Seeded graph matching maximizes CKA to align vision and language encoders, alongside a local CKA metric for retrieval between unaligned encoders.Source: §8 · p. 8
  • The methods outperform relative representations on cross-domain and cross-lingual caption matching and retrieval, facilitating zero-shot latent space communication between unaligned encoders.Source: §8 · p. 8

Supplementary Material

Figure A.1 reports accuracy and retrieval scores for QAP matching and local CKA-based retrieval while varying the number of base samples and keeping query samples fixed at 500.

Supplementary Material

A. Varying the Number of Samples

The supplement reports how sample counts affect matching and retrieval, and compares encoder combinations across training paradigms, datasets, and model sizes. It identifies high-CKA encoder pairs and notes that text model size has limited impact on high CKA with well-trained vision models.

A. Varying the Number of Samples

B. Vision and Text Encoders

  • The encoder analysis considers vision and text model combinations across model sizes, dataset regimes, and training paradigms to identify an optimal pair for caption matching.Source: B. Vision and Text Encoders · p. 11
  • All-Roberta-large-v1 paired with DINOv2 achieves a CKA of 0.706; All-Roberta-large-v1 is selected for main-paper tasks except cross-lingual experiments, which use paraphrasemultilingual-mpnet-base-v2.Source: B. Vision and Text Encoders · p. 11
  • Text model size has limited impact on high CKA, while well-trained vision models on large datasets consistently show high kernel CKA across text encoder sizes.Source: B. Vision and Text Encoders · p. 11

C. Layerwise CKA Analysis

Layerwise comparisons show that text–vision similarity and matching vary with encoder depth, with different patterns for CLIP and unaligned encoders. The appendix also examines method robustness, alternative text encoders, projection learning, unimodal task effects, and reverse retrieval.

C. Layerwise CKA Analysis

  • CLIP’s QAP matching accuracy jumps from 0.29 at the last layer to 0.79 after the projection head, while DINOv2–All-Roberta matching improves across successive layers.For DINOv2 with the last text layer fixed, matching accuracy increases with image depth; with the last image layer fixed, text-layer accuracy rises from 0.44 to 0.68.Source: C. Layerwise CKA Analysis · p. 13, Table A.3 · p. 13, Table A.4 · p. 13

D. Mathematical Relationship between Local CKA-based Retrieval and Relative Representations

  • At σ = 0.2, relative-representation matching accuracy drops 6.5% from its maximum of 47.3, while global CKA-based QAP drops 3.9% from its maximum of 53.9.The comparison uses 320 base samples and 500 query samples on COCO validation, with Gaussian noise added to both modalities.Source: D. Mathematical Relationship between… · p. 14, Table A.5 · p. 14

E. Other text encoders

F. Simple projection

  • QAP matching and local-CKA retrieval outperform projection learning, which requires hyperparameter tuning; both proposed methods are training-free.The projection baseline trains a two-layer MLP on frozen DINOv2-large representations.Source: F. Simple projection · pp. 14–15

G. Effect of unimodal tasks on alignment

  • Classification-trained vision models show higher semantic similarity to All-Roberta-large text representations than detection-, segmentation-, and depth-estimation-trained models.The comparison reports QAP accuracy and local-CKA scores for ViT, DETR, DPT, and SegFormer.Source: G. Effect of unimodal tasks on alignment · p. 15, Table A.8 · p. 15

H. Additional Retrieval Results

  • In reverse caption retrieval, matching accuracies change little, but retrieval accuracies show notable discrepancies.This setting retrieves a caption from N candidates given one image, with images shuffled rather than captions.Source: H. Additional Retrieval Results · p. 15

I. Additional Cross-Lingual Matching Results

The reverse cross-lingual experiments compare matching and retrieval across languages and encoder types. Qualitative examples show that local Kernel CKA retrieval can return caption-relevant images even when the original image is not in the top five.

I. Additional Cross-Lingual Matching Results

  • In reverse cross-lingual retrieval, CLIP’s retrieval@5 drops by over 4.5% on average, compared with a 2.1% drop for local CKA-based retrieval.Matching accuracy remains the same when the direction is reversed.Source: §I · p. 15
  • Language-specific BERT sentence encoders show lower CKA alignment and poorer matching and retrieval performance than the multilingual model.The vision encoder is fixed to OpenAI’s CLIP-ViT-L-14, trained on English image-caption pairs.Source: §I · p. 15, Table A.11 · p. 16

J. Qualitative results

  • In qualitative retrieval examples, images returned in the top five can remain semantically similar to the caption even when the original image is absent.The examples use All-Roberta-large-v1 text embeddings and DINOv2 ViT-L/14 image embeddings.Source: J. Qualitative results · p. 15, Table A.12 · p. 17

I. Additional Cross-Lingual Matching Results

  • For cross-lingual image matching and retrieval, QAP and local CKA perform comparably to CLIP on Latin languages and outperform it on non-Latin languages.The experiments use CLIP’s ViT-L vision encoder and a multilingual sentence transformer.Source: Table A.10 · p. 16
Loading 2401.05224v2…