PaperlayerView exact revision on arXiv

Source-linked AI summary

Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data

Dominik Schnaus, Thomas Dagès, Daniel Cremers, Xi Wang, Phillip Isola

arXiv:2610.09411v1cs.LGcs.AIcs.CV

TL;DR

Cross-modal alignment has generally relied on paired observations, while fully unpaired alignment of disjoint embedding sets remained unresolved. This paper uses Wasserstein Procrustes with a coarse geometric initialization to align independently trained spaces. It reports coarse alignment across datasets and domains, with success strongly associated with geometric similarity.

  • Problem

    Cross-modal alignment generally uses paired observations, while fully unpaired alignment of disjoint embedding sets remained an unsolved problem.

    Source: §1 · pp. 1–2

  • Method

    Wasserstein Procrustes jointly infers sample correspondences and an orthogonal map, initialized by matching the embedding spaces’ geometric structure.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Independently trained models often align without pairs across datasets and domains; with ≤100 pairs, the method substantially outperforms baselines and remains competitive with more pairs.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    In many cases, independently learned embedding geometry contains enough information to connect modalities without a multimodal Rosetta Stone.

    Source: §6 · p. 10

    The method cannot reliably determine from unpaired data alone whether two spaces are sufficiently similar, because usual alignment metrics need matching samples.

    Source: §5 · p. 9

Abstract

from arXiv · show

Multimodal representations enable zero-shot classification and retrieval, but aligning independently trained models usually requires large amounts of paired data. Yet, the Platonic Representation Hypothesis suggests that models trained on different modalities may converge spontaneously toward a shared representation geometry. But then, do we even need paired examples for cross-modal alignment? Remarkably, we show that paired examples are unnecessary for coarse cross-modal alignment. Our simple Wasserstein Procrustes method with a coarse geometric initialization aligns two disjoint embedding sets by estimating a single orthogonal map without seeing any pairs. Across datasets, modalities, and unimodal models, we show that we can consistently align independently trained representations without pairs, and standard geometric alignment metrics accurately predict when this is possible. Nevertheless, we can naturally benefit from paired examples. In the very few-pair regime, our method substantially outperforms existing ones, while staying competitive with pair-based methods with more added examples. Finally, we demonstrate that the resulting alignments can enable text-to-image generation without paired examples. These results show that independently trained models often share enough geometry to establish cross-modal correspondence with little or no paired data.

1 INTRODUCTION

Cross-modal alignment has traditionally relied on paired examples, leaving open whether independent embedding spaces can be aligned from geometry alone. The paper introduces a pair-free Wasserstein Procrustes approach and reports coarse alignment across models and datasets, with shared geometry predicting its success.

1 INTRODUCTION

  • Paired observations serve as a cross-modal “Rosetta Stone” in contrastive learning and large-scale multimodal models, while even reduced-supervision approaches use known pairings as anchors.The introduction frames this as the prevailing view that meaningful alignment requires observed correspondences.Source: §1 · p. 1
  • Whether two independently trained modalities can be aligned from embedding geometry alone, without a corresponding pair, remained unresolved for disjoint sample sets.Prior unpaired methods were restricted to small-scale datasets or class-level embeddings.Source: §1 · pp. 1–2
  • Wasserstein Procrustes jointly infers sample correspondences and an orthogonal map, initialized by independently matching the spaces’ geometric structure.The procedure uses only unimodal embeddings, without paired examples, class labels, or shared reference points.Source: §1 · p. 2
  • Across datasets and models, unpaired alignment captures coarse semantic structure, and its quality is strongly predicted by geometric similarity measured with scores such as CKA.The method also extends to scientific and medical domains, supports very few paired examples, and has a proof-of-concept text-to-image generation demonstration.Source: §1 · p. 2

2 RELATED WORK

Prior work establishes that geometry can support unpaired alignment within modalities and that different modalities can share geometric structure. This paper applies those ideas to pair-free cross-modal alignment of disjoint embedding sets.

2 RELATED WORK

  • Prior unpaired-alignment methods use adversarial mappings or jointly optimize correspondences and orthogonal transformations, but largely address representations within the same modality.Related optimization approaches include Wasserstein Procrustes and Gromov-Wasserstein formulations.Source: §2 · p. 3
  • Independently trained models can share concepts and geometric structure across modalities, including coarse neighborhood structure in vision and language representations.Prior studies measured this shared structure on paired representations; this work uses it to map between spaces without pairs.Source: §2 · p. 3
  • Few-pair methods use paired data to connect embedding spaces whose internal geometry is already informative, through relative representations, geometric regularization, or other alignment techniques.These approaches include LiT, ASIF, STRUCTURE, and methods using linear transformations, CKA, spectral representations, or optimal transport.Source: §2 · p. 3
  • Earlier fully unpaired cross-modal studies were limited to small-scale or class-aggregated settings; this work targets disjoint sets without pairs, labels, or shared reference points.The approach initializes by clustering and matching geometric structure, then alternates correspondence estimation with orthogonal Procrustes updates.Source: §2 · pp. 3–4

3 WASSERSTEIN PROCRUSTES WITH GEOMETRIC INITIALIZATION

The method jointly estimates a semi-orthogonal map and a transport correspondence, using coarse geometric matching to initialize a non-convex alignment problem. It refines sample-level matches and map estimates iteratively, while incorporating known pairs when available.

3 WASSERSTEIN PROCRUSTES WITH GEOMETRIC INITIALIZATION

  • Wasserstein Procrustes jointly estimates a semi-orthogonal map and a transport plan between centered, normalized embeddings with no known row or coordinate correspondences.Given one variable, the other is optimized by orthogonal Procrustes or linear assignment, respectively.Source: §3 · p. 4
  • Because the joint optimization is non-convex and random or uniform initialization yields little matching signal, the method initializes correspondence between clusters rather than individual samples.The shared cross-modal geometry is mainly coarse, rather than highly isometric.Source: §3 · p. 4
  • For each repetition, the method clusters both spaces independently and matches cluster centers by the permutation maximizing linear CKA, solving a quadratic assignment problem.It uses multiple MPOpt initializations with a GRASP primal heuristic because ablations identify QAP solution quality as the most important algorithmic choice.Source: §3 · p. 4
  • The matched clusterings yield a low-rank approximation to sample-level Gromov-Wasserstein correspondence without materializing an n × m transport matrix.The initialization relates to preserving intra-space similarities.Source: §3 · p. 4
  • A conditional-gradient step and blockwise linear assignments produce pseudo-pairs, which initialize the map by Procrustes; refinement alternates matching and map updates.Block partitioning avoids full assignment on large datasets, requiring only b × b score matrices.Source: §3 · pp. 4–5
  • Known pairs contribute to both cluster matching during initialization and the Procrustes update during refinement, allowing the same algorithm to interpolate between unpaired and few-pair settings.Paired examples are weighted so their total contribution is comparable to the pseudo-pairs from unpaired data.Source: §3 · p. 5

4 EXPERIMENTS

The experiments examine pair-free alignment across models and datasets and assess how shared geometry relates to alignment quality.

4 EXPERIMENTS

  • Across 7 vision models, 3 language models, and 4 datasets, the method consistently recovers coarse cross-modal structure, including for modalities drawn from different datasets.The experiments assess whether independent modalities can be aligned without pairs.Source: §4 · p. 5
  • Alignment quality depends strongly on shared geometry, which can be measured with common metrics.The experiments ask whether those metrics relate to pair-free alignment success.Source: §4 · p. 5
  • The reported experimental scope spans multiple vision and language models and four datasets rather than a single model pair or dataset.The method’s recovery of coarse structure also includes cross-dataset cases.Source: §4 · p. 5

2. Does our approach generalize to less common modalities for alignment?

The section tests unpaired alignment across modalities and domains, then measures how few paired examples improve it. Shared geometry predicts alignment success, and the resulting mappings capture coarse semantic correspondences, including in text-to-image generation.

4.1 UNPAIRED CROSS-MODAL ALIGNMENT IS POSSIBLE

  • Unpaired alignment transfers semantic information, reaching average zero-shot accuracies of 47.4% on CIFAR-10 (Top-1), 30.1% on CIFAR-100 (Top-5), and 26.5% on ImageNet-100 (Top-5).Generative token pooling performs substantially worse on detailed captioning datasets.Source: §4.1 · pp. 5–6
  • Without pairs, the method outperforms mini-vec2vec on 17 of 21 vision-language model pairs, with average FOSCTTM 0.154 versus 0.223.The evaluation spans vision and language models; Figure 3 reports the alignment comparison on MS COCO.Source: §4.1 · p. 5, Figure 3 · p. 6
  • Aligning MS COCO images with SPC captions yields average FOSCTTM 0.236 versus 0.154 when both modalities come from MS COCO, showing alignment does not require a hidden pairing from one dataset.Most model pairs remain similarly alignable across these disjoint datasets.Source: §4.1 · p. 6

4.2 UNPAIRED ALIGNMENT GENERALIZES ACROSS DOMAINS

  • Across language, biology, and neuroscience, the method is competitive with mini-vec2vec on NQ and achieves lower mean FOSCTTM than specialized baselines on PBMC (0.089 versus 0.121) and NSD fMRI (0.033 versus 0.035).Figure 4 summarizes NQ, PBMC RNA-to-ATAC, and NSD fMRI benchmarks.Source: Figure 4 · p. 7, §4.2 · p. 7

4.3 SHARED GEOMETRY PREDICTS ALIGNMENT ACROSS MODALITIES

  • Across seven scientific domains, CKA explains 69% of alignment-performance variance (R2 = 0.69) and correlates with FOSCTTM at ρs = −0.82.More similar geometric structure predicts easier unpaired alignment, while geometrically inconsistent spaces remain difficult to align.Source: §4.3 · p. 7

4.4 FEW-PAIR ALIGNMENT

  • With up to 20 pairs, FOSCTTM is 14× to 28× lower than the strongest baseline; at zero pairs, CIFAR-10 top-1 accuracy reaches 79.6%.At 50 and 100 pairs, FOSCTTM remains 5.4× and 3.2× lower, respectively; the 79.6% accuracy level is reached by only two baselines, each with 500 pairs.Source: §4.4 · pp. 7–8, Figure 5 · p. 8

4.5 VISUALIZING THE ALIGNMENT WITH TEXT-TO-IMAGE GENERATION

  • Retrieval and PCA visualizations show coarse semantic correspondence, and text-to-image generation produces reasonable unpaired results, often missing fine details.Adding a few pairs sharpens individual objects and scene attributes; the generator is trained on ImageNet-1K images, with paired data entering only through the aligner.Source: §4.5 · p. 8

5 DISCUSSION

Unpaired alignment depends on similar embedding geometry, but it typically recovers coarse semantics rather than fine details and is more constrained by noisy multimodal data.

5 DISCUSSION

  • Figure 6 shows coarse retrieval and class-level alignment, while unpaired generations capture broad scenes and added pairs improve finer details.The visualization uses DINOv2 ViT-B/14 and all-mpnet-base-v2 on MS COCO-train images and SPC captions, with retrieval on MS COCO-val.Source: Figure 6 · p. 9
  • Geometric similarity, rather than shared semantics alone, tracks alignment quality across modalities, but predicting success from unpaired data remains open.The usual alignment metrics require matching samples.Source: §5 · pp. 8–9
  • Coarse semantic structure can align without recovering modality-specific details or exact sample-wise correspondences.Coarse clusters align more consistently than local neighborhoods.Source: §5 · p. 9
  • Noisy web text that does not describe visual scenes may share less semantic structure with vision representations, limiting the approach's applicability.The discussion suggests better filtering and representations might extend alignment to noisier settings.Source: §5 · p. 9

6 CONCLUSION

The conclusion argues that coarse cross-modal correspondences can often be recovered without paired data, with geometric similarity predicting alignment success and paired supervision still useful in low-pair settings.

6 CONCLUSION

  • Independent models can often recover coarse cross-modal correspondences without paired data, including when modalities come from different datasets.The conclusion describes the method as Wasserstein Procrustes with a coarse geometric initialization.Source: §6 · p. 9
  • Alignment success is strongly predicted by CKA, and the conclusion reports gains over existing approaches in the low-pair regime while remaining competitive as more pairs are added.It identifies shared geometry as a condition for opportunities in modalities where paired information is scarce or absent.Source: §6 · p. 10

APPENDIX OVERVIEW

  • The appendix overview reports that cross-modal agreement lies in coarse rather than fine structures, and that generative token pooling helps short captions but hurts detailed ones.It also says the appendix includes generated images from the text-to-image experiment.Source: APPENDIX OVERVIEW · p. 18

A OUR ALIGNER: DETAILS AND ABLATION

A.1 OUR ALGORITHM

  • The algorithm has three stages: initialize a coarse transport, read it out as sample-wise correspondences, then refine with Wasserstein Procrustes alternation.Because random initialization fails in practice, the method uses shared coarse structure for initialization.Source: §A.1 · p. 18

A.2 ABLATION

  • For structured data, clustering and matching from mini-vec2vec gives a much better initialization and final optimum than the compared low-rank GW initializations.The ablation evaluates component choices and hyperparameter robustness using FOSCTTM across five seeds.Source: §A.1 · p. 20, §A.2 · p. 21

A.2.1 COMPONENT ABLATION

The ablations show that cluster-center matching is a strong initialization, and that the QAP solver, read-out, and refinement choices affect alignment quality. The method also performs well as a low-rank Gromov-Wasserstein solver on structured examples.

Initialization

  • Clustering and matching beats all other initializations; MPOpt beats mini-vec2vec’s 2-opt on NQ and MS COCO while remaining competitive on SNARE-seq.On NQ and MS COCO, matching cluster centers makes the problem solvable, and solver quality mainly determines alignment quality.Source: Table 1 · p. 22, §A.2.1 · p. 22

QAP solver

  • MPOpt with GRASP returns the certified optimum through 40 classes and, beyond 40, beats Hahn-Grant given equal runtime.With less runtime, it returns a permutation at least as good as Hahn-Grant’s in a fraction of the time.Source: Figure 7 · p. 22, §A.2.1 · p. 23

Low-rank Gromov-Wasserstein

  • On structured toy examples, cluster-center matching gives better initial and post-mirror-descent Gromov-Wasserstein costs, with its advantage growing as structure increases.The uniform toy example is the exception.Source: Table 2 · p. 23, §A.2.1 · p. 23

Read-out

  • Batched Hungarian matching is best among the three read-outs on all benchmarks, while direct read-out falls behind on NQ.Compared with top-k relative representations, it also removes one hyperparameter.Source: §A.2.1 · p. 23, Table 3 · p. 23

Refinement

  • On MS COCO, both refinements beat no refinement, and the paper’s simpler refinement beats mini-vec2vec; NQ and SNARE-seq are already saturated by read-out.The paper’s batch alternation has similar or better results on all three benchmarks without adding hyperparameters.Source: §A.2.1 · p. 24, Table 4 · p. 24

A.2.2 HYPERPARAMETER ABLATION

The hyperparameter sweeps show that larger settings generally improve performance, often at a speed cost, while the useful scale depends on the dataset. More initialization iterations also reduce variation across seeds.

Number of clusters

  • Increasing cluster count generally improves initialization, but gains saturate at different points across benchmarks; C = 30 balances performance and computational cost.Larger cluster counts can improve the Gromov-Wasserstein approximation while making the QAP harder to solve.Source: §A.2.2 · p. 25

Batch size

  • On NQ and MS COCO, performance saturates around batch size 5,000, while SNARE-seq performs well from 125 and changes little with larger batches.The b = 10,000 default is conservative; small batches cover only a small part of the larger NQ and MS COCO datasets.Source: §A.2.2 · p. 25

Initialization iterations

  • On MS COCO, FOSCTTM falls from 0.148 ± 0.088 at S = 1 to 0.076 ± 0.023 at S = 30 and 0.064 ± 0.006 at S = 100.More initialization iterations generally improve performance and reduce seed spread; at S = 30, FOSCTTM is 0.0003 ± 0.0002 on NQ and 0.216 ± 0.034 on SNARE-seq.Source: §A.2.2 · p. 25

Refinement iterations

  • Refinement improves MS COCO monotonically as R increases, makes no difference on NQ, and improves SNARE-seq only slightly within seed variation.R = 100 works well across all three datasets.Source: §A.2.2 · p. 25

B EXPERIMENTAL DETAILS

Across four vision-language corpora, the unpaired aligner generally beats vec2vec and mini-vec2vec, while geometric similarity predicts alignment quality. The appendix specifies the experimental protocol, model and dataset coverage, and evaluation measures.

B EXPERIMENTAL DETAILS

  • The appendix gives experiment models, data, splits, and metric parameters; FOSCTTM is evaluated on paired validation samples, with 0 for perfect alignment and 0.5 for random correspondence.The text-to-image setup maps caption embeddings into image space before diffusion generation; the diffusion model is trained on image data and never sees captions.Source: B EXPERIMENTAL DETAILS · pp. 25–27
  • Training sets are disjoint halves of a randomly permuted corpus; cross-dataset runs draw each modality independently, and most experiments average five seeds.The aligner uses C = 30 clusters, S = 30 restarts, batch size 104, and 100 refinement iterations; geometry scores and text-to-image retrieval use one seed.Source: B EXPERIMENTAL DETAILS · p. 25
  • The vision-language grid covers seven vision models, three language models, and four corpora, with MS COCO validation used for retrieval and cross-dataset evaluation.Other experiments cover Natural Questions, PBMC, fMRI, eleven additional modality pairs, and zero-shot classification on CIFAR-10, CIFAR-100, and ImageNet-100.Source: B EXPERIMENTAL DETAILS · pp. 26–27

C.1 UNPAIRED CROSS-MODAL ALIGNMENT

  • Across four corpora, our aligner outperforms vec2vec and mini-vec2vec on average, and most vision-language model pairs align well without pairs.Generative token pooling performs substantially worse on detailed-caption datasets; iBOT ViT-B/16 also generally performs worse than other vision models.Source: §C.1 · p. 28, Figure 9 · p. 29
  • CKA explains the most alignment variance across four corpora (R2 = 0.73), followed by mutual k-NN (0.40), TSI (0.37), and QSI (0.18).All four measures predict alignment, but differ in how tightly they track it and rank model pairs.Source: §C.1 · p. 28, Figure 14 · p. 33
  • In the cross-dataset setting, MPNet and Qwen3-Embedding-8B reach zero-shot accuracy comparable to single-dataset alignment, while generative token pooling falls to chance.Images come from MS COCO and captions from Stanford Paragraph Captioning; Figure 13 reports the zero-shot accuracy.Source: §C.1 · p. 28, Figure 13 · p. 32

C.2 TEXT-TO-IMAGE GENERATION

The generation experiments use an aligner to connect text embeddings to an image-only diffusion pipeline, and the aligner outperforms a linear map across reported measures. Domain-specific evaluations also show strong alignment on language and PBMC, with similar average fMRI performance and lower variability.

C.2 TEXT-TO-IMAGE GENERATION

  • The generation grids vary the number of known image-text pairs while holding captions, seed, and diffusion system fixed; the linear map is arbitrary with zero pairs.The diffusion model and decoder never see text, so the language-to-image map is the changing component; the grids show coarse semantic scenes without pairs and finer detail with added pairs.Source: §C.2 · p. 33, Figure 15 · p. 35, Figure 16 · p. 36
  • On CyclePrefDB, our aligner scores higher than the linear map on all four measures, for every pair count and both text encoders.With MPNet and no pairs, CLIPScore is 0.466, VQAScore 0.458, and TIFA 0.552, exceeding the linear map with 100 pairs at 0.411, 0.334, and 0.507, respectively.Source: Table 6 · p. 34, §C.2 · p. 34
  • On MS COCO, our aligner again beats the linear map on all four measures, but added pairs help more; MPNet TIFA rises from 0.589 without pairs to 0.686 with 100 pairs.With Contriever and no pairs, the aligner scores below the linear map with 100 pairs, including TIFA 0.435 versus 0.521.Source: §C.2 · p. 34, Table 7 · p. 39

C.3 UNPAIRED DOMAIN-SPECIFIC ALIGNMENT

  • On PBMC, our aligner reaches FOSCTTM 0.089±0.003 versus SCOT+ at 0.121±0.027, with label transfer accuracy of 94.2% versus 91.9%.The results are for RNA → ATAC and the nearest cross-modal neighbor; variability is also reported as an order of magnitude smaller for ours.Source: Table 9 · p. 39, §C.3 · p. 40
  • Across the ten Natural Questions encoder pairs, our aligner reaches FOSCTTM 0.0000 on nine pairs and 0.0002 on the tenth.On the gtr-to-gte pair, ours gives mean rank 1.1 ± 0.0, while mini-vec2vec gives 20.6 ± 36.5; the table reports per-pair results.Source: §C.3 · pp. 39–40, Table 8 · p. 39
  • Across 56 fMRI subject pairs, average FOSCTTM is similar for our aligner and the per-subject encoders, at 0.033 ± 0.025 and 0.035 ± 0.041, respectively.Our method varies less across pairs and seeds.Source: §C.3 · p. 40, Table 10 · p. 41

C.4 SHARED GEOMETRY ACROSS MODALITIES

Across fourteen modality pairs, unpaired alignment falls into near-solved, partial, and chance-level regimes that track shared geometry. When both modalities use the same encoder, centering and normalization alone often suffice.

C.4 SHARED GEOMETRY ACROSS MODALITIES

  • Four pairs align almost exactly, five recover at least some structure, and six are near chance; the aligned groups have higher geometric similarity than chance-level pairs.The well-aligned cases include Natural Questions, PBMC, and SNARE-seq despite different encoders or assays; the partial group remains far from identifying individual samples.Source: §C.4 · p. 42, Table 12 · p. 44
  • Across domains, CKA and TSI track alignment better than mutual k-NN, which is sensitive to differing dataset sizes and misranks SNARE-seq.SNARE-seq is the fifth best-aligned pair at 0.209 despite mutual k-NN of 0.043, lower than some chance-level pairs.Source: §C.4 · p. 42
  • When one encoder covers both modalities, the spaces share a basis by construction; centering and normalizing alone outperform the aligner on four of five pairs.The aligner still recovers coarse maps, but the authors identify centering and normalization as the method of choice in this shared-encoder setting.Source: §C.4 · p. 42, Table 11 · p. 44
  • Figure 19 visualizes good, partial, and failed alignments using union-projected PCA, unseen labels, nearest cross-modal neighbor agreement, FOSCTTM, and similarity scores.Its examples include a cell-line match with one label swap, partially recovered lineages, and a case where nothing is recovered.Source: Figure 19 · p. 43

D ALIGNMENT WITH GENERATIVE TOKEN POOLING

Generative token pooling is more helpful for short captions and less helpful for long, detailed ones. Comparisons on the same images with shortened captions support caption detail as a relevant condition for its alignment advantage.

D ALIGNMENT WITH GENERATIVE TOKEN POOLING

  • Generative pooling asks Qwen3-8B to generate a visual description from the caption and pools generated-token states rather than caption-token states.The experiment compares it with same-backbone mean pooling and Qwen3-Embedding-8B across captions of varying lengths.Source: D ALIGNMENT WITH GENERATIVE TOKEN… · p. 44
  • The authors hypothesize that generation adds visual detail missing from short captions but adds little to long captions that already contain it.Source: D ALIGNMENT WITH GENERATIVE TOKEN… · p. 44
  • Generative pooling exceeds mean pooling by +0.099 on MS COCO, +0.107 on CC12M, and +0.211 on WIT, but trails by −0.144 on DOCCI and −0.067 on DCI.The difference is near zero on Stanford Paragraph Captioning at +0.014; Figure 20 compares the three pooling strategies across six datasets.Source: D ALIGNMENT WITH GENERATIVE TOKEN… · p. 45, Figure 20 · p. 45
  • On the same DOCCI and DCI images, shortening captions shifts generative-minus-mean alignment from negative to positive.On DOCCI, the difference moves from −0.108 for full descriptions to +0.087 for the first sentence; on DCI, it moves from −0.071 for full captions to +0.131 for short human-written captions.Source: D ALIGNMENT WITH GENERATIVE TOKEN… · p. 45, Figure 21 · p. 45

E GRANULARITY OF ALIGNMENT

To compare alignment granularity under comparable evaluation conditions, the section examines cluster structure, principal components, and similarity ranges. Across these tests, shared alignment is strongest at a coarse scale rather than in fine-grained or extreme-similarity structure.

E GRANULARITY OF ALIGNMENT

  • The experiments use DINOv2 ViT-B/14, two language models, and MS COCO and WIT, reporting means and standard deviations across 10 random seeds.Alignment is measured with CKA, mutual k-NN, TSI, and QSI.Source: E GRANULARITY OF ALIGNMENT · p. 46
  • Across datasets, models, and granularities, alignment across cluster centers is substantially stronger than alignment within clusters, indicating stronger agreement on coarse organization.Random sampling is usually closer to fine alignment than to cluster-center alignment.Source: E GRANULARITY OF ALIGNMENT · p. 46, Figure 22 · p. 47
  • Cosine similarities from 0.0 to 0.6 carry most alignment, while very close and very distant points contribute little.Clipping similarities above 0.6 has limited effect, as does moving negative similarities to zero for CKA and mutual k-NN.Source: E GRANULARITY OF ALIGNMENT · p. 47, Figure 24 · p. 49
  • The first ten PCA dimensions capture most observed alignment; mutual k-NN needs around 100 components to approximate full-space alignment.Random subspaces require substantially more dimensions to reach comparable alignment.Source: E GRANULARITY OF ALIGNMENT · pp. 46–47, Figure 23 · p. 48
Loading 2610.09411v1…