PaperlayerView exact revision on arXiv

Source-linked AI summary

To Align or Not To Align: Check Your COMPASS Before You Train

Fabian Gröger, Petar Damjanović, Maria Brbić

arXiv:2610.18638v1cs.AIcs.CLcs.LG

TL;DR

Selecting compatible model pairs for alignment or merging is costly because compatibility is usually known only after transfer. COMPASS estimates compatibility beforehand from pair-specific ease and paradigm-level ceiling; it correlates with retrieval and merge outcomes across the evaluated tasks.

  • Problem

    Pair compatibility is typically assessed only after expensive alignment or merging, although choosing which models to combine is a critical decision.

    Source: §1 · p. 1

  • Method

    COMPASS combines pair-specific pre-transfer local-neighborhood agreement with a paradigm-level estimate of achievable agreement to rank candidate pairs.

    Source: §1 · pp. 2–3

  • Results

    COMPASS correlated with Flickr30k and COCO retrieval R@1 at 0.90 and 0.91, versus global CKA at 0.63 and 0.68, and with mean merge retention at 0.70 versus 0.25.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Using forward passes on a small probe set, COMPASS ranks candidate pairs to help allocate compute or reconsider the pretraining paradigm.

    Source: §6 · p. 10

    The finite-time theoretical guarantee assumes at least one model reaches its target neighborhood within tolerance ξ after T steps with probability at least pT.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Combining pretrained models is a common strategy for building AI systems, whether by aligning a vision encoder with a language model or merging fine-tuned checkpoints. However, finding compatible pairs typically requires costly empirical search, as compatibility is known only after alignment or merging. We propose COMPASS (COMPatibility ASsessment Score), a theoretically motivated diagnostic that estimates whether a model pair will be compatible prior to alignment or merging. Building on the observation that representational convergence is most robust at the level of local neighborhoods, COMPASS decomposes model compatibility into two cheap-to-compute axes: a pair-specific ease, defined as the $k$-NN overlap between the two encoders, and a paradigm-level ceiling, capturing the maximum overlap their training paradigms allow. Together, these two axes provide a ranking of candidate model pairs and prescribe how to act on each pair: (i) prioritize pairs that already agree and whose paradigms are compatible, (ii) invest additional compute on pairs with compatible paradigms but low initial agreement, (iii) switch the pretraining paradigm when it cannot support sufficient agreement, or (iv) deprioritize pairs that score poorly on both axes. We apply COMPASS to $72$ cross-modal alignment experiments and $392$ task-arithmetic model merges and show that compatibility estimates by COMPASS strongly correlate with downstream success, achieving average Pearson correlation of $0.91$ on cross-modal alignment tasks and $0.70$ on model merging.

1 Introduction

Choosing compatible model pairs is central to alignment and merging, yet their compatibility is usually known only after costly transfer runs. COMPASS estimates compatibility beforehand from local-neighborhood agreement and evaluates candidate pairs on pair-specific ease and paradigm-level ceiling.

1 Introduction

  • Compatibility is the key pair-selection decision, but prior transferability metrics select single models and pair compatibility is typically assessed only after expensive alignment or merging.A probe-based diagnostic could make evaluation cost scale with the number of models rather than the number of pairs.Source: §1 · p. 1
  • The paper builds on evidence that models often agree on local neighborhoods even when their global representation geometries differ.The Aristotelian Representation Hypothesis is described as emphasizing local rather than global agreement.Source: §1 · p. 1
  • COMPASS combines pair-specific initial local-neighborhood agreement, or ease, with a paradigm-level estimate of achievable agreement, or ceiling.The paper motivates these axes with analyses of finite-budget convergence and limits imposed by training paradigms.Source: §1 · pp. 2–3
  • Multiplying ease and ceiling ranks candidate pairs, while median-thresholded axes assign them to act-now, budget-limited, ceiling-limited, or incompatible regimes.The regimes suggest prioritizing strong pairs, investing more in budget-limited pairs, switching paradigms for ceiling-limited pairs, and deprioritizing pairs weak on both axes.Source: §1 · p. 2, Figure 1 · p. 2
  • COMPASS predicts retrieval and merge outcomes: alignment correlations are 0.90 on Flickr30k and 0.91 on COCO, while merge retention correlates at 0.70.Global CKA correlations are 0.63 on Flickr30k, 0.68 on COCO, and 0.25 for mean merge retention.Source: §1 · pp. 2–3

2 Related work

Prior work describes representational convergence as more evident in local neighborhoods than in global geometry. This paper applies local agreement to pair selection for alignment and merging, and relates training objectives to the neighborhood structures models learn.

Representational convergence

  • Representational convergence has been observed across independently trained models, but evidence at foundation-model scale favors local neighborhood alignment over global agreement.The paper positions its contribution as applying trained-model local agreement to selecting pairs for alignment and merging.Source: §2 · p. 3

Structure of training objectives

  • Training objectives shape representation geometry by specifying conditional relationships, so similar objectives may induce similar neighborhoods while different objectives may induce different ones.The I-Con framework unifies contrastive, non-contrastive, and clustering objectives through KL divergence over conditional distributions.Source: §2 · p. 3

Multi-modal alignment

  • Multimodal alignment seeks a shared space for semantically corresponding inputs, but image and text embeddings can retain a modality gap after contrastive training.This paper complements work on alignment methods and paired-data reduction with a pre-alignment diagnostic for choosing frozen encoder pairs.Source: §2 · p. 3

Weight-space transfer and model merging

  • Model merging reuses capabilities by combining weights, but incompatible checkpoints can produce an interpolated model that preserves neither model well.Existing approaches typically discover mergeability only after merging and evaluating; this work instead measures checkpoint neighborhood agreement beforehand.Source: §2 · p. 3

3 Preliminaries

The paper operationalizes local representational agreement as k-NN overlap on a probe set. It also represents neighborhoods as distributions, with smoothing to support its information-theoretic analysis.

Neighborhood overlap

  • Local k-NN overlap measures the fraction of nearest neighbors shared by two encoders for each probe sample, and the paper reports stronger correlation with transfer than global agreement such as CKA.The dataset-level average is mutual k-NN overlap; k is fixed at 10, with results reported unchanged at k values 20, 50, and 100.Source: §3 · p. 4

Setup

  • The setup uses a probe set of N samples embedded by two encoders, which may have different embedding dimensions; the theory is agnostic to probe-set choice.For each sample, each encoder's neighborhood consists of its k nearest other samples, with ties resolved deterministically.Source: §3 · p. 4

Neighborhood distribution

  • Each hard neighborhood distribution assigns uniform mass to the k nearest neighbors, and smoothing gives full support over non-self samples for KL calculations.The smoothed distribution mixes the hard neighborhood distribution with a uniform distribution over non-self indices.Source: §3 · p. 4

4 Theoretical motivation

The theoretical motivation links initial local overlap to finite-budget convergence and target discrepancy to a ceiling on achievable overlap. Across four settings, start-condition overlap is associated with the corresponding endpoint.

4.1 Pair-specific ease: per-sample start-condition overlap

  • Start-condition overlap tracks endpoints across four settings: vision r = 0.69 (n = 21), text r = 0.91 (n = 42), Flickr30k I2T R@1 r = 0.84 (n = 72), and merge quality r = 0.67 (n = 392).Figure 2 presents the unimodal, cross-modal alignment, and task-arithmetic merging comparisons.Source: Figure 2 · p. 5
  • Theorem 4.1 connects per-sample start-condition overlap to finite-time convergence under assumptions on the training trajectory.Higher initial overlap reduces the inter-model gap and tightens the finite-time guarantee, making pairs easier to align within a fixed training budget.Source: §4.1 · p. 5

4.2 Paradigm-level ceiling: target discrepancy

  • Target discrepancy captures how differently two paradigms define neighbors; even perfect optimization cannot remove this mismatch, which bounds achievable overlap.The mean overlap is bounded above by 1 − γ̄/(1 − ε) in the limit of perfect optimization.Source: §4.2 · p. 6

4.3 Implications from combining ease and ceiling

  • Start-condition overlap tightens the finite-time guarantee for reaching a common neighborhood structure, while differing targets add an obstruction to reaching each model's own target.When targets match, the triangle inequality bounds both models' distance to the shared target; when they differ, target discrepancy enters the bound.Source: §4.3 · p. 6

5 Cheap-to-compute scores for downstream problems

COMPASS ranks candidate model pairs using pair-specific neighborhood agreement and a paradigm-level ceiling, both measured before transfer. Across alignment and merging, the score tracks downstream outcomes and its phase diagrams distinguish different pair regimes.

5 Cheap-to-compute scores for downstream problems

  • For the 19 × 10 alignment grid, COMPASS requires 29 probe passes instead of 190 downstream alignment runs, with more than two orders of magnitude lower wall-clock cost in this setup.The alignment evaluation covers 72 vision-language pairs; merging evaluates 392 task-pair merges from 14 pretrained ViT backbones spanning five paradigms.Source: §5 · p. 7, §5.1 · p. 8, §5.2 · p. 9
  • COMPASS multiplies pair-specific ease by paradigm-level ceiling, ranks pairs, and uses median-split quadrants to distinguish act-now, budget-limited, ceiling-limited, and incompatible regimes.Both scores require only forward passes on a small probe set; the recommended actions include investing more for budget-limited pairs and switching paradigms for ceiling-limited pairs.Source: §5 · p. 7
  • |r| = 0.90 on Flickr30k and 0.91 on COCO for Ease × Ceiling predicting alignment R@1, compared with 0.84/0.83 for ease and 0.85/0.83 for ceiling alone.These are absolute Pearson correlations; Figure 3 compares candidate predictors with downstream outcomes.Source: Figure 3 · p. 7, §5 · p. 7
  • |r| = 0.70 for Ease × Ceiling predicting task-arithmetic merge quality, compared with 0.67 for ease and 0.64 for ceiling alone.The outcome is mean per-task accuracy retained after weight averaging across 392 merges.Source: Figure 3 · p. 7, §5 · p. 8

5.1 COMPASS for cross-modal alignment

  • Flickr30k mean I2T R@1 separates alignment regimes: act now reaches 0.46 ± 0.09, ceiling-limited 0.34 ± 0.05, budget-limited 0.30 ± 0.06, and incompatible 0.21 ± 0.11.The same ordering holds on COCO; Figure 4 plots ceiling against ease, with color and size encoding retrieval performance.Source: §5.1 · p. 8, Figure 4 · p. 8
  • Kruskal–Wallis tests reject equal alignment-regime R@1 distributions on Flickr30k (H = 45.3, p < 10−9) and COCO (H = 51.5, p < 10−10).Contrastive vision-language models and the largest DINOv2 variants populate act now; MAE and the smallest ImageNet-21K and DINOv2 models occupy incompatible.Source: §5.1 · p. 9
  • COMPASS remains predictive within each of six vision families (median Spearman ρ = 0.80) and across trained Linear, MLP, and LoRA-style adapters on 24 pairs.Across 18 alignment pairs, ceiling's correlation with performance rises from ρ = 0.79 after 10 epochs to 0.92 after 300 epochs.Source: §5.1 · p. 9

5.2 COMPASS for task-arithmetic model merging

  • Mean retention separates merging regimes: act now reaches 0.84 ± 0.16, budget-limited 0.57 ± 0.26, ceiling-limited 0.56 ± 0.23, and incompatible 0.35 ± 0.17.The phase diagram uses pre-merge k-NN overlap for ease and paradigm specialization for ceiling.Source: Figure 5 · p. 9, §5.2 · p. 10

6 Discussion

The discussion frames compatibility as a pre-transfer estimate based on local agreement and paradigm compatibility. It also cautions that phase labels summarize rather than replace continuous scores, while emphasizing their potential to guide compute allocation.

6 Discussion

  • Contrastive vision-language models favor cross-modal alignment but are unfavorable for task-arithmetic merging, while discriminative and self-distillation models show the reverse pattern.The relevant paradigm depends on the transfer operation rather than being universally good or bad.Source: §6 · p. 10
  • Leaving out individual model families or paradigms preserves correlations of 0.85–0.93 for alignment and 0.60–0.72 for merging on the remaining pools.Moderate ceiling noise or a shared ceiling per paradigm leaves rankings largely unchanged, whereas shuffling paradigm order removes this benefit.Source: §6 · p. 10
  • The four phase regimes are an interpretable summary, not strict categories; the continuous ease × ceiling score is more informative for finer-grained comparisons.Source: §6 · p. 10
  • Ranking pairs from forward passes on a small probe set can help practitioners direct compute toward promising combinations instead of aligning or merging every possible pair.The two-axis diagnostic also indicates whether to invest further in a pair or reconsider its pretraining paradigm.Source: §6 · p. 10

A Limitations

COMPASS’s scope and ceiling proxy impose practical boundaries, while the appendix specifies the model libraries, measurements, and alignment protocol used in its experiments.

A Limitations

  • The ceiling proxy is application-specific, requires domain knowledge, and cannot be reduced to one universal formula.The experiments instantiate it separately for cross-modal alignment and task-arithmetic merging.Source: A Limitations · p. 14
  • The phase diagram predicts efficiency under finite budgets, not impossibility; more data, larger adapters, longer training, or a different objective may still yield strong performance.The dynamical assumptions are sufficient rather than necessary, and may give loose bounds when optimizers, data distributions, or objectives differ substantially.Source: A Limitations · p. 14

B.1 Model libraries and measurement protocol

  • Three separate model libraries support convergence, cross-modal alignment, and merging experiments; measurements use calibrated mutual k-NN overlap and linear CKA on probe-set features.Features undergo quantile clipping at q = 0.95, with layer combinations selected to maximize calibrated overlap per pair.Source: §B.1 · p. 14

B.1.1 Convergence library

  • The convergence library spans 20 vision models across six paradigms and 17 language models across seven families, with three random initialization variants per model.The language models include varied architectural choices, enabling within- and across-family comparisons.Source: §B.1.1 · p. 14

B.1.2 Alignment library

  • The alignment library evaluates 72 pairs from 19 vision models and 10 text encoders, using a matchmaking design to sample the 19 × 10 grid.DINOv2-Giant is excluded because aligning its 1.1B-parameter encoder is too expensive for the per-pair adapter sweep.Source: §B.1.2 · p. 15

B.1.3 Merging library

  • The merging library contains 14 ViT backbones across five paradigms, with DINO and DINOv3 substituting for DINOv2 because of checkpoint patch-size compatibility.The substitution is operational: available DINOv2 timm base checkpoints use patch14, while the other merging base models use patch16.Source: §B.1.3 · p. 15

B.1.4 Operational computation of ease and ceiling

  • Operationally, ease is pair-specific k-NN overlap, while each vision encoder or pretrained merging base receives a ceiling shared across its candidate pairs or merges.For alignment, the ceiling is shared across pairs with the same vision encoder; for merging, it is shared across merges from the same pretrained base.Source: §B.1.4 · p. 15

B.2 Cross-modal alignment protocol

  • Cross-modal pairs are aligned using frozen encoders and a closed-form CSA adapter that maps their features into a shared 512-dimensional canonical subspace.For each pair, one vision-language layer combination is selected by mutual k-NN overlap at k = 10 using 5,000 MS-COCO training examples; retrieval is evaluated on COCO validation and Flickr30k test splits.Source: §B.2 · pp. 15–16

B.3 Task-arithmetic merging protocol

The merging protocol produces task-specialized checkpoints and evaluates pairwise weight averaging across eight classification tasks. The appendices also summarize pipeline hyperparameters, compute, datasets, and broader impact.

B.3 Task-arithmetic merging protocol

  • Across 14 backbones, the protocol fine-tunes one checkpoint per each of eight classification tasks and evaluates all within-model task pairs, yielding 392 pairwise merges.Pairwise merging averages the two backbone weights and restores each task’s original classifier head at evaluation.Source: §B.3 · p. 16
  • Merge quality is mean retention relative to each task’s fine-tuned baseline, with top-1 accuracy reported on each task’s standard test split.Fine-tuning uses AdamW, cross-entropy, cosine annealing, 20 epochs, and a fixed seed of 42.Source: §B.3 · p. 16

B.4 Hyperparameters

  • The hyperparameter tables summarize the cross-modal alignment and task-arithmetic fine-tuning and merging pipelines, and the authors state they suffice to reproduce both pipelines without source code.The supplied table captions identify their scope but do not provide individual settings.Source: §B.4 · p. 16, Table 1 · p. 16, Table 2 · p. 17

B.5 Compute resources

  • Experiments use eight RTX 3090 GPUs, with a total reported compute budget of approximately 100 GPU-hours.Each run uses one GPU, takes at most 30 minutes, and uses less than 4 GB of memory.Source: §B.5 · p. 17

B.6 Datasets and licenses

  • The datasets are publicly available and used in accordance with their terms; Table 3 lists them together with their licenses.Source: §B.6 · p. 17, Table 3 · p. 17

C Broader impact

  • COMPASS is a measurement framework for selecting model pairs before alignment or merging, intended to reduce wasted transfer compute.The authors report no new pretrained models or scraped data and foresee no direct negative societal impacts beyond those associated with the underlying models.Source: C Broader impact · pp. 17–18

D Why local: empirical evidence

The empirical comparison favors local k-NN overlap over global CKA for tracking alignment as model scale changes. The accompanying theory relates k-NN overlap to total variation between neighborhood distributions.

D Why local: empirical evidence

  • The local-global gap motivates using neighborhood overlap because local neighborhood recovery is a weaker requirement than global distance recovery.This allows local metrics to track alignment trends where global metrics flatten or diverge.Source: D Why local: empirical evidence · p. 18
  • For cross-modal alignment, local k-NN correlations average r ≈0.64 versus r ≈0.37 for global CKA, whose correlations are weaker and more dispersed.The figure plots Pearson correlation between calibrated alignment and partner-model scale for each fixed reference model.Source: D Why local: empirical evidence · p. 18, Figure 6 · p. 18

E.1 Overlap-TV identities

  • The bound in Theorem 4.2 relies on two exact identities connecting k-NN overlap with total variation between neighborhood distributions.Proposition E.1 concerns hard k-NN distributions, and Proposition E.2 concerns their smoothed counterparts.Source: §E.1 · p. 18

E.2 Proof of Proposition E.1

The proof identifies hard k-NN overlap with shared probability mass, then shows how uniform smoothing preserves that overlap relation while providing full support for the analysis.

E.2 Proof of Proposition E.1

  • For hard k-NN distributions, shared neighbors each contribute 1/k to the overlap, so the total shared mass equals the neighborhood-intersection size divided by k.Source: §E.2 · pp. 18–19

E.3 Proof of Proposition E.2

  • Mixing each hard neighborhood distribution with the same uniform background scales the total-variation gap by 1 − ε.The uniform distribution is over non-self indices.Source: §E.3 · p. 19

E.4 Why we ε-smooth hard distributions

  • The ε-smoothing gives conditional neighborhood distributions full support, making KL divergences finite while preserving hard-overlap semantics up to a factor of 1 − ε.The uniform background avoids assuming that distances between embeddings are comparable across views.Source: §E.4 · p. 19

F.1 Dynamical assumptions and proof variables

The appendix separates a finite-time convergence bound under reachability and discrepancy-decay assumptions from an eventual-convergence result requiring an additional basin condition. It also extends the finite-time bound to noisy dynamics, where a noise floor limits the guarantee.

F.1 Dynamical assumptions and proof variables

  • A1 requires a nonzero chance that at least one encoder reaches the target within finite time, while A2 controls the expected inter-encoder gap over time.A2 is problem-dependent and strongest when the encoders share training dynamics; it controls discrepancy, not whether either encoder reaches the target.Source: §F.1 · pp. 19–20

F.4 Convergence under basin condition

  • A3 is additionally required to extend the finite-time bound to eventual convergence; the inclusion applies when at least one encoder is near the target and the encoders are mutually close.The assumption is natural when the shared target is a stable fixed point of the training dynamics.Source: §F.1 · p. 20, §F.4 · p. 21

F.2 Finite-time control of the inter-encoder gap

  • Under A2, Lemma F.1 provides finite-time control of the inter-encoder gap.The proof invokes A2 directly.Source: §F.2 · p. 20

F.3 Two-form statement and proof of Theorem 4.1

  • Under A1–A2, the finite-time bound has equivalent discrepancy and start-condition-overlap forms, and becomes informative when the bound’s threshold is met.The overlap form directly connects the guarantee to the empirical ease quantity; outside the stated regime, the theorem gives no nontrivial guarantee.Source: §F.3 · pp. 20–21

F.5 Extension to noisy finite-time decay

  • Under noisy finite-time decay, the noise floor enters the bound additively, so a nontrivial guarantee requires η to exceed b(T)/pT.Even with perfect start-condition overlap, the bound is reduced by the noise-floor term; setting b(T)=0 recovers Theorem 4.1.Source: §F.5 · p. 21

G Paradigm-level ceiling on overlap

  • This section suppresses the time index in neighborhood distributions, overlaps, and k-NN sets.The convention applies throughout the discussion of the paradigm-level ceiling.Source: G Paradigm-level ceiling on overlap · p. 21

G.1 Proof of Theorem 4.2

The appendix relates target discrepancy to achievable local overlap and examines the assumptions and empirical proxies underlying COMPASS. It reports that initialization structure, larger neighborhood sizes, and an alternative merging probe preserve key qualitative patterns, though the alternative probe weakens the ease signal.

G.1 Proof of Theorem 4.2

  • For learned distributions whose two KL terms vanish, the sequence statement bounds limiting overlap in terms of target discrepancy.Theorem 4.2 applies to full-support targets; the bound remains informative when the target discrepancy is small.Source: §G.1 · p. 22, §G.2 · p. 22

H.1 Initialization structure

  • Vision within-family random overlap is 0.72–0.89 (mean 0.83), cross-family overlap averages ≈0.54, and pretrained-vs-random overlap is 0.09–0.28 (mean 0.19); text shows the same three-zone pattern.For text, within-family overlap is 0.4–0.8, cross-family overlap is ≈0.35, and pretrained-vs-random overlap is 0.2–0.4; Figure 2 confirms that the architectural proxy predicts family-level spread of final overlap.Source: §H.1 · p. 22, Figure 7 · p. 23

H.2 Robustness to neighborhood size

  • At k ∈ {20, 50, 100}, start-condition overlap continues to predict pretrained overlap and downstream transfer across the tested settings.Figures 8–10 report the same qualitative pattern as at k = 10.Source: §H.2 · p. 23, Figure 9 · p. 23, Figure 8 · p. 24

H.3 Probe-set sensitivity for merging ease

  • Replacing the MNIST probe with an 8-task pool weakens the merging-ease signal, but the product predictor remains positively correlated with merge quality.The pool uses 256 images per task across eight fine-tuning tasks, for 2,048 samples total.Source: §H.3 · p. 24, Table 4 · p. 25

H.4 Merging: paradigm-level results and multi-task extension

Across merging, the paradigm ordering persists when all eight task vectors are combined, while COMPASS also ranks and selects candidates across alignment and merging evaluations. Additional analyses test the diagnostic’s robustness across model groups, alignment procedures, ceiling estimates, and representation readouts.

H.4 Merging: paradigm-level results and multi-task extension

  • Discriminative and self-distillation paradigms retain the highest accuracy in full eight-task merging, while contrastive VL collapses; this ordering reverses the cross-modal alignment ordering.The multi-task accuracies are 0.52/0.37 for discriminative, 0.45/0.34 for self-distillation, and 0.10/0.09 for contrastive VL; these are not pairwise mean retentions.Source: §H.4 · p. 25, Figure 11 · p. 25

H.5 Regime-level distribution of downstream outcomes

  • The four COMPASS regimes have different downstream outcome distributions in both retrieval directions and merging, with Kruskal-Wallis tests rejecting equal distributions in all three settings.Regimes split ease and ceiling at their respective medians; the test is rank-based and used because sample sizes are unequal and outcomes are not normally distributed.Source: §H.5 · pp. 25–26, Figure 12 · p. 26

H.6 Ranking and budget model selection

  • COMPASS gives the strongest overall ranking; at k = 20, Flickr30k regret is 2.1% for COMPASS versus 12.7% for CKA, and it finds the best merge after 5 of 392 evaluations.Ceiling and task-vector cosine can select a strong first merge but rank the full pool less well.Source: §H.6.1 · p. 26, Table 5 · p. 26, §H.6.2 · pp. 26–27

H.6.3 Budget evaluation

  • On COCO, COMPASS reaches the candidate-pool optimum after one downstream run, versus random-order expectations of 0.42 at b = 1 and 0.68 at b = 5.Budget performance is the best downstream outcome among the top b ranked candidates, normalized by the pool optimum.Source: §H.6.3 · p. 27, Table 8 · p. 27

H.7 Additional baselines

  • Local k-NN overlap leads the tested representation-similarity metrics, while learned regressors require completed transfer outcomes as supervision and may fail on unseen groups.Within-family ease remains predictive, with median Spearman correlation 0.80; COMPASS requires no fitted parameters and is computed before transfer.Source: §H.7.1 · p. 27, §H.7.2 · p. 28, Table 11 · p. 28

H.8 Generalization across held-out families, backbones, and paradigms; H.9 Robustness to the alignment procedure

  • Alignment rankings remain stable across held-out model groups and trained adapter procedures; merging correlations remain 0.60–0.72 when complete paradigms are left out.Across alignment adapter types, COMPASS remains predictive and exceeds global CKA; the results include CSA-to-LoRA ranking agreement of ρ = 0.91 for I2T.Source: §H.8 · pp. 28–29, §H.9 · p. 29, Table 13 · p. 29

H.10 Cost and robustness of ceiling estimation

  • Coarsening ceiling estimates to one value per paradigm changes correlation negligibly, and COMPASS remains predictive with moderate noise; shuffling paradigm order removes or reverses the ceiling’s benefit.At noise equal to 50% of the ceiling’s empirical spread, correlations are 0.883 for alignment and 0.695 for merging.Source: §H.10 · p. 30, Table 18 · p. 30

H.11 Pair-specific signal within model families; H.12 Sensitivity to training stage within a fixed architecture; H.13 SigLIP text-tower readout

  • Ease predicts outcomes within vision families, ceiling changes track classification fine-tuning, and SigLIP-text results are specific to the tested mean-pooled readout.Within-family median Spearman correlation is 0.80; ease ranks SigLIP-text pairs internally at ρ = 0.75, while their measured overlap and retrieval are lower than for other text-model families.Source: §H.11 · p. 30, Table 19 · p. 31, §H.12 · p. 31

H.14 What the ceiling predicts under increasing training budget

This section examines how COMPASS’s ceiling predicts performance levels and rankings as alignment budgets increase, while noting that the paper reports no human-subjects research and only limited LLM use for writing, editing, formatting, or visualization.

H.14 What the ceiling predicts under increasing training budget

  • The experiment sweeps 10, 30, 100, and 300 alignment epochs across 18 vision-language pairs, using OLMo-1B, OLMo-7B, and BLOOMz-3B text towers with six vision encoders.The OLMo-1B sweep is reported in Table 21.Source: §H.14 · p. 31, Table 21 · p. 32
  • Spearman correlation between outcome and ceiling rises from 0.83 at 10 and 30 epochs to 0.94 at 100 and 300; the ceiling-limited MAE pair stops improving after 100 epochs.By contrast, high-ceiling CLIP and SigLIP pairs improve by 0.020 and 0.015, respectively, from 100 to 300 epochs. The sweep covers Flickr30k I2T R@1 with OLMo-1B held fixed.Source: §H.14 · p. 31, Table 21 · p. 32
  • Across all 18 pairs, outcome-ceiling correlation increases from 0.79 at 10 epochs to 0.92 at 300; at 300 epochs, tower-specific correlations are 0.94, 0.94, and 0.89.The tower-specific values are for OLMo-1B, OLMo-7B, and bloomz-3b, respectively.Source: §H.14 · p. 32
  • The ceiling predicts final plateau rather than budget-to-budget gain, which also depends on how close each pair is to saturation at 10 epochs.For OLMo-1B, ceiling and gain correlate at 0.89 because all pairs start far from saturation; for bloomz-3b, several high-ceiling pairs are near plateau while lower-ceiling MAE has more room, yielding a negative correlation.Source: §H.14 · p. 32

2. Limitations

  • The paper reports that it did not involve human-subjects research.The checklist marks the human-subjects answer as N/A.Source: §2 · p. 38
  • LLMs were not an important, original, or non-standard component of the method; their use was limited to writing, editing, formatting, or visualization and did not affect the methodology, experiments, or claims.Source: §2 · p. 38

Related papers

Loading 2610.18638v1…