PaperlayerView exact revision on arXiv

Source-linked AI summary

A Systematic Comparison of Multilingual Interpretability Methods Reveals Anisotropy-Driven Failures

Oskar Holmström, Marcel Bollmann, Marco Kuhlmann

arXiv:2609.04819v1cs.CL

TL;DR

Interpretability metrics for multilingual sharing have been developed in isolation, making disagreements hard to distinguish from measurement artifacts. This paper systematically compares four metrics across 21 models and five transfer tasks, finding that only ILO’s association with transfer survives controls, while recommending anisotropy diagnostics alongside sharing scores.

  • Problem

    Existing cross-lingual sharing metrics were developed largely in isolation, leaving unclear whether their disagreements reflect model differences or measurement artifacts.

    Source: §1 · pp. 1–2

  • Method

    The paper systematically compares four sharing metrics across 21 base models from five families and relates them to cross-lingual transfer on five tasks.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    Only ILO’s association with cross-lingual transfer survives all three controls, whereas ANC rests on between-family contrasts and CKA shows no association.

    Source: §4.3 · p. 8

  • Takeaways & Limitations

    Sharing metrics should be reported alongside anisotropy diagnostics because geometric scores cannot be interpreted independently of representation geometry.

    Source: §5 · p. 9

    The reported reliability ranking may change for models with substantially higher anisotropy or anisotropy modes absent from the 21-model study.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Multilingual language models develop shared cross-lingual representations, and various interpretability methods claim to quantify this sharing. These methods have been developed largely in isolation, and when they disagree, it is unclear whether the disagreement reflects a property of the model or an artifact of the measurement. We compare four sharing metrics (CKA, ANC, GMM dominance per token, and ILO) across 21 base models from five families (125M-14B parameters) and correlate each with cross-lingual transfer on five downstream tasks. We find that the metrics differ in their quantification of cross-lingual sharing in these models and suggest that the disagreement traces to anisotropy, the tendency of representations to cluster in a narrow cone of the embedding space. Only ILO's correlation with cross-lingual transfer (Spearman's $ρ= 0.90$) survives controls for model size, family, and per-task variation. We therefore recommend ILO as the primary sharing metric, to be reported alongside anisotropy diagnostics.

1 Introduction

This paper systematically compares four cross-lingual sharing metrics across multilingual models to determine whether their disagreements reflect model properties or anisotropy-driven measurement artifacts. It finds that ILO uniquely tracks cross-lingual transfer after controls and recommends pairing it with anisotropy diagnostics.

Introduction

  • Metric disagreements are difficult to interpret because existing methods were largely developed in isolation across only one or two model families.The paper therefore examines whether conclusions about architectures, training regimes, and scaling choices remain reliable under anisotropy.Source: §1 · p. 1
  • Anisotropy can inflate apparent similarity and make metrics sensitive to dominant low-dimensional directions rather than the full representational geometry.Representations occupy a narrow cone with variance concentrated in a few dimensions, creating risks for geometric measures of cross-lingual sharing.Source: §1 · p. 1
  • Across 21 base models from five families spanning 125M–14B parameters, the study compares four sharing metrics and asks when they agree and which best predicts transfer.The comparison also includes instruction-tuned variants and intermediate checkpoints.Source: §1 · p. 2
  • ILO (Wilie et al., 2025) reaches Spearman’s ρ = 0.90 with cross-lingual transfer, uniquely surviving controls for model size, family, and per-task variation.The paper recommends ILO as the primary sharing metric, reported alongside anisotropy diagnostics.Source: §1 · p. 2
  • ANC (Del and Fishel, 2022) forms a second tier based on between-family contrasts, while GMM dominance per token is too weak within families and CKA (Kornblith et al., 2019) shows no association.The reported disagreement traces to anisotropy, with language identity relocated to directions that some metrics discard or down-weight.Source: Figure 1 · p. 1, §1 · p. 2

2 Related Work

Prior work has proposed diverse metrics for cross-lingual sharing, but these methods are largely developed in isolation and may be affected by representational anisotropy. This paper addresses these gaps by comparing four representative metrics on the same models and examining how anisotropy affects their measurements.

Measuring cross-lingual sharing

  • Cross-lingual sharing metrics include CKA for global geometric similarity, ANC for per-neuron alignment, GMM-based language dominance, and ILO for neighborhood overlap.These span similarity-based, distributional, and neighborhood approaches; neuron-level analyses provide related but distinct perspectives.Source: §2 · p. 2
  • Unlike MEXA and DALI, which measure directed alignment with English, the four compared metrics quantify sharing among all languages.The paper relates these two views in Appendix C.4.Source: §2 · p. 2
  • Because existing methods are usually evaluated in isolation or on limited model families, metric disagreements may reflect measurement artifacts rather than substantive model differences.The paper compares four representative metrics on the same set of models in §4.Source: §2 · p. 2

Representational anisotropy

  • Representational anisotropy concentrates embeddings in a narrow cone, can mask genuine cross-lingual structure, and may distort geometric sharing metrics.Its causes include common shifts, self-attention dynamics, and optimizer effects; the paper studies its impact on sharing metrics rather than whether anisotropy is desirable.Source: §2 · pp. 2–3

3 Experimental Setup

The experiments compare four cross-lingual sharing metrics across 21 base models, using multilingual parallel data, zero-shot transfer, robustness controls, and anisotropy diagnostics. The setup distinguishes measurement families, model configurations, evaluation criteria, and statistical tests.

3 Experimental Setup

  • The main experiments compute four cross-lingual sharing metrics across 21 models and relate them to cross-lingual transfer.Source: §3 · p. 3

3.1 Sharing Metrics

  • The comparison spans global geometry with CKA, neuron correspondence with ANC, distributional structure with λ, and local neighborhood overlap with ILO.CKA compares pairwise distance structures; ANC averages absolute Pearson correlations across neurons; λ measures foreign-space likelihood dominance; ILO combines bridge and reachability scores from k-nearest-neighbor queries with k = 5.Source: §3.1 · p. 3

3.2 Models

  • The model suite contains 21 base models from five families, spanning 125M–14B parameters, with instruction-tuned variants and 59 intermediate checkpoints characterized separately.Transfer correlations use only the 21 base models because English fine-tuning and zero-shot evaluation are intended to assess pretraining geometry without confounding post-training recipes.Source: §3.2 · p. 4

3.3 Data

  • Interpretability analyses use 1,012 FLORES-200 parallel sentences across ten languages, four scripts, and diverse typological and morphological families.CKA and ANC require sentence-level aligned pairs, whereas λ and ILO operate on monolingual representations; λ and ILO use 10,000 tokens per language.Source: §3.1 · p. 3, §3.3 · p. 4

3.4 Cross-Lingual Transfer Evaluation

  • Transfer is evaluated by fine-tuning each base model on English and averaging non-English target-language accuracy across Belebele, XNLI, XCSR, SIB-200, and XQuAD.The transfer score is the unweighted mean of five task-level means, with coverage of up to nine non-English languages varying by task.Source: §3.4 · p. 4

3.5 Statistical Framework

  • Statistical analysis uses Spearman ρ with per-family, per-task, Benjamini–Hochberg, partial-size-controlled, and permutation-based robustness checks.Partial correlations control log10 parameter size, while 10,000-shuffle permutation tests assess both all-model and within-family associations.Source: §3.5 · p. 5

3.6 Additional Diagnostics

  • Effective dimensionality and mean random cosine similarity diagnose low-dimensional, narrow-cone representations where sharing metrics may be biased upward.Effective dimensionality counts PCA components retaining 98% variance; mean random cosine similarity averages 5,000 random hidden-state pairs.Source: §3.6 · p. 5

4 Results and Discussion

Across 21 models, ILO best predicts cross-lingual transfer and remains robust to controls, while metric disagreement reflects anisotropy-driven measurement failures; sharing can emerge abruptly during training or instruction tuning.

4.1 How Do Models Share Representations Across Languages?

  • Across models, concentrated, distributed, and minimal sharing patterns distinguish BLOOM/Qwen3/SmolLM3, XGLM, and OPT, respectively, with XGLM exposing λ’s failure to reflect transfer.XGLM has ILO 0.44–0.40 but near-zero λ, while transfer reaches 0.55 versus 0.60 for BLOOM-7.1B and 0.41 for OPT-6.7B; ANC discriminates better than CKA.Source: §4.1 · pp. 5–6, Table 1 · p. 6

4.2 Why Do Methods Disagree?

  • Anisotropy drives disagreement: variance-ranked PCA reduces high-λ models to two dimensions, discarding language identity into residual directions where probe accuracy reaches 96–99%.At high-λ layers, principal-subspace probe accuracy falls to chance while residual-subspace accuracy reaches 96–99%; thus λ can measure clustering after language information has been removed. Anisotropy harms these measurements, not transfer itself: Qwen3 and SmolLM3 combine npca = 2 and cos = 0.99 with strong transfer.Source: §4.2 · pp. 6–7

4.3 Do Interpretability Methods Predict Cross-Lingual Transfer?

  • ILO is the strongest transfer predictor (ρ = 0.90), surviving model-size, task-level, family-level, and anisotropy-centering checks, whereas ANC, λ, and CKA are less robust.[Figure 4] reports Spearman ρ = 0.90 (p < 0.001) and partial ρ = 0.84; ILO remains strongest without OPT (ρ = 0.78).Source: Figure 4 · p. 8, Table 2 · p. 8, §4.3 · p. 8

4.4 How Does Sharing Emerge During and After Pretraining?

  • Sharing appears through abrupt training transitions rather than gradual growth, with BLOOM’s threshold between 560M and 1.1B and Qwen3-0.6B contradicting a size-only explanation.BLOOM-1b1’s λ jumps from 0.13 to 0.79 between 10K and 100K steps, whereas BLOOM-560M peaks at 0.26; SmolLM3 begins at λ = 0.81 and later expands its high-λ zone.Source: §4.4 · p. 9
  • Instruction tuning can induce sharing in small models: BLOOMZ-560m raises peak λ from 0.21 to 0.84 and peak ILO from 0.25 to 0.37, while larger models shift modestly.The BLOOMZ-560m change creates a 10-layer collapsed-dimensionality zone with npca = 2, suggesting sharing is not fixed after pretraining.Source: §4.4 · p. 9

5 Conclusion

A systematic comparison across 21 models shows that anisotropy drives disagreements among sharing metrics, while only ILO remains associated with cross-lingual transfer after controls. The authors recommend pairing global sharing scores with anisotropy diagnostics and testing the conclusions through controlled interventions.

5 Conclusion

  • Anisotropy drives metric failures: GMM dominance can report relocated language information, miss distributed sharing, and collapse distributed and minimal sharing into near-zero scores.These failures arise because GMM dominance relies on PCA and anisotropic representations.Source: §5 · p. 9
  • Only ILO’s association with cross-lingual transfer survives controls for model size, family membership, and per-task decomposition.The comparison covers 21 base models from five families and four sharing metrics.Source: §5 · p. 9
  • Sharing scores should be reported with anisotropy diagnostics, including effective dimensionality and mean random cosine similarity, while global metrics remain appropriate for comparing models.Mechanistic methods are needed to explain how sharing is implemented, and they likewise benefit from anisotropy diagnostics (Tang et al., 2024; Kojima et al., 2024; Deng et al., 2025; Harrasse et al., 2026).Source: §5 · p. 9

Future work

  • Future work should test whether low npca at peak layers causes concentrated sharing, using anisotropy-suppressing optimizers and controlled training with matched architectures and varied data.The agenda also includes IsoScore (Rudman et al., 2022) and extending comparisons to multilingual encoders; Coupled Adam is proposed as an intervention (Stollenwerk and Stollenwerk, 2025).Source: §5 · p. 9

Limitations … A.4 Language Probe Setup

The study’s conclusions are limited by sample dependence, decoder-only coverage, language-selection bias, observational correlations, and untested hyperparameter and computational tradeoffs. The appendices specify the models, metrics, costs, and language-probe procedure used to support the analyses.

Limitations

  • The effective between-family sample is closer to five, limiting statistical power despite within-family, partial-correlation, and permutation analyses.SmolLM3 has one model size, and detailed per-language training-data proportions are publicly available only for BLOOM.Source: Limitations · pp. 9–10
  • The reliability ranking holds across 21 models but may change for models with substantially higher or unobserved forms of anisotropy, so diagnostics should accompany geometric sharing metrics.The study does not claim that ILO and ANC are universally safe.Source: Limitations · p. 10
  • The evidence is restricted to decoder-only models and comparatively high-resource languages, leaving transfer to multilingual encoders and thinly represented languages untested.Language coverage is deliberately restricted for like-for-like comparison, but task-specific omissions and three Romance languages may bias per-language patterns.Source: Limitations · p. 10
  • All reported correlations are associational, and common causes such as training-data diversity may jointly promote sharing and transfer.Controlled interventions would be needed to isolate causal mechanisms.Source: Limitations · p. 10

A.1 Models and Checkpoints

  • The study uses 21 base models from five families for transfer correlations, while instruction-tuned variants and intermediate checkpoints are used only for representational characterization.Checkpoint tables document the model inventory and exclude ten byte-identical BLOOM checkpoints as apparent duplicates.Source: §A.1 · p. 14, Table 3 · p. 14, Table 4 · p. 14

A.2 Metrics

  • The four sharing metrics are computed per layer on FLORES-200, with λ using PCA plus language-specific full-covariance GMMs, ILO using cosine kNN, and CKA and ANC using aligned sentence representations.The reported λ summary is its peak-layer value, while CKA and ANC aggregate across language pairs and layers; centered ILO removes the shared anisotropic direction.Source: §A.2 · pp. 14–15

A.3 Computational Cost

  • ILO is the most expensive metric: exact nearest-neighbor retrieval costs O(N^2d), compared with O(Nd^2) for λ, O(nd · min(n, d)) for CKA, and O(nd) for ANC.For N = 10^5 and d = 5,120, pooled hidden states require about 2 GB per layer at single precision.Source: §A.3 · p. 16

A.4 Language Probe Setup

  • The language probe is multinomial logistic regression over 10 FLORES languages, evaluated in PCA-principal and PCA-residual spaces using a stratified 80/20 split and test accuracy.The principal space retains at least 98% of variance with a minimum of two components, while features remain unstandardized for direct scale comparability.Source: §A.4 · p. 16

A.5 Cross-lingual Finetuning Datasets · A.6 Cross-lingual Finetuning Setup · B Results

Cross-lingual transfer is evaluated with five English-trained benchmarks spanning comprehension, inference, reasoning, classification, and question answering across the available target languages. Models are finetuned with a standardized AdamW protocol, task-specific budgets, and per-model/task learning-rate selection.

A.5 Cross-lingual Finetuning Datasets

  • Five benchmarks measure English-to-multilingual transfer across reading comprehension, natural language inference, commonsense reasoning, topic classification, and extractive question answering.Evaluation uses the benchmark-specific language coverage summarized in Table 5.Source: §A.5 · pp. 16–17, Table 5 · p. 17
  • Belebele trains on approximately 71K English examples and evaluates ten languages, while XNLI trains on approximately 392K English examples and evaluates eight available languages.Japanese and Portuguese are excluded from XNLI evaluation because they are absent from the benchmark coverage used here.Source: §A.5 · pp. 16–17
  • XCSR trains on approximately 9.7K English examples and evaluates nine target languages, whereas SIB-200 uses 701 examples per language and evaluates all ten languages.XCSR uses five-way commonsense reasoning, and SIB-200 uses seven-way topic classification.Source: §A.5 · p. 17
  • XQuAD trains on approximately 87.6K English SQuAD examples and evaluates F1 on seven covered target languages.The task is extractive question answering.Source: §A.5 · p. 17
  • The transfer score is the unweighted mean of five task means, with each task mean computed over its non-English target languages.Per-task language counts and overall coverage are summarized in Table 5.Source: §A.5 · p. 17

A.6 Cross-lingual Finetuning Setup

  • Every model-task pair uses AdamW with weight decay 0.01, gradient clipping at ℓ2 norm 1.0, 10% linear warmup, decay to zero, and maximum sequence length 512.Training uses bfloat16 mixed precision on A100/H200 GPUs, with all model parameters unfrozen and task-specific heads.Source: §A.6 · p. 17
  • Task budgets vary from three to twenty epochs, with validation-selected checkpoints where available and final-epoch models otherwise.Effective batch sizes are 32 for XNLI, XCSR, XQuAD, and SIB-200, and 16 for Belebele.Source: §A.6 · p. 17
  • For each model-task pair, 3–5 learning rates are swept and the rate maximizing mean target-language accuracy, or XQuAD F1, is reused for interpretability re-analysis.The sweep range is adapted to each model family’s stable regime, with boundary extensions when necessary.Source: §A.6 · p. 17, Table 6 · p. 18

B.1 Full Metric Results

Across 21 models, the metrics’ disagreement persists even when they share the same input unit, while λ’s behavior depends strongly on granularity and model family. Sample-balanced and word-level λ agree closely, but sentence-level pooling can erase or attenuate token-level cluster structure.

B.1 Full Metric Results

  • Figures 5–25 provide layer-wise profiles of sample-balanced λ, ILO, effective PCA dimensionality, pair-averaged CKA, and ANC for all 21 base models.CKA and ANC panels retain min–max variation across language pairs through shaded bands.Source: §B.1 · p. 18
  • Sample-balanced and word-level λ track closely across all 21 models, supporting the sampling choice used in the main analysis.Figures 26–29 show family-level comparisons of the three λ computations.Source: §B.1 · p. 18, Figure 26 · p. 26, Figure 27 · p. 26
  • Sentence-level λ is zero across BLOOM, OPT, and XGLM, but Qwen3 and SmolLM3 show early-layer peaks of 0.41–0.56.The Qwen3 and SmolLM3 peaks occur at relative depth ≤0.37, below sample-balanced and word-level peaks of ≈0.85–0.91; the peak narrows as Qwen3 scales from 0.6B to 14B.Source: §B.1 · p. 18
  • CKA and ANC disagree despite identical sentence embeddings, while λ and ILO give opposite XGLM verdicts from identical token samples, implicating metric geometry rather than input units.CKA and ANC correlate with transfer at ρ = 0.14 and 0.84, whereas λ = 0.03 and ILO = 0.44 on XGLM.Source: §B.1 · p. 18

B.2 Anisotropy Analysis

All five model families exhibit pronounced middle-layer anisotropy, with cosine similarity exceeding 0.85 in every model, while effective dimensionality is evaluated at peak-sharing layers under two PCA thresholds. The profiles generally rise, plateau, and drop near the output, with family-specific extremes and variation.

Cosine similarity profiles

  • Profiles in BLOOM, OPT, Qwen3, and SmolLM3 rise rapidly, remain highly similar through middle layers, and drop sharply in the final two layers.XGLM varies most, with XGLM-4.5B showing a mid-network dip to 0.57 absent from the other XGLM models.Source: §B.2 · p. 27, Figure 31 · p. 28, Figure 32 · p. 28

Effective dimensionality

  • Effective PCA dimensionality is reported at each model’s peak-sharing layer using 98% and 90% variance thresholds on a balanced 10,000-token-per-language subsample.Both thresholds use the pipeline’s floor of 2; OPT and XGLM use peak mean-ILO layers when λ remains near zero.Source: Table 7 · p. 27, §B.2 · p. 27

Cosine similarity profiles

B.3 Probe Results … B.6 Per-language perplexity

The appendix shows that language identity remains recoverable, but its location across PCA subspaces depends on model family and PCA-collapse behavior. Additional analyses validate ILO’s robustness to centering while showing that sentence-level ILO is more sensitive to anisotropy, and that high ILO does not imply uniformly low perplexity across languages.

B.3 Probe Results

  • Language identity remains recoverable at every layer, while BLOOM, Qwen3, and SmolLM3—but not OPT or XGLM—show PCA-subspace inversions linked to PCA collapse.Full-hidden-state probes achieve ≥0.96 accuracy throughout; in inversion layers, main-subspace accuracy falls to ≈0.09 while null-subspace accuracy reaches ≥0.99, whereas BLOOM-560M shows only a partial dip to 0.77.Source: §B.3 · p. 29, Figure 36 · p. 30, Figure 37 · p. 30

B.5 Sentence-Level ILO Variant

  • Sentence-level ILO correlates with transfer at ρ = 0.93 versus 0.90 for word-level ILO, while the variants agree on model ranks at ρ = 0.95.The paper nevertheless reports word-level ILO because it uses λ’s identical 10,000-token-per-language subsample and matches the main analysis.Source: §B.5 · p. 31
  • Sentence-level peak ILO shifts by up to 0.127 after mean-centering, compared with at most 0.010 for word-level ILO, especially in highly anisotropic models.Mean absolute shifts are 0.043 for sentence-level ILO and 0.003 for word-level ILO, with the largest sentence-level shifts in Qwen3-14B and SmolLM3.Source: §B.5 · p. 31

B.4 ILO Centering

  • Centered peak ILO differs from uncentered ILO by at most 0.010 across 21 models and by less than 0.005 in 17, preserving within-family ordering.The shift’s sign varies, indicating that centering does not systematically raise or lower ILO.Source: Table 8 · p. 31, §B.4 · p. 31

B.6 Per-language perplexity

  • High ILO does not guarantee uniformly low surprise across languages, because mean FLORES-200 devtest perplexity shows substantial cross-language variance.This supports interpreting ILO as crosslingual representational overlap rather than as a measure of uniformly low perplexity.Source: §B.6 · p. 31, Table 9 · p. 32
  • The per-language perplexity analysis therefore separates representational overlap from language-uniform modeling performance across the 10 evaluated languages.Table 9 reports mean FLORES-200 devtest perplexity per language for each base model, with lower values better.Source: §B.6 · p. 31, Table 9 · p. 32

B.7 Cross-Lingual Transfer Results

This section reports cross-lingual transfer after English-only fine-tuning, summarizing mean non-English performance and providing per-language results across five tasks with task-specific language coverage.

Per-language results

  • Per-language transfer results are provided separately for Belebele, XNLI, XCSR, SIB-200, and XQuAD after English-only fine-tuning.Tables 11–15 give the language-level breakdowns, with XQuAD reported using F1.Source: §B.7 · p. 33, Table 11 · p. 34, Table 12 · p. 35

Evaluation scope

  • Evaluation language coverage varies by task: Belebele and SIB-200 include all 10 languages, whereas XCSR, XNLI, and XQuAD omit specified languages.XCSR omits Turkish; XNLI omits Japanese and Portuguese; XQuAD omits Japanese, French, and Portuguese.Source: §B.7 · p. 33

Summary results

  • Mean accuracy, or F1 for XQuAD, is reported on non-English target languages after English-only fine-tuning at the best learning rate for each model–task pair.Table 10 provides the cross-lingual transfer summary, while per-pair learning rates are listed in Table 6.Source: §B.7 · p. 33, Table 10 · p. 34

B.8 Training Dynamics … C Statistical Analysis

Across training, instruction tuning, and finetuning, the sharing metrics show distinct dynamics and family-dependent shifts, while the statistical analysis evaluates correlations over 21 base models and five tasks. λ and ILO track sharing transitions more selectively than CKA and ANC, which can reflect anisotropy-related similarity.

B.8 Training Dynamics

  • λ and ILO reveal size- and stage-dependent sharing transitions, whereas CKA and ANC show substantial early middle-layer similarity consistent with anisotropy rather than a sharing transition.BLOOM-560M never develops a high-λ band; BLOOM-1.1B transitions between steps 10K and 100K, larger BLOOM models begin with a mid-layer plateau, and SmolLM3 establishes a high-λ band by step 40K that extends upward in Stage 3.Source: §B.8 · p. 37

B.9 Instruction Tuning Effects

  • BLOOMZ-560m shifts from λ = 0.21 and ILO = 0.25 in the base model to λ = 0.84 and ILO = 0.37 after instruction tuning.The other BLOOM sizes gain only +0.02 to +0.07 in peak λ and +0.09 to +0.15 in peak ILO; the cited visual is Figure 49.Source: §B.9 · p. 40, Figure 49 · p. 41
  • Qwen3-Instruct and SmolLM3-3B-Instruct show negligible instruction-tuning changes, with peak-λ and peak-ILO shifts within ±0.03, while CKA and ANC change only marginally across families.Their base models already operate near the upper end of the sharing range.Source: §B.9 · p. 40

B.10 Finetuning Effects

  • Finetuning analysis compares peak-metric shifts across all 105 model-task pairs, fixing each finetuned model’s peak layer to the corresponding base-model peak.The aggregate shifts are summarized in Table 16, with task-specific breakdowns for Belebele, SIB-200, XNLI, XCSR, and XQuAD in Tables 17–21.Source: §B.10 · p. 45, Table 16 · p. 45, Table 17 · p. 45
  • The finetuning results are reported as positive or negative peak-metric shifts relative to the base model, with positive values indicating that the finetuned peak exceeds the base-model peak.Per-task tables use the same conventions, while the analysis fixes the peak layer according to the base-model metric peak.Source: Table 16 · p. 45, Table 17 · p. 45

C Statistical Analysis

  • Statistical analysis uses Spearman’s ρ between each per-model peak or mean sharing metric and transfer scores across five tasks, using n = 21 base models from five families.The transfer scores are those reported in Appendix B.7.Source: C Statistical Analysis · p. 47

C.1 Primary Correlation Analysis … C.7 Within-Family Breakdown

Across correlation, robustness, and heterogeneity analyses, ILO most consistently tracks cross-lingual transfer. Its association persists across multilingual models, aggregation rules, languages, tasks, and within-family tests, whereas competing metrics show weaker or unstable behavior.

C.1 Primary Correlation Analysis

  • ILO and ANC are significant against transfer, λ is significant but weak, and CKA is non-significant in the primary four-metric comparison.The analysis covers 21 base models across five families.Source: Table 22 · p. 47, §C.1 · p. 47
  • After excluding English-only OPT, ILO remains strongest at ρ = 0.78, while ANC falls to 0.64, near λ at 0.66, and CKA remains non-significant.The remaining sample contains 16 multilingual models across four families.Source: Table 23 · p. 47, §C.1 · p. 47

C.2 Partial Spearman Correlations

  • Controlling for log10(parameter count) leaves ILO, ANC, and λ largely unchanged, while CKA increases modestly but remains weak.These partial Spearman correlations use all 21 base models.Source: §C.2 · p. 47, Table 24 · p. 47

C.3 Consistent Layer Aggregation

  • Under consistent all-peak and all-mean aggregation, metric rankings remain ILO strongest, ANC second, λ moderate, and CKA null; ILO and ANC vary by at most 0.04.λ weakens from 0.52 to 0.46 under all-mean, whereas CKA remains low at ρ = 0.12.Source: §C.3 · p. 47, Table 25 · p. 48

C.4 Per-Language Correlations

  • Per-language ILO correlates positively with transfer in 20 of 21 models, with median within-model ρ = 0.68, and remains representational after overlap control.Target-English alignment predicts transfer more strongly than average alignment with other languages for CKA and ANC, while ILO is source-language independent.Source: §C.4 · pp. 48–49, Table 27 · p. 49

C.5 Permutation Tests

  • ILO alone survives the stringent within-family permutation test, with p = .014; λ is borderline at p = .056, while ANC and CKA fail.The within-family null preserves between-family contrasts and tests whether associations exist inside families.Source: §C.5 · p. 49, Table 28 · p. 49

C.6 Per-Task Heterogeneity

  • ILO is significant on all five tasks with ρ ∈ [0.73, 0.88], ANC is significant on all five, λ on only Belebele and XQuAD, and CKA on none.λ is non-significant on XNLI, XCSR, and SIB-200.Source: §C.6 · p. 49, Table 29 · p. 50

C.7 Within-Family Breakdown

  • Within families, ILO is positive in three of four families, while λ flips sign on OPT, ANC on XGLM, and CKA is non-positive in three families.The cross-model positive CKA ordering is therefore attributed to between-family means rather than within-family relationships.Source: §C.7 · p. 49, Table 30 · p. 50

Related papers

Loading 2609.04819v1…