PaperlayerView exact revision on arXiv

Source-linked AI summary

A Systematic Comparison of Multilingual Interpretability Methods Reveals Anisotropy-Driven Failures

Oskar Holmström, Marcel Bollmann, Marco Kuhlmann

arXiv:2609.04819v1cs.CL

TL;DR

Different interpretability metrics can disagree about multilingual representation sharing, leaving unclear whether disagreement reflects models or measurement artifacts. The paper compares four metrics across 21 base models and five transfer tasks, finding that ILO is the only association that survives controls for model size, family, and per-task variation. It recommends reporting ILO alongside anisotropy diagnostics while treating the association as non-causal and not as a validated measure of sharing.

  • Problem

    Existing cross-lingual sharing metrics were developed largely in isolation, leaving unclear whether disagreements reflect genuine model differences or measurement artifacts.

  • Method

    The paper compares CKA, ANC, GMM dominance per token, and ILO across 21 base models from five families and relates them to cross-lingual transfer on five tasks.

  • Results

    ILO is the only metric whose association with cross-lingual transfer survives controls for model size, family, and per-task variation, reaching Spearman ρ = 0.90.

  • Takeaways & Limitations

    The paper recommends ILO as the primary sharing metric and reporting it alongside anisotropy diagnostics.

  • Takeaways & Limitations

    The reported associations are not causal, and the reliability ranking may shift for models with substantially higher or unencountered forms of anisotropy.

Abstract

from arXiv · show

Multilingual language models develop shared cross-lingual representations, and various interpretability methods claim to quantify this sharing. These methods have been developed largely in isolation, and when they disagree, it is unclear whether the disagreement reflects a property of the model or an artifact of the measurement. We compare four sharing metrics (CKA, ANC, GMM dominance per token, and ILO) across 21 base models from five families (125M-14B parameters) and correlate each with cross-lingual transfer on five downstream tasks. We find that the metrics differ in their quantification of cross-lingual sharing in these models and suggest that the disagreement traces to anisotropy, the tendency of representations to cluster in a narrow cone of the embedding space. Only ILO's correlation with cross-lingual transfer (Spearman's $ρ= 0.90$) survives controls for model size, family, and per-task variation. We therefore recommend ILO as the primary sharing metric, to be reported alongside anisotropy diagnostics.

1 Introduction

The paper asks whether disagreements among cross-lingual sharing metrics reflect model differences or measurement artifacts, focusing on anisotropy as a possible source of distortion. It compares four metrics across diverse multilingual models and evaluates which best tracks cross-lingual transfer.

  • Motivation: Existing sharing metrics were developed largely in isolation, making disagreements difficult to distinguish from genuine differences between models.The paper frames this as a measurement-reliability problem for conclusions about architectures, training regimes, and scaling.
  • Motivation: Anisotropy concentrates representation variance in a narrow cone and can inflate apparent similarity while making dominant directions control geometric measures.Figure 1 illustrates the narrow-cone geometry and the effect of removing top principal components.
  • Study Questions: The study systematically compares four sharing metrics across 21 base models from five families spanning 125M–14B parameters.The comparison is designed to test how metrics agree, disagree, and relate to cross-lingual transfer.
  • Findings: Models exhibit three sharing patterns rather than the binary split implied by metrics dominated by high-variance directions.The paper attributes metric disagreement to anisotropy and reports that language identity is relocated to directions some metrics discard or down-weight.
  • Findings: ILO is the only compared metric whose association with cross-lingual transfer survives controls for model size, family, and per-task variation, reaching Spearman ρ = 0.90.The paper recommends ILO as the primary sharing metric while treating anisotropy diagnostics as necessary context.

2 Related Work

Related work studies multilingual representation sharing, language alignment, and anisotropy from several methodological perspectives. These strands motivate comparing metrics directly rather than assuming that apparent alignment or similarity measures capture the same construct.

  • Multilingual Sharing: Multilingual models develop shared representations without explicit alignment supervision, but proposed drivers include positional embeddings, word order, parameter sharing, and training-data imbalance.The contributing factors remain debated across prior work.
  • Alignment Methods: Neuron-level and lens-based methods examine alignment or vocabulary projections, while later work questions whether apparent English reliance reflects genuine multilingual organization.Prior studies report that representations may mix several languages rather than pivoting toward English.
  • Alignment Methods: MEXA and DALI assess directed alignment with English, whereas the four metrics compared here target sharing among all languages.The paper relates these language- or instance-level views to its broader sharing measures in an appendix.
  • Representational Anisotropy: Transformer contextual embeddings often occupy a narrow cone, with high cosine similarity between vectors regardless of semantic content.Prior work attributes anisotropy to common shifts, self-attention dynamics, and optimizer effects, while post-hoc enhancement can improve isotropy.
  • Representational Anisotropy: Because anisotropy distorts geometric analyses, prior multilingual studies motivate testing whether sharing metrics remain reliable under altered representation geometry.This provides the methodological context for the paper’s systematic metric comparison.

3 Experimental Setup

The experimental setup compares four complementary sharing metrics across 21 base models and evaluates them against zero-shot transfer on five tasks. It also measures anisotropy and controls for confounds including model size and post-training differences.

  • Sharing Metrics: The study selects CKA, ANC, GMM dominance per token, and ILO to represent global geometry, neuron correspondence, distributional structure, and local neighborhood overlap.This design contrasts measurement families rather than variants of a single method.
  • Sharing Metrics: CKA compares pairwise distance structures between representation spaces, with values from 0 for no shared structure to 1 for identical structure up to isotropic scaling and orthogonal transformation.It serves as the established geometric-similarity baseline.
  • Sharing Metrics: ANC averages absolute Pearson correlations between corresponding neuron activations on parallel sentences across languages.Its value lies in [0, 1], with 1 indicating consistent responses from every neuron.
  • Sharing Metrics: GMM dominance per token fits a supervised Gaussian mixture model in a PCA subspace retaining 98% of variance and reports the fraction of tokens favored by a foreign-language space.The analysis uses 10,000 tokens per language to reduce tokenizer-fertility bias across families.
  • Sharing Metrics: ILO uses k-nearest-neighbor queries with k = 5 in the full representation space, combining bridge and reachability scores into an index from 0 to 1.Zero denotes complete separation and 1 denotes complete sharing.
  • Models and Data: Interpretability analyses use 1,012 FLORES-200 parallel sentences across ten languages spanning four scripts and diverse typological and morphological categories.This provides the multilingual representation data for the metric comparisons.
  • Transfer Evaluation: Each base model is fine-tuned on English and evaluated zero-shot across five non-English transfer tasks, while transfer is treated as validation rather than ground truth for sharing.The tasks are Belebele, XNLI, XCSR, SIB-200, and XQuAD; transfer has drivers beyond representational alignment.

4 Results and Discussion

Across 21 base models, four sharing metrics reveal three representation-sharing patterns, but anisotropy explains why they disagree. ILO is the only metric whose transfer association survives the paper’s robustness controls, while sharing can emerge abruptly during training and change after instruction tuning.

  • How Do Models Share Representations Across Languages?: Three sharing patterns emerge: concentrated, distributed, and minimal sharing.Concentrated sharing appears in BLOOM ≥1B, Qwen3, and SmolLM3; XGLM shows distributed sharing; OPT shows minimal sharing.
  • How Do Models Share Representations Across Languages?: XGLM exposes a metric failure: λ groups it with OPT despite transfer scores of 0.55 versus 0.41, while ILO and ANC track broader sharing patterns.XGLM’s transfer score approaches BLOOM-7.1B’s 0.60, whereas CKA remains narrowly ranged across families.
  • Why Do Methods Disagree?: Anisotropy drives disagreement by inflating CKA, leaving ANC relatively invariant to common shifts, and allowing ILO to remain stable under centering.Mean random cosine similarity ranges from 0.86 to 0.99; the centered ILO variant changes by mean |∆| = 0.003 across models.
  • Why Do Methods Disagree?: Variance-ranked λ projection collapses representations to two principal components in every high-λ model, excluding hundreds or thousands of other directions.At high-λ layers, language probes reach chance in the principal subspace but 96–99% accuracy in the residual subspace, indicating that language identity is relocated rather than destroyed.
  • Do Interpretability Methods Predict Cross-Lingual Transfer?: ILO is the strongest transfer predictor (ρ = 0.90), followed by ANC (ρ = 0.84), λ (ρ = 0.52), and CKA (ρ = 0.14).Only ILO survives controls for size, family, and per-task variation; excluding OPT leaves ILO strongest at ρ = 0.78.
  • How Does Sharing Emerge During and After Pretraining?: Sharing appears through abrupt training transitions and can be reshaped by instruction tuning.BLOOM-1b1’s λ jumps from 0.13 to 0.79 between 10K and 100K steps, while instruction tuning creates a collapsed-dimensionality zone in BLOOMZ-560m.

5 Conclusion

The comparison shows that anisotropy can make sharing metrics misrepresent how multilingual representations are organized, while ILO remains the only metric whose transfer association survives the tested controls. The authors therefore recommend pairing sharing scores with anisotropy diagnostics and using mechanistic methods to explain implementation.

  • Anisotropy causes GMM dominance per token to report sharing when language identity has shifted into discarded dimensions, while missing sharing distributed across thousands of directions.
  • Only ILO’s association with cross-lingual transfer survives controls for model size, family membership, and per-task decomposition.
  • The metrics conflate concentrated, distributed, and minimal sharing by collapsing distributed and minimal sharing into one near-zero score.
  • Geometric sharing scores should be reported alongside anisotropy diagnostics, including effective dimensionality and mean random cosine similarity.
  • The observational design cannot determine whether the low-npca bottleneck actively forces language-agnostic computation or is incidental.

Limitations

The study’s reliability ranking is bounded by its sample, model scope, language coverage, and methodological assumptions. The authors therefore caution against treating ILO or ANC as universally safe metrics.

  • Sample and design: The 21 models share architecture, data, and tokenizer within families, making the effective between-family sample closer to five and limiting statistical power.The authors address this with within-family analyses, partial correlations, and permutation tests.
  • Generalization: The reliability ranking may shift for models with substantially higher anisotropy or anisotropy modes absent from this study.The reported ranking holds across the 21 studied models, but the authors do not claim universal safety for ILO or ANC.
  • Model scope: Whether the ranking transfers to multilingual encoders is untested because all 21 studied models are decoder-only.The authors make no claim about encoder models, which differ in training objective and attention pattern.
  • Language coverage: The ten evaluation languages are comparatively high-resource, include three Romance languages, and provide incomplete benchmark coverage.The conclusions therefore concern sharing among well-represented languages, while reliability for thinly represented languages remains unknown.
  • Interpretation and implementation: All reported correlations are associational, so plausible common causes such as training-data diversity remain unresolved without controlled interventions.The authors also follow fixed PCA and ILO hyperparameters without systematic sensitivity analysis, and ILO is the most computationally expensive metric.

A.5 Cross-lingual Finetuning Datasets

The evaluation uses five cross-lingual benchmarks with English training and non-English evaluation, averaging task-level transfer across the languages each benchmark covers. Fine-tuning uses standardized optimization, task-specific budgets, and per-model learning-rate selection.

  • Benchmarks: Five benchmarks evaluate reading comprehension, natural language inference, commonsense reasoning, topic classification, and extractive question answering.The benchmarks are Belebele, XNLI, XCSR, SIB-200, and XQuAD.
  • Language protocol: Training is English-only, while evaluation uses the covered non-English languages, with benchmark coverage ranging from seven to ten languages.XNLI excludes Japanese and Portuguese; XCSR covers nine target languages; XQuAD covers seven.
  • Aggregate metric: The transfer score is the unweighted mean of five per-task means, each computed over that task’s non-English target languages.This preserves each task’s available-language coverage while giving the five tasks equal weight.
  • Optimization: Models are fine-tuned with AdamW, 10% linear warmup, linear decay, gradient clipping, and maximum sequence length 512.Classification uses a fresh task-specific linear head, while question answering uses standard span-prediction heads.
  • Selection procedure: For each model-task pair, 3–5 learning rates are swept and the rate maximizing mean target-language accuracy or XQuAD F1 is selected.Validation-based tasks use the lowest validation cross-entropy checkpoint; tasks without validation use the final epoch.

B.1 Full Metric Results

The appendix reports layer-wise profiles for four sharing metrics across 21 base models and compares how sampling granularity affects their measurements. CKA, ANC, λ, and ILO can disagree even when computed from identical input units.

  • Full metric profiles: Layer-wise profiles cover 21 base models, with GMM λ, ILO, effective PCA dimensionality, CKA, and ANC reported for each model.Figures 5–25 show λ, ILO, and effective PCA dimensionality alongside CKA and ANC, including pairwise ranges where applicable.
  • Granularity of λ: Sample-balanced and word-level λ track each other closely across all 21 models, supporting the paper’s sampling choice.The sample-balanced computation uses 10,000 tokens per language.
  • Cross-metric disagreement: CKA and ANC disagree despite identical sentence embeddings, while λ and ILO disagree despite identical token samples.Their transfer correlations are ρ = 0.14 and 0.84 for CKA and ANC, while XGLM yields λ = 0.03 and ILO = 0.44.

B.2 Anisotropy Analysis

The anisotropy analysis finds strong middle-layer concentration across all model families and uses effective dimensionality and cosine similarity to characterize it. Anisotropy is therefore widespread rather than confined to models with high λ.

  • Effective dimensionality: Effective PCA dimensionality is measured at each model’s peak-sharing layer using both 98% and 90% variance thresholds, with a floor of 2.Both thresholds use the same balanced 10,000-tokens-per-language subsample.
  • Cosine similarity: All five families exhibit pronounced middle-layer anisotropy, with average random-pair cosine similarity exceeding 0.85 at mid-depth in every model.OPT and Qwen3 reach ≥0.99 in some layers, while BLOOM-560M peaks at 0.86.
  • Layer profiles: BLOOM, OPT, Qwen3, and SmolLM3 show rapid rises, long high-similarity plateaus, and sharp drops in the final two layers.XGLM varies more, including a 0.57 mid-network dip in XGLM-4.5B.
  • Scope: High anisotropy is a generic property of the studied families, not a property limited to the high-λ subset.The conclusion follows from the cosine-similarity profiles across all five families.

B.3 Probe Results

Probe results show that language identity remains recoverable even when it disappears from the principal subspace, while centered ILO remains stable across models. These findings distinguish relocated language information from information loss and support ILO’s robustness to common anisotropic offsets.

  • Language probes: Full-state language probes achieve ≥0.96 accuracy at every layer of every model, including layers where λ is high.Language identity is therefore retained in the full representation.
  • Principal and residual subspaces: In middle layers, BLOOM, Qwen3, and SmolLM3 show main-subspace accuracy near 0.09 while residual-subspace accuracy reaches ≥0.99.The main–null inversion follows the same family structure as λ.
  • Centered ILO: Centered peak ILO differs from uncentered peak ILO by at most 0.010 across all 21 base models and by less than 0.005 in 17 models.The within-family ordering is preserved, and the shift’s sign varies.
  • Sentence-level ILO: Sentence-level ILO correlates with transfer at ρ = 0.93 versus ρ = 0.90 for word-level ILO, with model-rank agreement of ρ = 0.95.The paper reports no changed conclusion under either variant.
  • Perplexity check: Cross-language perplexity remains substantially variable even for high-ILO models, indicating that ILO measures representational overlap rather than uniformly low surprise.Table 9 reports mean FLORES-200 devtest perplexity by language and model.

B.7 Cross-Lingual Transfer Results

The paper evaluates cross-lingual transfer after English-only finetuning using five downstream tasks and reports mean performance on non-English target languages. Task language coverage differs, so comparisons must account for the languages included in each benchmark.

  • Evaluation setup: Cross-lingual transfer is summarized as mean accuracy, or F1 for XQuAD, on non-English targets after English-only finetuning.Learning rates are selected separately for each model–task pair.
  • Tasks: The five evaluated tasks are Belebele, SIB-200, XCSR, XNLI, and XQuAD.Per-language results are provided in Tables 11–15.
  • Language coverage: Belebele and SIB-200 cover all 10 evaluation languages, whereas XCSR, XNLI, and XQuAD omit specified languages.XCSR drops Turkish; XNLI drops Japanese and Portuguese; XQuAD drops Japanese, French, and Portuguese.

B.8 Training Dynamics

Training dynamics show that λ and ILO identify sharing transitions differently from CKA and ANC. Instruction tuning mainly changes smaller BLOOM models, while other families show little metric movement.

  • Base-model dynamics: BLOOM-560M never develops a high-λ band, BLOOM-1.1B transitions between steps 10K and 100K, and larger BLOOM models begin with a mid-layer plateau.For SmolLM3-3B, a high-λ band appears by step 40K and expands into upper layers during Stage 3.
  • Base-model dynamics: CKA and ANC report substantial middle-layer similarity from the earliest checkpoints, including BLOOM-560M, where λ never reaches threshold.This pattern is attributed to anisotropy rather than a sharing transition.
  • Base-model dynamics: ILO follows BLOOM’s λ trajectory but rises more gradually across checkpoints.
  • Instruction-tuned variants: BLOOMZ-560m shifts from λ = 0.21 in the base model to λ = 0.84 after instruction tuning, while ILO rises from 0.25 to 0.37.The remaining BLOOM sizes gain only modest peak-λ increments, while peak ILO rises by +0.09 to +0.15.
  • Instruction-tuned variants: Qwen3-Instruct and SmolLM3-3B-Instruct show negligible changes, with peak-λ and peak-ILO shifts both within ±0.03.CKA and ANC profiles change only marginally across all families.

C.1 Primary Correlation Analysis

Across models and tasks, ILO is the strongest predictor of cross-lingual transfer, while CKA is weak or null and λ is less stable. This ranking persists under controls for model size, aggregation rules, language level, and task variation.

  • Primary correlations: ILO and ANC significantly correlate with transfer, λ is significant but weak, and CKA is not significant in the primary analysis.
  • Primary correlations: After excluding OPT, ILO remains strongest at ρ = 0.78, while ANC reaches ρ = 0.64 and λ reaches ρ = 0.66.OPT is the English-only baseline, whereas the remaining 16 models are multilingual.
  • Robustness checks: Controlling for model size leaves ILO, ANC, and λ largely unchanged, while CKA increases modestly but remains weak.
  • Consistent aggregation: The metric ranking is identical under all-peak and all-mean aggregation: ILO strongest, ANC second, λ moderate, and CKA null.ILO and ANC vary by no more than 0.04 across rules; CKA remains weak with peak treatment (ρ = 0.12).
  • Per-language analysis: Per-language ILO correlates positively with transfer within 20 of 21 models, with a median within-model ρ of 0.68.Controlling for English-target subword overlap changes ILO and ANC partial correlations by no more than 0.04.
  • Task and family variation: ILO alone survives the within-family permutation test with p = .014; λ is borderline at p = .056, while ANC and CKA fail.Within-family results therefore distinguish ILO’s association from between-family ordering effects.
  • Task and family variation: ILO is significant on all five tasks with ρ ∈ [0.73, 0.88], whereas λ is significant on only Belebele and XQuAD and CKA reaches no task.ANC is significant on all five tasks, though weakest on SIB-200.
Loading 2609.04819v1…