Cross-lingual Similarity of Multilingual Representations Revisited

1 Introduction

• The paper investigates whether “first align, then predict” generalizes from mBERT to causal language models and larger-scale MLMs and CLMs.The prior finding discussed in the introduction concerned mBERT, an MLM with 110M parameters.Source: §1 · pp. 1–2
• The paper critiques CKA/CCA conceptually and empirically for cross-lingual similarity analysis.Its motivating case study shows that CKA can fail to capture cross-lingual similarity considered useful in this context.Source: §1 · pp. 1–2
• ANC is proposed as a straightforward alternative that exploits a priori-aligned neurons in representations from the same model.The paper presents this neuron alignment as a property suited to cross-lingual comparison.Source: §1 · pp. 1–2

2 Related work

• Prior studies disagreed on whether language representations diverge or align with network depth.Singh et al. (2019a) reported divergence, whereas Conneau et al. (2020) and Muller et al. (2021) reported alignment followed by some recovery of language specificity.Source: §2 · p. 2
• Del and Fishel (2021) attributed differing similarity findings to pooling choices and reported mean-pooling as the better option.Singh et al. (2019a) used CLS-pooling, while Muller et al. (2021) used mean-pooling.Source: §2 · p. 2
• Li et al. (2015) aligned correlated neurons across separate networks, while this paper needs no neuron-alignment step because it compares languages within one multilingual model.The paper focuses on models trained with CLM and MLM objectives and leaves seq2seq models for future work.Source: §2 · p. 2
• The paper builds on prior work by demonstrating CKA’s cross-lingual failure, proposing another comparison method, and examining the pattern across objectives and model sizes.Earlier work applied CCA variants and CKA to multilingual representations, including within multilingual models.Source: §2 · p. 2

3 Similarity Indexes Background

• A neuron is represented by its activation values across dataset examples, while a layer is the matrix of its neuron vectors or the subspace they span.The paper records each unit’s activations over the dataset to form its neuron vector.Source: §3 · pp. 2–3
• CCA and CKA compare subspaces spanned by neurons, allowing different neuron counts and avoiding any requirement that neurons align.They use neurons to describe the vector subspaces rather than compare the neurons themselves.Source: §3 · p. 3
• CCA and CKA derive second-order descriptions, such as Gram matrices or canonical vectors, and compare those representations.The methods differ in which second-order information they use and how they compare it.Source: §3 · p. 3
• CCA averages canonical-direction correlations, while Linear CKA weights correlations by eigenvalues; SVCCA first applies SVD and uses top components.PWCCA weights correlations according to how much data their canonical directions capture.Source: §3 · p. 3
• CCA and SVCCA scores remain unchanged under invertible linear transformations of either representation, in addition to the invariances shared by these methods.The described shared invariances include data scaling, centering, and orthogonal transformations.Source: §3 · p. 3

4 Problems With CKA/CCA

• The case study presents an empirical and conceptual case against CKA/CCA: CKA misses a relationship after an architectural change that does not hurt performance, and the indexes have interpretability and assumption concerns.The setup compares three XLM-RoBERTa base models pretrained with different normalization schemes and fine-tuned on English XNLI.Source: §4 · p. 4, §4.1 · p. 4

4.1 Experiments Setup

• CKA scores for Normformer layers 6–12 stay around zero despite no substantial zero-shot transfer difference from the other normalization models.The table reports a 7–9% cross-lingual transfer gap across the three models; tuned Normformer layers 6–11 also lack revealed similarity.Source: Figure 1 · p. 5, §4.2 · p. 5
• Deeper Normformer layers retain cross-lingual sentence-matching scores, exceeding 50% for French, despite the near-zero CKA scores.The matching results show only a slightly decreasing trend across layers 6–12.Source: Figure 2 · p. 5, §4.2 · p. 5

4.3 Downsides of CCA

• CCA-like indexes are hard to interpret because their second-order comparisons obscure neuron-level explanations, and the section also identifies robustness concerns.The authors note that CCA/CKA combine neurons into linear combinations, making it difficult to trace which individual neurons drive a score.Source: §4.3 · p. 6
• Changing the index used to analyze mBERT produced conflicting conclusions in prior work, with SVCCA or CKA results aligning with other studies where PWCCA had suggested early divergence.The cited studies report behavioral evidence in the opposite direction from Singh et al. (2019a)’s PWCCA conclusion.Source: §4.3 · p. 6
• Similarity indexes capture different aspects of representations, so the analysis goal must determine what kind of similarity is sought.The paper introduces ANC as an alternative aligned with cross-lingual similarity analysis goals.Source: §4.3 · p. 6

5.1 Definition

• The neuron-wise score supports interpretation by showing which neurons contribute most or least and are most language-specific or language-neutral.The paper connects aligned neurons to zero-shot transfer: an English-tuned linear head can work on French when corresponding neurons are similar.Source: §5.1 · p. 6
• The alignment assumption reflects objectives in which representations of the same sentences should match; residual connections strengthen it for hidden layers.Source: §5.1 · p. 6
• ANC averages absolute correlations between corresponding neurons in two language representations, relying on their a priori one-to-one alignment.Absolute values account for negative correlations that a following layer can convert with a negative weight.Source: §5.1 · pp. 6–7
• ANC’s one-to-one alignment assumption is available for representations from a single network, but the method does not work across different networks.For layers from different networks, the authors say CCA-like indexes are likely a better fit.Source: §5.1 · p. 7

5.2 Sanity Checks

• ANC reveals the “first align, then predict” pattern in mBERT and XLM-R, and captures deeper-layer similarity in scale_normformer unlike CKA.For mBERT, correlations at the most language-natural layers are about 0.7.Source: §5.2 · p. 7, Figure 3 · p. 7, Figure 4 · p. 8
• ANC passes sanity checks against related-work findings and the Section 4 case where CKA fails, supporting its use for comparing languages within one model.Source: §5.2 · p. 8

6 Scaling Laws of Cross-lingual Representational Similarity in Multilingual Models

• The analysis tests whether the pattern generalizes to CLMs and persists in large-scale multilingual MLMs and CLMs.The motivation is that prior analyses often focused on a single model, while scaling has been associated with downstream performance improvements.Source: §6 · p. 8
• The study covers encoder-only MLMs and decoder-only CLMs, with models within each group varying in parameter counts and neurons per layer.Source: §6 · p. 8, Table 2 · p. 8
• All studied networks follow the “first align, then predict” pattern despite differences in training objective, number of languages, and model size.Source: §6 · p. 8, Figure 5 · p. 8, Figure 6 · p. 9

• The motivating example shows CKA fails to reveal deeper-layer cross-lingual similarity, while CCA methods have related problems despite passing that sanity check.Source: §7 · p. 9
• ANC passes the sanity check CKA fails and produces results consistent with evidence from related work.It is based on the assumption of neuron alignment in cross-lingual representations.Source: §7 · p. 9
• ANC shows that the “first align, then translate” pattern generalizes to CLMs and larger-scale MLMs and CLMs.Source: §7 · p. 9

• Experiments sample languages from Germanic, Romance, Slavic, Baltic, and Finno-Ugric families.Source: Ethical Considerations · p. 9
• The study does not cover other language families or extremely low-resource languages, which may be underrepresented in pretrained LMs and require different analysis tooling.Source: Ethical Considerations · p. 9
• For those cases, the authors recommend using their method together with previous methods for more reliable results.Source: Ethical Considerations · p. 9

• The appendix contains supplementary figures supporting auxiliary claims made throughout the paper.Source: A Appendix · p. 11
• Figure 7 shows CKA scoring deeper-layer similarity at about zero despite contrary evidence from Section 4.2.Source: Figure 7 · p. 11
• Figure 8 shows CKA captures last-layer CLS representation similarity but misses layers 6–11, whereas ANC captures similarity across all layers.The last layer uses CLS-pooled embeddings tuned for XNLI; other layers use mean pooling.Source: Figure 8 · p. 11
• Figure 9 shows PWCCA and SVCCA results are more intuitive for this XLM-Normformer case but struggle in other scenarios.Source: Figure 9 · p. 11