PaperlayerView exact revision on arXiv

Source-linked AI summary

The Geometry of Multilingual Language Model Representations

Tyler A. Chang, Zhuowen Tu, Benjamin K. Bergen

arXiv:2205.10964v2cs.CL

TL;DR

How multilingual language models integrate language-sensitive information with cross-lingual alignment remains unclear. This paper analyzes XLM-R through language-modeling predictions, subspace comparisons, and visualizations. It finds similar mean-centered subspaces alongside orthogonal axes encoding language-sensitive and language-neutral features.

  • Problem

    How multilingual language models’ representation geometry integrates language-sensitive information with cross-lingual alignment remains unclear.

    Source: §1 · p. 1

  • Method

    The paper analyzes XLM-R using language-modeling predictions, direct comparisons between language subspaces, and low-dimensional visualizations.

    Source: §1 · p. 1

  • Results

    XLM-R language subspaces are similar after mean-centering, while orthogonal language-sensitive and language-neutral axes encode features including vocabularies, token positions, and part-of-speech.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The authors conclude that multilingual models encode varied features along orthogonal axes in representation space for downstream tasks and multilingual learning.

    Source: §7 · p. 9

    The 88-language sample was skewed toward Indo-European languages, with no Native American languages, so the results may overestimate cross-language subspace similarity.

    Source: Limitations · p. 9

Abstract

from arXiv · show

We assess how multilingual language models maintain a shared multilingual representation space while still encoding language-sensitive information in each language. Using XLM-R as a case study, we show that languages occupy similar linear subspaces after mean-centering, evaluated based on causal effects on language modeling performance and direct comparisons between subspaces for 88 languages. The subspace means differ along language-sensitive axes that are relatively stable throughout middle layers, and these axes encode information such as token vocabularies. Shifting representations by language means is sufficient to induce token predictions in different languages. However, we also identify stable language-neutral axes that encode information such as token positions and part-of-speech. We visualize representations projected onto language-sensitive and language-neutral axes, identifying language family and part-of-speech clusters, along with spirals, toruses, and curves representing token position information. These results demonstrate that multilingual language models encode information along orthogonal language-sensitive and language-neutral axes, allowing the models to extract a variety of features for downstream tasks and cross-lingual transfer learning.

1 Introduction

The geometry of multilingual representations remains poorly understood, including how language-sensitive and language-neutral information coexist. This work analyzes XLM-R and finds similar mean-centered language subspaces alongside distinct axes that encode language-sensitive and language-neutral features.

1 Introduction

  • The study examines how multilingual representation geometry integrates language-sensitive information with cross-lingual alignment, a relationship whose underlying structure remains unclear.Prior work had examined these kinds of information but left the broader geometry and feature-encoding axes open questions.Source: §1 · p. 1
  • The analysis combines language-modeling predictions, direct comparisons between language subspaces, and low-dimensional visualizations using XLM-R as a case study.Source: §1 · p. 1
  • XLM-R language subspaces are similar after mean-centering, while their means encode language-sensitive information and distinct axes encode language-neutral features.The identified features include token vocabularies, token positions, and part-of-speech; the axes remain relatively stable across layers.Source: §1 · p. 2

2 Related work

Prior work has examined language-sensitive and language-neutral information in multilingual models, but has focused mainly on specific features or probing tasks. This paper instead considers how the broader geometry of the original representation space supports feature encoding and interactions.

2 Related work

  • Earlier studies linked language representations to typological and phylogenetic features, while also identifying shared subspaces for syntactic information and cross-lingual retrieval.These studies generally focused on specific features or probing tasks rather than the broader geometry of the original representation space.Source: §2 · p. 2

3 Language subspaces

The paper analyzes XLM-R language subspaces across 88 languages using contextualized representations, perplexity-based projections, and direct subspace comparisons. The results indicate that mean-centered subspaces are especially similar in middle layers, while language means account for substantial projection differences.

3 Language subspaces

  • For each language, the authors used SVD on contextualized token representations to identify affine subspaces that account for language-modeling performance and become similar after mean-centering.They used 262K representations per language and selected dimensionality to account for 90% of its variance.Source: §3 · p. 2, §3.2 · p. 2

3.1 Model and dataset

  • The study uses XLM-R representations from 88 languages shared by its pretraining corpus and OSCAR, excluding languages with fewer than 100 OSCAR sequences.Representations came from Transformer layers one through twelve and uncontextualized embeddings at layer zero.Source: §3.1 · p. 2

3.2 Affine language subspaces

  • Projecting onto each language’s own affine subspace caused only minor perplexity increases despite retaining less than half the original dimensionality.The median subspace dimensionality was 335 of 768 dimensions, and the authors conclude that these subspaces capture much of the information relevant to language modeling.Source: §3.2 · p. 3
  • Projecting onto another language’s subspace after shifting it to the evaluation language mean produced only moderately higher perplexity than using the evaluation language’s subspace, especially in middle layers.The widening gap between shifted and unshifted projections in deeper layers suggests that language means account for much of the performance degradation there.Source: §3.2 · p. 3, Figure 2 · p. 3

3.3 Subspace distances

  • In layers six through eleven, mean-centered language subspaces differed by rotations of less than five degrees and scalings of less than 1.6x.Figure 3 presents analogous rotations and scalings across layers, with shaded regions showing one standard deviation.Source: §3.3 · p. 4, Figure 3 · p. 4

4 Language-sensitive axes

XLM-R languages occupy similar subspaces after mean-centering, while differences between language means encode language-sensitive information. Mean-shifting and subspace projection induce target-language vocabulary predictions, and language-sensitive axes cluster languages by family and remain stable across middle layers.

4 Language-sensitive axes

  • After mean-centering, language subspaces are similar, but their means differ along axes that encode language-sensitive information such as vocabularies.The section assesses subspace similarity through language-modeling predictions and direct subspace comparisons.Source: §4 · p. 4

4.1 Inducing target language vocabulary

  • The vocabulary analysis defined language tokens by frequency of at least 1e-6 and excluded 945 common tokens appearing in at least 90% of 88 languages.The resulting language vocabularies ranged from 3K to 24K tokens out of XLM-R’s original 250K-token vocabulary.Source: §4.1 · pp. 4–5
  • Combined mean-shifting and subspace projection produced 4.7x more target-language tokens (10% →47%) and 3.5x fewer evaluation-language tokens (75% →21%) than the unmodified model.The combined transformation outperformed either operation individually; Figure 4 shows token proportions after the different transformations.Source: §4.1 · p. 5, Figure 4 · p. 5

4.2 Linear discriminant analysis (LDA)

  • Language-sensitive axes yielded nearly identical projections across layers five through nine and remained similar from layers three through eleven.The projections suggest representations change little along these axes through middle and late-middle layers.Source: Figure 5 · p. 6, §4.2 · p. 6
  • LDA projections loosely clustered languages by family, while earlier layers and the final layer appeared to cluster more by script.The first axes separated language representations; the script clustering appeared closer to token inputs or output predictions.Source: §4.2 · pp. 5–6

5 Language-neutral axes

The paper uses LDA to identify axes for token position and part-of-speech, then evaluates whether these features are encoded independently of input language. Position and POS information are largely language-neutral, and their representations remain stable across layers.

5.1 Token positions

  • Token position was identifiable from position axes without knowing the source language, indicating that it remains largely language-neutral through the model.The axes were identified by applying LDA to representations for every sixteen token positions, using 8K samples per position from all languages.Source: §5.1 · p. 7
  • Position information appeared in nonlinear structures: toruses encoded relative positions, while curves encoded absolute positions and spirals combined both.The paper hypothesizes that the structures reflect the model’s use of both relative and absolute position information.Source: §5.1 · p. 7, Figure 7 · p. 7
  • Position axes were stable from the second hidden layer through the last, with nearly identical curves when representations were projected onto axes identified in other layers.This suggests minimal transformations along language-neutral position axes as representations pass through the model.Source: §5.1 · p. 7

5.2 Part-of-speech

  • POS representations clustered roughly by tag independently of input language, using LDA axes identified from token representations mapped to Universal Dependencies tags.The UD corpus covered 61 of the 88 languages in OSCAR and XLM-R; visualizations using all 17 tags were omitted because early axes did not separate all tags.Source: §5.2 · p. 8
  • POS axes were relatively stable across layers one through ten, with similar projections from axes identified in other layers.The authors note this is consistent with middle layers processing higher-level information while retaining low-level information along stable axes.Source: §5.2 · p. 8

6 Multilingual structure

The section describes language subspaces as similar after mean-shifting but distinct along language-sensitive axes, while shared language-neutral axes encode features such as token positions and part-of-speech. It also notes that feature axes were often orthogonal, though the observed orthogonality may reflect the high-dimensional setting and feature selection.

6 Multilingual structure

  • Language subspaces differ along language-sensitive axes such as those encoding token vocabularies, while language-neutral axes encode token positions and part-of-speech.Representations in different languages do not directly occupy a shared multilingual subspace; extracting these signals requires projection onto specific axes and subspaces.Source: §6 · p. 8
  • The authors leave rigorous quantification of the language-sensitivity of individual features and axes to future work.They describe different types of language-sensitive and language-neutral axes in Appendix C.Source: §6 · p. 9
  • Axes encoding different features were often orthogonal and independent, allowing projections with minimal influence from extraneous noise and other dimensions.The authors caution that this may reflect low-dimensional subspaces being likely to be orthogonal in high-dimensional spaces and their selection of features likely to be minimally correlated.Source: §6 · p. 9
  • Language families, token positions, and part-of-speech were initially found to remain stably encoded across middle and late-middle layers.The authors suggest these layers may retain low-level feature structures for later token reconstruction, while representations change along higher-level feature axes.Source: §6 · p. 9

7 Conclusion

The conclusion identifies language subspaces and language-sensitive and language-neutral axes in XLM-R using several analytical methods. It suggests that representations encode features through projections onto orthogonal axes, supporting simultaneous encoding of varied signals for downstream tasks and multilingual learning.

7 Conclusion

  • The study identifies language subspaces and individual language-sensitive and language-neutral axes in XLM-R.The analyses use causal effects on language modeling predictions, direct subspace comparisons, and low-dimensional visualizations.Source: §7 · p. 9
  • The results suggest multilingual language models encode features by projecting representations onto orthogonal axes in the representation space.Source: §7 · p. 9
  • This organization supports efficient, simultaneous encoding of varied signals for downstream tasks and multilingual learning.Source: §7 · p. 9

Limitations

The findings are bounded by the linguistic coverage of XLM-R and OSCAR, variation in corpus size and quality, and experiments on only one pretrained model. The authors caution that these constraints may affect cross-language similarity estimates and the generalizability of the results.

Limitations

  • The 88-language sample is skewed toward Indo-European languages, and its limited linguistic diversity may lead to overestimates of cross-language subspace and representation similarity.52 languages are Indo-European; the sample includes six Austronesian languages, two African languages, and no Native American languages.Source: Limitations · p. 9
  • OSCAR and XLM-R pretraining corpora vary substantially in size and quality, potentially yielding less-nuanced representations and less-representative subspaces for some languages.The authors also note that language-sensitive axes may encode topic-distribution shifts across languages.Source: Limitations · p. 9
  • Because experiments used only XLM-R, other architectures, hyperparameter settings, or parameter initializations could produce different results.The authors call for future assessments covering a wider variety of models and languages.Source: Limitations · pp. 9–10

A Experimental details

Experiments use the base 270M-parameter XLM-R model and representations extracted from sampled OSCAR text sequences of up to 512 tokens. The reported experiments required approximately 17.5 days across eight Titan Xp GPUs.

A Experimental details

  • All experiments use the base-size XLM-R model with 270M parameters, implemented in Huggingface Transformers.Representations were extracted from unmodified text in the September 2021 OSCAR release.Source: A Experimental details · pp. 11–12
  • Each input sequence contained 512 tokens, formed by concatenating OSCAR lines, and samples came from up to the first 8M sequences per language.The sequences were randomly sampled.Source: A Experimental details · p. 12
  • Experiments ran on eight Titan Xp GPUs for approximately 17.5 days, with most computation spent evaluating perplexities and in-language token proportions.These evaluations covered projection types, language pairs, and layers in Sections 3.2 and 4.1.Source: A Experimental details · p. 12

B.1 Subspace distance metric

The section estimates language subspaces from contextualized representations and compares their covariance matrices with a distance metric. It calibrates observed distances against controlled rotations and scalings of each language subspace.

B.1 Subspace distance metric

  • 262K contextualized token representations per language were used to estimate subspaces for 88 languages, with Javanese and Sundanese as the only languages below that sample size.The authors found that this sample size produced relatively stable subspaces across random subsets.Source: §B.1 · p. 12
  • Each language covariance matrix is treated geometrically as an ellipsoid whose axes encode representation directions and whose scales encode variance in those directions.The covariance is computed from the right singular vectors and singular values of the mean-centered representation matrix.Source: §B.1 · p. 12
  • The distance between language covariance matrices uses a Riemannian metric that is symmetric and invariant to linear transformations.The metric defines distance through the shortest geodesic between positive definite matrices.Source: §B.1 · p. 12

B.2 Rotation and scaling comparisons

  • Rotation degrees and scaling multipliers are calibrated by transforming each language subspace and averaging its self-distance across languages and 16 random transformations.Rotations span 0–90 degrees; scaling multipliers span 1.00–4.00, with variance scaled by the multiplier squared.Source: §B.2 · p. 12
  • Each observed language-pair distance is mapped to the lowest rotation degree or scaling multiplier producing an equal or greater mean transformed-subspace distance.The mapping uses the monotonic increase in distance observed as rotation or scaling grows.Source: §B.2 · pp. 12–13

C Language-sensitive vs. language-neutral axes

The section characterizes language sensitivity through projected representation distributions and relates it to differences in means and variances. It also reports stable language-sensitive, token-position, and part-of-speech axes, while noting limits on interpreting axes from distributions alone.

C Language-sensitive vs. language-neutral axes

  • A subspace is language-sensitive when its projection has high mutual information with input-language identity, and language-neutral when that mutual information is low.The section distinguishes these cases by whether projected distributions differ across languages.Source: C Language-sensitive vs.… · p. 13
  • Language-neutral axes can have equal means and similarly high variance across languages, whereas high variance in one language and low variance in another is a language-sensitive pattern.Axes with equal means and low variance are also described as language-neutral, though they are unlikely to contain much relevant information; variance is not a perfect indicator of downstream importance.Source: C Language-sensitive vs.… · p. 13
  • Language-sensitive axes with unequal means and similarly high variance may represent languages as shifted distributions, while axes with unequal means and low variance in each language have little supporting evidence.The shifts could reflect true feature differences, imperfect alignment, or different features across languages.Source: C Language-sensitive vs.… · pp. 13–14

C.1 Extracting features

  • Representation distributions alone cannot establish whether an axis encodes the same feature across languages, so the analysis grounds interpretation in features and downstream consequences.The authors note that causal effects on predictions require independent verification.Source: §C.1 · p. 14

D Additional plots

  • The additional visualizations use the global representation mean as the origin and orthogonalize and normalize axes before projection.Changing the origin shifts projected representations by a constant vector without affecting the visualizations.Source: D Additional plots · p. 14

D.1 Language-sensitive axes

  • Language-separating LDA axes from layer eight yield similar projections across middle layers, with visualizations including all 88 language means.The language-separating axes in each layer are also projected for comparison.Source: §D.1 · p. 14

D.2 Token positions

  • Layer-eight token-position axes produce similar projections for representations from layers two through eleven, indicating stable position information across those layers.Source: §D.2 · p. 14

D.3 Part-of-speech

  • LDA identifies part-of-speech axes that remain stable across most layers, with similar projections from layers one through ten onto axes identified for layer eight.Additional plots show separations among several groups of parts of speech.Source: §D.3 · p. 14

D.4 Multiple features

The figures combine language-sensitive and language-neutral dimensions in shared projections. Viewing the projections from different angles separates language identity from part-of-speech and token-position structure.

D.4 Multiple features

  • The figures project representations onto LDA axes separating language, token position, or part of speech, including language-family groupings and multiple part-of-speech groups.Figure 11 identifies language families and isolates; Figure 15 displays several POS groupings.Source: Figure 11 · p. 15, Figure 13 · p. 16, Figure 14 · p. 17
  • Layer-two projections combine two part-of-speech axes with one language-sensitive axis; side and top-down views distinguish language sensitivity from part-of-speech.Source: Figure 16 · p. 18
  • Layer-eight projections combine a part-of-speech axis, a token-position axis, and a language-sensitive axis, with side and top-down views exposing these features separately.Source: Figure 17 · p. 18
Loading 2205.10964v2…