PaperlayerView exact revision on arXiv

Source-linked AI summary

High-Dimensional Interlingual Representations of Large Language Models

Bryan Wilie, Samuel Cahyawijaya, Junxian He, Pascale Fung

arXiv:2503.11280v5cs.CL

TL;DR

Whether multilingual LLMs form a unified interlingual representation or align only partially across languages remains unsettled. The paper proposes a framework for shared and fragmented components and an ILO measure of local-neighborhood alignment. It reports that single-language fine-tuning disrupts early-layer alignment, while selective freezing preserves alignment and improves cross-lingual generalization.

  • Problem

    Evidence is mixed on whether multilingual LLMs converge to a unified interlingual representation or remain partially aligned across language groups.

    Source: §1 · pp. 1–2

  • Method

    The paper proposes a framework for shared and fragmented interlingual components and introduces ILO to quantify alignment through local neighborhood structures in high-dimensional representations.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    Single-language fine-tuning disrupts early-layer interlingual alignment, while selectively freezing parameters preserves alignment and improves cross-lingual generalization.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings underscore interlingual semantic alignment’s role in scalable multilingual learning and offer a quantitative view of alignment in multilingual representations.

    Source: §1 · p. 2, §6 · p. 9

    ILO captures generic interlingual mixing rather than task-vector alignment and does not predict downstream cross-lingual transfer performance.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) trained on massive multilingual datasets hint at the formation of interlingual constructs--a shared subspace in the representation space. However, evidence regarding this phenomenon is mixed, leaving it unclear whether these models truly develop unified interlingual representations, or present a partially aligned constructs. We explore 31 diverse languages varying on their resource-levels, typologies, and geographical regions; and find that multilingual LLMs exhibit inconsistent cross-lingual alignments. To address this, we propose an interlingual representation framework identifying both the shared interlingual semantic subspace and fragmented components, existed due to representational limitations. We introduce Interlingual Local Overlap (ILO) score to quantify interlingual alignment by comparing the local neighborhood structures of high-dimensional representations. We utilize ILO to investigate the impact of single-language fine-tuning on the interlingual representations in multilingual LLMs. Our results indicate that training exclusively on a single language disrupts the alignment in early layers, while freezing these layers preserves the alignment of interlingual representations, leading to improved cross-lingual generalization. These results validate our framework and metric for evaluating interlingual representation, and further underscore that interlingual alignment is crucial for scalable multilingual learning.

1 Introduction

The introduction frames whether multilingual LLMs form a unified interlingual representation or only partially align some languages as an open question. It presents a framework and ILO metric for studying this alignment, then reports that selective freezing during single-language fine-tuning preserves alignment and improves cross-lingual generalization.

1 Introduction

  • The paper asks whether multilingual LLMs develop a unified interlingual representation or remain partially aligned across language groups.Prior evidence is mixed, leaving the extent of alignment across typologies, geographical regions, and resource levels unclear.Source: §1 · pp. 1–2
  • The framework distinguishes a shared-semantic core from fragmented components, and ILO measures alignment by comparing high-dimensional representations’ local neighborhood structures.ILO combines measures of whether a language’s representations neighbor diverse languages individually and connect to them collectively.Source: §1 · p. 2
  • Preliminary experiments find that parallel semantic inputs are represented differently across languages, with better neuron-activation alignment among high-resource pairs and languages sharing family or regional roots.These observations motivate examining alignment across diverse language groups.Source: §1 · p. 2
  • Selective freezing during single-language fine-tuning preserves interlingual alignment and improves cross-lingual generalization, whereas single-language training disrupts alignment in early layers.The analysis uses multilingual mathematical reasoning, selected for its language-agnostic properties.Source: §1 · p. 2
  • The analysis covers 31 languages sampled from Flores-200 and distributed across language families, regions, and resource levels.The supplied table caption identifies these three dimensions of language coverage.Source: Table 1 · p. 2

2 Related Works

Earlier interlingua work developed syntactic and semantic representations, while recent studies indicate that LLMs inherit shared interlingual representations. The characteristics of those representations in LLMs remain unexplored.

2 Related Works

  • Earlier syntactic interlingua approaches used language-specific delexicalized grammars or shared abstract syntax, but the latter could be incomplete and difficult to scale to new languages.Examples include Spoken Language Translator, PARC’s XLE, and Verbmobil.Source: §2 · p. 2
  • Semantic interlingua work moved toward latent semantic vectors and specialized objectives for constructing cross-lingual representations.The passage situates these methods alongside statistical machine translation and cross-lingual alignment.Source: §2 · p. 2
  • Studies indicate that current LLMs inherit shared interlingual representations, but their characteristics remain unexplored.In Aya Expanse (8B), early and late layer embeddings cluster by resource levels and linguistic features and minimally overlap in the supplied visualization.Source: §2 · p. 3, Figure 2 · p. 3

3 Interlingual Representations in Multilingual LLMs

The section treats multilingual representations as language-specific encodings in a shared space, while allowing their alignment to vary. It distinguishes a core region of shared semantics from fragmented regions where some representations align only partially.

3 Interlingual Representations in Multilingual LLMs

  • The section models each language’s input through a language-specific encoding function in a shared high-dimensional space and defines semantic alignment for parallel inputs.The alignment definition uses a similarity function over semantically equivalent input pairs; higher α indicates better alignment.Source: §3 · p. 3

3.1 Multilingual Shared Representation Space

  • The framework partitions multilingual representations into an aligned core that captures shared semantics and fragmented regions where alignment with that core is challenging.It allows alignment quality to vary with representational limitations and language-specific properties.Source: §3.1 · p. 3
  • Figure 3 reports stronger per-layer ANC correlations for high-resource language pairs than for pairs involving low-resource languages.The figure compares per-layer scores on Aya Expanse (8B).Source: Figure 3 · p. 4

3.2 Core Interlingual Region

  • The core interlingual region predominantly encodes universal semantic structures and syntactic abstractions.The paper associates this region with semantic emphasis and retaining language signals for language-specific predictions.Source: §3.2 · p. 4

3.3 Fragmented Region

  • Languages with less overlap in the core may occupy fragmented regions because sparse data, typological distance, or morphological complexity can hinder partial alignment.The paper says such representations may be weakly aligned to the universal semantics in the core, potentially lowering performance on tasks using those languages.Source: §3.3 · p. 4

4 Semantic Alignment of Multilingual LLMs Representations

Across 31 languages and several multilingual LLMs, this section examines hidden-state alignment by layer and language properties. It finds intermediate-layer overlap but uneven alignment by resource level and linguistic grouping.

4 Semantic Alignment of Multilingual LLMs Representations

  • The analysis projects hidden states to 2D to inspect parallel-language proximity, then measures activation consistency across 31 Flores-200 languages and four multilingual models.Languages vary in resource level, region, and family.Source: §4 · p. 4

4.2 Cross-lingual Alignments Depend on Resource-level and Linguistic Properties

  • High-resource pairs correlate more strongly, and within-group pairs align better; gaps are linked to sparse data, typological distance, and morphological complexity.Source: Figure 4 · p. 5, §4.2 · p. 5

4.1 Inherent Regional Clustering with Mid-Layers High-Resource Alignment

  • Intermediate layers show cross-family and cross-region overlaps, including some low-resource languages, while early and late layers cluster by resource and linguistic features with minimal overlap.Some parallel representations remain fragmented outside the intermediate-layer overlaps.Source: §4.1 · p. 5

4.2 Cross-lingual Alignments Depend on Resource-level and Linguistic Properties

  • They quantify layer-wise alignment with Average Neuron-wise Correlation (ANC), correlating corresponding neuron activations for semantically identical inputs across languages.Source: §4.2 · p. 5

5 Intrinsic Interlinguality of LLMs

This section defines Interlingual Local Overlap (ILO) to assess multilingual alignment, then studies how single-language fine-tuning and selective freezing affect alignment and cross-lingual transfer. Fine-tuning disrupts early-layer alignment, while selective freezing preserves alignment and improves cross-lingual generalization.

5.1 Interlingual Local Overlap Score

  • ILO compares local neighborhoods of semantically parallel inputs, and a high aggregated score signals aligned interlingual semantics across diverse language inputs.The score combines local mixing and cross-language reachability using a harmonic mean.Source: §5 · pp. 5–6, §5.1 · p. 6
  • Bridge measures the share of samples whose k-nearest neighbors include at least τ other languages; reachability measures the fraction of unique other languages encountered across a language’s samples.The multilingual representation space is treated as a graph whose hidden-state embeddings connect to their k-nearest neighbors.Source: §5.1 · p. 6

5.2 Experiment Design

  • The selective-freezing experiment freezes the first 4, 8, 12, or 16 layers, as well as token embeddings, final layer normalization, and the language-modeling head.These parameters are identified as language aligners.Source: §5.2 · p. 6
  • Experiments use Llama-3.1 (8B) and Gemma-2 (9B), GSM8KInstruct and MGSM, greedy zero-shot accuracy, and ILO with k = 10, τ = 5, and Euclidean distance.The paper reports consistent ILO trends for other tested neighborhood sizes and distances.Source: §5.2 · p. 7

5.3 Results and Analysis

  • Fine-tuning transfers to target languages, especially high-resource and same-family or same-region languages; Spanish-to-English reaches 38.4%, on par with Spanish-to-Spanish.Some target languages nevertheless show performance degradation after fine-tuning.Source: §5.3 · pp. 7–8
  • Selective freezing sustains pretrained semantic alignment and improves cross-lingual generalization; freezing the first 12 layers gives the best aggregated improvements.The reported transfer gains extend to low-resource, cross-family, and cross-regional languages, with nearly no degradation.Source: Figure 5 · p. 8, §5.3 · p. 8
  • Single-language fine-tuning decreases alignment as early as layer 4 in Llama-3.1 and layer 6 in Gemma-2, without later recovery to pretrained levels.The authors also report reduced overlap between language clusters after fine-tuning.Source: §5.3 · p. 8

6 Conclusion

The paper frames multilingual representations as a shared semantic core alongside fragmented components, and introduces ILO to quantify local interlingual alignment. It presents alignment as important to understanding and optimizing multilingual LLMs.

6 Conclusion

  • The framework distinguishes a shared semantic core from fragmented components that reflect representational limitations.The core captures shared semantics across languages; ILO measures alignment in local neighborhoods of high-dimensional representations.Source: §6 · p. 8
  • Interlingual Local Overlap (ILO) quantifies alignment through local neighborhood structures in high-dimensional representations.Source: §6 · pp. 8–9
  • The study emphasizes semantic alignment as important for analyzing and optimizing multilingual LLMs across diverse linguistic tasks.Source: §6 · p. 9

Limitations

The study identifies limitations in language coverage, task scope, and what its alignment framework captures. It also treats expansion of the shared core and integration of fragmented regions as future work.

Limitations

  • The 31-language sample has an Indo-European predominance among high-resource languages, so stronger HRL correlations may partly reflect data-availability bias.The passage links this imbalance to the disproportionate representation of Indo-European languages in CommonCrawl.Source: Limitations · p. 9
  • Cross-lingual transfer is evaluated primarily on multilingual mathematical reasoning, and the authors call for evaluation on other tasks.They choose the task for its largely language-agnostic nature and its joint assessment of linguistic understanding and logical reasoning.Source: Limitations · p. 9
  • The framework presumes a shared core interlingual region, leaving its existence and broader language coverage as assumptions and future research questions.The authors suggest larger, more heterogeneous datasets and linguistic priors could strengthen the core and its universality.Source: Limitations · p. 9
  • Underrepresented or typologically distant languages may form fragmented regions rather than integrate fully with the core, including under extremely limited data.Targeted strategies to bridge those regions are proposed as future work.Source: Limitations · p. 9
  • The proposed measures capture generic hidden-state mixing, not task-vector alignment, and do not predict downstream cross-lingual transfer performance.The authors note that strong interlingual alignment may coexist with loss of task-relevant nuances.Source: Limitations · p. 9
  • The study focuses on textual embeddings; extending local-neighborhood alignment analysis to multimodal settings remains future work.The authors suggest such work could help separate semantic content from modality-specific biases.Source: Limitations · p. 9

Ethical Considerations

The paper presents its framework and metrics as tools for inspecting interlingual alignment and representational gaps. Its appendices provide language-sample details and additional ANC and t-SNE visualizations.

Ethical Considerations

  • The authors frame the framework and metrics as tools for inspecting alignment and disparities, using public parallel corpora and excluding sensitive or personally identifiable information.They connect recognition of shared semantics with linguistic diversity and fairer representation.Source: Ethical Considerations · p. 10

A Details on Linguistic Properties

B Further Details on ANC Scores

  • The ANC appendix reports aggregate peak scores as means over each pair’s peak layers, selected from the top three layers by 75th-percentile ANC.It reports that top correlated pairs among LLMs are similar on HRLs; instruction-tuned models show similar top-pair sets to their pretrained counterparts despite ranking differences.Source: B Further Details on ANC Scores · p. 14

C.1 ANC Comparisons from Other LLMs

  • The appendix provides ANC visualizations for Aya Expanse, Llama-3.1, Llama-3.1-Instruct, Gemma-2, Gemma-2-Instruct, and Qwen.Source: §C.1 · p. 14

C.2 T-SNE Visualizations from Other LLMs

  • The appendix provides t-SNE visualizations of hidden-state embeddings for Aya Expanse, Qwen, Llama-3.1, Llama-3.1-Instruct, Gemma-2, and Gemma-2-Instruct.Source: §C.2 · p. 14

C.3 Reports on Cross-Lingual Transfer Experiments for Gemma-2 (9B)

The appendix points to Gemma-2 (9B) cross-lingual transfer results and layer-wise ILOL scores across pretrained, fine-tuning, and selective-freezing modes.

C.3 Reports on Cross-Lingual Transfer Experiments for Gemma-2 (9B)

  • For Gemma-2 (9B), the appendix reports cross-lingual transfer performance on MGSM and layer-wise ILOL scores across pretrained, fine-tuning, and selective-freezing modes.The transfer results are in Table A3 and the layer-wise ILOL scores are in Figure A1.Source: §C.3 · p. 14

D Interlingual alignments of various multilingual LLMs

Multilingual LLMs share a middle-layer interlingual region, but alignment strength varies across models. ILO scores correspond to t-SNE overlap, and the section also reports ablation analyses.

D Interlingual alignments of various multilingual LLMs

  • Gemma-2 shows stronger cross-lingual alignment than Aya Expanse: its intermediate-layer mean correlations exceed 0.5, while Aya’s above-0.5 correlations are limited to HRL pairs and in-region pairs.Top correlated language pairs are similar among LLMs, and instruction-tuned models show similar top-pair sets to their pretrained counterparts.Source: D Interlingual alignments of various… · p. 14, Table A2 · p. 15

E Observation of Interlingual Alignment Preservation in T-SNE Projections

  • High ILO scores correspond to interlingual overlap in t-SNE projections, while lower scores correspond to more fragmented representations.The authors present this correspondence as validation of ILO as a measure of local structure in the multilingual shared embedding space.Source: E Observation of Interlingual… · p. 15

F Ablation Studies

  • The section reports comprehensive ablations of the study’s hyperparameters and analyzes their effects.The supplied passage does not specify the ablation outcomes.Source: F Ablation Studies · p. 15

F.1 t-SNE perplexity

The broad t-SNE overlap patterns persist across perplexity settings despite some language-pair differences. ILO trends also remain consistent across tested k-NN parameters and distance metrics.

F.1 t-SNE perplexity

  • Across tested perplexities, broad layerwise overlap patterns remained consistent despite some language-pair differences.Early and late layers generally showed minimal overlap, while intermediate-layer overlaps mainly involved high-resource languages and some regionally overlapping low-resource languages.Source: §F.1 · p. 15

F.2 k-NN parameters of the ILO score

  • About 0.03–0.05 higher aggregated ILO across layers with k=5, τ=3, while k=20, τ=10 lowered it roughly 0.1–0.15 versus Figure 5.With cosine distance, changes from varying k were less pronounced and aggregated ILO scores remained in a similar range.Source: §F.2 · p. 16
  • Across k-NN settings and distance metrics, ILO shows alignment decreases in the same layers, while selective freezing sustains prior alignment levels across layers.This includes cosine and Euclidean distances and the tested k and τ settings.Source: §F.2 · p. 16

F.3 Layer selection for selective freezing

In Llama-3.1 (8B) MGSM experiments, freezing early layers improves cross-lingual generalization, with the first 12 layers yielding the best overall performance. Alignment outcomes vary with how many layers are frozen.

F.3 Layer selection for selective freezing

  • Across settings, transfer improved within language families and regions, with negligible degradation—and sometimes improvement—in low-resource, cross-family, and cross-regional scenarios.With 12 frozen layers, performance improved for English, Russian, French, German, and Bengali, but decreased for Spanish, Chinese, Japanese, Swahili, and Thai.Source: §F.3 · pp. 16–17
  • Freezing the first 12 layers gave the best cross-lingual generalization in Llama-3.1 (8B) MGSM ablations.The study compared freezing the first 4, 8, 12, and 16 layers.Source: §F.3 · p. 16, Table A4 · p. 17
  • Freezing the first 4, 8, or 12 layers maintained and improved alignment across layers, whereas freezing 16 preserved alignment only in frozen layers and yielded lower alignment in subsequent layers than fine-tuning.The ablation figure reports this pattern across the tested layer selections.Source: §F.3 · p. 16, Figure A24 · p. 34

Related papers

Loading 2503.11280v5…