PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Lingual Representation Alignment by Token-Level Optimal Transport in a Language-Agnostic Space

Taisei Yamamoto, Ryoma Kumon, Danushka Bollegala, Hitomi Yanaka

arXiv:2609.06381v1cs.CL

TL;DR

Prior cross-lingual alignment methods may overlook language-specific information or rely on sentence-level alignment, limiting how they handle token-level differences across languages. CAROT decomposes representations and aligns language-agnostic components token by token while preserving language-specific components. CAROT-trained models outperform existing CLA methods in 11 of 18 evaluation settings.

  • Problem

    Prior CLA methods often ignore language-specific information or align only at sentence level, while languages can differ in token order and grammatical markers.

    Source: §1 · p. 2

  • Method

    CAROT uses LEACE to separate language-specific from language-agnostic representations, aligns the latter across languages at the token level with optimal transport, and preserves the former.

    Source: §1 · p. 2

  • Results

    Models trained with CAROT outperform existing CLA methods in 11 of 18 model-task-language settings, though their gains do not match those observed with steering.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CAROT provides an interpretable framework and insights into ideal multilingual representations for cross-lingual alignment.

    Source: §8 · p. 9

    Because LEACE considers only linear features, CAROT may overlook nonlinear language features used by LLMs, potentially limiting its effectiveness.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Cross-lingual alignment (CLA) aims to align the representations of large language models (LLMs) across languages, enabling cross-lingual transfer to improve multilingual capabilities. Previous CLA methods often ignore language-specific information encoded in representations and only consider sentence-level alignment, which may lead to suboptimal performance and input-output language mismatch. We propose CAROT (Cross-Lingual Alignment of Representations in a Language-Agnostic Space via Optimal Transport), which consists of two steps: identifying language-specific representations in LLMs' internal states and aligning language-agnostic representations across languages at the token level by optimal transport, while explicitly preserving language-specific representations. Inference-time steering experiments show that the representations computed by CAROT are effective alignment targets, improving multilingual performance by up to 11.2 points in accuracy while maintaining input-output language consistency. We further use the representations obtained by CAROT as training targets, internalizing the aligned representations. The trained models outperform existing CLA methods in 11 of 18 evaluation settings (3 models $\times$ 3 tasks $\times$ ID/OOD languages). Our work provides insights into what constitutes effective alignment targets for CLA in LLMs. Code is available at https://github.com/ynklab/CAROT

1 Introduction

Cross-lingual transfer can help address LLMs’ limited abilities in low-resource languages, but prior alignment methods may overlook language-specific information or operate only at sentence level. CAROT aligns language-agnostic representations token by token while preserving language-specific representations; its steering targets improve multilingual performance without language mismatch, and its trained models outperform existing CLA methods in 11 of 18 settings.

1 Introduction

  • Cross-lingual transfer—the ability to generalize knowledge and skills across languages—is important because LLMs have limited understanding of low-resource languages with insufficient training data.CLA aligns representations across languages so semantically similar inputs map to similar representations.Source: §1 · p. 1
  • Prior training methods often align parallel sentences without preserving language-specific information, while many steering methods use global shifts rather than token-level alignment.Different word orders and language-specific markers, such as Japanese “wa” and “ga,” motivate finer-grained alignment.Source: §1 · pp. 1–2
  • CAROT uses LEACE to isolate language-specific information, then aligns source and target language-agnostic representations token by token with optimal transport while preserving the language-specific component.The source is the model’s dominant language; the target is the language to improve.Source: §1 · p. 2, Figure 1 · p. 2
  • CAROT-derived representations improve multilingual performance in steering experiments without inducing input-output language mismatch.These experiments support using the representations as alignment targets.Source: §1 · p. 2
  • Models trained with CAROT outperform existing CLA methods in 11 of 18 model-task-language settings, though their gains do not match those observed with steering.The settings span three models, three tasks, and in-distribution and out-of-distribution languages.Source: §1 · p. 2

2 Related Work

Prior work describes multilingual LLMs as using a shared semantic space and links cross-lingual representation similarity with task performance. Existing CLA approaches include steering and training, but often align only sentence-level representations or fail to preserve language-specific information; CAROT addresses these gaps with token-level alignment of language-agnostic representations.

Multilingual Mechanisms of LLMs

  • Logit-lens studies find that the top-ranked language in middle layers is often the model’s dominant language, and other work finds shared linear subspaces across languages.These findings support the hypothesis that multilingual inputs map to a shared semantic space for inference.Source: §2 · p. 2

Cross-Lingual Alignment (CLA)

  • Prior CLA work relates stronger performance to representations closer to the dominant language and applies steering or contrastive training to increase cross-lingual similarity.Several training approaches align sentence-level representations while ignoring language-specific information; another considers it but still aligns at sentence level.Source: §2 · p. 3

Concept Erasure

  • LEACE removes linear features representing a target concept while minimizing distance from the original representations; CAROT uses it to isolate language-specific features in LLMs.Earlier multilingual methods removed language-specific components using per-language centroids, mean-and-variance normalization, or low-rank subspaces.Source: §2 · p. 3

3 CAROT

CAROT decomposes token representations into language-specific and language-agnostic components, aligns the latter across languages with optimal transport, and restores the target-language component. Its outputs can guide inference-time steering or serve as training targets.

3 CAROT

  • CAROT has two stages: LEACE-based decomposition of hidden representations, followed by token-level optimal-transport alignment of language-agnostic components.The target-language-specific component is preserved.Source: §3 · p. 3

3.1 Language-Specific and Language-Agnostic Decomposition

  • Sentence-level weighted averages train layer-specific multiclass LEACE to identify language features; the eraser is then applied tokenwise to separate language-specific and language-agnostic components.Averaging is intended to expose dominant sentence semantics while making language direction more salient to LEACE.Source: §3.1 · p. 3

3.2 Token-Level Alignment with Optimal Transport in a Language-Agnostic Space

  • CAROT aligns target and source language-agnostic tokens using unbalanced optimal transport with Sinkhorn, uniform marginals, and squared Mahalanobis distance as cost.Unbalanced transport accommodates differing sequence lengths and tokens without clear counterparts; Mahalanobis distance accounts for anisotropic hidden vectors.Source: §3.2 · pp. 3–4
  • Perplexity filtering removes diffuse transport matches, retaining tokens confidently matched to a focused subset of the other language’s tokens.The threshold controls which target and source tokens are zeroed out.Source: §3.2 · p. 4
  • Each target token’s aligned representation interpolates between transported source and original target features according to received transport mass; low-mass tokens retain their original features.CAROT then adds back the preserved target-language-specific component to form the output representation.Source: §3.2 · p. 4

3.3 Using CAROT Representations

  • CAROT representations serve as inference-time steering targets or training targets; steering needs source-language translations, whereas training requires parallel data only during training.Training uses alignment and language-modeling losses, alternating them in the first half before using only the language-modeling loss in the second half.Source: §3.3 · pp. 4–5

4 Experimental Setup

Experiments use three multilingual LLMs and evaluate knowledge retrieval, reasoning and knowledge, reading comprehension, and open-ended question answering across selected target languages. Hyperparameters are selected using GMMLU development accuracy, and evaluation tracks accuracy, language fidelity, and cross-lingual agreement.

Models and Datasets

  • Experiments use Llama3.1-8B, gemma3-4b, and Qwen3-4B, with 12 ID languages and 2,560 sentences per language for LEACE training.LEACE data come from WMT24++ and NTREX.Source: §4 · p. 5
  • Evaluation covers KLAR knowledge retrieval, GMMLU knowledge and reasoning, BELEBELE reading comprehension, and XQuAD open-ended question answering.English is the source; target-language counts are seven for KLAR, nine for GMMLU and BELEBELE, and five for XQuAD.Source: §4 · p. 5

Hyperparameter Search

  • Hyperparameters are selected by grid search using GMMLU development accuracy, with training learning rates searched over 5e-4, 1e-4, 5e-5, and 1e-5.CAROT training reuses each model’s best steering layer and filtering threshold.Source: §4 · p. 5

Metrics

  • Evaluation uses accuracy, fidelity—the share of answers in the input language—and cross-lingual agreement—the share of target-language answers matching source-language answers.GlotLID identifies the output language for fidelity.Source: §4 · p. 5

5 Analysis of Language-Specific and Language-Agnostic Representations

LEACE separates language-specific from language-agnostic sentence representations: remaining representations mix across ID languages, while erased representations separate by language. The similarity comparison supports this distinction.

5 Analysis of Language-Specific and Language-Agnostic Representations

  • For ID languages, remaining representations mix across languages, while LEACE-erased representations separate by language; OOD erased representations also separate, with some residual language clustering in remaining representations.The same trend holds across other layers and models; LEACE captures language-specific representations that generalize to OOD languages to some extent.Source: §5 · pp. 5–6, Figure 2 · p. 6
  • In remaining representations, inter-language similarity is higher than intra-language similarity, indicating language-agnostic representations; in LEACE-erased representations, intra-language similarity is consistently close to 1, indicating language-specific components.Source: §5 · p. 6

6 Steering Evaluation

CAROT outperforms steering baselines in most cases while largely maintaining language fidelity. Its results also show early-layer accuracy gains, increased source–target similarity, and robustness contributions from its decomposition and transport-plan filtering.

6.1 Baselines

  • Lang Vec uses a diff-in-means vector offset and MLRS suppresses then restores language-specific components; both align at sentence level, unlike CAROT’s token-level alignment.CAROT uses LEACE for a theoretically grounded language-specific decomposition.Source: §6.1 · p. 6

6.2 Results and Analysis

  • CAROT outperforms steering baselines in most cases, with up to 11.2 points higher accuracy than Unsteered and no substantial fidelity degradation.Table 1 averages task accuracy and language fidelity across target languages; accuracy improvements over Unsteered are statistically significant in marked cases.Source: §6.2 · p. 6, Table 1 · p. 7
  • CAROT’s best steering layers are early-to-middle, and its accuracy improves most in early layers while fidelity stays high across languages and layers.Later-layer steering may be weaker because the intervention targets question tokens after relevant information has propagated to the answer-generating position.Source: §6.2 · pp. 6–7, Figure 4 · p. 7
  • Source–target similarity of language-agnostic representations rises at the steering layer and remains higher later; inspected transport plans are sparse, low-noise, and align semantically corresponding tokens.These observations support successful alignment of language-agnostic representations across source and target languages.Source: §6.2 · pp. 7–8, Figure 5 · p. 8
  • LEACE decomposition and transport-plan filtering contribute to robustness; removing filtering can reduce Qwen3-4B/KLAR accuracy by 14.0 points and lowers fidelity in many settings.CAROT also outperforms a last-token-only swap variant in most cases, supporting the effectiveness of token-level alignment for accuracy.Source: §6.2 · p. 8
  • CAROT adds a source-language forward pass and OT computation, roughly doubling Spanish prefill cost, but this once-per-prefill overhead is amortized over generation.Table 2 reports the forward-pass timing setup for Llama3.1-8B on GMMLU test samples.Source: §6.2 · p. 8, Table 2 · p. 8

7 Training Experiments

CAROT training is evaluated as a way to internalize language-agnostic representations computed by CAROT. It achieves the best accuracy in 11 of 18 settings, while training gains remain smaller than steering gains and may be affected by representation drift.

7 Training Experiments

  • The training objective targets language-agnostic representations computed by CAROT, with models trained to match those representations.The experiments apply instruction-tuning-style training with cross-lingual alignment objectives to base, non-instruction-tuned models.Source: §7 · p. 8, §7.1 · p. 8

7.1 Training Settings

  • Training uses WMT24++ and NTREX for alignment data and Aya Dataset for instruction tuning, with five high-resource training languages and ID/OOD evaluation.XQuAD is omitted because models did not consistently produce the required extractive-generation format, making accuracy uninformative.Source: §7.1 · p. 8

7.2 Results and Analysis

  • CAROT achieves the best accuracy in 11 of 18 model-task-language settings across three models, three tasks, and ID/OOD languages.The evaluation covers GMMLU, KLAR, and BELEBELE; Table 3 aggregates accuracy by task and ID/OOD language group.Source: §7.2 · p. 9, Table 3 · p. 9
  • CAROT training increases source-target language-agnostic representation similarity, but its overall performance gains are smaller than those from steering.The representation-similarity analysis uses the FLORES+ devtest set and reports Llama3.1-8B results.Source: §7.2 · p. 9
  • Precomputed LEACE erasers and covariance matrices may become inaccurate as representations change during training; average principal angles reach 40.3° and 32.8°, respectively.The paper proposes periodically updating these matrices as future work.Source: §7.2 · p. 9
  • CAROT training yields smaller improvements on GMMLU, which includes reasoning-heavy problems; the experiments consider SFT objectives, while RL-based training is left for future work.GRPO is given as an example of a recent RL approach aimed at improving reasoning capabilities.Source: §7.2 · p. 9

8 Conclusion

CAROT separates language-specific and language-agnostic components, then aligns the latter across languages at the token level while preserving the former. Steering and training experiments report multilingual improvements, with trained models outperforming previous CLA methods in 11 of 18 settings.

8 Conclusion

  • CAROT separates language-specific and language-agnostic components in hidden states, then aligns language-agnostic representations across languages at the token level.The method preserves language-specific representations during alignment.Source: §8 · p. 9
  • CAROT-trained models outperform previous CLA methods in 11 of 18 evaluation settings, although their gains do not fully match steering gains.The settings span three models, three tasks, and ID/OOD languages.Source: §8 · p. 9
  • Steering with CAROT improves multilingual performance without compromising input-output language consistency, but requires a source-language translation at inference.Training with CAROT uses its representations as objectives so translation is needed only during training.Source: §8 · p. 9

Limitations

The paper identifies limits in its evaluation and training setup, including model-size scope, LEACE’s linear-feature assumption, and untested cultural understanding. It also describes the datasets used and how LEACE separates language-specific from language-agnostic components.

Limitations

  • Experiments cover 4–8B parameter models; effectiveness on larger models remains uninvestigated, and the evaluation does not assess cultural understanding.The authors also leave incorporating CAROT into RL frameworks and periodically updating LEACE and covariance matrices during training for future work.Source: Limitations · pp. 9–10
  • LEACE considers only linear features, so CAROT may overlook nonlinear language features used by LLMs and thereby limit its effectiveness.The authors note that representations drift during training while precomputed LEACE and covariance matrices remain fixed; periodic updates may provide a more accurate training signal.Source: Limitations · p. 10

A Details on Models and Datasets

  • Experiments use instruction-tuned Llama3.1-8B, gemma3-4b, and Qwen3-4B for steering, and their base variants for training.The study uses WMT24++ and NTREX to train LEACE, FLORES+ to analyze representations, four evaluation datasets, and Aya Dataset for instruction tuning.Source: A Details on Models and Datasets · p. 13

B LEACE

  • LEACE is applied to hidden states using input language as the concept, yielding a language-agnostic component and a language-specific residual.Its affine transformation aims to erase linear predictability of the concept while minimizing expected squared distance from the original vectors.Source: B LEACE · p. 13

C Additional Results for Language-Specific and Language-Agnostic Representations

The additional analyses examine LEACE’s representation separation, its language-classification behavior, and CAROT’s token transport plans. They also specify evaluation protocols, steering baselines, and hyperparameter selection.

C Additional Results for Language-Specific and Language-Agnostic Representations

  • At layers 6 and 25, LEACE-erased Llama3.1-8B representations separate more clearly by language, while remaining representations mix across languages; gemma3-4b and Qwen3-4B show a similar trend.Gemma3-4b’s original representations appear more mixed, possibly due to anisotropy; its top principal component contributes 89.6% at layer 13.Source: C Additional Results for… · p. 13, Figure 6 · p. 15, Figure 7 · p. 15
  • After LEACE, language-classification accuracy drops clearly for ID languages but remains almost 100% for OOD languages, indicating residual language-specific information.The authors note that remaining ID accuracy around 50–80% may reflect domain overlap or classifier overfitting.Source: C Additional Results for… · pp. 13–14

D Optimal Transport Plans

  • CAROT’s perplexity-filtered transport plans are sparse and align semantically corresponding tokens, such as “(Tur)quie” and “Turkey.”The authors state that filtering can improve alignment robustness by suppressing noise.Source: D Optimal Transport Plans · p. 14

E Evaluation Protocols

  • Evaluation uses KLAR Continent, GMMLU, BELEBELE, and XQuAD, selected for parallel translations and language- and culture-neutral evaluation.GMMLU, BELEBELE, and XQuAD prompts are translated from English; the study uses test sets except for XQuAD, which uses its validation set.Source: E Evaluation Protocols · p. 14

F Baseline Steering Methods

  • Lang Vec steers by adding a source–target mean hidden-vector difference to target-language hidden states, with strength and layer selected by search.The search tests α values from 0.5 to 2.0 and every third layer for each model.Source: F Baseline Steering Methods · pp. 14–15
  • MLRS decomposes concatenated language mean vectors into language-agnostic and language-specific components, then intervenes on the language-specific component at inference.Its coefficient rank is set to L − 1, and the optimal objective solution is obtained using singular value decomposition.Source: F Baseline Steering Methods · pp. 15–16

G Selected Hyperparameters

  • Steering and training hyperparameters are selected using GMMLU dev accuracy; training CAROT uses its steering-selected layer and θ values.The steering and training settings are reported in Tables 5 and 6, respectively.Source: G Selected Hyperparameters · p. 16

H Additional Steering Results

CAROT is reported as stable across models and benchmarks, with accuracy and fidelity advantages over ablations in many cases. Its steering results also show higher agreement and generally better fidelity than Lang Vec.

Agreement Ratio of Steering Results

  • CAROT achieves the highest agreement ratio in most cases, and its agreement-ratio increase exceeds its accuracy increase for Llama3.1-8B on GMMLU: 2.4 versus 1.4 points.Agreement ratio measures the percentage of instances where target-language answers match source-language answers.Source: H Additional Steering Results · p. 16

Detailed Layer-wise and Language-wise Steering Results

  • CAROT improves accuracy across languages, especially in early layers, while maintaining high fidelity in most cases; Lang Vec improves accuracy less and often has low early-to-middle-layer fidelity.For gemma3-4b, the authors suggest low fidelity in some languages may relate to its tendency to generate English reasoning steps when prompted in other languages.Source: H Additional Steering Results · p. 16

Ablation of Each Component of CAROT

  • CAROT outperforms No LEACE in 9 of 12 accuracy cases and Last Token in 10 of 12; No Filter is sometimes 14.0 points lower on Qwen3-4B KLAR and has lower fidelity in 9 of 12 cases.The authors interpret these results as evidence that LEACE decomposition and OT-plan filtering improve robustness, while token-level OT alignment improves accuracy.Source: H Additional Steering Results · p. 16

I Training Details

The training details describe MidAlign’s alternating contrastive and language-modeling steps, training settings, and CAROT’s post-training representation analysis. They also report computational costs and disclose LLM use.

I Training Details

  • MidAlign alternates contrastive and language-modeling steps; its contrastive step mean-pools hidden states, while the LM step uses task-specific data.The study trains for one epoch with batch size 16 and applies LoRA to all linear modules.Source: I Training Details · pp. 16–17
  • After CAROT training, source–target language-agnostic representation similarity rises most around the training layer and remains elevated across layers.For Llama3.1-8B, the reported training layer is layer 12.Source: I Training Details · pp. 17–18, Figure 12 · p. 20

J Computational Resources

  • All experiments run on one NVIDIA H100 GPU; per-model steering hyperparameter searches take about two hours, while training searches take about 1.5 hours for LM and CAROT and four hours for MidAlign.Evaluation takes 2.5 hours for GMMLU and 20 minutes each for KLAR, BELEBELE, and XQuAD.Source: J Computational Resources · p. 18

K LLM Usage

  • Claude Code assisted experiment-script and visualization implementation, while ChatGPT proofread the paper and translated evaluation prompt templates.Supplementary figures report LEACE language-classification accuracy and steering fidelity and accuracy across models.Source: K LLM Usage · p. 18, Figure 8 · p. 19, Figure 10 · p. 20
Loading 2609.06381v1…