Steering beyond Language: Improving Multilingual Training-Free Text Embeddings via Paired-Residual Subspace Removal

1 Introduction

• Existing training-free embedding methods are developed and evaluated predominantly in English, leaving their multilingual behavior largely unexamined.The paper’s analysis finds that translations with the same meaning tend to occupy distinct regions of the last-token latent space.Source: §1 · pp. 1–2
• The study asks whether language-specific variation can be suppressed without sacrificing semantic content in training-free text embeddings.Source: §1 · p. 2
• PRSR estimates low-rank language-sensitive subspaces from paired residuals of parallel sentences and removes their components from last-token hidden states.The modified representations continue through the remaining frozen layers, and the method requires no model parameter updates.Source: §1 · p. 2
• PRSR improves existing training-free baselines across monolingual and cross-lingual evaluations on different LLMs.The evaluations comprise more than 200 monolingual tests across 18 languages and bitext mining between English and over 200 languages.Source: §1 · p. 2

Training-Free Text Embeddings

• Training-free embedding methods use prompt designs or internal interventions, but have been evaluated primarily in English.Examples include PromptEOL, Pretended CoT, ECHO, ReverseEOL, Contrastive Prompting, Token Prepending, and KV-Embedding.Source: §2 · p. 2

Language-Agnostic Representations

• Language-agnostic representation methods suppress language signals during training or remove language-associated subspaces from extracted embeddings.PRSR instead modifies intermediate hidden states within the frozen LLM, allowing the intervention to affect subsequent computation.Source: §2 · p. 3

Activation Steering in LLMs

• Prior multilingual activation-steering work targets reasoning, while this work applies activation steering to text embeddings in monolingual and cross-lingual settings.Source: §2 · p. 3

3.1 Language-Conditioned Embedding Geometry

• Translations of the same sentences remain separated in language-specific regions of the embedding space.The t-SNE visualization uses 1,500 parallel sentence groups across eight languages.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2 Language Variation Rivals Semantic Variation

• The comparison tests whether translations or same-language negatives are more similar to each anchor, using 1,500 FLORES+ groups across 18 languages.Negatives are selected through length-stratified sampling; positive similarity margins indicate higher similarity to the translation.Source: §3.2 · p. 3, Figure 2 · p. 3
• 36.2% of comparisons favor a same-language negative over the corresponding translation in similarity to the anchor.The mean anchor–translation cosine distance is 85.6% of the mean anchor–negative distance.Source: §3.2 · p. 3, Figure 2 · p. 3
• PRSR’s overview removes projections onto layer-specific language-sensitive subspaces to steer representations toward a more language-invariant semantic space.The subspaces are estimated by centering parallel-sentence last-token representations within groups, stacking residuals, and applying SVD.Source: Figure 3 · p. 4

4 Paired-Residual Subspace Removal

• PRSR addresses multilingual embedding misalignment by suppressing language-specific variation while preserving shared semantics.The section frames the intervention as inference-time steering for training-free text embeddings.Source: §4 · p. 4

4.1 Preliminaries

• Training-free embeddings use the last-token hidden state, which can attend to the full input sequence under causal attention, followed by output normalization.The text embedding is extracted at the output layer.Source: §4.1 · p. 4

4.2 Paired-Residual Subspace Estimation

• Paired residuals subtract each parallel group's mean representation from its sentences; their stacked residuals define a low-rank subspace capturing dominant language-sensitive variation.PRSR estimates a separate basis at each selected layer and reuses those bases across evaluation languages.Source: §4.2 · p. 4

4.3 Layer-Specific Activation Steering

• From a designated block through the output layer, PRSR removes each last-token state’s projection onto the corresponding language-sensitive subspace before extracting the embedding.Because the intervention changes internal representations while keeping the LLM frozen, it can be integrated with existing training-free embedding methods.Source: §4.3 · p. 5

5.1 Experimental Setup

• The evaluation includes 208 monolingual task–language evaluations across 18 languages and bitext mining between English and over 200 languages.The cross-lingual benchmarks are BUCC.v2, Tatoeba, and FLORES; monolingual scores are drawn from MTEB and MMTEB.Source: §5.1 · p. 5
• Experiments use frozen LLaMA-2-7B, Mistral-7B, and Qwen3-8B models, with layer-specific subspaces estimated from 1,500 parallel sentence groups in 15 XNLI languages.Separate subspace bases are estimated for each model and baseline, then shared across evaluation languages.Source: §5.1 · p. 5

5.2 Results

• PRSR raises monolingual scores by 7.3 points for PromptEOL, 6.2 for Pretended CoT, 6.0 for ECHO, 6.7 for TP, and 6.8 for CP.The comparisons cover 208 task–language evaluations across 18 languages; Table 1 reports significant gains over each corresponding baseline (paired t-test, p < 0.01).Source: Table 1 · p. 6, §5.2 · p. 6
• PRSR improves all five training-free baselines on cross-lingual bitext mining, with gains of 18.51–23.96 points on FLORES, 25.89–34.90 on Tatoeba, and 31.46–55.25 on BUCC.v2.On FLORES, gains extend to both English-to-X and X-to-English directions.Source: Table 2 · p. 6, §5.2 · p. 6
• PRSR also improves performance in eight evaluation languages that were not used to estimate the paired-residual subspaces.The authors suggest this generalization reflects broadly transferable language-sensitive structure.Source: §5.2 · p. 6

6.1 Ablations in Subspace Construction

• Correctly paired residuals yield the best performance: Shuffled-Pair beats Language-Mean, while Random slightly degrades performance versus Baseline.The authors attribute PRSR's additional gain to shared semantics within parallel groups, which reduces unrelated semantic differences in estimating language-sensitive directions.Source: Table 3 · p. 7, §6.1 · p. 7

6.2 Removal versus Amplification

• Amplifying the estimated subspace reduces the MTEB (eng, v2) score from 46.82 to 24.16, whereas PRSR removal raises it to 53.81.Amplification also significantly degrades Tatoeba, FLORES, and BUCC.v2 performance, while removal substantially improves all three.Source: Table 4 · p. 7, §6.2 · p. 7

6.3 Robustness across Different Backbones

• PRSR improves PromptEOL across three LLM backbones on both monolingual and cross-lingual evaluations.The result suggests that PRSR's effectiveness is not specific to one LLM family.Source: Table 5 · p. 7, §6.3 · p. 8

6.4 Effect of Subspace Rank

• Performance peaks around q = 14, then declines at higher ranks; the authors use q = K −1 as the default because it is near the peak.They hypothesize that higher ranks begin to include semantic information useful for embedding extraction.Source: Figure 4 · p. 8, §6.4 · p. 8

6.5 Impact of Intervention Layers

• The best intervention starting blocks are 17 for LLaMA-2-7B, 14 for Mistral-7B, and 33 for Qwen3-8B.Extending removal from final to intermediate layers generally benefits the first two models, while Qwen3-8B fluctuates at earlier starting layers.Source: Figure 5 · p. 8, §6.5 · p. 8

• The study examines multilingual training-free text embeddings and identifies language-conditioned geometry that can leave semantically equivalent texts misaligned across languages.Source: §7 · p. 8
• PRSR estimates language-sensitive subspaces from paired residuals of multilingual parallel texts and suppresses their components during embedding extraction.Source: §7 · p. 8
• Experiments on monolingual and cross-lingual benchmarks show consistent improvements over existing training-free baselines across different LLM backbones.Further ablations and analyses validate the effectiveness and necessity of paired-residual subspace removal.Source: §7 · p. 8

Limitations

• The study tests three decoder-only LLM families from 7B to 8B parameters; applicability to larger models remains unexamined.Cross-lingual evaluation pairs English with over 200 other languages and does not directly assess semantic matching between non-English languages.Source: Limitations · p. 9
• The findings do not establish generalizability to larger-scale LLMs or non-English-to-non-English matching.The authors identify evaluation in those settings as a way to further establish generalizability.Source: Limitations · p. 9

Ethical Considerations

• Representations may retain societal biases from the underlying models and data.The paper encourages users to assess biases in intended applications and follow model and dataset usage conditions.Source: Ethical Considerations · p. 9
• Improved semantic alignment does not necessarily ensure fairness across languages or communities.The paper recommends assessing potential biases in intended applications.Source: Ethical Considerations · p. 9
• The study uses existing pretrained models and benchmark datasets, whose usage conditions downstream users are encouraged to follow.The paper also recommends assessing potential biases in intended applications.Source: Ethical Considerations · p. 9

A.1 Monolingual Benchmark Details

• The monolingual evaluation comprises 208 evaluations across 18 languages, with scores averaged across evaluations and 41 English datasets spanning seven embedding task categories.The benchmark sources and dataset counts are summarized in Table 6.Source: §A.1 · p. 12, Table 6 · p. 12

A.2 Cross-Lingual Benchmark Details

• BUCC.v2, Tatoeba, and FLORES assess translation retrieval for English paired with over 200 non-English languages, using macro-averaged F1 across language subsets or directions.For FLORES, the overall score is the arithmetic mean of its En→X and X→En directional scores.Source: §A.2 · p. 12, Table 7 · p. 12

A.3 Implementation Details

• Layer-specific PRSR bases are estimated separately for each backbone and training-free baseline from the first 1,500 complete XNLI validation parallel groups across 15 languages, and remain fixed and shared across evaluation languages.The maximum input sequence length is 512 tokens for all methods except ECHO, which uses 1,024 tokens.Source: §A.3 · p. 12

B.1 Effect of PRSR on Language-Conditioned Geometry

• PRSR reduces language clustering (NMI 0.569 to 0.148; ARI 0.396 to 0.108) and same-language-negative preference 36.24% to 10.06%, while the translation-to-negative distance ratio falls 85.56% to 50.98%.The authors report that the greater increase in distance to same-language negatives than to translations makes similarity more reflective of semantic correspondence.Source: Table 8 · p. 13, §B.1 · p. 13

B.2 Effect of Construction Sample Size

• All six construction sizes outperform the baseline similarly; 100 groups perform comparably to 2,000, with the highest score at 1,500 groups.The results suggest effective language-sensitive subspaces can be estimated from relatively few parallel groups.Source: Figure 6 · p. 13, §B.2 · pp. 13–14

B.3 Effect of Subspace Construction Data

• XNLI and FLORES+ construction yield comparable MTEB scores (54.62 and 54.98); two languages raise the score from 46.93 to 52.18, peaking at 54.98 with eight.The authors use all 15 XNLI languages to avoid adding a language-selection hyperparameter.Source: §B.3 · p. 14

B.4 Compatibility with Mean Pooling

• With mean pooling, PRSR raises ECHO’s 208-evaluation monolingual average from 41.5 to 44.5; bitext scores rise for Tatoeba (1.44 to 20.93), FLORES (1.63 to 19.56), and BUCC (0.50 to 91.79).The authors state that the results show PRSR can extend to different pooling strategies through corresponding adaptations to subspace construction and inference-time steering.Source: Table 9 · p. 14, §B.4 · pp. 14–15

B.5 Full Monolingual Results

• Tables 11–14 report per-dataset comparisons of training-free baselines with and without PRSR across 41 English datasets and 167 non-English evaluations.The English datasets are grouped into seven task categories, and non-English evaluations are grouped by language; better baseline or PRSR category and language averages are highlighted.Source: §B.5 · p. 15, Table 11 · p. 17, Table 12 · p. 18