PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-lingual Representation Learning via Centroid Intervention Fusion

Wei Sun, Marie-Francine Moens

arXiv:2608.26357v1cs.CL

TL;DR

Uneven multilingual performance and costly adaptation motivate a lightweight alternative to pairwise cross-lingual intervention. CIF fuses multilingual projections into one trimmed, language-shared operator, improving performance over pairwise intervention across tasks and model families while leaving multimodal transfer and adaptive settings unresolved.

  • Problem

    Pairwise cross-lingual intervention requires separate operators and cannot explicitly share useful knowledge across language pairs, limiting scalability.

  • Method

    CIF rewrites language-specific projections as identity-centered perturbations, trims outliers around a centroid, and averages the retained perturbations into one shared intervention operator.

  • Results

    CIF reaches 51.77% macro-average performance and improves over INCLINE by up to +1.73 percentage points across multilingual benchmarks.

  • Takeaways & Limitations

    CIF provides a scalable alternative to pairwise intervention while preserving shared multilingual structure and improving transfer across reasoning, understanding, editing, and translation tasks.

  • Takeaways & Limitations

    The experiments focus on text-based multilingual understanding and generation, while adaptive intervention strength and filtering remain unexplored.

Abstract

from arXiv · show

Large language models (LLMs) exhibit uneven multilingual performance, especially when dealing with low-resource languages. Inference-time intervention offers a lightweight way to improve cross-lingual transfer by modifying the hidden states produced by the LLMs during the forward pass, without updating model parameters. However, existing cross-lingual intervention methods typically learn separate projections from source to target languages, which limits scalability and prevents knowledge sharing across languages. We propose Centroid Intervention Fusion (CIF), a projection fusion framework that consolidates multiple multilingual intervention projections into a single language-shared operator. Across multilingual commonsense reasoning, natural language inference, factual editing, and machine translation benchmarks, CIF outperforms the strongest prior pairwise intervention baseline by up to +3.378 pp on average across four model backbones, while supporting performance gains for low resource languages. The code is available at https://github.com/VRCMF/CIF.git.

1 Introduction

LLMs perform unevenly across languages, while existing multilingual interventions use separate pairwise operators that limit scalability and knowledge sharing. CIF fuses these projections into one language-shared operator and improves cross-lingual intervention across diverse tasks and models.

  • Low-resource languages often lag behind high-resource languages, while continued pretraining and supervised multilingual fine-tuning can be costly.
  • Inference-time intervention modifies hidden states during the forward pass without updating model parameters.
  • Pairwise intervention requires a separate operator for each transfer representation and cannot explicitly share intervention knowledge across language pairs.
  • CIF estimates shared cross-lingual structure by rewriting pairwise projections as identity-centered perturbations and fusing them with centroid-guided trimming.
  • CIF is evaluated on multilingual reasoning, natural language inference, factual editing, and machine translation across four LLM families, with consistent improvements over pairwise intervention.

2 Related Work

Related work combines inference-time steering with cross-lingual representation alignment, building on evidence that multilingual models encode partially shared semantic spaces. CIF addresses the remaining limitation of independent pairwise intervention by consolidating operators across languages.

  • Inference-time intervention steers LLM behavior by modifying hidden states during inference rather than updating model parameters.
  • Existing steering methods construct signals through attention-head editing, contrastive prompts, activation differences, or input-adaptive vectors.
  • Cross-lingual representation alignment seeks shared semantic spaces that support retrieval, similarity, and transfer across languages.
  • INCLINE learns source-to-target alignment matrices and applies them separately at inference time, limiting knowledge sharing as the language set grows.
  • CIF instead studies whether cross-lingual intervention operators can be consolidated across languages.

3 Method

CIF learns language-specific projections, converts them into identity-centered residual interventions, trims outliers around a centroid, and averages the retained residuals into a shared operator applied to hidden states.

  • 3.1 Learning Language-Specific Projection Matrices: Each source sentence is paired with its anchor-language translation, and both are processed by the frozen model to obtain layer-specific representations.
  • 3.1 Learning Language-Specific Projection Matrices: CIF learns one linear projection per source language from paired source and anchor-language representations.
  • 3.2 Learning Language-Shared Projection via Centroid-Guided Intervention Fusion: CIF fuses residual interventions rather than full projections by subtracting the identity matrix from each learned projection.
  • 3.2 Learning Language-Shared Projection via Centroid-Guided Intervention Fusion: The method computes a centroid of residual perturbations and measures each residual’s distance from that centroid using the Frobenius norm.
  • 3.2 Learning Language-Shared Projection via Centroid-Guided Intervention Fusion: CIF retains the r = |S| − drop languages closest to the centroid and averages their residuals to form the fused intervention.
  • 3.2 Learning Language-Shared Projection via Centroid-Guided Intervention Fusion: The intervention strength α controls the fused matrix, with α = 0 reducing CIF to the identity transformation.
  • 3.3 Inference-Time Intervention: During inference, the shared matrix replaces selected hidden representations before forward computation and removes the need for a language identifier or separate projection.

4 Experiments

CIF consolidates language-specific intervention projections into one shared operator and improves performance across multilingual understanding, generation, and factual-editing benchmarks. Its gains over pairwise intervention are supported by both main results and ablations.

  • Main Results: CIF reaches a 51.77% macro-average and improves over INCLINE by up to +1.73 pp across benchmarks.It also exceeds SADI, ITI, and CAA by +5.71, +5.53, and +6.31 pp, respectively.
  • Reasoning and Understanding: CIF improves XCOPA by +3.38 pp, XStoryCloze by +1.61 pp, XWinograd by +1.88 pp, XCSR by +1.59 pp, and XNLI by +1.80 pp over INCLINE.The largest reasoning gain is on XCOPA, increasing the score from 53.98% to 57.36%.
  • Factual Editing and Generation: CIF improves MzsRE by +0.52 pp, WMT23 by +0.68 pp, and FLORES-101 by +1.22 pp over INCLINE.CIF achieves the best intervention-based performance on both WMT23 and FLORES-101.
  • Comparison with Non-Intervention Baselines: CIF provides a translation-free alternative to Translation-NLLB by operating directly in model representation space.Translation-NLLB remains preferable when an error-free translation system is available for the target language.
  • Ablation Study: Centroid fusion raises XCOPA from 53.98% to 55.81%, XStoryCloze by +1.96 pp, and XNLI by +0.77 pp over independent projections.The independent-projection variant follows the INCLINE-style strategy and has the weakest overall performance.
  • Ablation Study: Trimmed aggregation further improves XCOPA by +1.55 pp, XNLI by +1.03 pp, and WMT23 and FLORES-101 by +0.84 BLEU points.It removes projection matrices farthest from the centroid before constructing the final operator.

5 Analysis

CIF’s analysis finds shared, compact structure in multilingual projection space and shows that centroid-guided dropping removes geometric outliers. Applying CIF also brings English and Hindi hidden representations closer and changes prediction outcomes beneficially.

  • Shared Structure in Projection Space: Language-specific residual projections align more with a shared centroid than with one another.The authors characterize this as a descriptive diagnostic because the centroid is estimated from the same languages.
  • Shared Structure in Projection Space: A small number of singular vectors explain a large fraction of projection energy, indicating a low-dimensional shared subspace.This pattern is especially evident on FLORES, XStoryCloze, WMT, and XWinograd.
  • Centroid-Guided Drop: CIF removes projections with larger centroid distances than retained projections across all datasets.The result is especially clear on FLORES, XNLI, XCSR, and XCOPA, supporting geometric rather than arbitrary filtering.
  • Centroid-Guided Drop: The languages removed most often vary by dataset: Tamil on XCOPA, Hindi on several understanding and FLORES tasks, and Japanese on WMT.This indicates that drop decisions adapt to task-specific projection geometry.
  • Representation-Level Alignment: After CIF, Hindi hidden representations move toward English representations and can overlap them in English–Hindi FLORES examples.The analysis uses 260 parallel sentence pairs from Qwen2.5-7B-Instruct and visualizes the intervention-layer MLP states with PCA.
  • Representation-Level Alignment: English–Hindi centroid Euclidean distance decreases from 5.45 to 2.53, while average cross-language pairwise distance decreases from 5.50 to 2.54.Both distances decrease by about 54%.
  • Error Analysis: CIF corrects 66 examples mispredicted by INCLINE and 59 examples mispredicted by CIF without dropping in the sampled XCOPA analysis.Three examples are wrong under both baselines but corrected by CIF, although CIF also introduces some new errors.

6 Conclusion

CIF fuses pairwise projection matrices into a shared crosslingual intervention operator, replacing separate language-pair operators. Across multiple tasks and LLM families, it improves transferable intervention representations and performance over prior intervention methods.

  • 6 Conclusion: CIF constructs one shared crosslingual intervention operator by fusing projection matrices learned from multiple language pairs.It estimates a robust multilingual centroid in perturbation space and uses centroid-guided trimming to reduce outlying or conflicting projections.
  • 6 Conclusion: CIF improves over previous intervention methods across multilingual reasoning, natural language inference, factual editing, and machine translation tasks.The improvements hold across multiple LLM families.
  • 6 Conclusion: Projection matrices from different languages contain stable shared structures that CIF uses to produce better transferable intervention representations.These findings support modeling crosslingual intervention in a shared projection space.
  • 6 Conclusion: CIF provides a scalable alternative to pairwise intervention while preserving the model’s original multilingual representations.

7 Limitations

CIF’s evaluation is limited to text-based multilingual understanding and generation, and its intervention settings use fixed strength and centroid-based filtering within each task.

  • 7 Limitations: CIF has been evaluated only in text-based multilingual understanding and generation settings.Its transfer to multimodal or interactive settings remains unclear.
  • 7 Limitations: CIF uses a fixed intervention strength and fixed centroid-based filtering strategy within each task setting.Adaptive strategies accounting for language- or task-specific variation may further improve robustness.

A.1 Case Study

The XCOPA case study shows that fusion can preserve correct predictions in Estonian and Indonesian, while centroid-guided dropping corrects a conflicting Swahili prediction.

  • A.1 Case Study: Table 4 compares CIF without dropping, INCLINE, and CIF on three XCOPA examples, with English translations added for readability.
  • A.1 Case Study: CIF preserves correct fusion predictions in Estonian and Indonesian, while INCLINE selects the distractor.The examples involve deleting spam from a spamfilled inbox and taking out a ticket stub from an emptied pocket.
  • A.1 Case Study: CIF without dropping selects the wrong Swahili continuation, whereas INCLINE predicts correctly.
  • A.1 Case Study: After centroid-guided dropping, CIF removes conflicting multilingual perturbations and recovers the correct Swahili answer.The cases suggest that CIF retains reliable projected signals and trims misleading ones.

A.2 Implementation Details

The implementation learns language-specific projections from parallel English–non-English sentences, applies inference-time intervention across transformer layers, and evaluates multiple intervention methods under matched settings.

  • A.2 Implementation Details: For each English–target-language pair, the setup randomly samples 500 parallel sentence pairs.News Commentary v16 is used when available, and CCAligned covers languages not included there.
  • A.2 Implementation Details: Intervention is applied once at the last input token across all transformer layers, with α tuned on a fixed grid from −1 to 1 in steps of 0.1.One α is selected per dataset–backbone pair using average validation performance across target languages.
  • A.2 Implementation Details: CIF tunes the number of removed projection matrices over drop ∈ {0, 1, 2}, selecting one dataset–backbone-level value shared across target languages.drop = 0 corresponds to CIF without trimming.
  • A.2 Implementation Details: Classification tasks score fixed answer candidates, generation tasks use greedy decoding at temperature 0, and machine translation uses SacreBLEU.The evaluated tasks include XCOPA, XStoryCloze, XWinograd, XCSR, XNLI, MzsRE, WMT23, and FLORES-101.
  • A.2 Implementation Details: All intervention baselines use the same languages, backbones, tuning budget, decoding settings, and frozen backbone parameters as CIF.This isolates performance differences to the intervention operator rather than training, validation, tuning, or decoding choices.

A.3 Per-language Effects

CIF improves many low-resource and typologically distant languages, including languages not retained during fusion, but gains are uneven in generation and depend on validation-tuned intervention settings. Its shared operator reduces deployment cost independently of the number of target languages.

  • Per-language gains: Hindi and Arabic gain 12.30 and 11.45 points on XCSR, while Arabic gains 5.25 BLEU on FLORES.These are representative improvements among challenging languages examined in the per-language analysis.
  • Limitations: Chinese on FLORES decreases by 3.21 BLEU, showing that trimming can remove language-specific information useful to individual languages.In leave-one-language-out evaluation, Chinese on FLORES also drops by −0.75 BLEU when its projection is excluded from fusion.
  • Leave-one-language-out transfer: LOO-CIF remains close to full CIF on most held-out languages, with small improvements after removing Tamil, Vietnamese, Hindi, or Vietnamese in selected settings.On XCOPA, removing Tamil and Vietnamese improves held-out scores by +1.40 and +0.60 points; on FLORES, removing Hindi and Vietnamese improves them by +0.63 and +0.47 points.
  • Per-language gains: CIF improves many typologically distant or lower-resource languages, with XCSR gains ranging from +7.65 to +12.30 points for Arabic, Hindi, Japanese, Vietnamese, Urdu, and Chinese.The analysis also reports improvements for Arabic, Hindi, Swahili, Thai, Turkish, Urdu, Vietnamese, and Chinese on XNLI.
  • Sensitivity: Performance depends on validation-tuned trimming and intervention strength: drop = 1 is best on XCOPA, XNLI, and WMT23, whereas drop = 2 is slightly better on FLORES.For example, WMT23 changes from 32.44 at α = −1.0 to 42.24 at α = 0.4, while FLORES changes from 28.46 to 38.57 across the reported α settings.
  • Scalability: CIF requires L deployed projection matrices per layer instead of INCLINE’s N × L matrices, reducing deployment operators by a factor of N.Its final inference-time memory and deployment cost remain independent of the number of target languages used during fusion.
Loading 2608.26357v1…