PaperlayerView exact revision on arXiv

Source-linked AI summary

Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline

Meng Lu, Ruochen Zhang, Carsten Eickhoff, Ellie Pavlick

arXiv:2505.20546v2cs.CL

TL;DR

Multilingual LLMs exhibit cross-lingual factual inconsistencies, while the mechanisms causing these failures have not been functionally unified or systematically linked to specific errors. The paper uses mechanistic analyses to characterize English-centric recall and late translation, then applies language- and dataset-independent vector interventions to those stages. Targeted interventions effectively reduce cross-lingual factual recall inconsistencies, validating the multilingual LLM processing pipeline observed in previous research.

  • Problem

    Multilingual LLMs exhibit cross-lingual factual inconsistencies, while the mechanisms causing these failures have not been functionally unified or systematically linked to specific errors.

  • Method

    The paper uses mechanistic analyses to characterize English-centric recall and late translation, then applies language- and dataset-independent vector interventions to those stages.

  • Results

    Targeted interventions effectively reduce cross-lingual factual recall inconsistencies, validating the multilingual LLM processing pipeline observed in previous research.

  • Takeaways & Limitations

    Mechanistic insights can guide targeted interventions that reduce cross-lingual factual inconsistencies and unlock latent multilingual capabilities.

  • Takeaways & Limitations

    The study covers five non-English languages, ten relations, and a single model, so broader language, relation, and model coverage remains necessary.

Abstract

from arXiv · show

Multilingual large language models (LLMs) often exhibit factual inconsistencies across languages, with significantly better performance in factual recall tasks in English than in other languages. The causes of these failures, however, remain poorly understood. Using mechanistic analysis techniques, we uncover the underlying pipeline that LLMs employ, which involves using the English-centric factual recall mechanism to process multilingual queries and then translating English answers back into the target language. We identify two primary sources of error: insufficient engagement of the reliable English-centric mechanism for factual recall, and incorrect translation from English back into the target language for the final answer. To address these vulnerabilities, we introduce two vector interventions, both independent of languages and datasets, to redirect the model toward better internal paths for higher factual consistency. Our interventions combined increase the recall accuracy by over 35 percent for the lowest-performing language. Our findings demonstrate how mechanistic insights can be used to unlock latent multilingual capabilities in LLMs.

1 Introduction

Multilingual LLMs show cross-lingual factual inconsistencies, but prior work had not unified the language-specific, English-centric, and output-translation stages into a mechanism explaining these failures.

  • 1 Introduction: Cross-lingual factual inconsistency undermines multilingual LLM reliability because identical questions can produce conflicting answers across languages.Non-English inputs also have higher reported untruthfulness rates.
  • 1 Introduction: Prior studies indicate that multilingual models often use English in intermediate layers while language-specific signals control early and late processing.Factual-recall studies further suggest that retrieval is close to English and answer formation is language-specific.
  • 1 Introduction: The paper unifies these findings into a pipeline in which facts are retrieved through intermediate English-centric mechanisms and translated into the target language in later layers.This characterization connects multilingual processing stages to factual recall behavior.

2. Error analysis of multilingual inconsisten-

The error analysis identifies two failure points in multilingual factual recall and motivates language- and dataset-independent interventions targeting translation and English-centric recall.

  • 2. Error analysis of multilingual inconsisten-: The first failure occurs when the model forms a correct intermediate English answer but produces an incorrect target-language answer.This indicates an error in late-stage translation or conversion.
  • 2. Error analysis of multilingual inconsisten-: The second failure occurs when the model does not retrieve the correct intermediate English answer, so improving translation alone cannot fix the final output.The recall intervention therefore targets the earlier English-centric stage.
  • 2. Error analysis of multilingual inconsisten-: The interventions use representation differences to promote accurate conversion and an in-context-learning signal to enhance English-centric recall.Together, they redirect processing toward more reliable internal paths.
  • 2. Error analysis of multilingual inconsisten-: Combining both interventions boosts factual-recall accuracy by up to 37.6 percentage points in the lowest-performing language and by 19.04 points on average.The combined method also outperforms baselines such as explicit translation on held-out tasks.

2 Multilingual Factual Recall Pipeline

The paper studies factual recall with a multilingual parallel-fact dataset and Logit Lens, finding that models retrieve facts in an English-centric space before producing target-language answers in late layers.

  • 2 Multilingual Factual Recall Pipeline: The dataset contains 2,862 manually validated fact triples across six languages and ten relation types.The languages include English, Chinese, Japanese, Korean, French, and Spanish.
  • 2 Multilingual Factual Recall Pipeline: Logit Lens projects each layer’s final-token representation into vocabulary space to track how candidate answers change during recall.The analysis compares answer-rank trajectories across layers and error types.
  • 2 Multilingual Factual Recall Pipeline: 21.3% of examples are correct, with English answers becoming top-ranked around layer 21 before target-language answers take over in the final layers.This pattern supports English-centric retrieval followed by late target-language decoding.
  • 2 Multilingual Factual Recall Pipeline: Among the 78.7% incorrect cases, 21.7% have correct intermediate English answers without a top-ranked target-language answer, while 78.3% fail to retrieve the correct English answer.These cases suggest separate late translation and earlier English-centric recall failures.
  • 2 Multilingual Factual Recall Pipeline: The paper presents the integrated pipeline as motivation, then validates its failure points and develops interventions for translation and recall.The subsequent analyses test whether multilingual prompts activate suboptimal components.

3 Fixing Incorrect Translation Errors

The model can often form the correct intermediate English answer but fails to convert it accurately into the input language. A translation-vector intervention steers factual recall toward more effective translation components and improves conversion and recall accuracy.

  • 21.7% of errors arise when the model knows the answer in English but produces the wrong target-language answer.Explicit translation reaches 56.1 accuracy versus 21.3 accuracy for factual recall, indicating that translation capability is not fully engaged in recall.
  • The recall and explicit-translation tasks use the same late layers but only 0.5 average MLP cosine similarity, revealing distinct internal translation pathways.Both processes shift the English answer around layer 17 and use MLP layers 22–27, but their neuron activations only partially overlap.
  • The translation-difference vector is computed from mean residual-stream activations across translation and recall prompts, then injected at a selected layer.Layers 21–27 are evaluated on validation data, with layer 21 producing the largest improvement.
  • The intervention increases activation similarity to explicit translation and improves factual recall, especially for Chinese, Japanese, and Korean.Its gains are more modest for French and Spanish because those languages already have higher pre-intervention conversion rates.
  • 67.74% average conversion correctness after intervention, up from 39.56% originally.Conversion correctness measures correct final answers conditioned on correct intermediate English answers.
  • Translation failures reflect insufficient engagement of effective translation components during multilingual recall, not an absence of translation capability.A single general-purpose translation signal recovers much of the lost performance.

4 Fixing Incorrect English Recall Errors

A second failure occurs before translation: multilingual prompts often fail to retrieve the correct intermediate English answer because the English-centric recall pathway is underused. A recall vector derived from in-context learning activates this pathway across languages and tasks.

  • 4.1 English Factual Recall Components are Insufficiently Activated: An earlier failure persists when multilingual prompts do not retrieve the correct English answer, even if late-layer translation is improved.The recall intervention targets this underused English-centric factual recall stage.
  • 4.1 English Factual Recall Components are Insufficiently Activated: Multilingual prompts propagate relation information less effectively than English prompts, reaching the final position at 32.65% versus 43.30% at layer 16.This discrepancy indicates weaker activation of an otherwise similar recall substage.
  • 4.1 English Factual Recall Components are Insufficiently Activated: English prompts also achieve substantially higher correct-answer extraction rates than non-English prompts, despite both increasing from layer 15 and peaking around layer 21.The gap implicates earlier task-recognition and relation-identification stages in multilingual recall inconsistency.
  • 4.2 Recall Task Vector: The recall vector averages final-token hidden activations from five-shot English factual recall and injects the result into the residual stream.It is extracted and applied across relation datasets and languages, making it task-independent and language-independent.
  • 4.2 Recall Task Vector: Candidate vectors are selected by validation gains in intermediate English-answer accuracy and relation propagation across intervention layers and scaling factors.Scaling factors above 5 add noise and reduce answer quality; the reported best factor is i = 2.
  • 4.2 Recall Task Vector: The recall vector triggers more relation propagation than five-shot ICL and activates attention heads important for English factual recall during multilingual processing.Its task-independent and language-independent signal improves zero-shot performance across ten relations and five languages.

5 Intervention Effects

The combined translation and recall interventions provide the strongest final accuracy for several languages and outperform prompting-based translation baselines. Fine-tuning remains stronger overall, while the training-free method is competitive, especially for non-Latin scripts.

  • The combined intervention yields the highest final accuracy for Chinese, Japanese, Korean, and French.
  • The intervention consistently outperforms the translate-recall-translate baseline across three random data splits.The baseline explicitly translates the question into English, performs the task in English, and translates the answer back.
  • Fine-tuning achieves higher overall accuracy, but the intervention remains competitive, particularly for languages using non-Latin scripts.The intervention is training-free and avoids additional training resources.

6 Related Work

The paper builds on mechanistic studies of factual recall, multilingual representations, steering vectors, and component reuse. It differs by jointly targeting translation and earlier English-centric recall failures with language- and dataset-independent vectors.

  • Prior work characterizes factual recall mechanisms and multilingual processing, while concurrent work also studies translation failures at final generation.The paper positions its contribution as linking these stages to specific cross-lingual factual inconsistencies.
  • Unlike concurrent work, the approach activates more translation neurons rather than learning a linear mapping between languages and also targets an earlier recall failure.
  • The interventions extend steering-vector, function-vector, and task-vector ideas by constructing dataset-independent and language-independent signals.These vectors strengthen latent pathways already present in the model rather than focusing on one specific task.

7 Discussion

The discussion connects the proposed multilingual factual-recall pipeline to targeted interventions, while identifying unresolved questions about early-layer translation and the practical trade-offs against in-context learning.

  • Discussion: The interventions use mechanistic insights to redirect multilingual models toward better internal paths, supporting the proposed English-centric processing account.The discussion presents this as an initial step toward using mechanistic analysis to uncover and influence latent multilingual capabilities.
  • Understanding Early Layers: The intervention improves propagation of English relations, but the connection between early-layer translation and subject enrichment remains unexplored.Preliminary analysis finds that approximately 13% of subject tokens also undergo translation to English in intermediate layers.
  • Understanding Early Layers: The reliability of the logit lens for analyzing early-layer behavior is questionable, motivating alternative analysis strategies.The paper cites Belrose et al. (2023) and Ghandeharioun et al. (2024) in support of this concern.
  • In-context learning vs. Interventions: Standard 5-shot in-context learning outperforms the intervention-based method because it encodes more direct language- and task-specific information.The authors note that multiple in-context examples may be impractical when users need to retrieve a single fact.

8 Conclusion

The conclusion presents targeted vector interventions as a way to reduce cross-lingual factual inconsistencies and use mechanistic insights to improve multilingual model behavior.

  • Conclusion: Targeted vector-based interventions reduce cross-lingual factual recall inconsistencies and validate the multilingual processing pipeline.The interventions are framed as an initial step toward directing models along better internal paths.
  • Conclusion: The approach offers opportunities to uncover latent multilingual capabilities and enable more modular control of LLM behavior.The paper proposes automated identification of weaknesses and corresponding solutions as a direction for future work.

Limitations

The study’s evaluation is limited to five non-English languages and ten relations, with stronger gains observed for Chinese, Japanese, and Korean.

  • Scope: The study evaluates only five non-English languages and ten relations, limiting the demonstrated scope of its findings.The authors call for evaluation across more diverse language families and syntactic structures.
  • Scope: Improvements are more significant for languages using non-Latin scripts, including Chinese, Japanese, and Korean.The paper recommends testing additional language families and syntactic structures to assess generality.

Ethical Considerations

The study investigates multilingual factual-recall mechanisms and proposes targeted interventions, using a manually curated dataset reviewed for privacy and sensitivity concerns.

  • Ethical Considerations: The manually curated dataset was thoroughly reviewed to exclude personally identifiable information and sensitive data.The paper also frames the interventions as potentially useful for improving fairness and reducing cross-language bias.
  • Ethical Considerations: The proposed interventions may provide actionable insights for improving fairness and reducing bias across languages in existing multilingual LLMs.The authors state that broader evaluation across languages, tasks, and model architectures is still needed to assess robustness and generalizability.

A Dataset Construction

The study constructs a multilingual factual-recall benchmark and uses mechanistic analyses and interventions to examine how models retrieve, translate, and produce factual answers across languages.

  • Dataset Construction: The dataset uses manually verified English fact pairs, semantically equivalent multilingual prompts, and relation-token annotations to support precise cross-language comparisons.Candidate relation translations are filtered with lemmatization, WordNet similarity, and GPT-4o scoring above 0.8.
  • Dataset Construction: 2,862 verified fact triplets span six languages, ten relations, and geographically diverse facts for multilingual factual-recall evaluation.The parallel prompts preserve the underlying entity while varying only its surface linguistic form.
  • Mechanistic Analysis: Translation and factual recall share model regions, with translation beginning around layers 17–25 and late MLPs showing higher average AIE than attention heads.Average AIE is 9.82% for MLP components versus 2.74% for attention heads in layers 22–25.
  • Interventions: The recall intervention reactivates attention heads important for English answer extraction and uses layer 3 with scaling factor 2 in the best configuration.Both non-English ICL examples and the language- and dataset-independent vectors reactivate these heads.
  • Interventions: The combined intervention configuration applies translation and recall vectors at layers 25 and 3, respectively, both with scaling factor 2, yielding the highest validation final accuracy.The study also evaluates whether multilingual gains harm original English fact-recall performance.
  • Baselines: The translate-recall-translate baseline explicitly translates the question into English, performs the task in English, and translates the answer back, but accumulated translation errors can keep zero-shot performance poor.Its evaluation counts an answer as correct when one of the first five generated tokens contains the answer token.
Loading 2505.20546v2…