Source-linked AI summary
Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models
Mingyang Wang, Heike Adel, Lukas Lange, Yihong Liu, Ercong Nie, Jannik Strötgen, Hinrich Schütze
TL;DR
MLMs can answer equivalent factual prompts inconsistently across languages, but the causes of this behavior were not previously established. Using mechanistic interpretability, the paper identifies a language-transition failure and introduces a linear shortcut that improves prediction accuracy and cross-lingual consistency.
Problem
MLMs store factual knowledge across languages but often respond inconsistently to semantically equivalent prompts, while the underlying causes remain insufficiently explored.
Method
The paper traces multilingual factual recall across layers with mechanistic analysis and proposes a linear shortcut that bypasses final-layer computations.
Results
MLMs encode knowledge in a largely language-independent concept space before transitioning to language-specific processing, and shortcut mapping improves prediction accuracy and cross-lingual consistency.
Takeaways & Limitations
Language transition is a critical mechanism for cross-lingual consistency, and a lightweight linear shortcut offers a practical way to adapt concept-space knowledge to target languages.
Takeaways & Limitations
The analysis uses English as its pivot language, and KLAR’s 17 languages do not fully represent the diversity of world languages.
Abstract
from arXiv · showhide
Multilingual language models (MLMs) store factual knowledge across languages but often struggle to provide consistent responses to semantically equivalent prompts in different languages. While previous studies point out this cross-lingual inconsistency issue, the underlying causes remain unexplored. In this work, we use mechanistic interpretability methods to investigate cross-lingual inconsistencies in MLMs. We find that MLMs encode knowledge in a language-independent concept space through most layers, and only transition to language-specific spaces in the final layers. Failures during the language transition often result in incorrect predictions in the target language, even when the answers are correct in other languages. To mitigate this inconsistency issue, we propose a linear shortcut method that bypasses computations in the final layers, enhancing both prediction accuracy and cross-lingual consistency. Our findings shed light on the internal mechanisms of MLMs and provide a lightweight, effective strategy for producing more consistent factual outputs.
1 Introduction
MLMs often answer semantically equivalent factual prompts inconsistently across languages, while the causes of these failures remain insufficiently understood. This work uses mechanistic analysis to identify language-transition failures and proposes a shortcut that improves accuracy and consistency.
- Cross-lingual factual inconsistency limits the robustness and fairness of MLMs despite their ability to store and retrieve knowledge across languages.
- Failures in the final language transition can produce an incorrect target-language answer even when the correct fact is encoded in the concept space.In the illustrated Chinese query, the model identifies Ottawa in English but outputs Toronto.
- The study traces information flow in consistent and inconsistent predictions to investigate where multilingual factual recall succeeds or fails.
- MLMs process factual knowledge in a largely language-independent concept space through most layers before transitioning to language-specific spaces in the final layers.
- The study introduces KLAR, a 17-language, 20-relation dataset used to evaluate cross-lingual consistency in two state-of-the-art MLMs.
- The analysis identifies language transition as the main failure point and proposes bypassing final-layer computations with a shortcut to improve prediction accuracy and cross-lingual consistency.
2 Related Work
Prior work studied multilingual model representations and factual recall, but did not explain inconsistent cross-lingual predictions. This paper addresses that gap through representation-level mechanistic interpretability and comparison of consistent and inconsistent cases.
- Mechanistic interpretability decomposes model computations into interpretable components, including features and circuits responsible for specific behaviors.
- This work uses Logit Lens to project intermediate latent states into vocabulary space and track how information evolves across layers.
- Earlier studies examined multilingual model internals and factual knowledge, while research on inconsistent predictions remained limited.
- The analysis builds on findings that LLaMA2 relies on English as a pivot and separates language and concept representations.
- The study examines both consistent and inconsistent predictions to reveal how MLMs transition from language-independent to language-specific processing.
3 KLAR Dataset
KLAR is constructed to make autoregressive multilingual factual probing more compatible and evaluations more unambiguous. It uses parallel multilingual triples, end-position answers, and filtered relations with clear correct answers.
- KLAR represents facts as subject-relation-object triples and prompts models to predict the object, such as Ottawa for Canada’s capital.
- KLAR addresses weaknesses in BMLAMA17 by placing objects at the end of autoregressive prompts and excluding relations with multiple correct answers.
- Templates follow the form “<Question> The answer is:” and are translated from English into 16 other languages.
- The construction excludes ambiguous relations and checks template clarity manually and/or through back-translation.
- The dataset contains 2,619 parallel factual triples across 17 languages and 20 relation types.
4 Cross-lingual Consistency Evaluation
The evaluation measures full-answer factual correctness and cross-lingual consistency for LLaMA2 and BLOOM on multilingual factual prompts. Both models struggle with consistency across languages, especially linguistically diverse and non-Latin-script pairs.
- The evaluation uses the model’s complete factual answer rather than only its first predicted token.This avoids falsely counting outputs as correct when their first token matches but the full answer is wrong.
- Both models face low cross-lingual consistency, particularly across linguistically diverse language pairs and non-Latin scripts.LLaMA2 generally performs better than BLOOM, but consistency remains challenging for both.
- Figure 2 compares language pairs using the overlap ratio of correct predictions.
5 Analyzing Multilingual Factual Recall
The models recall facts through language-independent concept-space processing before adapting representations to the target language. Layer-wise ranks, latent-state similarities, and language composition reveal transitions from relation processing to object extraction and final language-specific adaptation.
- The analysis uses Logit Lens to track target-language and English answer ranks and compare latent-state similarities across layers.
- Correct-answer ranks drop together during object extraction, then English and target-language ranks diverge as target-language processing begins.Object extraction begins around layer 15 in BLOOM and layer 12 in LLaMA2, while later divergence marks language-specific adaptation.
- Latent-state similarity peaks around 0.8 in middle layers across language pairs, including linguistically diverse pairs, indicating a shared concept space.Similarity decreases in the final layers as representations transition toward language-specific processing.
- MLMs progress from relation processing to object extraction and then language-specific adaptation of concept-space representations.The transition occurs around layer 28 in LLaMA2 and layer 19 in BLOOM.
- LLaMA2’s middle-to-upper latent representations are English-dominant, whereas BLOOM shows a more diverse language composition.Similar compositions across input languages support a concept space largely independent of the input language.
6 Examining the Cause of Cross-Lingual Inconsistency
Incorrect target-language predictions arise despite correct factual knowledge being represented in middle-layer concept space. The decisive error appears during final-layer language transition, where an incorrect target-language answer overtakes the correct answer.
- The incorrect answer surpasses the correct answer during final-layer language transition, although both correct-answer ranks improve in middle layers.
- Spanish and Chinese examples show the English equivalent reaching rank 0 before the incorrect target-language answer rises in the final layers.
- These findings identify language transition as the critical failure point causing cross-lingual factual inconsistency.
- LLaMA2’s English-biased latent space makes the switch to target-language processing clearer than BLOOM’s less interpretable latent space.
7 Linear Shortcut for Improving Cross-Lingual Consistency
The proposed linear shortcut learns to adapt earlier concept-space representations to each target language while bypassing error-prone final-layer computations. It improves accuracy and cross-lingual consistency, outperforming translation-based baselines and fine-tuning in the reported comparisons.
- The method learns one language-specific linear mapping from an intermediate layer to the final representation and applies it instead of the original final layers.The mapping is estimated from correctly predicted samples and shared across relations for each language.
- The shortcut improves prediction accuracy and cross-lingual consistency across models and languages.Figure 8 reports results for LLaMA2 and BLOOM with and without the shortcut method.
- Translation-based baselines perform poorly, indicating that existing translators are insufficient for cross-lingual factual prediction.
- Fine-tuning underperforms the shortcut method for LLaMA2 and underperforms the original BLOOM model except for English improvements.
- The shortcut is lightweight because it directly adapts earlier representations using only linear operations.
8 Conclusion
The study identifies a three-stage multilingual knowledge-recall process and shows that transition errors can cause incorrect target-language predictions despite accurate object extraction. A linear shortcut bypassing final-layer computations improves prediction accuracy and cross-lingual consistency.
- A three-stage recall process comprises language-independent relation processing, object extraction, and final language-specific adaptation.
- Transition errors can produce incorrect predictions even when object extraction is accurate.
- The proposed shortcut bypasses final-layer computations to improve prediction accuracy and cross-lingual consistency.
- Future work should examine more languages, additional models, and non-linear shortcuts for broader applicability and stronger mitigation.
Limitations
The study’s consistency analysis uses English as a pivot, covers a limited portion of world-language and model diversity, and relies on a linear shortcut. Downstream applications are identified but remain outside the study’s scope.
- Using English as the pivot may limit applicability to language pairs that do not involve English.
- KLAR covers 17 languages but does not fully capture world-language diversity, while the analysis spans limited model architectures and sizes.
- The shortcut relies on linear approximation, which may not capture complex transformations during language switching as well as non-linear approaches.
- Multilingual knowledge localization and cross-lingual knowledge editing are relevant downstream applications but fall outside this study’s scope.
Ethical considerations
The study is diagnostic, but incorrect multilingual model predictions may propagate misinformation or reflect underlying model biases.
- Incorrect model predictions may propagate misinformation or reflect biases present in multilingual language models.
A Appendix
The appendix details the KLAR evaluation setup, analyzes multilingual latent representations, and reports a linear shortcut designed to reduce cross-lingual factual inconsistency.
- A.1 KLAR Dataset Details: The KLAR dataset spans 17 languages and 20 relation types, with model-specific analyses covering 12 LLaMA2 languages and 7 BLOOM languages.It uses relation-specific templates, parallel factual knowledge, and filters out relations with multiple answers or fewer than 30 samples.
- A.2 Latent Representation Analysis: Latent-state similarity rises from early to middle layers before declining in final layers, indicating a transition from language-independent concepts to language-specific representations.The trend also holds for linguistically diverse language pairs.
- A.2 Latent Representation Analysis: Across input languages, both models show similar middle-to-upper-layer language distributions, supporting a concept space largely independent of the input language.LLaMA2 is dominated by English in these layers, whereas BLOOM has a more diverse linguistic composition.
- A.2 Latent Representation Analysis: Correct English answers can rank highest in middle-to-upper layers but be overtaken by incorrect target-language answers during the language transition.This pattern provides a concrete mechanism for cross-lingual factual failures.
- A.3.1 Method: The linear shortcut approximates the mapping from an intermediate latent state to the final state, using a slope correction and grid-searched layer, correction factor, and sample count.The method bypasses the language-transition computation, while the selected extraction layer and sample count remain consistent across languages but β varies.
- A.3.3–A.3.4 Performance: Translation-based methods perform poorly, whereas the shortcut improves predictions across a wide range of relation types.The reported analysis attributes translation-en failures partly to entity translation difficulty and translation-early-exit failures to unreliable token-level translations.