Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models

1 Introduction

• Shared universal fact sets make cross-language comparisons possible but may overrepresent some regions and underrepresent facts relevant elsewhere.This bias makes evaluation results harder to interpret.Source: §1 · p. 1
• BLOOM-3b answers consistently in English, Spanish, and Vietnamese but not Hungarian and Greek, including when its shared answer is wrong.The example suggests the first three languages share relevant knowledge representations.Source: §1 · pp. 1–2
• Cross-lingual consistency matters for similar user experiences across languages and for understanding whether knowledge transfers implicitly between languages.The authors also note that prior model-editing work had not examined effects on languages other than the directly edited language.Source: §1 · p. 2
• The paper proposes RankC, which measures consistency independently of accuracy, and builds BMLAMA by filtering existing datasets for a multi-parallel benchmark.It applies RankC across multilingual PLMs and studies language properties associated with consistency; a model-editing case study provides preliminary evidence that CLC predicts cross-language transfer.Source: §1 · p. 2

2 Related Work

• Prompt-based factual probing fills a relation template with a subject and checks whether the model ranks the known object highest among candidates.LAMA introduced the approach, which became the main technique for assessing factual knowledge in PLMs.Source: §2 · p. 2
• X-FACTR (Jiang et al., 2020) and MLAMA (Kassner et al., 2021) report strong cross-language variation in factual-probing accuracy, while GeoMLAMA (Yin et al., 2022) probes region-sensitive commonsense facts.These studies primarily assess the amount of knowledge encoded per language, rather than how it percolates among languages.Source: §2 · p. 2
• Self-consistency measures whether a model gives the same answer to meaning-preserved paraphrases; multilingual results reported poor self-consistency in every language.Fierro and Søgaard (2022) measure this separately by language.Source: §2 · pp. 2–3
• Earlier cross-lingual factual-consistency work found low overlap in correct predictions, whereas this study also treats shared incorrect answers as consistent.Jiang et al. (2020) reported a 34% peak for English-Dutch and analyzed only mBERT; concurrent work by Ohmer et al. (2023) studied consistency in paraphrase identification and natural language inference.Source: §2 · p. 3

3 Measuring Cross-Lingual Consistency

• For each query, the PLM ranks candidate words by prediction score; cross-lingual comparison uses the ranked candidates for corresponding queries.Source: §3 · p. 3

3.1 Prior Work: Correct Predictions Overlap

• Jiang et al. (2020) compare overlapping correct predictions after pairwise filtering, which creates different query sets and limits comparison across language pairs with very different filtered samples.Source: §3.1 · p. 3

3.2 This Work: RankC Metric

• Requiring queries and candidates translated across all languages gives each language pair equal query-set lengths and candidate counts.Source: §3.2 · p. 3
• RankC assesses cross-lingual consistency independently of accuracy by considering rankings of all candidates rather than only correct predictions.Source: §3.2 · pp. 3–4
• For each translated query pair, RankC averages top-j candidate-overlap scores, weighting higher-ranked candidates more heavily with normalized softmax weights.Source: §3.2 · p. 4
• Nearly all language pairs with high COverlap also score high on RankC, while RankC identifies additional high-consistency pairs whose COverlap was low because probing accuracy was low.Source: §3.2 · p. 4

• BMLAMA-17 contains 6.7k queries in 17 languages, while BMLAMA-53 contains 3k queries in 53 languages.Source: §4 · p. 4
• The model set includes XLM-RoBERTa-large, mT5-large, and four BLOOM sizes from 560m to 3b parameters.Source: §4 · p. 4
• Figure 2 reports factual probing accuracy on BMLAMA-17; Figure 3 shows pairwise RankC, with darker shading for higher consistency and green bars for language accuracy.Source: Figure 2 · p. 5, Figure 3 · p. 5

5 Main Consistency Results

• Increasing BLOOM model size slightly and consistently improves factual probing accuracy, while XLM-RoBERTa-large and mT5-large outperform the BLOOM series on average.The BMLAMA-17 results are consistent with Jiang et al. (2020) on X-FACTR.Source: §5 · p. 5

5.1 Consistency in Different PLMs

• Average consistency is low across models, reaching its lowest reported level of 25% in BLOOM-3b.Figure 4 reports little RankC fluctuation across PLMs of different scales.Source: §5.1 · p. 5, Figure 4 · p. 6
• English, French, Dutch, Spanish, and Catalan show considerable consistency in mT5-large and XLM-RoBERTa-large, while BLOOM-3b follows a similar pattern except for Dutch.Vietnamese and Turkish are notably consistent with the European languages across PLMs; Russian and Ukrainian also form a high-consistency pair.Source: §5.1 · pp. 5–6

5.2 Effect of Model Size

• Average RankC changes by only +2% across a fivefold increase in BLOOM parameters, suggesting that scaling may leave CLC unresolved.The authors caution that this pattern cannot safely be generalized to other models.Source: §5.2 · p. 6

6.1 Typological Similarity

• RankC has a moderate correlation with genetic similarity, weak correlations with geographic similarity and, on BMLAMA-53, syntactic similarity, and no significant correlation with phonological similarity.Genetic and geographic correlations decrease slightly on BMLAMA-53; the authors suggest noise in low-resource typological vectors may contribute.Source: §6.1 · p. 6

6.2 Subword Vocabulary Overlap

• The authors measure vocabulary overlap by segmenting parallel-corpus text with each model’s tokenizer, making the overlap measure model-dependent.They use BMLAMA and Flores-200; the authors note that BMLAMA-based correlations may be less generalizable because it is also the evaluation corpus.Source: §6.2 · p. 7
• Subword vocabulary overlap correlates significantly more strongly with CLC than genetic similarity, and Flores-based overlap remains significantly positively correlated.The Flores comparison indicates that the finding is not limited to BMLAMA.Source: §6.2 · p. 7
• The authors propose that factual knowledge may propagate shallowly through shared subword embeddings, and that limited overlap can hamper transfer even between related languages.They speculate that larger subword vocabularies may reduce consistency by decreasing the chance that languages share word parts; they leave this hypothesis for future work.Source: §6.2 · pp. 7–8

Method

• The study uses ROME, which modifies weights in early feed-forward layers where a factual association has been located through causal interventions.The case study uses counterfactual facts that were not observed during pre-training.Source: §7 · p. 8

Case study

• The evaluation uses BLOOM-3b, inserts facts in English, and compares two target languages with high English RankC against two with low RankC.Six queries are selected where the original answer was highly probable before editing; subject and object tokens are kept the same across languages.Source: §7 · p. 8
• After English facts are edited, Spanish and Vietnamese show consistent propagation, whereas Hungarian and Greek remain more likely to favor the original correct answers.The same pattern appears in three additional queries reported in Appendix J.Source: §7 · p. 9
• The authors characterize the evidence as a small study and suggest CLC may reflect languages’ susceptibility to knowledge perturbations, not only existing knowledge.They present this as a potential direction for multilingual model editing.Source: §7 · p. 9

8 Conclusion

• Across PLMs, average CLC is low and is not visibly affected by model size.The authors report that consistency grows extremely slowly as model scale increases.Source: §8 · p. 9
• Language-pair CLC correlates with genetic similarity, but its correlation with vocabulary overlap is notably stronger.The authors suggest factual knowledge may propagate primarily through shared word embeddings rather than deeper language-agnostic representations.Source: §8 · p. 9
• Facts inserted in one language are more likely to propagate to languages with higher CLC scores with that language.Source: §8 · p. 9

Limitations

• GPU limits restricted the tested models to BLOOM-7.1b and below.The authors encourage testing larger models to see whether the same conclusions hold.Source: Limitations · p. 9
• BMLAMA’s facts are likely more relevant to the Western world, potentially biasing evaluation.The authors note this issue is inherited from the source benchmarks and is difficult to fix when comparing the same facts across languages.Source: Limitations · p. 9

Ethics Statement

• BMLAMA queries may inherit gender and racial biases from Wikidata, while the paper focuses on consistency across languages rather than the specific knowledge in any one language.Source: Ethics Statement · p. 9

A GenBench Evaluation Card

• The evaluation card presents the BMLAMA experiments using the GenBench generalization taxonomy.Source: A GenBench Evaluation Card · p. 12, Table 4 · p. 12

B Prompt-based Probing

• Prompt-based scoring ranks candidates using average subword log probabilities for encoder-only models, sequence outputs for encoder-decoder models, and next-token probabilities for decoder-only models.Source: B Prompt-based Probing · p. 12

C Ranking-Based Weight

• Softmax weighting was selected for RankC because it best highlights the most consistent language pairs in the weighting comparison.Source: C Ranking-Based Weight · p. 13, Figure 6 · p. 13

D RankC Computation Example

• The English–Spanish example sorts candidates by their probe scores, compares translated rankings, and averages consistency across query pairs to obtain RankC.Source: D RankC Computation Example · p. 13

E Range of RankC Values

• 63% is the approximate RankC level when two languages give the same top answer for every query, and the paper treats scores around or above it as highly consistent.Source: E Range of RankC Values · p. 13

F Comparison of RankC and COverlap

• RankC retrieves nearly all high-consistency pairs found by COverlap and identifies seven additional pairs that COverlap underestimates because of low probing accuracy in one language.Three high-consistency pairs found by COverlap are missed.Source: F Comparison of RankC and COverlap · p. 13, Figure 7 · p. 14

G Effect of Non-Supported Languages in BLOOM

• BLOOM’s non-supported languages were retained because their probing accuracy exceeded random guessing; excluding them strengthened the correlation between CLC and subword overlap.The authors suggest these languages may occur in BLOOM’s training data.Source: G Effect of Non-Supported Languages in… · p. 14, Table 6 · p. 14

H Additional Results on BMLAMA-53

• On BMLAMA-53, average consistency aligns with BMLAMA-17, except for a 2% decrease on XLM-RoBERTa-large as the number of languages increases.Source: H Additional Results on BMLAMA-53 · p. 14, Figure 8 · p. 15
• High subword-vocabulary overlap accompanies high RankC in several language pairs across BLOOM-3b, mT5-large, and XLM-RoBERTa-large; Figure 11 plots overlap–CLC correlations for the latter two models.The Russian–Ukrainian pair has 80% genetic similarity and high CLC in all three models.Source: H Additional Results on BMLAMA-53 · pp. 14–16, Figure 10 · p. 15, Figure 9 · p. 15

I Knowledge Consistency on More Models

• BLOOM-7.1b gains 1.8% probing accuracy over BLOOM-3b, while average CLC increases only 1.1%.Source: I Knowledge Consistency on More Models · p. 16
• BLOOMZ-3b has lower average probing accuracy and CLC than BLOOM-3b, but its CLC is less affected and its heatmaps are nearly identical.The models use the same tokenizer; the authors say this supports CLC’s independence from accuracy and its reliance on subword overlap.Source: I Knowledge Consistency on More Models · p. 16
• LLaMA-7b reaches 28.5% average CLC versus 26.3% for BLOOM-7.1b, and the reported correlations support a positive relationship between CLC and vocabulary overlap.Source: I Knowledge Consistency on More Models · p. 16, Table 7 · p. 16

J Additional Cases of Counterfactual Knowledge Incorporation

• After English model editing, Spanish and Vietnamese (high RankC with English) change synchronously, while low-RankC languages are less affected; similar results across paraphrases and subjects suggest little prompt influence.Source: J Additional Cases of Counterfactual… · p. 16, Table 8 · p. 16

K Raw Logits Before/After Model Editing

• The appendix provides raw, unnormalized logits before and after editing for all examined queries.Source: K Raw Logits Before/After Model Editing · p. 16, Table 9 · p. 17