Polyglot or Not? Measuring Multilingual Encyclopedic Knowledge in Foundation Models

• Foundation models generalize to some new tasks but can inherit inaccuracies from training corpora.These inaccuracies may be propagated to models fine-tuned from them.Source: §1 · p. 1
• LLaMA devotes two-thirds of its training dataset to an English-only Common-Crawl subset, potentially limiting non-English performance.Source: §1 · p. 1
• The study therefore treats multilingual knowledge recall as an open question for widely adopted foundation models.Source: §1 · p. 1

• Language models can store and recall factual associations, while also ingesting dubious or harmful associations from poor or unrepresentative corpora.Source: §2 · p. 1
• Earlier studies evaluated encyclopedic knowledge in BERT and RoBERTa using rank-based approaches.The cited work includes Jiang et al. (2020) and Kassner et al. (2021).Source: §2 · p. 1
• This study uses contrastive scoring and evaluates diverse causal and masked language models rather than a single architecture.The design aims to assess conceptual understanding more rigorously and representatively than rank-based evaluation of one architecture.Source: §2 · p. 1

3 Task

• Facts are represented as triples ⟨s, r, o⟩, with the subject and object linked by a relation.For example, “Paris is the capital of France” becomes ⟨Paris, capital of, France⟩.Source: §3 · p. 1
• The model receives a natural-language sentence with the object masked and is scored by P_M(o | s, r).Source: §3 · p. 2
• CKA compares the true object probability with probabilities assigned to erroneous counterfactual objects.A model demonstrates understanding when the true fact receives greater probability than its counterfactual.Source: §3 · p. 2
• CKA_M(s, r, o) > 1 indicates that the model understands the factual association.Source: §3 · p. 2
• The contrastive approach avoids vocabulary-wide ranking limitations and provides a more nuanced confidence measure across languages and contexts.The study applies CKA to a multilingual dataset for the first time.Source: §3 · p. 2
• Language-level accuracy is the share of cloze completions with CKA_M(s, r, o) > 1, using tokenizer adjustments to align probing with the first completion token.The evaluated models are fully open-source because token probabilities are required for the test.Source: §3 · p. 2

4 Dataset

• The dataset contains 303k knowledge statements in 20 languages, with true and false objects consistently placed on the right side.A given fact appears in 12 of the 20 languages on average.Source: §4 · p. 2
• Its records include dataset identifiers, stems, true and false objects, relations, subjects, and objects across 31 relation categories.The dataset contains 76,036 unique subjects and 18,837 unique objects.Source: §4 · p. 2
• Construction merged two English datasets, filtered inaccuracies and grammatical errors, removed duplicate triples, and produced 26,254 English statements before translation.Source: §4 · p. 2
• The English data was translated with Google Translate into 19 target languages spanning Latin and Cyrillic scripts.The approach follows prior multilingual studies, and prior work reported minimal practical differences between machine- and manually translated cloze statements.Source: §4 · p. 2

5 Results

• LLaMA-33B outperforms the other four evaluated foundation models on mean accuracy across 20 languages.The multilingual leaderboard averages performance across the 20-language test, with uncertainty estimated using 10k bootstrap iterations per language.Source: §5 · p. 2, Table 1 · p. 3
• Tested models struggle more to recall facts in multilingual settings than in English-only evaluation.The multilingual leaderboard caption explicitly contrasts multilingual performance with English-only results.Source: Table 1 · p. 3
• LLaMA models of varying sizes are the front-runners on the English-only leaderboard.The English-only table reports model accuracy on English data and provides 95% confidence intervals from 10k bootstrap iterations.Source: Table 2 · p. 3
• LLaMA-33B performs better on Latin-script languages than on Cyrillic-script languages, with language-script dependence confirmed statistically.A chi-squared test reports χ2 = 3570.58 and p < 0.001.Source: §5 · p. 3

Training Data and Model Parameters

• LLaMA-7B with 1T tokens outperformed OPT-30B with 180B tokens on the English-only test, while 110M-parameter mBERT outperformed two 7B models on the multilingual test.Within the LLaMA family, the largest English-only performance differential occurred from 13B to 33B, alongside a 1T-to-1.4T training-token increase.Source: Training Data and Model Parameters · p. 3

Subject Entity Error Analysis

• Geographic recall averaged 89.94%, ranging from 93.31% for Asian locations to 80.65% for Antarctica, with the Asia–Europe difference significant.The comparison used 48,606 prompts covering 3,213 geographic entities across 20 languages (χ2 = 66.408, p < 0.001).Source: Subject Entity Error Analysis · p. 3
• LLaMA answered 75.87% of gender-labeled questions correctly and performed slightly worse on male-subject facts despite male entities being nearly five times more common.The model’s superior performance on female-subject facts was statistically significant (χ2 = 69.096, p < 0.001).Source: Subject Entity Error Analysis · p. 3

Wikipedia’s Role in LLaMA Performance

• Wikipedia pages with more unique target entities were associated with better LLaMA recall in that language.The correlation between average unique target entities per page and performance was strong and significant (Pearson’s r = 0.78, p < 0.001).Source: Wikipedia’s Role in LLaMA Performance · p. 4

Qualitative Insights

• Qualitative errors suggest frequency bias: LLaMA often mishandled Antarctica, and it favored Europe over Antarctica for Cape Monaco because of European references in the page.Performance outside English was degraded for prompts involving English-speaking countries, with larger deviations in Slavic languages, indicating unreliable cross-lingual transfer.Source: Qualitative Insights · p. 4

7 Future Work

• Future work includes multilingual model weight editing, evaluation of future open-source models, and adding languages that use neither Cyrillic nor Latin scripts.The authors mention ongoing work with native Hindi and Japanese speakers.Source: §7 · p. 4
• A future CKA iteration could model fact difficulty from language-specific training-data availability and connect individual errors causally to training-corpus artifacts.The authors propose using Wikipedia analysis values as prior probabilities and examining additional training-corpus metadata.Source: §7 · p. 4

8 Conclusion

• The study evaluates multilingual encyclopedic knowledge with a contrastive assessment, finding LLaMA strongest in both multilingual and English-only settings but limited in non-English recall.The authors argue that multilingual datasets can support training and transparent evaluation of foundation models across linguistic contexts.Source: §8 · p. 4

Assessing Open vs. Proprietary LLMs

• Full vocabulary-level token probabilities are required for the test, but GPT-4 and several largest foundation models do not provide them.GPT-3 exposes probabilities for only its five most likely next tokens, while GPT-4 reports none.Source: Assessing Open vs. Proprietary LLMs · p. 5

GPU Resources

• Approximately 500 hours of GPU usage and single-GPU workstations constrained the study’s ability to evaluate truly massive models.The authors used fixed workstation resources and therefore could not evaluate the full range of very large models.Source: GPU Resources · p. 5

Ethics Statement

• None of the tested models exceeded 90% in any evaluated language, so the authors caution against using LLMs as authoritative sources of facts.The test instead helps identify languages, topics, and contexts where factual errors are more likely.Source: Ethics Statement · p. 5
• The test could help malicious actors identify facts stored in model memory and insert damaging information through other methods.The authors specifically mention MEMIT as an example of a method that could be used for such insertion.Source: Ethics Statement · p. 5

Reproducibility

• The code and data are openly released, and experiments used Nvidia GPUs ranging from T4 to H100 for different model sizes.Larger models used an H100, medium models an A100, and smaller models a T4.Source: Reproducibility · p. 9

Dataset Preprocessing

• After merging two datasets into 33,870 items, preprocessing removed flagged errors, grammatical errors, trivial completions, inaccurate relations, and sensitive topics, leaving 29,156 items.This included removing 3,088 pairs whose correct fact was explicitly stated in the stem, making completion trivial.Source: Dataset Preprocessing · p. 9

Geographic Labeling

• Geographic labeling filtered relation types for location entities, identified 3,427 entities, and obtained continent labels for 3,213 through ChatGPT.A random 10% validation sample found every checked continent label correct.Source: Geographic Labeling · pp. 9–10

Gender Labeling

• Gender labeling filtered person-related relations, sampled 1,200 entities, and obtained gender labels for 1,057 through ChatGPT.A random 10% validation sample found every checked gender label correct.Source: Gender Labeling · p. 10

Wikipedia Entity Analysis

• Wikipedia analysis sampled 10,000 articles per language, applied SpaCy named-entity recognition to 14 languages, and counted entities and dataset-target entities.The analysis tracked both overall and distinct entity counts per article.Source: Wikipedia Entity Analysis · pp. 10–11

Text Generation Configuration

• Using 8-bit quantization reduced memory roughly by half while changing accuracy by only 0.35–0.47%, so the authors used 8-bit precision throughout.The comparison covered LLaMA-7B and LLaMA-13B on the English-only subset.Source: Text Generation Configuration · p. 11, Table 8 · p. 15

Testing New Models

• The reported results cover models released before June 2023, while a project leaderboard continues testing newer releases.The authors mention Llama-2, Mistral-7B, and Falcon-180B as subsequent releases.Source: Testing New Models · p. 11
• LLaMA-33B performs better on Latin-script than Cyrillic-script languages, with script dependence confirmed by χ2 = 3570.58, p < 0.001.The comparison concerns Ukrainian, Bulgarian, Russian, and Serbian; Table 3 reports per-language accuracy and evaluated pairs.Source: Table 3 · p. 12, Figure 1 · p. 16