PaperlayerView exact revision on arXiv

Source-linked AI summary

X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models

Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, Graham Neubig

arXiv:2010.06189v3cs.CL

TL;DR

Prior work largely examined factual knowledge retrieval in English, leaving multilingual performance insufficiently studied. X-FACTR addresses this gap with a multilingual, multi-token benchmark and decoding methods, finding that retrieval is harder in low-resource languages and remains challenging overall.

  • Problem

    Prior studies of factual knowledge retrieval in language models have focused almost invariably on English, despite factual knowledge being written and queried in many languages.

  • Method

    The paper introduces X-FACTR, a multilingual benchmark spanning 23 languages, expands probing to multi-token entities, and develops decoding algorithms for multi-token predictions.

  • Results

    Retrieval is easier in high-resource languages than low-resource languages, but overall performance remains relatively low; advanced decoding raises English multi-token accuracy from 5.57% to 11.06%.

  • Takeaways & Limitations

    X-FACTR shows that factual knowledge in multilingual language models varies across languages, motivating methods that improve multilingual retrieval and type-appropriate entity prediction.

  • Takeaways & Limitations

    English results are not directly comparable to the 31.1% BERT result from prior work because that result used a different fact set constrained by shared vocabularies.

Abstract

from arXiv · show

Language models (LMs) have proven surprisingly successful at capturing factual knowledge by completing cloze-style fill-in-the-blank questions such as "Punta Cana is located in _." However, while knowledge is both written and queried in many languages, studies on LMs' factual representation ability have almost invariably been performed on English. To assess factual knowledge retrieval in LMs in different languages, we create a multilingual benchmark of cloze-style probes for 23 typologically diverse languages. To properly handle language variations, we expand probing methods from single- to multi-word entities, and develop several decoding algorithms to generate multi-token predictions. Extensive experimental results provide insights about how well (or poorly) current state-of-the-art LMs perform at this task in languages with more or fewer available resources. We further propose a code-switching-based method to improve the ability of multilingual LMs to access knowledge, and verify its effectiveness on several benchmark languages. Benchmark data and code have been released at https://x-factr.github.io.

1 Introduction

X-FACTR extends factual knowledge probing beyond English by creating a multilingual benchmark for 23 languages and accommodating morphological variation and multi-token entities. Experiments show that retrieval is harder in low-resource languages, multilingual knowledge is largely language-specific, and code-switching can improve retrieval with low-resource prompts.

  • X-FACTR studies how multilingual language models retrieve factual knowledge across languages.
  • The benchmark covers 23 languages with varying data availability and uses manually created prompts adapted to language-specific morphology.
  • Multi-token entities comprise more than 75% of Wikidata facts, motivating decoding algorithms that generate multi-token predictions.
  • Factual retrieval is easier in high-resource languages but remains relatively low overall, making the task challenging.
  • Almost 50% of facts are recalled in only one language, indicating that knowledge memorized by multilingual models is largely distinct across languages.
  • Code-switching-based fine-tuning successfully improves knowledge retrieval with low-resource language prompts in French, Russian, and Greek experiments using English.

2 Retrieving Facts from LMs

The paper follows the LAMA cloze-probing setup: a model receives a subject-relation prompt with a masked object and predicts the missing fact. X-FACTR extends this setup to multiple languages and multi-token predictions.

  • LAMA probes whether a language model memorized a subject-relation-object fact by predicting the object in a cloze-style prompt.
  • The subject replaces [X], the object remains masked as [Y], and the prediction is compared with the ground-truth object.
  • X-FACTR extends the original single-token English setting to more languages and multiple predicted tokens.

3 Multilingual Multi-token Factual Retrieval Benchmark

X-FACTR constructs a multilingual benchmark from Wikidata-linked T-REx facts across diverse languages, data conditions, scripts, and grammatical systems. Its prompts encode morphological requirements, are automatically inflected, and receive 88% average naturalness and grammaticality judgments.

  • 3.1 Languages: The benchmark includes 23 languages from 11 families and 10 scripts, spanning substantial variation in Wikipedia presence as a proxy for data availability.
  • 3.2 Facts: The benchmark includes multi-token entities because many popular entities cannot be represented by a single token, supporting a less biased evaluation.
  • 3.2 Facts: X-FACTR samples 1000 subject-object pairs for each of 46 T-REx relations, with probabilities proportional to fact frequency.
  • 3.3 Prompts: Prompts handle language-specific case requirements and agreement features such as person, gender, and number.
  • 3.3 Prompts: Morphological features are inferred using Wikidata information and language-specific heuristics, then converted into surface forms with an inflection package.
  • 3.3 Prompts: 88% of evaluated prompt completions were judged natural and grammatically correct on average by native speakers.

4 Multi-token Decoding

This section develops methods for generating multi-token predictions with masked language models, combining initial prediction strategies with iterative refinement. Final predictions are selected by confidence across candidate mask lengths, with length normalization and confidence recomputation as additional options.

  • Motivation: Multi-token entity generation remains challenging for masked LMs, which predict individual words from bidirectional context rather than decoding naturally left to right.The benchmark therefore requires methods for generating multiple predictions from masked language models.
  • Initial Prediction: Masked positions can be filled independently in parallel or sequentially, conditioning each prediction on previously generated tokens.These approaches parallel masked-LM prediction and autoregressive-style conditioning.
  • Refinement: After initial filling, predictions can be iteratively modified one token at a time until convergence or a maximum iteration limit.The framework includes independent and autoregressive approaches for initial prediction or refinement.
  • Final Prediction: Because the ground-truth length is unknown, the method enumerates mask lengths from 1 to M and selects the prediction with highest confidence.The confidence is computed for each candidate length before choosing the final output.
  • Final Prediction: Length normalization divides confidence by the number of mask tokens, while recomputation updates stale token probabilities after surrounding tokens change.These components address the tendency of summed probabilities to favor short predictions and account for changed context.

5 X-FACTR Benchmark Performance

X-FACTR evaluates multilingual factual retrieval across 23 languages and multiple pretrained LMs, finding strong resource-level differences and generally low accuracy. Decoding choices affect single- and multi-token predictions differently, while model comparisons vary by language.

  • Performance on Different Languages: Less than 15% accuracy was achieved on high-resource languages and less than 5% on some low-resource languages in favorable settings.Examples of low-resource languages include Marathi and Yoruba.
  • Performance on Different Languages: High-resource languages usually outperform middle- and low-resource languages across multilingual and monolingual LMs.The paper attributes this pattern possibly to greater pretraining data, memorization differences, or shared Latin script among some languages.
  • Performance of Different LMs: M-BERT outperforms XLM and XLM-R on high-resource languages, while their performance is similar on low-resource languages.The authors conjecture that factual retrieval requires finer-grained lexical distinctions than some other cross-lingual tasks.
  • Performance of Different LMs: M-BERT outperforms monolingual BERT on Dutch, Spanish, and Greek but underperforms it on English, Russian, Chinese, and Turkish.The paper hypothesizes that training-corpus differences contribute substantially to final performance.
  • Performance of Different Decoding Methods: Advanced decoding raises multi-token accuracy but lowers single-token accuracy; on English, the best method increases multi-token accuracy from 5.57% to 11.06%.The authors suggest that advanced methods may select higher-confidence multi-token predictions over correct single-token predictions.

6 Improving Multilingual LM Retrieval

The authors investigate cross-lingual differences in multilingual factual retrieval and propose code-switching to share entity knowledge across languages. Code-switched fine-tuning improves retrieval when English answers are accepted, especially for Greek and Russian.

  • Motivation: Correctly retrieved facts often appear in only one language, with language-specific training exposure helping explain these differences.Tencent is correctly predicted in Chinese but incorrectly predicted in English, suggesting stronger representation in the Chinese corpus.
  • Method: Code-switching replaces entity mentions with counterparts in another language so entities can serve as pivots for sharing knowledge.The method uses bi-directional replacement in Wikipedia sentences before fine-tuning pretrained multilingual LMs.
  • Experimental setup: 560k, 396k, and 129k code-switched sentences are produced for French, Russian, and Greek paired with English, respectively.The experiments compare code-switched fine-tuning with original M-BERT and raw-text fine-tuning, using single- and double-evaluation settings.
  • Results: Code-switched fine-tuning performs slightly worse under original-language-only matching but significantly better when English and prompt-language targets are accepted.This pattern indicates that code-switching helps retrieve facts, often through English rather than the prompt language.
  • Results: The improvement is larger for Greek and Russian, while raw-text fine-tuning already outperforms original M-BERT.A Greek prompt retrieves “mathematics” in English after code-switched fine-tuning, instead of the non-informative “οποίων” without it.

7 Related Work

Prior work probes factual knowledge in pretrained language models and develops multilingual benchmarks for other NLP tasks. X-FACTR combines these lines by targeting multilingual factual retrieval, a setting the authors identify as previously uncovered.

  • Factual knowledge retrieval: Factual knowledge has been probed either through cloze prompts that treat LMs as static knowledge sources or through question-answer fine-tuning.These approaches show that large-scale LMs can contain substantial factual knowledge, sometimes outperforming systems using external resources.
  • X-FACTR: X-FACTR extends prior factual probing by combining multilingual evaluation with multi-token entity prediction.This positions the benchmark at the intersection of multilinguality and factual knowledge retrieval.
  • Multilingual benchmarks: Multilingual benchmarks cover tasks such as question answering, language understanding, and syntactic prediction, but not multilingual factual retrieval from LMs.The authors state that, to their knowledge, this retrieval setting had not been covered by previous work.

8 Conclusion

The paper introduces X-FACTR as a multilingual, multi-token benchmark and evaluates factual retrieval across languages and language models. The results show that retrieval is difficult and that the knowledge represented in LMs varies by language.

  • Conclusion: X-FACTR examines multilinguality and LM factual knowledge through a benchmark containing multilingual and multi-token probes.The study compares retrieval across languages and models.
  • Conclusion: Experiments demonstrate that multilingual factual retrieval is difficult and that knowledge contained in LMs varies across languages.The conclusion frames these findings as the principal results of the cross-language and cross-model comparisons.
  • Future directions: Future work includes alternative pre-training or fine-tuning methods and techniques encouraging LMs to predict entities of the correct types.These directions follow the paper’s reported retrieval challenges.

A Benchmark Details

Benchmark composition differs across languages: high-resource languages contain a larger share of single-token object facts, whereas low-resource languages contain relatively more multi-token facts.

  • Benchmark composition: High-resource languages such as English, French, Dutch, and Spanish have a larger proportion of single-token facts than low-resource languages.The benchmark details report object-token ratios across languages and the number of facts per language.

B Benchmark Prompt Quality

The benchmark evaluates multilingual factual retrieval with prompts that may contain multi-token entities, using several masked-language-model decoding strategies. Prompt quality varies across languages, especially where inflection and grammatical gender are difficult, but the authors judge most prompts high quality.

  • B Benchmark Prompt Quality: Prompt evaluation uses native-speaker judgments on sampled sentences to assess whether multilingual relational prompts are natural and grammatical.The evaluation examines prompts after automated grammatical-attribute inference and inflection.
  • B Benchmark Prompt Quality: Russian and Marathi show comparatively high inflection-related error rates, arising from grammatical-gender prediction in Russian and limited inflection-model training data in Marathi.The Russian percentage includes gender and number errors, while Marathi errors are linked to scarce training data.
  • B Benchmark Prompt Quality: Most prompts are judged high quality despite occasional inflection or grammatical-gender mistakes that should remain intelligible to native speakers.The authors also note that some prompts may be awkward for particular relation senses because of the original Wikidata annotations.
  • C Multi-Token Decoding: Independent decoding predicts all mask tokens in parallel without conditioning on other newly predicted tokens.This method provides the initial prediction stage for multi-token entities.
  • C Multi-Token Decoding: Order-based decoding predicts mask tokens left to right, conditioning each step on previously generated tokens.It is one of the autoregressive approaches considered for initial prediction or refinement.
  • C Multi-Token Decoding: Order-based refinement replaces tokens sequentially with masks and re-predicts them until a left-to-right scan makes no changes.The procedure modifies predicted tokens from left to right.
  • C Multi-Token Decoding: Confidence-based decoding predicts the currently highest-confidence masked position first, allowing an arbitrary prediction order.Its refinement stage re-predicts the lowest-confidence token and stops when the replacement is unchanged.
  • C Multi-Token Decoding: Figure 6 reports the ratio of facts by object-token count, while Table 5 summarizes benchmark statistics and Table 6 reports prompt-error categories.Table 6 categories are not mutually exclusive, and its Russian inflection percentage uses a broader error definition.

C.2 Additional Decoding Components

Additional decoding components address score bias, stale token confidences, and search over alternative predictions. These additions increase computational cost when confidence values are repeatedly recomputed.

  • C.2 Additional Decoding Components: Length normalization divides the decoding score by the number of mask tokens because the unnormalized sum may favor short predictions.The normalization targets multi-token prediction length bias.
  • C.2 Additional Decoding Components: Confidence re-computation updates token probabilities after surrounding predictions change, reflecting masked LMs’ bidirectional conditional distributions.Without recomputation, previously assigned confidences become stale.
  • C.2 Additional Decoding Components: O(M^2BT) time complexity results when beam search, confidence recomputation, and the other decoding components are combined.Beam search retains B plausible predictions and requires deduplication when different decoding paths converge.
  • C.2 Additional Decoding Components: Table 7 presents M-BERT cases where the best decoding method succeeds although independent prediction fails, while Table 8 lists models and training corpora.The supplied passages identify these table contents but do not report additional numerical outcomes.

E Detailed Experimental Results

The paper reports detailed performance across languages and language models, alongside error analyses for Spanish and Greek. Results use different mask-token lengths for language groups, and unsupported models are marked explicitly.

  • E Detailed Experimental Results: Detailed performance across language models and languages is reported together with Spanish and Greek error cases.The supplied passage points to Tables 10 and 9 for these details.
  • E Detailed Experimental Results: Table 8 identifies multilingual and monolingual language models by their HuggingFace shortcut names and lists their training corpora.Its notes distinguish corpora including OSCAR, TwNC, SoNaR-500, OPUS, and Europarl.
  • E Detailed Experimental Results: Table 9 reports M-BERT error cases for Spanish and Greek as percentages.The supplied passage does not specify the individual error categories or values.
  • E Detailed Experimental Results: Table 10 reports accuracy by language and language model, using M = 5 mask tokens for English, French, Dutch, Spanish, and Vietnamese.The table uses M = 10 for the other languages and marks missing or unsupported models with “-”.
Loading 2010.06189v3…