Source-linked AI summary
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
Nora Kassner, Philipp Dufter, Hinrich Schütze
TL;DR
Prior multilingual evidence about language models as knowledge bases was limited, although extending evaluation beyond English matters for diversity and accessibility. The paper creates multilingual benchmarks and evaluates mBERT with typed queries across 53 languages, finding language-dependent performance and bias while pooling predictions improves performance. LAMA also contains easy-to-guess triples, and translation constraints limit the multilingual dataset's coverage.
Problem
Prior research on language models as knowledge bases focused exclusively on English, leaving multilingual performance and language dependence insufficiently studied.
Method
The authors translate TREx and GoogleRE into 53 languages and evaluate mBERT using typed candidate ranking and precision at one.
Results
mBERT's performance varies across languages, query language induces language-specific bias, and pooling predictions across languages improves performance.
Takeaways & Limitations
Multilingual probing reveals that mBERT does not store entity knowledge in a language-independent way, while multilingual prediction pooling can improve retrieval.
Takeaways & Limitations
LAMA contains many easy-to-guess triples, and translation coverage constraints reduce the studied multilingual dataset to 53 languages with incomplete translated-triple coverage.
Abstract
from arXiv · showhide
Recently, it has been found that monolingual English language models can be used as knowledge bases. Instead of structural knowledge base queries, masked sentences such as "Paris is the capital of [MASK]" are used as probes. We translate the established benchmarks TREx and GoogleRE into 53 languages. Working with mBERT, we investigate three questions. (i) Can mBERT be used as a multilingual knowledge base? Most prior work only considers English. Extending research to multiple languages is important for diversity and accessibility. (ii) Is mBERT's performance as knowledge base language-independent or does it vary from language to language? (iii) A multilingual model is trained on more text, e.g., mBERT is trained on 104 Wikipedias. Can mBERT leverage this for better performance? We find that using mBERT as a knowledge base yields varying performance across languages and pooling predictions across languages improves performance. Conversely, mBERT exhibits a language bias; e.g., when queried in Italian, it tends to predict Italy as the country of origin.
1 Introduction
This paper extends probing of pretrained language models as knowledge bases from English to multilingual settings, using mBERT across 53 languages. It finds varying language performance, query-language bias, and improved results when predictions are pooled across languages.
- Research questions: The study investigates whether mBERT can serve as a multilingual knowledge base across 53 languages.It addresses prior work's exclusive focus on English and links multilingual evaluation to diversity and accessibility.
- Contributions: The authors automatically create multilingual versions of TREx and GoogleRE and use typed entity ranking to investigate knowledge in pretrained language models.Typed querying ranks candidates of the entity type required by each template.
- Findings: mBERT answers queries with varying performance: it works reasonably for 21 languages and worse for 32 languages.
- Findings: Query language affects predictions, with Italian queries more likely to produce Italian entities such as Italy for country-of-origin questions.
- Findings: Pooling predictions across languages improves performance by large margins and even outperforms monolingual English BERT.
2 Data
The study builds mLAMA by translating LAMA's TREx and GoogleRE triples and templates into multiple languages, while evaluating knowledge with typed queries. Translation coverage and LAMA's easy-to-guess facts constrain the resulting multilingual dataset.
- LAMA: LAMA combines TREx and GoogleRE triples queried through manually created masked templates without finetuning.TREx contains 34,039 triples across 41 relations, while GoogleRE contains 5,528 triples across 3 relations.
- LAMA: The dataset includes many easy-to-guess triples, such as inferring that a person with an Italian-sounding name was born in Italy.
- Translation: The authors translate entities and templates using Google Translate, Wikidata, and Google Knowledge Graphs, then check templates for validity and correct them when necessary.German, Hindi, and Japanese templates receive additional native-speaker checks.
- Translation: mBERT covers 104 languages, but translation resources support only some languages and entities; the study therefore retains 53 languages with coverage above 20%.The macro average of translated triples across these languages is 43%.
3 Experiments
The experiments compare mBERT and BERT under typed and untyped querying, including multilingual and multitoken settings. Typed querying constrains predictions to relation-specific candidates and improves evaluation relative to untyped querying, especially for multitoken objects.
- 3.1 Model: mBERT is pretrained on the 104 largest Wikipedias, while English BERT-Base provides the comparison model.
- 3.2 Typed and Untyped Querying: Typed querying selects the highest-probability entity from a relation-specific candidate set, whereas untyped querying selects from the full vocabulary.For example, the candidate set for a birth-place relation contains cities.
- 3.2 Typed and Untyped Querying: Typed querying prevents type errors such as predicting years instead of cities for a template asking where someone was born.
- 3.3 Singletoken vs. Multitoken Objects: Multitoken entities are evaluated by averaging their tokens’ log probabilities across masked templates.The method considers templates with one through the maximum number of tokens required by any candidate entity.
- 3.4 Evaluation: The final metric is precision at one, macro-averaged over relations and, for multilingual results, over languages.
4 Results and Discussion
The authors evaluate typed versus untyped querying, then analyze mBERT across languages, finding substantial variation, language-specific biases, and gains from pooling predictions.
- 4.1 UnTyQ vs. TyQ: TyQ is better suited than UnTyQ for evaluating knowledge in language models because it reduces errors from type constraints, fluency, and multitoken answers.UnTyQ has mean p1 .03 versus .17 for TyQ on multitoken objects.
- 4.1 UnTyQ vs. TyQ: Manual template corrections produce only small effects, so template modification is not an effective solution to the typing problem.This conclusion follows from corrected and paraphrased templates evaluated for German, Hindi, and Japanese.
- 4.1 UnTyQ vs. TyQ: Machine-translated templates outperform manual translations by 1 percentage point in averaged p1, 0.23 versus 0.24, on a common 14-language subset.The comparison uses TyQ on the TREx subset.
- 4.3 Multilingual Performance: mBERT performs reasonably in 21 languages, but rel-p1 is below 0.6 for 32 languages, showing unstable performance across languages.rel-p1 compares each language's p1 with mBERT's English p1 on the same triples.
- 4.4 Bias: Query language affects predictions: Italian queries tend to produce Italy as the country of origin, demonstrating language-specific bias.The effect is reported across several relations, with additional evidence in the supplementary material.
- 4.5 Pooling: Pooling predictions by selecting the majority-predicted object improves over mBERT queried in English by 6 percentage points on LAMA and over BERT by 3 percentage points on LAMA-UHN.The authors attribute part of the pooled improvement to eliminating language-specific bias.
5 Related Work
Related work established language models as knowledge bases and explored multilingual language modeling, but had not analyzed multilingual pretrained models as knowledge bases. The paper addresses this gap with translated, comparable benchmarks across languages.
- Prior knowledge-base research: Earlier LAMA research used masked templates to probe factual knowledge in pretrained language models, but addressed only English.The cited work also examined issues such as negation, names, adapters, and alternatives to fill-in-the-blank querying.
- Multilingual models: Although multilingual models support zero-shot crosslingual transfer, prior work had not examined how well they function as knowledge bases.Existing multilingual question-answering datasets do not substitute for this specific analysis.
- Benchmark choice: The authors use translated TREx and GoogleRE because manually created templates and entities preserve comparability across languages better than crowd-sourced databases.They contrast this approach with resources such as ConceptNet and BabelNet.
- Position relative to contemporaneous work: Contemporaneous work translated LAMA templates for 23 languages, whereas this paper emphasizes typed querying, comparability, pooling across languages, and language biases.The distinction identifies the paper's focus relative to nearby multilingual LAMA research.
6 Conclusion
The paper presents mLAMA for multilingual investigation of language-model knowledge across 53 languages. Results indicate that entity knowledge retrieval varies by language, with performance gaps and language bias showing it is not language-independent.
- mLAMA investigates knowledge in language models across 53 languages.
- Correct entities can be retrieved for many languages, but performance lags behind English for languages such as Japanese and Thai.
- Language-bias experiments indicate that mBERT does not store entity knowledge in a language-independent way.
A Language Bias
The querying language clearly affects mBERT's predictions, with especially strong effects for relations asking about countries, while some other relations remain stable.
- Querying language changes the most common predicted entities across the evaluated relations.The analysis aggregates predictions across all triples for 10 relations.
- Country-related relations such as P495 and P1001 show drastic language-bias effects.
- Relations P463 and P178 are relatively stable, whereas P39 varies substantially without a clear pattern.
B Data Samples
Tables 4 and 5 present randomly sampled entries from the data.
- Tables 4 and 5 show randomly sampled data entries.
C Pretraining Data
The paper examines whether language performance relates to the amount of pretraining data, using Wikipedia article counts and TyQ performance. The analysis notes that these counts may differ from mBERT's actual training data, and Figure 6 shows no clear trend.
- The analysis relates language performance to the number of Wikipedia articles available for each language.It uses article counts as of January 2021 and p1 for TyQ.
- The article counts may differ from mBERT's actual pretraining data because the original training data were unavailable.
- Figure 6 shows no clear trend between TyQ p1 and the number of corresponding Wikipedia articles.