PaperlayerView exact revision on arXiv

Source-linked AI summary

MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering

Shayne Longpre, Yi Lu, Joachim Daiber

arXiv:2007.15207v2cs.CL

TL;DR

MKQA addresses limited, non-comparable, and passage-dependent multilingual QA evaluation by introducing a language-independent, retrieval-independent benchmark aligned across 26 languages. It is challenging in English and especially difficult for lower-resourced languages.

  • Problem

    Existing multilingual QA datasets lack linguistic diversity, comparable evaluation between languages, and retrieval-independent annotations, limiting their use across retrieval, knowledge-graph, and generative approaches.

  • Method

    MKQA is a multilingual open-domain QA evaluation set whose language-independent answers support predictions extracted from documents, generated, or derived from knowledge graphs.

  • Results

    The best English system reaches only 52.3% F1, while performance remains substantially weaker in lower-resourced languages such as Korean and Arabic.

  • Takeaways & Limitations

    MKQA provides a challenging benchmark for measuring cross-language progress in multilingual open-domain question answering.

  • Takeaways & Limitations

    Alternative setups such as intentionally mismatched question and answer languages or propagating English answers into other languages are left for future work.

Abstract

from arXiv · show

Progress in cross-lingual modeling depends on challenging, realistic, and diverse evaluation sets. We introduce Multilingual Knowledge Questions and Answers (MKQA), an open-domain question answering evaluation set comprising 10k question-answer pairs aligned across 26 typologically diverse languages (260k question-answer pairs in total). Answers are based on a heavily curated, language-independent data representation, making results comparable across languages and independent of language-specific passages. With 26 languages, this dataset supplies the widest range of languages to-date for evaluating question answering. We benchmark a variety of state-of-the-art methods and baselines for generative and extractive question answering, trained on Natural Questions, in zero shot and translation settings. Results indicate this dataset is challenging even in English, but especially in low-resource languages

1 Introduction

MKQA addresses multilingual QA evaluation gaps with aligned, retrieval-independent annotations across 26 diverse languages. Its curated setup supports fair comparison across QA approaches while remaining challenging, including in English.

  • Dataset contribution: MKQA introduces 10k Natural Questions queries human-translated into 25 additional languages, producing aligned multilingual evaluation examples.Answers replace passage-dependent spans with language- and retrieval-independent annotations linked to Wikidata entities and defined value types.
  • Evaluation design: Fully aligned examples enable direct comparison across all 26 languages, including for cross-lingual and zero-shot systems.The evaluation reserves assessment for predominantly geographically invariant queries to improve fairness across languages.
  • Annotation design: Retrieval-independent annotations avoid tying answers to particular passage phrasing, supporting retrieval, knowledge-graph, and generative QA methods.The authors argue this design also minimizes translation artifacts associated with translating passage-bound annotations.
  • Dataset contribution: 26 languages and dialects span 14 language-family branches, covering half of the world’s native-language population.The authors describe MKQA as the largest and most linguistically diverse open-domain QA evaluation set available at the time.
  • Difficulty and results: 52.3% F1 is achieved by the best English model, while the best result is only 5.7% above a naive baseline on the lowest-resource language.The authors conclude that MKQA is more challenging than Natural Questions and supports broad multilingual QA evaluation.

2 Related Work

Prior multilingual QA resources trade off linguistic diversity, cross-language comparability, or retrieval independence. MKQA combines broad typological coverage with aligned questions and document-independent answers.

  • Existing resources: English QA has many established resources, whereas non-English QA datasets remain comparatively rare and usually cover only one additional language.Existing examples include SQuAD, TriviaQA, Natural Questions, and several Chinese-focused datasets.
  • Multilingual QA resources: MLQA and XQuAD provide parallel examples but use extractive answers derived from supplied documents.XQA offers retrieval-independent entity answers, while TyDi emphasizes typological diversity but uses retrieval-system-based annotations.
  • Comparison with native-speaker datasets: TyDi and MKQA both target typological diversity, but TyDi uses different questions per language, limiting direct comparability.TyDi’s answerable-passage rates also vary substantially across languages, from 22% in Korean to 69% in Arabic.
  • Comparison with native-speaker datasets: MKQA’s predominantly geographically invariant questions reduce variation caused by cultural or geographic differences in answerability.This design helps isolate multilingual system-performance differences more fairly across languages.
  • Answer annotation: 67.58% answer coverage exceeds TyDi’s 38% language average because MKQA answers are not limited by selected passage quality.MKQA defines unanswerability by whether a human can find a succinct web answer, rather than whether a passage contains one.
  • Answer annotation: Document-independent answers let MKQA evaluate approaches without requiring answers to come from a single retrieved source document.This contrasts with TyDi’s extractive spans tied to particular Wikipedia documents and indices.

3 Dataset Collection

MKQA constructs reliable, flexible answers through multi-stage curation, normalization, entity linking, agreement checks, and localization into 26 languages. The pipeline begins with realistic Natural Questions queries and produces structured answer labels usable across QA approaches.

  • Design goals: The collection pipeline targets realistic questions, reliable annotations, and minimal assumptions about the underlying QA technique.These goals support fair comparison among different multilingual open-domain QA methods.
  • Query selection: 10,000 Natural Questions queries are sampled because they originate from realistic information-seeking users.The six-stage pipeline then performs four answer-curation steps followed by question-and-answer localization into 26 languages.
  • Answer curation: Five annotators independently search the web to copy or generate ideal answers and assign each answer a typed category.Categories include atomic values, entities, yes/no, short answer, long answer, and unanswerable.
  • Answer curation: Answer formats are normalized and short answers are resolved against Wikidata to disambiguate entities and collect aliases.Three graders review entity-linking suggestions, with domain-expert adjudication when agreement is insufficient or the correct entity is absent.
  • Answer curation: Answers accepted by at least two annotators enter the gold set, while unresolved cases receive domain-expert decisions using all five preliminary annotations.This agreement-based process is the fourth answer-curation stage.
  • Answer localization: Verified answers are localized using Wikidata names and aliases where available, with professional bilingual translators handling missing links or language entries.Localization combines translation, transliteration, and unchanged forms, while query translators preserve meaning and use localized entity names when available.

4 Dataset Quality and Analysis

MKQA evaluates multilingual QA with geographically broad, parallel languages and retrieval-independent annotations. Quality analysis finds translation and annotation artifacts, while its answer-type design makes evaluation more challenging than NQ.

  • Linguistic Diversity: 90.62% of the world population lives in a country with an official language included in MKQA.The percentage is calculated using country populations and official-language links from Wikidata.
  • Translation and Answer Quality: A grading experiment covering just above 1% of the data assesses query-translation acceptability and answer preservation across languages.Graders also judge original English queries as a baseline.
  • Translation and Answer Quality: Chinese translations receive lower quality judgments than German, Spanish, and Thai, while answer acceptability remains at or above 90% even for Thai.Reported error sources include cultural-context differences, generic annotation issues, incorrect transliteration, and translation artifacts.
  • Answer Type Analysis: 32.4% of MKQA examples are Unanswerable or Long, compared with 63% of NQ short-answer questions.MKQA defines unanswerability by whether a human can find a succinct answer, rather than whether a retrieved passage contains one.
  • Dataset Analysis: Retrieval-independent annotations support challenging evaluation, supplementary metadata, and interpretable error analysis across QA methods.The framework provides answer types and Wikidata QIDs alongside the annotations.

5 Experiments

MKQA benchmarks extractive and generative QA systems across 26 aligned languages using retrieval, translation, and reader baselines. Results identify retrieval as a central bottleneck and show a substantial performance gap between English and lower-resource languages.

  • 5.1 Task Definition: MKQA predictions may be No Answer or a target-language Text Answer produced by extraction, generation, or a knowledge graph.The task is defined for any QA method rather than a single retrieval architecture.
  • 5.1 Task Definition: Per-language F1 is averaged across examples, while Macro Average F1 is the mean of the 26 per-language F1 scores.Evaluation reports exact match and token-overlap F1 after language-specific normalization.
  • 5.2 Baseline Approaches: Baselines combine off-the-shelf retrieval with readers fine-tuned on Natural Questions, including XLM-R, M-BERT, RoBERTa, and MT5.Settings include multilingual retrieval, English retrieval with translation, gold NQ passages, extractive QA, and generative QA.
  • 5.3 Results: Translation followed by English DPR outperforms multilingual sparse retrievers, while Gold NQ documents contain a valid answer 80.22% of the time.The retrieval comparison is averaged across all 26 languages.
  • 5.3 Results: 32.42% F1 is the lower bound from predicting No Answer for every example.MKQA combines Unanswerable and Long Answers into No Answer to focus on automatically and robustly evaluated short factoid answers.
  • 5.3 Results: Retrieval remains a multilingual open-retrieval QA bottleneck, and readers and translation still fail to consistently produce localized answers even with gold passages.Answerable-question performance is higher when retrieval succeeds, but answerable questions are more difficult overall than unanswerable ones.
  • 5.3 Results: 52.3 F1 is achieved by XLM-R on English, while performance is poor for lower-resource language families.The minimum score is 32.42%, obtained when the threshold predicts No Answer for every question.

6 Discussion

MKQA is challenging even in English because retrieval-independent, higher-coverage annotations reclassify difficult questions as answerable and reduce the dominance of easy unanswerable examples. Its parallel design makes difficulty comparable across 26 languages while exposing a substantial gap for lower-resourced languages.

  • Difficulty of MKQA: 52.3% F1 is the best English baseline score on MKQA, below performance on Natural Questions, TriviaQA, and other standard benchmarks.The authors characterize MKQA as highly challenging in English and as leaving room for improvement across English and the long tail of natural languages.
  • Annotation differences: MKQA’s retrieval-independent, high-coverage re-annotation shifts the most challenging questions from unanswerable in NQ to answerable in MKQA.NQ can miss answers through retrieval failure, unavailable Wikipedia answers, or annotator misses; MKQA is not susceptible to the first two and is likely less affected by the third.
  • Annotation differences: 70.4% of documents for NQ-labeled unanswerable queries contain a valid MKQA answer, suggesting annotator error is the largest source of those NQ labels.This is also identified as the largest source of improvement in MKQA’s label quality.
  • Answer distribution: 69.38% F1 for NQ versus 52.08% for MKQA at the best threshold is primarily explained by NQ’s higher proportion of easier unanswerable examples.MKQA has 32% unanswerable examples compared with 63% in NQ, making its task more oriented toward answering than abstaining.
  • Cross-lingual difficulty: MKQA is similarly challenging in all 26 languages, while Korean and Arabic reach only 6% above the 32.42% always-unanswerable lower bound.The reported English-to-lower-resource gap indicates substantial deficits for current transfer-learning methods in low-resource multilingual QA.
  • Implications: MKQA’s parallel questions and language-independent annotations support fair comparison among knowledge-graph, retrieval, and generative QA methods across languages.The benchmark is intended to let practitioners and researchers test and compare techniques more fairly, comparably, and precisely than previous benchmarks.

7 Conclusion

MKQA is presented as a multilingual open-domain QA evaluation set whose geographical invariance, parallel questions, retrieval-independent annotations, and linguistic diversity distinguish it from existing resources. Baseline evaluations show substantial room for improvement in English and lower-resource languages.

  • MKQA combines geographical invariance, language-parallel questions, retrieval-independent annotations, and linguistic diversity.
  • These properties distinguish MKQA from existing resources in annotation quality, difficulty, and flexibility for evaluating new approaches.
  • Baseline results show ample room for improvement in both English and lower-resource languages.
Loading 2007.15207v2…