PaperlayerView exact revision on arXiv

Source-linked AI summary

MLQA: Evaluating Cross-lingual Extractive Question Answering

Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, Holger Schwenk

arXiv:1910.07475v3cs.CLcs.AIcs.LG

TL;DR

Multilingual QA lacks large, high-quality evaluation data because such datasets are difficult and costly to collect. The paper introduces MLQA, a highly parallel seven-language extractive QA benchmark with cross-lingual tasks and baselines. Results demonstrate a substantial gap between transfer and training-language performance, motivating further work in cross-lingual QA.

  • Problem

    Multilingual QA datasets are scarce because collecting them at sufficient scale and quality is difficult and costly, limiting evaluation of cross-lingual QA progress.

  • Method

    MLQA constructs a highly parallel seven-language extractive QA benchmark from aligned Wikipedia contexts, with translated questions, target-language answer spans, and two cross-lingual evaluation tasks.

  • Results

    Baselines show significant room for improvement, with transfer performance substantially behind training-language performance; G-XLT trails XLT by 8.2 mean F1.

  • Takeaways & Limitations

    MLQA provides a common, parallel testbed intended to catalyse research toward closing the gap between training- and testing-language performance.

  • Takeaways & Limitations

    Wikipedia contexts may sometimes derive from machine-translated article seeds or human translations rather than independently written content.

Abstract

from arXiv · show

Question answering (QA) models have shown rapid progress enabled by the availability of large, high-quality benchmark datasets. Such annotated datasets are difficult and costly to collect, and rarely exist in languages other than English, making training QA systems in other languages challenging. An alternative to building large monolingual training datasets is to develop cross-lingual systems which can transfer to a target language without requiring training data in that language. In order to develop such systems, it is crucial to invest in high quality multilingual evaluation benchmarks to measure progress. We present MLQA, a multi-way aligned extractive QA evaluation benchmark intended to spur research in this area. MLQA contains QA instances in 7 languages, namely English, Arabic, German, Spanish, Hindi, Vietnamese and Simplified Chinese. It consists of over 12K QA instances in English and 5K in each other language, with each QA instance being parallel between 4 languages on average. MLQA is built using a novel alignment context strategy on Wikipedia articles, and serves as a cross-lingual extension to existing extractive QA datasets. We evaluate current state-of-the-art cross-lingual representations on MLQA, and also provide machine-translation-based baselines. In all cases, transfer results are shown to be significantly behind training-language performance.

1 Introduction

MLQA addresses the scarcity of high-quality multilingual QA data by introducing a highly parallel extractive benchmark and two cross-lingual evaluation tasks. It also provides baselines showing substantial room to improve transfer across languages.

  • Motivation: Multilingual QA datasets remain scarce because collecting them at sufficient scale and quality is difficult and costly.This limits both progress measurement and straightforward supervised training for QA in other languages.
  • Motivation: Existing cross-lingual datasets often cover few languages, lack parallel instances, or use less practical QA settings.Highly parallel data supports fairer language comparisons, fewer source-language annotations, and additional evaluation setups without extra annotation cost.
  • Benchmark construction: MLQA is a multi-way parallel extractive QA benchmark covering English, Arabic, German, Vietnamese, Spanish, Simplified Chinese, and Hindi.The benchmark is constructed from aligned Wikipedia sentences and paragraphs, with English questions translated professionally and target-language answer spans annotated in aligned contexts.
  • Benchmark construction: Over 46,000 QA annotations comprise 5,000–6,000 instances per language, more than 12,000 in English, and predominantly four-way-aligned instances.Each instance has aligned equivalents in multiple other languages, always including English.
  • Evaluation tasks: MLQA evaluates cross-lingual transfer from English and generalized transfer with questions and contexts written in different languages.Generalized transfer is possible because the benchmark is highly parallel.
  • Baselines: Zero-shot XLM transfers best among the reported baselines, but all models lag well behind training-language performance.The baselines use English SQuAD training data and include machine-translation span mapping, multilingual BERT, and XLM.
  • Contributions: The paper contributes a multilingual annotation pipeline, the MLQA dataset, two cross-lingual QA tasks, and baseline evaluations demonstrating significant room for improvement.These contributions position MLQA as an evaluation resource for advancing cross-lingual QA.

2 The MLQA corpus

MLQA is designed as a multilingual, parallel extractive QA benchmark whose construction supports fair cross-language comparison and multiple transfer settings. Its corpus combines aligned Wikipedia contexts with English crowdsourced questions, professional translations, and target-language answer annotations.

  • Design goals: MLQA targets parallel evaluation across diverse languages, enabling fairer transfer comparisons, additional cross-language setups, and lower annotation cost.The dataset also seeks natural documents, extractive QA coverage, and compatibility with existing training resources.
  • Data source: The corpus uses Wikipedia because its multilingual scale supports collection across languages and allows reuse of existing QA training resources.English is selected as the source language because it has the largest Wikipedia and an accessible worker pool.
  • Annotation pipeline: The three-step pipeline extracts aligned paragraphs, crowdsources English questions with answers inside the aligned sentence, then translates questions and annotates target-language spans.Constraining English answers to the aligned sentence increases the likelihood that questions remain answerable across languages.
  • Parallel sentence mining: 385,396 four-way parallel sentences were selected after incremental alignment with English, balancing language parallelism against sentence quantity and diversity.Seven-way aligned sentences were found less linguistically diverse and often concentrated near article beginnings.
  • Annotation quality: 82% inter-annotator agreement was obtained for English answers, comparable to 84% on the SQuAD v1.1 development set.Two additional answer annotations were collected for each question to calculate mean token F1 agreement.
  • Corpus composition: MLQA contains 12,738 English instances, 5,029–6,006 instances per target language, and 9,019 four-way parallel instances across a broad topical range.The corpus also includes 2,930 three-way and 789 two-way parallel instances.

3 Related Work

Related work spans extensive English QA resources, emerging multilingual benchmarks, cross-lingual QA modeling, and parallel QA datasets. MLQA extends these efforts with broader multilingual and parallel coverage using real Wikipedia contexts.

  • Monolingual QA Data: English has a wide variety of QA datasets, with large span-based resources helping establish extractive QA as a dominant paradigm.Examples include SQuAD, TriviaQA, NewsQA, and Natural Questions.
  • Monolingual QA Data: Non-English QA datasets remain comparatively limited, although resources have expanded to languages including Chinese, Korean, and Arabic.The passage contrasts this broader but still smaller collection with the abundance of English data.
  • Cross-lingual QA Modelling: Cross-lingual QA modeling has explored RDF question answering, English-to-Korean transfer, and English-question-based Hindi generation.These approaches use English resources in different ways, sometimes alongside in-language seed data.
  • Cross-lingual QA Data: Existing parallel QA datasets include English–Hindi, English–Bulgarian, Chinese–English cloze QA, and English–Chinese novel-based resources, but are generally limited to two languages.A manually translated extractive resource provides 1,190 SQuAD instances in 10 languages.
  • Positioning MLQA: MLQA uses real Wikipedia contexts and provides more than 5,000 QA pairs per language across seven languages.This distinguishes it from manually translated resources with fewer instances per language.
  • Aggregated Cross-lingual Benchmarks: XGLUE and XTREME incorporate MLQA as part of broader unified multilingual benchmark suites.These projects aggregate multiple high-quality multilingual tasks rather than focusing only on QA.

4 Cross-lingual QA Experiments

The experiments evaluate cross-lingual QA through transfer tasks and zero-shot or translation-based baselines. Models are trained with English data and assessed on multilingual MLQA settings using language-aware evaluation procedures.

  • Evaluation tasks: MLQA evaluates cross-lingual transfer, where English-trained models answer questions in a different target language, and generalized transfer, where question and context languages differ.Parallel instances permit question-language swapping while preserving question meaning.
  • Translation baselines: Translate-Train machine-translates SQuAD instances into each target language, encloses answers in quotes, and discards approximately 5% of instances when span extraction fails.The translated corpus is then used to train a target-language model.
  • Translation baselines: Translate-Test translates target-language contexts and questions into English, predicts an English span, and maps it back by selecting the original span with maximum F1.Attention scores from the translation model support answer-span mapping for languages other than Hindi.
  • Representation baselines: The zero-shot experiments train multilingual BERT and XLM on SQuAD, select models using only English development data, and test one model across all seven languages.No target-language training or development data is used.
  • Evaluation metrics: Multilingual evaluation adapts SQuAD preprocessing by stripping Unicode punctuation and language-specific articles, while using mixed segmentation for Chinese.Whitespace tokenization is retained for the other MLQA languages.

5 Results

MLQA evaluation shows that XLM transfers best overall, but cross-lingual performance remains substantially below English training-language performance. Results also vary by question type, language pairing, and whether the question is answered well in English.

  • XLT Results: XLM transfers best overall on XLT, while back-translated answers perform poorly because they may not map back to spans in the original context.The Translate-Test baseline is specifically weakened by span-mapping failures after back-translation.
  • XLT Results: 39% of XLM exact-match errors are completely wrong answers, 49% partially overlap the gold span, 7% are acceptable non-overlapping answers, and 5% are annotation errors.Error proportions vary little across languages.
  • XLT Results: “When” questions are easiest across languages, whereas “Where” questions are challenging in most target languages.F1 is stratified by common English wh-words relative to overall XLM performance.
  • XLT Results: Transfer performance is better when XLM answers the English question well, but remains above zero when its English F1 is zero.This suggests that some questions may be easier to answer in certain languages than in English.
  • G-XLT Results: In G-XLT, performance is usually best when question and context languages match, while English questions usually work best for a given context language.Hindi and Arabic are exceptions to the matching-question pattern, while Chinese and Vietnamese are exceptions to the English-question pattern.
  • English Comparison: MLQA-English scores are on average 5.8% below adjusted SQuAD scores, with longer contexts and slightly longer answers contributing to the discrepancy.MLQA-English contexts are 28% longer than SQuAD contexts, and answers average 3.1 rather than 2.9 tokens.

6 Discussion

The discussion highlights limitations in MLQA’s contexts, annotation procedure, and evaluation design, while finding that cross-lingual models’ word-matching behavior is more robust than expected. The authors also identify target-language development data and alternative training languages as future directions.

  • Context Quality: MLQA contexts may include human-edited translations or machine-translation seeds rather than independently written multilingual content.The parallel sentence-mining method can therefore source existing translations instead of fully independent documents.
  • Annotation Constraints: Restricting answers to specified sentences may encourage single-sentence questions, although this mirrors a known SQuAD annotation issue.The authors note that MLQA-English scores are similar to SQuAD, suggesting the restriction does not make questions less challenging.
  • Evaluation Scope: The experiments use only English development data under strict zero-shot conditions, leaving target-language fine-tuning and other training datasets for future work.Using target-language development sets for optimization or fine-tuning could support higher transfer performance, but those experiments are not conducted here.
  • Cross-lingual Robustness: An 8.2 mean-F1 drop from XLT to G-XLT suggests cross-lingual models’ word-matching behavior is more nuanced and robust than simple language mismatch would imply.G-XLT makes symbolic question-context matching less straightforward because the languages differ.

7 Conclusion

The paper introduces MLQA, a highly parallel seven-language QA benchmark, and evaluates state-of-the-art baselines on two cross-lingual tasks. The results demonstrate substantial room to narrow the gap between training- and testing-language performance.

  • Conclusion: MLQA is a highly parallel multilingual QA benchmark covering seven languages.The benchmark supports evaluation across English, Arabic, German, Vietnamese, Spanish, Simplified Chinese, and Hindi.
  • Conclusion: State-of-the-art baselines are evaluated on two cross-lingual understanding tasks, demonstrating significant room for improvement.The stated goal is to catalyse work that closes the training-language versus testing-language performance gap.

A.1 Annotation Interface

The annotation interface asks workers to write questions and highlight answer spans within bold parallel sentences, with validation features and instructions supporting data entry.

  • Annotation Interface: Workers write each question and highlight its answer span in the bold sentence.The interface includes data-input validation features and a drop-down window with detailed instructions.

A.2 Additional MLQA Statistics

Additional statistics show that MLQA resembles SQuAD in question-type distribution while covering more topics and exhibiting substantial variation in sequence lengths across languages.

  • MLQA and SQuAD have very similar distributions of question types by wh-word.This supports using SQuAD as the English training dataset.
  • MLQA averages 1.9 context paragraphs per Wikipedia article, compared with 43 in SQuAD, yielding broader topic coverage.
  • Vietnamese has the longest average contexts and German the shortest, while every language has a substantial tail of long contexts.
  • Except in Chinese, answers average 3 to 4 tokens across languages.

A.3 QA Performance stratified by question and answer types

The appendix stratifies MLQA performance by question words, answer named-entity types, and English question difficulty, revealing systematic variation across languages and model settings.

  • By wh-word: “When” questions are consistently easier than average across languages, while “how” questions are more challenging in English.
  • By wh-word: “Where” questions are challenging for Spanish, German, Chinese, and Hindi, but not for Arabic or Vietnamese.
  • By Named-Entity type: Answer spans containing named entities are easier than those without them across languages, with the largest difference in German.
  • By Named-Entity type: Temporal answers are consistently easier than average, especially in German, Spanish, Hindi, and Vietnamese.
  • By English Question Difficulty: The difficulty split evaluates language performance according to whether XLM had zero word overlap with the correct English answer.
  • XLM achieves 53.4 mean G-XLT F1 versus 47.2 for Multilingual BERT, with a smaller XLT-to-G-XLT drop of 8.2 versus 10.5.
Loading 1910.07475v3…