XOR QA: Cross-lingual Open-Retrieval Question Answering

Akari Asai; Jungo Kasai; Jonathan H. Clark; Kenton Lee; Eunsol Choi; Hannaneh Hajishirzi

Source-linked AI summary

XOR QA: Cross-lingual Open-Retrieval Question Answering

Akari Asai, Jungo Kasai, Jonathan H. Clark, Kenton Lee, Eunsol Choi, Hannaneh Hajishirzi

arXiv:2010.11856v3Oct 22, 2020cs.CL

TL;DR

XOR QA addresses the gap created when multilingual questions lack answers in their own language because of information scarcity or asymmetry. The paper introduces a cross-lingual open-retrieval dataset and three tasks spanning retrieval, answer-span extraction, and end-to-end generation. The best baseline averages 18.7 F1 on XOR-FULL, showing the benchmark remains challenging.

Problem
Multilingual QA often assumes same-language answers, although non-English questions may require information from other languages because of information scarcity or asymmetry.
Method
The paper constructs XOR-TYDI QA with 40k native-speaker questions across seven languages and introduces three tasks using English and multilingual Wikipedia resources.
Results
18.7 F1 points is the best baseline’s average on XOR-FULL, and experiments characterize XOR-TYDI QA as a challenging benchmark.
Takeaways & Limitations
XOR-TYDI QA provides a benchmark for advancing multilingual representation learning and multilingual open-retrieval QA.
Takeaways & Limitations
Estimated translation costs for three other non-English languages were considerably higher because translation cost depends on translator availability.

Abstract

from arXiv · show

Multilingual question answering tasks typically assume answers exist in the same language as the question. Yet in practice, many languages face both information scarcity -- where languages have few reference articles -- and information asymmetry -- where questions reference concepts from other cultures. This work extends open-retrieval question answering to a cross-lingual setting enabling questions from one language to be answered via answer content from another language. We construct a large-scale dataset built on questions from TyDi QA lacking same-language answers. Our task formulation, called Cross-lingual Open Retrieval Question Answering (XOR QA), includes 40k information-seeking questions from across 7 diverse non-English languages. Based on this dataset, we introduce three new tasks that involve cross-lingual document retrieval using multi-lingual and English resources. We establish baselines with state-of-the-art machine translation systems and cross-lingual pretrained models. Experimental results suggest that XOR QA is a challenging task that will facilitate the development of novel techniques for multilingual question answering. Our data and code are available at https://nlp.cs.washington.edu/xorqa.

1 Introduction

XOR QA addresses multilingual questions whose answers may be unavailable in the question language by enabling cross-lingual retrieval from English and target-language resources. The paper introduces a 40k-question dataset across seven languages, three diagnostic-to-end-to-end tasks, and baselines whose best XOR-FULL result is 18.7 F1.

Motivation: Most prior multilingual QA work focused on English, motivating a dataset that combines native-speaker information-seeking questions, open retrieval, and multilingual QA.The goal is to enable cross-lingual answer retrieval rather than assume answers exist in the question language.
Dataset: XOR-TYDI QA contains 40k annotated questions across seven typologically diverse languages, drawn from native-speaker questions originally unanswerable because of information scarcity or asymmetry.Its reference documents are multilingual, supporting cross-lingual retrieval.
Dataset construction: The annotation pipeline translates questions into English, retrieves and verifies answers in English Wikipedia, then translates verified answers back to the target languages.A model-in-the-loop framework is used during answer finding to reduce annotation errors.
Tasks: The three tasks progress from retrieving English evidence paragraphs, to extracting minimal English answer spans, to generating target-language answers using English and target-language Wikipedia.The first two tasks diagnose retrieval and reading failures before the end-to-end XOR-FULL task.
Results: 18.7 F1 points is the best baseline’s average on XOR-FULL, indicating that XOR-TYDI QA remains challenging for real-world multilingual open-retrieval QA.The baselines extend state-of-the-art open-retrieval QA systems to multilingual retrieval.

2 The XOR-TYDI QA Dataset

XOR-TYDI QA builds a cross-lingual open-retrieval dataset from native-speaker questions that lacked same-language answers. Its annotation pipeline translates questions, retrieves and verifies answers using English Wikipedia, then translates answers back across seven languages.

Dataset design: XOR-TYDI QA combines cross-lingual retrieval with native-speaker information-seeking questions across seven languages.The dataset contains questions inherited from TYDI QA that were originally unanswerable because of information scarcity or asymmetry.
Annotation pipeline: The annotation pipeline selects unanswered TYDI QA questions, translates them into English, retrieves answers from English Wikipedia, verifies them, and translates answers back.This four-step process supports cross-lingual retrieval while preserving the original target-language question setting.
Annotation pipeline: A model-in-the-loop framework balances retrieval and annotation errors by ranking candidate paragraphs before annotators inspect them.Google Search retrieves up to ten Wikipedia articles, Path Retriever ranks their paragraphs, and annotators first inspect the five highest-scoring paragraphs.
Dataset results: 40% is the maximum reported answer-coverage improvement from cross-lingual retrieval, with answers found for more than 50% of questions in six of seven languages.The improvement is measured for questions becoming answerable through English Wikipedia search.
Dataset results: The dataset includes manually translated training questions and examples showing that answers may require English or another language's Wikipedia.English Wikipedia supplies information absent from Korean Wikipedia in one example, while another target-language-specific case requires a different language resource.
Dataset limitations: Telugu questions about local administrative details remain often unanswerable because English Wikipedia lacks them and Telugu Wikipedia coverage is low.This identifies a concrete coverage boundary for cross-lingual retrieval.

3 XOR QA Tasks and Baselines

XOR QA defines three increasingly complex tasks for answering non-English questions with English and target-language Wikipedia, supported by translation and multilingual baselines.

XOR-RETRIEVE: XOR-RETRIEVE retrieves English Wikipedia paragraphs containing enough information to answer a target-language question.
Evaluation: Retrieval is evaluated by recall at 2k and 5k selected tokens, while answer extraction uses Exact Match and F1.
Baselines: Baselines translate queries before monolingual retrieval or directly apply multilingual pretrained models for cross-lingual retrieval.
XOR-ENGLISHSPAN: XOR-ENGLISHSPAN retrieves English paragraphs and extracts a minimal answer span for the target-language question.
XOR-FULL: XOR-FULL generates an answer in the target language using both English and target-language Wikipedia without knowing where the answer appears.

4 Experiments and Analysis

Experiments show that all three XOR QA tasks remain challenging, with translation quality strongly affecting retrieval and downstream answering, while multilingual resources help XOR-FULL.

All three XOR QA tasks remain challenging even for strong baseline models.
XOR-RETRIEVE: 72.1 R@5kt is achieved by DPR with human translations, versus 67.2 with GMT and 50.0 with the authors’ MT.
XOR-RETRIEVE: BM25 substantially underperforms the other retrieval models, while DPR generally matches or exceeds Path Retriever.
XOR-ENGLISHSPAN: 38.2 macro F1 with human translations exceeds 32.9 for GMT and 20.5 for the authors’ MT in XOR-ENGLISHSPAN.
XOR-FULL: 18.7 F1, 12.1 EM, and 16.8 BLEU are achieved by the best XOR-FULL pipeline using GMT, Google Search, and DPR.
XOR-FULL: Using both English and target-language Wikipedia generally outperforms restricting retrieval to either collection alone.

5 Related Work

XOR-TYDI QA extends multilingual QA and cross-lingual information retrieval by combining native-speaker questions with large, multilingual answer collections and unknown answer languages.

Multilingual QA: XOR-TYDI QA contains large-scale native-speaker questions designed for multilingual QA rather than translated English-centric questions.
Multilingual QA: Unlike QA@CLEF, XOR-TYDI QA includes enough questions for training current state-of-the-art QA systems and targets non-parallel Wikipedia collections.
These design differences better simulate real-world scenarios and introduce challenges that remain understudied.
Cross-lingual Information Retrieval: XOR QA differs from standard CLIR because it uses question-form queries, considers English and query-language documents, and searches much larger collections.

6 Conclusion

The paper presents XOR QA and XOR-TYDI QA as a cross-lingual benchmark for answering non-English information-seeking questions through multilingual document retrieval and reading.

XOR-TYDI QA provides 40k newly annotated open-retrieval questions across seven typologically diverse languages.
Experiments identify XOR-TYDI QA as a challenging benchmark for both QA and multilinguality research.

Legal and Ethical Considerations

The dataset addresses ethical and annotation challenges in multilingual open-retrieval QA, including potential harm from incorrect information and limited human verification of massive document collections.

Risks and safeguards: Wikipedia may contain incorrect information about people, creating a potential risk of harm or legal action.The authors report extensive quality control and answer verification to minimize this risk.
Representation: Existing QA benchmarks often overrepresent English-centric questions and topics, potentially disadvantaging people with diverse languages and backgrounds.The paper identifies questions written mainly by native English speakers as a fundamental benchmark problem.
Annotation scope: Open-retrieval annotation cannot establish that an answer does not exist across millions of documents, so evaluation discards questions without found answers.This setting requires reconsidering conventional notions of human performance.
Annotation scope: Traditional single-article annotation can incorrectly label questions unanswerable because of annotation errors, beyond information scarcity or asymmetry.The paper contrasts this procedure with open-retrieval annotation challenges.

B Additional Details of Dataset Creation

The dataset creation pipeline combines ranked-paragraph annotation, translator qualification, worker screening, and language-specific answer normalization to improve coverage and consistency.

Answer annotation: The interface ranks paragraphs from the top five retrieved articles and presents them first to annotators.Articles ranked sixth through tenth are excluded from paragraph ranking because additional articles introduce noise, although annotators may still find them relevant.
Answer annotation: Annotators can inspect full articles or collapse them, while blue paragraphs indicate high BERT-ranker scores and orange marks the clicked paragraph.These interface choices support focused paragraph selection while retaining access to article context.
Translation: Translators qualify through 20 sample questions, native-speaker quality assessment, and filtering for entity errors or excessive copying from machine translation.The process removes translators whose outputs fail the stated standards.
Quality control: QA workers must have approval rates of at least 96% and pass calibration, expert-judgment, agreement, and ongoing manual-monitoring procedures.Qualified workers are sampled weekly for annotation review.
Translation: Answer translation uses metric units, Wikipedia’s official proper-noun translations when available, transliteration otherwise, and language-specific numeral and measure-word conventions.These instructions aim to keep translated answers consistent with the original dataset.
Dataset composition: The corpus excludes evaluation examples with long-answer annotations but no short-answer annotations.Table 9 defines long answers as paragraph annotations and short answers as short-answer annotations.

C Training details

Training uses replicated transformer translation and neural retrieval-reader pipelines, with language-specific BM25 configurations and annotated positive and negative contexts.

Machine translation: The translation baselines use mostly base-sized autoregressive transformers, with pretrained large models for English↔Russian.Inputs are encoded with Moses tokenization, BPE, or SentencePiece, and models are trained with fairseq.
Neural retrieval: English DPR and Path Retriever models are initialized from Natural Questions Open models and fine-tuned on human-translated questions with gold paragraphs.This transfers pretrained open-retrieval QA parameters to XOR-TYDI QA.
Neural retrieval: Retriever training follows prior work’s hyperparameters and sampling strategies for positive and negative contexts.The paper emphasizes context selection as crucial for competitive neural retrievers.
Neural retrieval: DPR uses annotated long-answer paragraphs as positives, splits them into 100-token units, and treats units containing short answers as positive context.Negative contexts include sampled paragraphs from the top-five preselected paragraphs and in-batch negatives.
Neural retrieval: Path Retriever uses annotated long answers as positives and randomly samples top-50 paragraphs from the top-10 retrieved articles as negatives.This creates retrieval examples from the annotation candidate pool.
Sparse retrieval: BM25 uses Elasticsearch analyzers for six languages, with Kuromoji and Nori plugins for Japanese and Korean, but no Telugu implementation.Telugu lacks a supported default analyzer or official plugin in the described setup.
Machine reading: Reader training samples three negatives for Path Retriever and BM25, while DPR uses 24 negative paragraphs with distant supervision.Readers jointly predict answer spans and paragraph relevance scores.

D.1 Additional Experimental Results

Additional experiments compare translated and multilingual retrieval and reading pipelines across XOR tasks, showing consistent advantages for GMT and mixed results for the authors’ translation baseline.

XOR-RETRIEVE: Given human translations, DPR generally outperforms the other retrieval baselines on XOR-RETRIEVE.Retrieval results are reported with R@2kt and R@5kt metrics.
XOR-RETRIEVE: GMT-translated queries outperform other models in all XOR-TYDI QA languages on the development set.The same performance trend is observed on the test set.
XOR-RETRIEVE: Our MT outperforms the multilingual approach in Arabic, Bengali, Finnish, Japanese, and Korean but performs poorly in Telugu.The paper states that this pattern is consistent with the test-data results.
XOR-ENGLISHSPAN: GMT significantly outperforms Our MT and the multilingual model on XOR-ENGLISHSPAN development data.The authors attribute low Telugu performance for Our MT probably to error propagation.
XOR-FULL: XOR-FULL evaluation reports F1, BLEU, and EM scores across separate tables, including development-set averages.These metrics evaluate the end-to-end setting from multiple perspectives.

D.2 Additional Analysis

Additional analyses show that retrieval difficulty varies substantially across questions and that combining English and target-language Wikipedia usually improves XOR-FULL performance, though not uniformly.

Single-language Wikipedia ablations: 20.1 vs. 14.3 F1 in Korean shows that using both English and target-language Wikipedia can outperform a single-language ablation.In a majority of languages, full models using both resources outperform single-language systems.
Single-language Wikipedia ablations: English-only baselines outperform full models in Japanese and Finnish, where monolingual answer aggregation is affected by poor monolingual-model performance.This is an exception to the majority-language pattern for multilingual document pools.
Retrieval by difficulty: 65.3 for easy versus 59.9 for hard subsets in R@2kt with human translation and DPR indicates a substantial retrieval difficulty gap.The hard subset is defined by whether the annotation retriever selected a gold paragraph.

D.3 Qualitative Analysis on Translation Errors

The qualitative analysis illustrates how mistranslating a key entity can redirect retrieval to an unrelated topic, while standard translation metrics may overlook the mistake.

Entity translation errors: Translating “Almond Eye” as “almond” caused retrieval errors and selected Wikipedia articles about almonds.The example contrasts the correct translation produced by the authors’ MT system with Helsinki’s incorrect translation.
Entity translation errors: A single entity error can change the retrieved evidence topic even when the rest of a question is translated fluently.The Almond Eye example demonstrates this retrieval consequence.
Metric limitations: BLEU and other intrinsic metrics may miss the importance of translation mistakes involving key words such as entities and years.The paper identifies precise translation of key words as a primary challenge in question translation.