Source-linked AI summary
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, Jennimaria Palomaki
TL;DR
Multilingual models need evaluation data that represents the broad typological variety of the world’s languages, not just English. TyDi QA presents a large-scale multilingual corpus of information-seeking question-answer pairs collected through a model-free and translation-free procedure. Baseline mBERT performance shows a large gap from human performance across languages, while English SQuAD training yields very low F1 on TyDi QA.
Problem
Multilingual models need evaluation data that represents the broad typological variety of the world’s languages, not just English.
Method
TyDi QA presents a large-scale multilingual corpus of information-seeking question-answer pairs collected through a model-free and translation-free procedure.
Results
Baseline mBERT performance shows a large gap from human performance across languages, while English SQuAD training yields very low F1 on TyDi QA.
Takeaways & Limitations
TyDi QA provides a challenging multilingual evaluation intended to support conclusions aligned with information-seeking needs and diverse language phenomena.
Takeaways & Limitations
Scores should not be compared directly across languages because their questions, Wikipedia content, annotator quality, and other variables differ.
Abstract
from arXiv · showhide
Confidently making progress on multilingual modeling requires challenging, trustworthy evaluations. We present TyDi QA---a question answering dataset covering 11 typologically diverse languages with 204K question-answer pairs. The languages of TyDi QA are diverse with regard to their typology---the set of linguistic features each language expresses---such that we expect models performing well on this set to generalize across a large number of the world's languages. We present a quantitative analysis of the data quality and example-level qualitative linguistic analyses of observed language phenomena that would not be found in English-only corpora. To provide a realistic information-seeking task and avoid priming effects, questions are written by people who want to know the answer, but don't know the answer yet, and the data is collected directly in each language without the use of translation.
1 Introduction
TyDi QA addresses the need for reliable multilingual question answering by introducing a large-scale, information-seeking corpus designed around linguistic diversity and realistic user needs. It supports progress through linguistic analysis, data-quality evaluation, baseline modeling, and public benchmarking.
- Motivation: TyDi QA responds to the challenge that many potential QA users do not speak English and that languages differ substantially in typology, resources, writing systems, and spacing conventions.The introduction highlights variation in word order, reduplication, morphosyntax, case, plurality, question marking, relativization, data availability, and standardization.
- Contribution: TyDi QA presents the first public large-scale multilingual corpus of information-seeking question-answer pairs, collected with a model-free, translation-free procedure.The dataset is intended to support question answering research beyond English and align evaluation with real users’ information needs.
- Motivation and goals: The benchmark targets roughly the world’s top 100 languages and encourages models that work across diverse linguistic phenomena and data scenarios.Although it contains 11 languages, the dataset is described as covering a large variety of linguistic phenomena and data scenarios.
- Contribution: The paper provides typological descriptions, glossed examples, an open-source baseline, and a public leaderboard with a hidden test set to track multilingual modeling progress.These resources are intended to enable intrinsic and extrinsic analyses and more reliable conclusions than artificially easy datasets would support.
2 Task definition
TyDi QA asks models to answer questions using a Wikipedia article’s content through passage selection and minimal answer-span prediction. The formulation captures uncertainty about where an answer appears and whether the question is answerable.
- TyDi QA presents a question with the content of a Wikipedia article and requires two predictions.
- Passage selection returns the index of the answering passage or NULL when no passage answers the question.
- Minimal answer span prediction returns byte indices for the minimal complete answer, YES or NO for answerable yes/no questions, or NULL otherwise.
- This formulation reflects that users may not know where an answer appears or whether their question is answerable.Figure 1 illustrates an example question-answer pair.
3 Data collection procedure
TYDI QA elicits genuinely information-seeking questions from annotators who have not seen the answers, retrieves a Wikipedia article for each question, and labels answer passages, minimal spans, or unanswerability. This design preserves natural question–answer mismatches and supports reasoning over full articles where many questions have no answer.
- Question elicitation: Annotators write questions they genuinely want answered without seeing the answer, using only short Wikipedia prompts for inspiration and freedom to ask unrelated topics.This avoids priming and encourages questions about topics that interest them, including topics not covered by the prompt article.
- Article retrieval: Each question is paired with the top-ranked result from a Google search restricted to the relevant language’s Wikipedia domain, using an atomic language-specific snapshot.Tables, long lists, and infoboxes were removed from articles to focus the challenge on multilingual text.
- Answer labeling: Annotators select the best answer passage or indicate unanswerability, then mark the shortest satisfactory character span, ideally 1–3 words, or YES/NO for boolean questions.Minimal-answer guidelines required additional training, with annotators qualifying at 90%+ and repeated training used to limit drift.
- Article retrieval: 46%–82% of questions cannot be answered by their retrieved article, reflecting an information-seeking setting with lengthy 5K–30KB documents rather than answer-containing paragraphs.Unlike SQuAD-style datasets, TYDI QA provides full documents and does not guarantee that an answer is present.
- Quality validation: Native-speaker checks verified fluent questions and that prompts did not already answer them, while collection discouraged opinion and conversational questions.The resulting unseen-answer questions can be underspecified and lexically or morphosyntactically mismatched with their answers.
4 Related Work
Prior multilingual QA datasets typically covered only one or two languages or used translation, while other multilingual QA benchmarks targeted different task formulations. QA research also spans broad formats, including generation, corpus retrieval, reading comprehension, and expert-authored information verification.
- Multilingual QA datasets: Multilingual QA benchmarks also addressed different tasks, such as retrieving noun phrases from question-like prompts or locating document spans describing properties.XQA and XCMRC use syntactically question-like prompts for noun-phrase retrieval, while Kenter et al. locate spans providing information about properties such as location.
- Multilingual QA datasets: Earlier non-English multilingual QA datasets generally covered one or two languages, including Chinese, Arabic, and translated French, Japanese, Korean, and Italian resources.These include DuReader and DRCD, translated SQuAD evaluation sets, Korean SQuAD translations, semiautomatic Italian translation, and ARCD.
- QA task formulations: Question answering encompasses diverse formats, from word-by-word answer generation to finding answers across entire corpora.Examples include generative QA, TREC, and DrQA.
- QA task formulations: Another QA paradigm verifies experts’ knowledge through carefully authored trivia or reading-comprehension questions with intended correct answers.This information-verifying framing includes TriviaQA, Quizbowl/Jeopardy!, NewsQA, CoQA, and RACE, with later work emphasizing difficult examples or particular answer types.
5 Typological diversity
TyDi QA prioritizes typological and data diversity, selecting languages from many families whose linguistic and resource characteristics challenge translation-dependent modeling. The dataset spans phenomena including rich morphology, flexible or nonconfigurational syntax, varied writing systems, and inconsistent spacing, making seemingly simple tasks such as string matching difficult.
- Language selection: TyDi QA selects languages for typological diversity across linguistic families and for data characteristics such as limited monolingual or parallel data, reducing reliance on high-quality machine translation.The selection aims to cover different linguistic devices and languages with limited economic incentives for large-scale parallel data production.
- Language examples: The dataset also represents varied word formation and grammatical encoding, including Arabic root-pattern morphology and clitics, Indonesian reduplication and affixation, and Kiswahili prefixal agreement and pro-drop.These languages illustrate diversity in how inflectional, derivational, number, person, and semantic relations are expressed.
- Language examples: The 11 languages cover diverse morphological and syntactic systems, including Finnish’s extensive inflection, Japanese particles and final verbs, Korean case and honorific systems, and Thai’s analytic structure.Finnish nouns typically have approximately 140 forms and verbs about 260 forms; Korean nominal particles express up to 15 cases.
- Language examples: Figure 2 shows Finnish compounding, genitive and plural inflection, and consonant gradation interacting within a question-answer example.The example contrasts inflected forms inside a compound with a standalone nominative answer form.
- Modeling challenges: Examples from Finnish, Russian, and Arabic show that morphophonological alternations, compounding, spelling variation, diacritization, whitespace, and gender variation make English-trivial string matching complex.Arabic examples include name spelling variation, selective diacritization, inconsistent whitespace, and gender variation; Figure 3 presents non-trivial Russian inflectional changes.
6 A quantitative analysis
TyDi QA contains 204K examples supported by 277K annotations, with balanced English question-word distributions and evidence that elicitation and annotation procedures produce reliable data. Error analysis identifies false-positive NULL judgments and ambiguity-driven minimal-answer mistakes as important residual issues.
- Dataset size: 204K examples comprise 166K one-way training annotations and 37K three-way development/test examples, totaling 277K annotations.These totals are reported in Table 4.
- Question types: TyDi QA’s English development data has a more balanced distribution of question words than SQuAD.The comparison is presented in Table 2; cross-language question-word comparisons are difficult because question words function differently by context.
- Protocol adherence: In a 100-pair sample, every question overlapped its prompt by 1–2 words, none was answered by the prompt, and discarded prompts reduced question–answer lexical overlap.The overlap typically involved an entity or word of interest, while prompts were discarded from the final dataset.
- Annotation quality: Expert judgments of 200 Finnish and Russian development-set pairs found high annotation accuracies, supporting TyDi QA as a signal for evaluating multilingual question-answering models.Experts assessed article answerability, three passage-answer annotations, and minimal-answer correctness.
- Error analysis: NULL-related errors were entirely false positives across article lengths, whereas minimal-answer errors often arose when multiple dates appeared plausible.Three answer annotations would largely mitigate NULL errors, but matching difficulty or subtlety remains a likely alternative cause beyond article length.
7 Evaluation
TYDI QA evaluates passage selection and minimal-answer prediction primarily with language-macro-averaged F1, using special handling for NULL answers and partial span overlap. mBERT remains substantially below the pessimistic human estimate, while scores should not be compared directly across languages.
- 7.1 Evaluation: TYDI QA’s primary measure is F1, computed from precision and recall within each language and then macro-averaged over the non-English languages.English measurements are treated mainly as a debugging aid rather than the task’s goal.
- 7.1 Evaluation: NULL consensus is thresholded to prevent always predicting NULL from exploiting the dataset’s imbalance in answerable questions.Passage-selection answers for NULL-consensus questions receive credit when matching any annotator-selected passage; precision and recall denominators depend on consensus and prediction status.
- 7.1 Evaluation: For minimal answers, systems predict NULL, YES, NO, or a contiguous answer span, receiving partial credit for overlapping spans and full credit for an annotator-supported YES/NO answer.A non-NULL prediction receives credit only when the consensus is non-NULL.
- 7.2 Human performance: Human performance is estimated by holding out one annotation and evaluating it against the other two with bootstrap resampling, making the estimate pessimistic when annotators disagree.The authors note that this disagreement-based estimate may understate human performance because multiple answers can be correct.
- 7.3 Baselines and results: mBERT shows a large performance gap below the lesser estimate of human performance across all languages, with results reported alongside first-passage and human baselines in Table 5.The mBERT model is jointly trained on all languages, while the first-passage system is an untrained baseline.
- 7.3 Baselines and results: Scores should not be compared across languages because question sets, Wikipedia coverage, annotation quality, and other conditions differ by language.The authors argue that engaging directly with these differences exposes important aspects of the multilingual problem space.
8 Gold passage: A simplified task
The simplified GOLDP task provides gold passages for compatibility with SQuAD-style evaluation, but TYDI QA retains substantially lower lexical overlap and very low F1, including in English-only zero-shot transfer.
- Task design: GOLDP supplies only the gold answer passage, discards unanswerable questions, uses SQuAD 1.1 metrics, and removes Thai and Japanese because limited whitespace breaks existing tools.The task is intended to enable direct comparison with prior work and compatibility with SQuAD, XQuAD, and MLQA code.
- Lexical overlap: TYDI QA shows substantially lower question-answer lexical overlap than MLQA and XQuAD across all languages.Overlap is measured using tokens shared with a 200-character answer window, excluding the 100 most frequent tokens.
- Results: Very low F1 scores persist when mBERT is jointly fine-tuned on multilingual TYDIQA-GOLDP data, despite removing long articles and unanswerable questions.The simplified setting still leaves substantial headroom for improvement.
- Results: English-only SQuAD fine-tuning also yields very low TYDIQA-GOLDP zero-shot F1, somewhat below XQuAD results reported by Artetxe et al. (2019).English performance is significantly lower, indicating that SQuAD question-answer style transfers poorly to this task.
9 Recommendations and future work
The authors propose research on multilingual QA phenomena, transfer and translation, zero-shot evaluation, and deeper linguistic analysis. They also call for compatible datasets while noting that low-resource languages may lack both written answers and parallel data for trustworthy translation.
- Research directions: Future work should study morphology–question-answer matching, transfer learning, machine translation for augmentation and runtime QA, and zero-shot QA across held-out languages.These directions address varying data availability and linguistic challenges.
- Data understanding: Further linguistic analysis is needed to identify modeling techniques suited to the full variety of phenomena observed across the world’s languages.The authors argue that deeper understanding of the data itself is key.
- Dataset expansion: The authors invite compatible datasets for additional languages because no single effort can cover the world’s languages, citing the Universal Dependencies treebank’s breadth as evidence of community potential.They note that the Universal Dependencies treebank had over 70 languages.
- Low-resource languages: Many languages lack written content needed to answer questions, making cross-language retrieval and translation necessary despite limited parallel data for trustworthy translation systems.This creates a particular challenge for low-resource languages, where parallel data is also likely to remain scarce.
10 Conclusion
The conclusion argues that question answering on a typologically diverse dataset can provide a challenging, trustworthy evaluation for multilingual models whose progress may generalize across the world’s languages. It also notes that the authors do not release automatic translations because machine translation may be a fruitful research direction.
- 10 Conclusion: Question answering on typologically diverse languages is presented as a challenging, trustworthy evaluation whose progress is more likely to generalize across the world’s linguistic phenomena.The authors argue that avoiding translation-based data collection and multilingual modeling mitigates sampling risks, although the supplied passage is truncated before specifying them.
- 10 Conclusion: The authors do not release automatic translations because machine translation may be a fruitful research direction for TyDi QA.They state that translation has previously stymied innovation around multilingual datasets.
- 10 Conclusion: The authors will share their annotation protocol on request.