PaperlayerView exact revision on arXiv

Source-linked AI summary

Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answering

Priyanka Sen, Alham Fikri Aji, Amir Saffari

arXiv:2210.01613v1cs.CL

TL;DR

Complex QA lacks datasets that are simultaneously large, naturally elicited, and multilingual. Mintaka supplies a crowd-elicited dataset linked to Wikidata and translated into eight languages; its best baseline scores 38% hits@1, indicating room for improvement.

  • Problem

    Complex QA datasets have gaps in scale and naturalness, and multilingual complex datasets are rare.

    Source: §1 · p. 1

  • Method

    The authors collect 20,000 English QA pairs from crowd workers, annotate them with Wikidata entities, and translate them into eight languages.

    Source: §1 · p. 1

  • Results

    38% hits@1 is the best score reported among the existing Mintaka baselines, showing room for improvement.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Mintaka addresses an identified QA dataset gap with a large-scale, complex, naturally elicited, and multilingual resource.

    Source: §1 · p. 1, §7 · p. 9

    None of the baselines explicitly handle all Mintaka complexity types; language models especially struggle with numerical operations such as counting.

    Source: §6.2 · p. 9

Abstract

from arXiv · show

We introduce Mintaka, a complex, natural, and multilingual dataset designed for experimenting with end-to-end question-answering models. Mintaka is composed of 20,000 question-answer pairs collected in English, annotated with Wikidata entities, and translated into Arabic, French, German, Hindi, Italian, Japanese, Portuguese, and Spanish for a total of 180,000 samples. Mintaka includes 8 types of complex questions, including superlative, intersection, and multi-hop questions, which were naturally elicited from crowd workers. We run baselines over Mintaka, the best of which achieves 38% hits@1 in English and 31% hits@1 multilingually, showing that existing models have room for improvement. We release Mintaka at https://github.com/amazon-research/mintaka.

1 Introduction

Mintaka targets the lack of natural, multilingual datasets for complex question answering, where existing resources trade off scale, complexity, or naturalness. The authors introduce a crowd-elicited dataset linked to Wikidata and report that baseline models still have room to improve.

1 Introduction

  • Complex questions require operations beyond a single fact lookup, such as multi-hop reasoning, comparison, or set intersection.The introduction illustrates comparison with a question that requires looking up two movie budgets and selecting the higher one.Source: §1 · p. 1
  • Existing datasets often offer scale without complexity or complexity without scale, while automatically generated questions may sound less natural.The authors say this may mismatch QA models' training data with real-world use cases.Source: §1 · p. 1
  • Mintaka contains 20,000 English crowd-worker question-answer pairs linked to Wikidata and translated into eight languages, for 180,000 samples.The translations were produced by professional translators.Source: §1 · p. 1
  • 38% hits@1 is the best score among the existing Mintaka baselines, showing room for improvement.The passage reports the 38% result without specifying a language.Source: §1 · pp. 1–2
  • The paper releases Mintaka with train, development, and test sets of 14,000, 2,000, and 4,000 samples.The release is publicly available on the Mintaka GitHub repository.Source: §1 · p. 2

2 Related Works

Prior QA datasets vary in complexity, naturalness, scale, and language coverage. Mintaka is presented as a large-scale, naturally elicited, complex, multilingual dataset that brings these characteristics together.

2 Related Works

  • Reading-comprehension datasets such as SQuAD and Natural Questions are often large, but typically answer questions from a single text passage rather than targeting complexity.XQuAD, MLQA, and TyDi QA are multilingual examples in this group.Source: §2 · p. 2
  • HotpotQA and MuSiQue include multi-hop reading-comprehension questions, while DROP covers operations such as addition, counting, and sorting.These datasets do not naturally elicit questions; the passage describes questions tied to selected passages or composed from existing questions.Source: §2 · p. 2
  • WebQuestionsSP and ComplexQuestions use real user questions, but contain 5K and 2K question-answer pairs, respectively.ComplexQuestions includes multi-hop questions, temporal constraints, and aggregations.Source: §2 · p. 2
  • ComplexWebQuestions expands WebQuestionsSP to 35K pairs and adds complex operations, but uses generated questions paraphrased by crowd workers.The operations include multi-hop, comparative, and superlative questions.Source: §2 · p. 2
  • LC-QuAD 2.0 contains 30K questions based on Wikidata and DBpedia, while GrailQA contains 64K questions based on FreeBase.Both are described as larger-scale complex KGQA datasets.Source: §2 · p. 2
  • Large complex datasets such as KQA Pro and GrailQA use automatically generated questions followed by human paraphrasing, which can reduce naturalness.The passage notes examples of unnatural question phrasing from these datasets.Source: §2 · p. 3
  • QALD 2018 combines multilingual coverage and complex operations across 11 languages, but contains only 400 questions.Its operations include counts and comparatives.Source: §2 · p. 3
  • Mintaka's questions are naturally elicited without restrictions on the facts or articles they can address, and translated into eight languages.The authors describe it as one of the first large-scale complex and multilingual QA datasets.Source: §2 · p. 3

3 Mintaka

Mintaka is a 20,000-question complex QA dataset collected in English from crowd workers, annotated with Wikidata entities, and translated into eight languages. It spans eight topics and includes seven specified complexity types plus generic questions.

3 Mintaka

  • Mintaka contains 20,000 English questions translated into eight languages, yielding 180,000 questions annotated with Wikidata entities in both questions and answers.Questions were collected from crowd workers.Source: §3 · p. 3
  • The dataset covers movies, music, sports, books, geography, politics, video games, and history, with 2,500 questions collected per topic.The topics were chosen as broadly appealing and suitable for complex questions.Source: §3 · p. 3
  • Its question types include counting, comparison, superlatives, ordinal positions, and multi-hop reasoning.The examples include counting astronauts elected to Congress, comparing mountain heights, and identifying a quarterback through a Super Bowl-winning team.Source: §3 · p. 3
  • The remaining types are intersection, difference, yes/no, and generic questions, with generic prompts given only a topic and no complexity constraint.For each topic, the collection includes 250 questions per complexity type and 500 generic questions.Source: §3 · p. 3
  • Professional translators translated the English questions, creating a parallel dataset, but collecting them from U.S. workers may introduce U.S. entity bias.The authors state that parallelism allows models to be compared across languages.Source: §3 · p. 3

4 Dataset Collection

Mintaka was built through crowd-worker question elicitation, Wikidata entity-linking tasks, and professional translation. Workers created unrestricted questions, while annotation procedures and coverage differed for answer and question entities.

4 Dataset Collection

  • MTurk workers in the United States completed three dataset tasks and were required to have a 98% approval rating and at least 5,000 approved tasks.Source: §4 · pp. 3–4

4.1 Question Elicitation

  • For each topic-complexity pair, workers wrote five free-text questions and answers without restrictions on sources or facts.Topics were left broad, allowing questions on subjects such as Ancient Egypt or World War II.Source: §4.1 · p. 4
  • Count and superlative answers included numerical values, but count answers with more than five entities could provide only a sample rather than a complete list.For multi-answer count questions, workers were asked to list at least five entities.Source: §4.1 · p. 4

4.2 Answer Entity Linking

  • Answer Entity Linking asked workers to identify answer entities and link them to Wikidata; number and yes/no answers were excluded from this task.Each answer was annotated by two workers, with disagreements sent to a third annotator.Source: §4.2 · p. 4

4.2 Answer Entity Linking; 4.3 Question Entity Linking

  • Answer-entity agreement was 82% after two annotators and 97% after three; question-entity agreement was 68%/78% for new entities and 80%/98% for seen entities.Remaining answer-entity disagreements were verified by the authors; the same was done for question entities.Source: §4.2 · p. 4, §4.3 · p. 5

4.3 Question Entity Linking

  • Question linking used worker-verified or modified entity spans because question texts can contain multiple plausible entities; English questions retain spans, while translations retain only entity IDs.For new question entities, annotation proceeded to a third annotator when the first two disagreed; seen entities could be string-matched to prior answer annotations.Source: §4.3 · pp. 4–5

4.4 Translations

  • Professional translators translated the 20,000 questions into Arabic, French, German, Hindi, Italian, Japanese, Portuguese, and Spanish after MTurk translation trials had poor results and limited worker availability in some languages.Source: §4.4 · p. 5

5 Dataset Analysis

Mintaka received higher naturalness ranks than the comparison datasets. Its linked question-answer pairs often have Wikidata paths, while the MTurk worker demographics differed from U.S. Census estimates.

5.1 Dataset Statistics

  • Mintaka contains 13,232 unique Wikidata entities, and answers are predominantly entities (72%), followed by booleans (14%) and numbers (7%).Source: §5.1 · p. 6
  • For Wikidata-linked questions, 62% had a question-to-answer entity path within 1 hop and 97% within 2 hops; these are upper bounds because query paths are unknown.The dataset contains question-answer pairs but does not identify the correct query path.Source: §5.1 · p. 6

5.2 Naturalness Evaluation

  • Mintaka ranked higher in average naturalness than all comparison datasets, with differences significant at p < 0.001.Workers ranked questions from 1 (least natural) to 5 (most natural).Source: Figure 1 · p. 6, §5.2 · p. 6

5.3 Demographics of MTurk Workers

  • The passage reports a worker split of 58% versus 42%; 72% of workers were between ages 25 and 44, while only 2% were between ages 18 and 24.Source: §5.3 · p. 7
  • Workers identifying as Asian were more represented than in U.S. Census estimates of the general population: 10% of workers identified as Asian, compared with 6% in the Census.Source: §5.3 · p. 7
  • Workers were more educated than the U.S. population (61% held a bachelor's degree or higher vs. 32%), and 91% were employed and 85% lived in urban or suburban areas.More workers lived in the South (35%).Source: §5.3 · p. 7

6 Baselines

The English baseline leader reaches 38% hits@1, while multilingual results are generally lower. Across the baselines, complex questions remain difficult, particularly when solving them requires operations beyond retrieving or following stated facts.

6.1 Models

  • The eight end-to-end baselines comprise three language models, three KG-based models, and two retriever-readers; three baselines are also evaluated across all languages.Language and retriever-reader models use crowd-worker answers as labels, while the KG models use an October 18, 2021 Wikidata snapshot.Source: §6.1 · p. 7

6.2 Analysis

  • 38% hits@1: fine-tuned T5 for Closed Book QA is the best-performing English baseline on Mintaka.The English results are reported in Table 5.Source: Table 5 · p. 7, §6.2 · p. 8
  • 31% hits@1: trained DPR is second-highest and handles complex questions when the retrieved passage already contains the reasoning, but struggles when it does not.For the chronological Marvel question, DPR fails to find a passage explicitly stating the order.Source: §6.2 · pp. 8–9
  • 20% hits@1: Rigel is the strongest KGQA baseline, but its entity-answer and relation-following design cannot sort or filter; its paths can aid interpretation and debugging.The paper notes that KGQA methods can be updated through the external knowledge graph, unlike static knowledge stored in large language-model parameters.Source: §6.2 · p. 9
  • Almost all multilingual model results are slightly lower than their English results, and the authors report that work remains for parity across languages.The English-translation T5 for CBQA outperforms MT5, but still scores below the original English questions; automatic translation degrades performance.Source: §6.2 · p. 9
  • No baseline explicitly handles every complexity type; language models especially struggle with numerical operations, while KGQA relation following misses answers requiring more complex operations.The authors suggest adding operations and selecting the appropriate operation per question, potentially combining language models with interpretable KG facts and operations.Source: §6.2 · p. 9

7 Conclusions

Mintaka is an end-to-end question-answering dataset linked to Wikidata, designed to combine large scale, complex questions, natural elicitation, and multilingual coverage. Its baselines leave room for improvement, especially across languages, and the authors hope the release encourages further work on complex multilingual QA.

7 Conclusions

  • Mintaka is an end-to-end question-answering dataset linked to Wikidata.Source: §7 · p. 9
  • The dataset addresses a QA-dataset gap with large-scale, complex, naturally elicited, and multilingual data.Source: §7 · p. 9
  • The baselines show room for improvement in handling complex questions, especially across languages, and the authors hope Mintaka encourages continued research.Source: §7 · p. 9

A MTurk Tasks

MTurk workers elicited questions and annotated question and answer entities using Wikidata search. The tasks included question elicitation, answer-entity annotation, and question-entity annotation.

A MTurk Tasks

  • For question elicitation, MTurk workers saw examples and wrote five questions for a topic and complexity type.The illustrated task concerned movies and comparative questions.Source: A MTurk Tasks · p. 11
  • For answer-entity annotation, workers identified answer entities and used Wikidata search to find and enter the correct entity URL.The question provided context for identifying which awards the answer referred to.Source: A MTurk Tasks · p. 11
  • For question-entity annotation, workers focused on one entity at a time, verified or corrected its text, then searched Wikidata for its URL.The example highlights “Oscars” in the question about Argo, even though “Argo” is also a valid entity.Source: A MTurk Tasks · p. 11

B Model Training Details

The training setup uses the full Mintaka splits for most trained models, with separate handling for zero-shot and knowledge-graph models.

B Model Training Details

  • T5, fine-tuned T5 for CBQA, KVMemNet, and trained DPR use all 14,000 train, 2,000 dev, and 4,000 test examples.MT5 is fine-tuned on all languages simultaneously; individual-language fine-tuning gave similar scores at higher computational cost.Source: B Model Training Details · p. 12
  • Zero-shot T5 CBQA and DPR are evaluated directly on the test set without train or dev examples.Source: B Model Training Details · p. 12
  • EmbedKGQA and Rigel use only examples with linked entities in both question and answer, excluding number, boolean, date, and string answers and unlinked entities.EmbedKGQA also loses examples whose Wikidata entities lack entity-to-entity facts.Source: B Model Training Details · p. 12

C Mintaka Examples

Table 7 provides additional Mintaka question-answer examples.

C Mintaka Examples

D Model Training Details

The results show that generic and complex questions remain challenging, and scores near 50% on comparative or yes/no questions may reflect guessing rather than reasoning. Evaluation rules and tables describe answer matching and results by complexity type.

D Model Training Details

  • Count answers may be returned as the counted entities; entity-answer order is ignored, while text answers require exact string matching.Source: D Model Training Details · p. 12
  • Around 50% on comparative or yes/no questions does not necessarily show reasoning, since random guessing can score 50%.Both generic and complex questions remain challenging for the models.Source: D Model Training Details · p. 12
  • Table 8 distinguishes Hits@1 Subset on the available test set from Hits@1 Adjusted across the full 4,000-question test set.Source: Table 8 · p. 15
  • Table 9 breaks down trained-baseline performance across generic, multihop, intersection, difference, comparative, superlative, and ordinal questions.Source: Table 9 · p. 16

Related papers

Loading 2210.01613v1…