AmQA: Amharic Question Answering Dataset

• QA datasets distinguish factoid from non-factoid questions and extractive from abstractive answers.Source: §1 · p. 1
• Amharic was absent from the growing map of multilingual QA datasets, despite existing datasets for many other languages.Source: §1 · p. 1
• The dataset uses context spans as answers, including examples such as ቤተ ጊዮርጊስ for a question about Lalibela’s cross-shaped church.Source: §1 · p. 2
• AmQA contains 2,628 question-answer pairs from 378 Amharic Wikipedia documents, with contexts, questions, and answers in each triplet.Source: §1 · p. 2
• AmQA can evaluate end-to-end, reading-comprehension, retriever-reader, and full-Wikipedia open-domain QA settings.Source: §1 · p. 2

• Existing datasets such as SQuAD use crowd workers to create answer spans from Wikipedia articles, while other languages also have translated versions.Source: §2 · p. 2
• Amharic QA research included factoid and non-factoid systems, but these efforts did not produce a published dataset for evaluation.Source: §2 · p. 2
• The lack of public Amharic QA data and supporting NLP tools motivated AmQA as a testbed for Amharic, cross-lingual, and multilingual QA models.Source: §2 · p. 3

3 The AmQA Dataset

• The dataset was created in three phases: article gathering, question-answer crowdsourcing, and question-answer validation.Source: §3 · p. 3

3.1 Article collection and cleaning

• 378 cleaned Wikipedia articles were randomly selected after size filtering, category removal, preprocessing, and chunking by subtopics.Source: §3.1 · p. 3

3.2 Question-Answer Pair Crowdsourcing

• Native Amharic-speaking crowd workers with at least bachelor’s degrees received training and formulated questions from the cleaned contexts.Source: §3.2 · p. 3

3.3 Question-Answer Pair Validation and Annotation

• Validation removed ambiguous, overly long, and non-consecutive-answer questions while requiring precise answers supported by the context.Source: §3.3 · p. 3
• Validated pairs were annotated with Haystack and exported in SQuAD JSON format, with stray newline characters removed.Source: §3.3 · p. 3

4.1 Data statistics

• 172 words is the average context length, while average questions contain 9.22 words and answers contain 2.66 words.Source: §4.1 · p. 3

4.2 Questions Expected Answer Type

• 300 randomly selected questions were categorized by expected answer type using interrogative terms and answer phrases.Source: §4.2 · pp. 3–4
• More than 18% of questions each seek locations, numbers, or times, while descriptions account for 13% and persons for 14.38%.Source: §4.2 · p. 4
• Other entities comprise 10.7% of questions, whereas list and organization questions are least common at 3.01% and 2.67%.Source: §4.2 · p. 4

5.1 Baseline Model

• The retriever-reader model uses BM25 to retrieve passages and XLM-RLarge to predict answer start and end positions.Source: §5.1 · p. 4

5.2 Evaluation

• The baseline evaluation measures F-score and exact match for reading-comprehension and retriever-reader settings.Source: §5.2 · p. 4
• 71.74 F1 for XLM-RLarge exceeds 64.69 F1 for XLM-RBase in reading comprehension.The reported XLM-RLarge score is comparable to the 70.7 average F1 for XLM-R across seven MLQA languages.Source: §5.2 · p. 4, Table 3 · p. 4
• Model outputs sometimes contain the gold answer with affixes, extra strings, spaces, or punctuation, motivating character normalization and punctuation, quotation-mark, and space removal.Source: §5.2 · p. 5
• The retriever-reader model improves after preprocessing normalizes returned answers.Source: §5.2 · p. 5

6 Summary & Outlook

• AmQA contains document, question, and answer triplets curated from Amharic Wikipedia.Source: §6 · p. 5
• The paper sets baseline values for both reading comprehension and retriever-reader QA settings.Source: §6 · p. 5
• The authors hope AmQA will stimulate monolingual and multilingual QA research, with equivalent translations enabling cross-lingual QA use.Source: §6 · p. 5

Limitations

• AmQA is small because creating it required expensive labor, disadvantaging data-intensive methods.Source: Limitations · p. 5
• The limited number of human annotators may introduce inherent biases or systematic annotation errors.Source: Limitations · p. 5
• Available computing resources limited baseline selection, and stronger out-of-the-box baselines may exist.Source: Limitations · p. 5

• Figure 2 presents the distribution of interrogative terms in the AmQA dataset.Source: Figure 2 · p. 7
• Together, the figures address AmQA question terminology and dataset example structure.Source: Figure 2 · p. 7, Figure 3 · p. 7
• Figure 3 presents an SQuAD-like example from the AmQA dataset.Source: Figure 3 · p. 7