Source-linked AI summary
Neural Arabic Question Answering
Hussein Mozannar, Karl El Hajal, Elie Maamary, Hazem Hajj
TL;DR
The paper addresses open-domain factual Arabic QA, where answers must be spans from Arabic Wikipedia and progress is constrained by data, retrieval, and reading-comprehension challenges. It introduces crowdsourced and translated datasets and builds SOQAL with hierarchical TF-IDF retrieval and BERT reading comprehension. The BERT reader reaches 61.3 F1 on ARCD, while SOQAL reaches 27.6 F1 in open-domain ARCD QA.
Problem
Arabic open-domain QA lacks sufficient annotated datasets and requires efficient retrieval and accurate answer extraction over Arabic Wikipedia.
Method
The paper creates ARCD and Arabic-SQuAD and combines hierarchical TF-IDF retrieval, BERT reading comprehension, and answer ranking in SOQAL.
Results
61.3 F1 was achieved on ARCD by the BERT reader, while SOQAL achieved 27.6 F1 on open-domain ARCD.
Takeaways & Limitations
The paper provides datasets and an evaluated end-to-end pipeline for Arabic open-domain QA using Wikipedia.
Takeaways & Limitations
SOQAL’s deployment requires paragraph selection because the optional 4-gram retrieval step was not used in evaluation.
Abstract
from arXiv · showhide
This paper tackles the problem of open domain factual Arabic question answering (QA) using Wikipedia as our knowledge source. This constrains the answer of any question to be a span of text in Wikipedia. Open domain QA for Arabic entails three challenges: annotated QA datasets in Arabic, large scale efficient information retrieval and machine reading comprehension. To deal with the lack of Arabic QA datasets we present the Arabic Reading Comprehension Dataset (ARCD) composed of 1,395 questions posed by crowdworkers on Wikipedia articles, and a machine translation of the Stanford Question Answering Dataset (Arabic-SQuAD). Our system for open domain question answering in Arabic (SOQAL) is based on two components: (1) a document retriever using a hierarchical TF-IDF approach and (2) a neural reading comprehension model using the pre-trained bi-directional transformer BERT. Our experiments on ARCD indicate the effectiveness of our approach with our BERT-based reader achieving a 61.3 F1 score, and our open domain system SOQAL achieving a 27.6 F1 score.
1 Introduction
The paper addresses open-domain factual Arabic QA using Arabic Wikipedia, focusing on the lack of datasets and the challenges of retrieval and answer extraction. It introduces Arabic QA datasets and an end-to-end system combining hierarchical TF-IDF retrieval with neural reading comprehension.
- Open-domain Arabic QA requires efficient large-scale information retrieval, accurate answer extraction, and sizable training and testing data.
- Datasets for Arabic QA: ARCD contains 1,395 crowdsourced questions with accompanying answer segments from Arabic Wikipedia.
- Datasets for Arabic QA: Arabic-SQuAD contains 48k machine-translated paragraph-question-answer tuples from SQuAD.
- Open-domain Arabic QA system: SOQAL uses a hierarchical TF-IDF retriever and a BERT-based reader for open-domain Arabic questions.
2 Related Work
Prior Arabic QA research used limited datasets and classical or specialized methods, while broader QA progress has been driven by large datasets and neural reading-comprehension models. The paper situates its approach within this shift toward scalable data and modern architectures.
- Arabic QA approaches: Earlier Arabic QA approaches commonly relied on classical information retrieval and targeted specific question types.
- Arabic QA datasets: Arabic QA datasets were generally small, often lacked accompanying text segments, and larger datasets were synthetically generated.
- QA datasets: Large datasets and advances in deep learning and language representation models have driven progress in question answering.
- Reading comprehension and QA: Machine reading comprehension has advanced through recurrent models, attention mechanisms, convolutions, and self-attention.
- Reading comprehension and QA: Prior open-domain systems combined Wikipedia retrieval with reading models, including a TF-IDF retriever and RNN reader achieving 29.8% exact-match accuracy on open-SQuAD.
3 Dataset Collection
The paper constructs ARCD through crowdsourcing proficient Arabic speakers to create extractive questions from Arabic Wikipedia, and creates Arabic-SQuAD by machine-translating SQuAD for training.
- ARCD collection: Crowdworkers wrote Arabic question-answer pairs from Wikipedia paragraphs, with answers required to be exact text spans.
- Crowdsourcing: Workers were screened for Arabic proficiency and task quality, producing 1,395 questions from 465 paragraphs.
- Article curation: The collection used 155 randomly sampled highly viewed Arabic Wikipedia articles covering diverse topics.
- Arabic-SQuAD: Arabic-SQuAD was created by translating SQuAD version 1.1 with Google Translate’s neural machine translation API.
- Arabic-SQuAD: The translation produced 48,344 questions on 10,364 paragraphs from the first 231 SQuAD training articles.
4 Our System: SOQAL
SOQAL answers Arabic open-domain questions with a pipeline that retrieves relevant Wikipedia content, extracts candidate spans with BERT, and ranks the answers. Its hierarchical TF-IDF retriever uses staged n-gram search to balance retrieval accuracy, computational cost, and reader workload.
- System Architecture: SOQAL combines document retrieval, machine reading comprehension, and answer ranking to extract an answer span from Arabic Wikipedia.The system receives a question and Arabic Wikipedia, then returns a small text span as the answer.
- Hierarchical TF-IDF Document Retriever: TF-IDF retrieval scores questions and documents by cosine similarity after tokenization, stemming, stopword removal, and n-gram feature construction.The retriever returns the top k documents; larger k improves the chance of including relevant documents but makes extraction slower and more error-prone.
- Hierarchical TF-IDF Document Retriever: Hierarchical TF-IDF first retrieves about 1,000 documents with bigrams, then reranks them with 4-grams to return about 15 candidates efficiently.The second stage preserves much of the first stage’s accuracy while keeping the final retrieval set small.
- BERT Document Reader: BERT encodes question–paragraph pairs with a bidirectional Transformer and predicts answer start and end positions from token representations.Inference selects a span of at most 15 tokens by maximizing the product of start and end probabilities.
- Answer Ranking: The pipeline feeds retrieved paragraphs to the reader, normalizes document and candidate scores separately, and combines them with a tuned linear weighting.The weighting parameter β is selected by line search on a development set.
- Answer Ranking: For deployment on large articles, an additional 4-gram retriever can reduce the paragraph set and reader load, although this step was not used in evaluation.The paper identifies this additional retrieval step as crucial for usage because articles can be very large.
5 Dataset Analysis
The dataset analysis examines answer diversity, reasoning requirements, and machine-translation issues affecting Arabic-SQuAD. It identifies substantial translation inconsistencies between answers and paragraphs.
- Answer diversity: ARCD answer categories include numerical answers, verbs, adjectives, noun phrases, and named entities.The authors manually categorized 100 sampled questions and used MADAMIRA to help identify named entities.
- Question reasoning: ARCD questions require word matching, world knowledge, syntactic variation, or reasoning across multiple sentences.These categories were adapted from prior reading-comprehension analyses.
- Arabic-SQuAD: Arabic-SQuAD was created through machine translation, with paragraphs and questions generally maintaining their original meaning.The authors separately analyzed issues arising from translating SQuAD.
- Arabic-SQuAD: 25,490 Arabic-SQuAD answers, almost 47.3% of the total, were not found in their corresponding translated paragraphs.Reported causes included context-sensitive named-entity transliteration and minor typographic or tense differences.
6 System Experiments
The experiments evaluate retrieval, reading comprehension, transfer learning, and the end-to-end SOQAL system on Arabic QA datasets. Results show strong neural reading performance but a larger bottleneck in selecting the correct paragraph.
- Retriever: 75.6% is achieved by Google Search with k = 10, while bigram TF-IDF with k = 350 reaches 73.5% on ARCD retrieval.The hierarchical retriever reaches 65.3%, improving on the single-bigram configuration by 17.6%.
- Retriever: ARCD questions may be ambiguous without their source paragraphs, making the Google Search baseline difficult to beat.The questions were written with specific paragraphs in mind.
- Reader: BERT reaches 44.4 F1 and QANet 48.6 F1 on the Arabic-SQuAD test set, while BERT reaches 90.08 SM accuracy on ARCD without ARCD training.The authors attribute the relatively low Arabic-SQuAD results partly to faulty translated answers.
- Transfer Learning: 61.3 F1 is obtained when BERT is trained on combined Arabic-SQuAD and ARCD data, an 8.3% improvement over the separate training regimes.The authors state that ARCD training helped adapt to its differing answer distribution.
- End-to-end system: 27.6 F1 and 29.8 SM are achieved by SOQAL on ARCD-Test.The close scores indicate that, when the correct paragraph is retrieved, the remaining issue is ranking that paragraph highly enough.
7 Conclusion
The paper introduces Arabic QA datasets and an open-domain system combining hierarchical TF-IDF retrieval with BERT reading comprehension. It reports strong ARCD reading results, lower end-to-end open-domain performance, and identifies paragraph selection as future work.
- Contributions: The paper introduces ARCD, Arabic-SQuAD, hierarchical TF-IDF retrieval, and BERT-based document reading for open-domain Arabic QA.The datasets and system address Arabic QA data, retrieval, and reading-comprehension needs.
- Results: 61.3 F1 and 90.0% sentence match are achieved on ARCD, while SOQAL achieves 27.6 F1 on open-domain ARCD.The reported results distinguish document reading performance from end-to-end open-domain performance.
- Future work: Future work will expand ARCD and improve the end-to-end system by focusing on paragraph selection.The conclusion identifies paragraph selection as the targeted improvement area.