PaperlayerView exact revision on arXiv

Source-linked AI summary

From RAG to Agentic RAG for Faithful Islamic Question Answering

Gagan Bhatia, Hamdy Mubarak, Mustafa Jarrar, George Mikros, Fadi Zaraket, Mahmoud Alhirthani, Mutaz Al-Khatib, Logan Cochrane, Kareem Darwish, Rashid Yahiaoui, Firoj Alam

arXiv:2601.07528v2cs.CLcs.AI

TL;DR

Islamic QA evaluation must capture hallucination and abstention because conventional MCQ/MRC formats can miss unsupported free-form answers. The paper introduces a bilingual generative benchmark and grounded training and retrieval resources, then shows that retrieval—especially agentic RAG—improves correctness and bilingual robustness, including for smaller models.

  • Problem

    Standard MCQ/MRC evaluations do not directly measure free-form hallucination or abstention in Islamic question answering, where unsupported responses raise reliability concerns.

  • Method

    The paper builds ISLAMICFAITHQA alongside SFT, preference-alignment, and verse-level Quran retrieval resources, and uses structured tool calls for iterative agentic grounding.

  • Results

    Retrieval improves correctness across models, while agentic RAG produces the largest gains beyond standard RAG; Qwen3-4B-2507 reaches 48.90 accuracy versus 21.85 base and 38.85 with RAG.

  • Takeaways & Limitations

    Tool-mediated grounding supports state-of-the-art performance and stronger Arabic–English robustness, including with smaller backbones.

  • Takeaways & Limitations

    The benchmark under-represents multiple valid answers across interpretive traditions, while evaluation depends on LLM-judge correctness and limited human calibration; grounding is primarily Quran-centric.

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly used for Islamic question answering, where ungrounded responses may carry serious religious consequences. Yet standard MCQ/MRC-style evaluations (MCQ: Multiple choice questions, MRC: Machine Reading Comprehension) do not capture key real-world failure modes, notably free-form hallucinations and the ability to abstain when evidence is insufficient. To address this gap, we introduce IslamicFaithQA, a 3,810-item bilingual (Arabic/English) generative benchmark with atomic single-gold answers, which enables direct measurement of hallucination and abstention. We additionally developed an end-to-end grounded Islamic modeling suite consisting of (i) 25K Arabic text-grounded SFT reasoning pairs, (ii) 5K bilingual preference samples for reward-guided alignment, and (iii) a verse-level Qur'an retrieval corpus of ~6k atomic verses (ayat). Building on these resources, we develop an agentic Quran-grounding framework (agentic RAG) that uses structured tool calls for iterative evidence seeking and answer revision. Experiments across Arabic-centric and multilingual LLMs show that retrieval improves correctness and that agentic RAG yields the largest gains beyond standard RAG, achieving state-of-the-art performance and stronger Arabic-English robustness even with a small model (i.e., Qwen3 4B). We made the datasets are publicly available. https://huggingface.co/datasets/QCRI/IslamicFaithQA

1 Introduction

Islamic QA requires evaluation beyond fluent generation because unsupported answers, language variation, and jurisprudential nuance challenge reliable grounding. The paper introduces a bilingual benchmark and grounded modeling resources, then reports that retrieval—especially agentic RAG—improves faithful performance.

  • Islamic QA systems can produce confident but unsupported responses, while Arabic and culturally situated jurisprudential settings demand careful grounding and uncertainty handling.
  • Existing evaluation gaps motivate a benchmark measuring hallucination and abstention through bilingual, generative questions with atomic single-gold answers.
  • The modeling suite combines 25K Arabic text-grounded SFT reasoning pairs, 5K bilingual preference samples, and a 6,236-verse Quran retrieval corpus.
  • Agentic Quran grounding turns retrieval into iterative evidence seeking through structured tool calls, including semantic search, verse reading, and metadata lookup.
  • 48.90 accuracy for Qwen3-4B-2507 with Agentic RAG exceeds 38.85 with RAG and 21.85 for the base model, while narrowing the Arabic–English gap.
  • 57.30 average accuracy is achieved by Fanar-2-27B with Agentic RAG, the strongest reported overall result.

2 Related Work

Related work identifies persistent reliability problems in multilingual and Islamic QA, including hallucination, language-sensitive alignment, and inadequate answer-only evaluation. It motivates retrieval and agentic evidence seeking as alternatives to relying on longer reasoning traces alone.

  • Multilingual evaluations report that moral judgments and alignment behavior vary with language and data provenance, alongside recurring Western-dominance effects and representational bias.
  • Islamic QA benchmarks increasingly target fiqh reasoning, abstention, and culturally faithful evaluation, but continue to report gaps in Arabic performance and jurisprudential nuance.
  • Arabic-focused studies report elevated factuality and faithfulness errors, motivating evaluation beyond answer-only metrics toward attribution and joint reasoning-output assessment.
  • Longer reasoning traces do not reliably improve grounding and can increase overconfident errors, so additional thinking is not a substitute for evidence.
  • Prior work presents iterative search, tool use, and verification as mechanisms that can improve faithfulness when retrieval and orchestration are reliable.
  • These findings motivate comparing standard RAG with agentic RAG under a strict generative, abstention-aware Islamic QA protocol.

3 Datasets

The paper constructs a unified Islamic modeling and evaluation suite spanning training, preference alignment, Quran retrieval, and generative benchmark resources. ISLAMICFAITHQA emphasizes text-grounded atomic answers while covering varied difficulty, topics, and reasoning demands.

  • The data suite contains instruction-tuning data, preference-alignment data, a retrieval corpus, and the ISLAMICFAITHQA evaluation benchmark.
  • Training and alignment resources: The SFT reasoning dataset contains 25,000 bilingual instruction-response pairs derived from Quranic verses and Hadith, requiring grounded reasoning rather than simple extraction.
  • Training and alignment resources: The RL preference dataset contains 5,000 bilingual samples with canonical-text questions, gold answers, and evaluation parameters for reward-based alignment.
  • Training and alignment resources: The Quran RAG dataset contains 6,236 verse-level retrieval units with standardized metadata enabling canonical citation and deterministic evidence mapping.
  • ISLAMICFAITHQA benchmark: ISLAMICFAITHQA addresses MCQ and MRC limitations with 3,810 bilingual generative questions designed to measure hallucination using LLM-as-a-Judge assessment.
  • ISLAMICFAITHQA curation pipeline: The curation pipeline selects difficult annotated examples, reformulates them into short fact-based questions with atomic gold answers, and adds difficulty, reasoning, and topic metadata.
  • Question profiling analysis: Difficulty peaks at level 3 (31.2%), while inheritance law (26.4%) and jurisprudence (17.4%) are the largest question categories.
  • Question profiling analysis: 70.7% of questions require active reasoning and 55.4% require multi-step reasoning, extending evaluation beyond direct factual recall.

4 Experiments

The experiments evaluate Arabic-centric and multilingual LLMs through standardized benchmarking, fine-tuning, reward alignment, retrieval, and tool-mediated evidence seeking. Models are assessed on IslamicFaithQA using consistent prompting and a three-label correctness framework.

  • Development workflow: The development workflow spans corpus processing, reasoning-oriented supervision, base-model benchmarking, supervised fine-tuning, reward-guided alignment, and inference-time agentic retrieval.The agentic environment provides access to Qur’an and Hadith databases through dedicated tools and retrieval steps.
  • Models: The study benchmarks Arabic-centric and multilingual instruction-tuned LLMs under a unified prompting and grading setup.Base models are evaluated zero-shot with consistent prompts, response formats, output limits, and decoding settings.
  • Retrieval settings: Standard RAG retrieves relevant Qur’anic verse units before answer generation, whereas Agentic RAG uses tool-mediated evidence seeking.The retrieval corpus contains 6,236 ayat embedded with mE5-base for Arabic and English querying.
  • Evaluation: The evaluation uses ISLAMICFAITHQA and labels predictions as Correct, Incorrect, or Not_Attempted according to semantic alignment with atomic gold answers.The study reports %Correct as the primary metric and also analyzes label-wise performance.
  • Evaluation settings: RAG and Agentic RAG are applied to Fanar-1-9B, ALLaM-7B, and Qwen3-4B-2507 after training, while Fanar-2-27B is evaluated without additional fine-tuning.Fanar-2-27B is not fine-tuned because it already shows strong baseline performance.
  • Evaluation reliability: The judge is calibrated on 200 held-out bilingual instances, reaching 79% human–LLM agreement and Cohen’s κ of 0.51 for interannotator agreement.The subset is balanced across Arabic and English and across difficulty levels.

5 Results

ISLAMICFAITHQA reveals a large gap between fluent instruction following and grounded Islamic correctness. Domain supervision, reward alignment, retrieval, and especially Agentic RAG improve performance, while iterative grounding increases inference cost.

  • Baseline results: 48.05 average accuracy is achieved by Fanar-2-27B, with 48.20 in Arabic and 47.90 in English, leading the base-model results.ALLaM-7B is the next strongest model in the reported baseline ranking.
  • Baseline results: 25.70, 19.40, and 18.95 average accuracy are reported for EuroLLM-9B, Llama-3.1-8B, and Mistral-7B-v0.2, respectively, all below 30%.These results show that broad instruction-following ability does not ensure fine-grained, text-grounded Islamic correctness.
  • SFT: 21.85 to 30.55 is the Qwen3-4B-2507 accuracy increase from base performance to SFT, while ALLaM-7B rises from 37.75 to 38.30 and Fanar-1-9B from 35.40 to 36.45.The smaller gains for stronger in-domain baselines suggest diminishing returns when domain priors are already stronger.
  • Reward-guided alignment: 38.30 to 39.55 and 36.45 to 38.18 are the average-accuracy gains from SFT to reward-guided alignment for ALLaM-7B and Fanar-1-9B.The alignment objective favors short, atomic answers with fewer risky additions.
  • Agentic RAG: 38.85 to 48.90 is Qwen3-4B-2507’s increase from standard RAG to Agentic RAG, a gain of +10.05 points and the largest reported intervention gain for that backbone.The agent gathers evidence iteratively, reads specific ayat for disambiguation, refines queries, and revises answers.
  • Error analysis: 24 of 36 recovered cases are Not_Attempted → Correct transitions and 12 are Incorrect → Correct transitions under Agentic RAG.The gains therefore include both answering previously unresolved questions and correcting some prior errors.
  • Efficiency trade-off: 502 to 1,520 average tokens and 0.52s to 3.45s latency are required by Qwen3-4B when moving to Agentic RAG, with 6.63× higher latency and 6 tool calls per query.The method shifts part of the computational cost from model size to inference-time orchestration.
  • Bilingual robustness: 49.60 Arabic versus 48.20 English is Qwen3-4B-2507’s Agentic RAG performance, indicating more balanced bilingual results.The paper associates this balance with iterative evidence seeking and explicit verse inspection over a shared canonical retrieval base.

6 Conclusion

The paper introduces a benchmark and grounded modeling suite for evaluating and reducing hallucinations in open-ended Islamic generation. Retrieval improves correctness, while Agentic RAG produces the largest gains beyond standard RAG through iterative evidence seeking and tool use.

  • Conclusion: ISLAMICFAITHQA and the grounded modeling suite directly evaluate and reduce hallucinations in open-ended religious generation.The suite combines supervised domain reasoning, judge-guided preference alignment, and Islamic-centric retrieval.
  • Conclusion: Retrieval substantially improves correctness, while Agentic RAG yields the largest gains beyond standard RAG through iterative evidence seeking and disambiguation.The conclusion attributes these gains to explicit tool use.
  • Implications: Tool-mediated grounding delivers state-of-the-art performance and improved Arabic-English robustness even with smaller backbones.The paper presents this as a practical path toward more trustworthy Islamic assistants.

Limitations

The benchmark and evaluation scope constrain how broadly the reported faithfulness and abstention results should be interpreted.

  • Scope boundaries: Single-gold atomic questions under-represent cases where multiple answers are valid across madhāhib or interpretive traditions.The benchmark also relies on an LLM judge and limited human calibration, while Quran-centric grounding may disadvantage hadith-, fiqh-, or consensus-supported questions.
  • System limitations: Agentic RAG adds latency and failure modes including tool-use errors and misleading citation attribution.The benchmark focuses on short-form question answering rather than long-form religious guidance.

Ethical Considerations

The dataset construction pipeline used human annotation and LLM assistance, with safeguards governing annotator participation and LLM use.

  • Annotators were recruited through a third-party provider, paid standard local hourly rates, and required to sign nondisclosure agreements.
  • LLMs standardized phrasing, supported metadata annotation, and were not treated as sources of religious knowledge.

Broader Impact

The work provides resources for more trustworthy Islamic question answering by measuring correctness, hallucination, and abstention and by grounding answers in Islamic sources. Its broader scope includes domain-specific question generation, profiling, retrieval, and grading procedures.

  • The benchmark targets trustworthy Arabic–English Islamic QA by evaluating correctness, hallucination, and abstention in grounded generation.
  • Question generation: Question generation emphasizes concise factual questions and gold answers grounded in Islamic sources, including Qur'an verses, Hadith, and scholarly consensus.
  • Question profiling: Question profiling assigns difficulty, reasoning, multi-step reasoning, and one category label using fixed annotation rules.
  • Grounded answering: The Quran-answering setup supports semantic verse search, surah lookup, ayah reading, and within-surah search before producing cited answers.
  • Evaluation: The grader uses semantic alignment with gold targets to distinguish correct, incorrect, and not-attempted answers, while examples specify how partial or contradictory responses are treated.

B Experimental Details

Experiments use controlled inference, training, retrieval, and agentic interaction settings, alongside examples and annotation protocols for the benchmark and data resources.

  • Inference: Benchmark inference uses vLLM with temperature T = 0.7 and otherwise standard generation settings.
  • Evaluation: Automatic grading uses GPT-4.1 at temperature T = 0 to reduce sampling variance for identical evaluation inputs.
  • Training: SFT uses a learning rate of 5 × 10^-5, while RL uses the GSPO objective with a learning rate of 3 × 10^-6.
  • Retrieval: Standard RAG embeds queries and Qur'anic verses with mE5-base, indexes them in ChromaDB, and supplies retrieved verse units as evidence context.
  • Agentic RAG: Agentic RAG uses two turns: evidence seeking with retrieval and verse inspection, followed by final answer generation conditioned on that evidence.
  • Examples: The benchmark includes examples of plausible but semantically incorrect answers, grounded SFT pairs, and preference instances with canonical questions and concise gold answers.
  • Annotation: Annotation assigns 1–5 difficulty, reasoning and multi-step indicators, and one category from a fixed 12-category taxonomy.

E.7 Annotation Results and Agreement

Annotation results show broad difficulty coverage and substantial category agreement, while retrieval evaluation is explicitly framed as preliminary rather than a full benchmark.

  • Annotation results: Category assignment achieves 82.96% agreement and Cohen's κ = 0.62, indicating substantial agreement under the annotation guidelines.
  • Retrieval comparison: mE5-base achieves MAP_Q@5 = 0.138 versus 0.094 for BM25 on IslamicEval 2025 Shared Task 2.
  • Limitations: The retrieval comparison is a preliminary sanity check rather than a full benchmark, and broader hybrid retrieval remains future work.

G Label-wise Results

Table 9 separates correct, incorrect, and not attempted predictions, revealing both performance differences and distinct response behaviors across models. Fanar-2-27B leads overall, while other systems trade higher attempt rates for more errors or abstain frequently.

  • 48.05% average correct rate makes Fanar-2-27B the best overall performer, ahead of ALLaM-7B at 37.75% and Fanar-1-9B at 35.40%.
  • ALLaM-7B and Fanar-1-9B attempt most questions but also produce relatively high incorrect rates.
  • Several Qwen3 and DeepSeek variants leave more than 70% of questions not attempted, yielding low overall correctness despite fewer incorrect responses.
  • Many models perform better in English than Arabic, whereas Fanar-2-27B remains comparatively balanced across both languages.
  • Separating correct, incorrect, and not attempted predictions provides a more informative view of model behavior than accuracy alone.
Loading 2601.07528v2…