Source-linked AI summary
Ansari: A Retrieval-Grounded Islamic AI Assistant -- Architecture, Deployment, and Lessons from 140,000 Conversations
M Waleed Kadous, Amr Elsayed, Abdullah Al Nahas, Ashraf Haress
TL;DR
General-purpose LLMs can fabricate Islamic sources and present contested rulings too confidently. Ansari addresses these risks with retrieval-grounded, policy-governed responses, and reports leading IslamicMMLU performance while highlighting persistent social and technical limits.
Problem
General-purpose LLMs may fabricate Qur’anic verses or hadith and flatten centuries-old juristic disagreement into confident rulings when answering Islamic questions.
Method
Ansari uses an agentic retrieval-augmented design in which a tool-using language model searches authenticated Islamic corpora and answers only from retrieved text with citations, across multiple deployment surfaces.
Results
Ansari ranks first on the public IslamicMMLU leaderboard with 94.2% average accuracy, ahead of the strongest frontier systems, while its live-leaderboard lead is not a durable claim of superiority.
Takeaways & Limitations
Grounding and openly published policy can make a values-sensitive assistant more authentic and inspectable, but community absence in model formation remains a hard gap beyond the model alone.
Takeaways & Limitations
Grounding reduces but does not eliminate residual errors, and the zero-hallucination result comes from a small validation set rather than a universal guarantee.
Abstract
from arXiv · showhide
General-purpose large language models (LLMs) are increasingly used to answer religious questions, but for Islamic content they carry two serious risks: factual fabrication (inventing Qur'anic verses or hadith) and subtle value misalignment. We present Ansari, a deployed, retrieval-grounded Islamic AI assistant that has handled more than 140,000 conversations across 25+ languages since June 2023. Ansari is built around an agentic retrieval loop: a tool-using language model issues searches against authenticated Islamic corpora -- the Qur'an, hadith collections, a multi-volume jurisprudence (fiqh) encyclopedia, and exegetical (tafsir) sources -- and answers only on the basis of what it retrieves, with citations attached for verification. We describe the system's architecture (the agent loop, the retrieval tools, the corpora, and the system prompt that encodes editorial and theological policy), its multi-platform deployment (web, mobile, WhatsApp, and as a Model Context Protocol server and an Agent Skill), and what 140,000 real conversations reveal about how Muslims actually use such a tool. We report results on several complementary evaluations -- zero-shot performance on accredited institutional exams, a human-rated validation during Ramadan, and two independent, externally run benchmarks on which Ansari currently tops the public IslamicMMLU leaderboard ahead of frontier models and is competitive on Islamic legal reasoning (IslamicLegalBench) while strongly resisting false premises -- and draw out lessons that generalize beyond Islam to any faith- or values-sensitive deployment of LLMs: grounding is necessary but not sufficient, the system prompt is a theological as much as a technical artifact, and the absence of community in how models are formed remains a hard gap.
1 Introduction
Ansari addresses the risks of fabricated or misattributed Islamic content with a deployed assistant whose authenticity and accuracy are built into its architecture and policy. The system has served more than 140,000 conversations across over 25 languages and multiple platforms while combining grounded retrieval, citations, and evaluations across several lenses.
- Ansari is a deployed Islamic AI assistant designed to make authenticity and accuracy structural system properties rather than hopes about the model.
- More than 140,000 conversations in over 25 languages have been answered since June 2023 across web, mobile, WhatsApp, MCP, and Agent Skill surfaces.
- The paper combines a usage study, multiple evaluations, and transferable lessons about theological policy, grounding, and community formation.
- Its agentic retrieval design searches authenticated Islamic corpora and answers only from retrieved text with native citations.
- Authenticity and user trust depend on policy and process as much as model quality, including no fabricated sources, faithful disagreement handling, transparent citation, and honest uncertainty.
2 Background and related work
Ansari builds on retrieval augmentation while adapting it to Islamic content, where canonical wording, attribution, transmission chains, and legitimate juristic disagreement matter. Its deployed-system perspective complements benchmarks that measure knowledge, legal reasoning, values, representation, or user-facing counsel.
- Retrieval augmentation grounds generation in an authoritative corpus so models can cite rather than confabulate.
- Islamic applications require preserving canonical wording and attribution because losing a transmission chain or elevating a minority opinion can be substantively wrong.
- RAG makes the retrieval corpus the boundary of what the system is allowed to assert, directly addressing fabricated religious authority.
- Existing work benchmarks Islamic knowledge, legal reasoning, values, and cross-faith representation, whereas JaleesBench measures what an assistant’s counsel does to users.
- Ansari’s contribution is complementary: a deployed system, its architecture, and lessons from real usage rather than another benchmark.
3 Design goals and constraints
Ansari was shaped by authenticity, scholarly pluralism, and multilingual reach, with grounding and citation costs accepted as the price of trust. These goals constrain deployment through higher latency and expense than an ungrounded chatbot.
- Authenticity by construction requires the assistant to avoid asserting religious content it cannot ground in an authenticated source.
- Respect for scholarly pluralism requires surfacing consensus and presenting legitimate disagreements fairly rather than adjudicating between schools.
- Reach requires serving multilingual communities distributed across different devices and platforms.
- Grounding every answer makes conversations more expensive, while faithful citation makes answers longer and slower than an ungrounded chatbot.
4 System architecture
Ansari is a production service that uses a tool-using model in a bounded search–read–refine loop, retrieves from authenticated Islamic sources, and returns cited answers. Its retrieval tools support scriptural, hadith, jurisprudence, and tafsir searches, with Qur’an and hadith tools dominating observed usage.
- 4.1 Overview and stack: The production system adapts one agent across web, mobile, WhatsApp, MCP, and Agent Skill front ends while persisting users, threads, and messages in PostgreSQL.
- 4.2 The agentic retrieval loop: The agent receives the user thread and tool definitions, chooses whether to answer or search, and receives retrieval outputs as function responses.
- 4.2 The agentic retrieval loop: The search–read–refine loop repeats until the model produces a final answer, which is streamed with citations from retrieved documents.
- 4.2 The agentic retrieval loop: The facilitator bounds each query at ten tool calls, three consecutive uses of one tool, and ten loop iterations to keep latency and cost predictable.
- 4.3 Retrieval tools and corpora: The tools are thin clients over four classes of authenticated sources, returning structured metadata for precise citation.
- 4.3 Retrieval tools and corpora: Qur’an search appeared in 51.6% of threads, hadith search in 43.8%, fiqh in 34.1%, and tafsir in 3.2%.
4.4 The system prompt
Ansari’s editable system prompt encodes its editorial and theological policy, while configurable model selection and production constraints shape how that policy is deployed. The architecture treats sourcing, disagreement, pastoral safety, accountability, and cost as connected design concerns.
- The system prompt: The production system prompt is the primary location for Ansari’s editorial and theological policy.Its behavior is governed not only by model weights but by an editable prompt.
- The system prompt: Ansari identifies as a multilingual Sunni Islamic assistant and requires Qur’an, hadith, and fiqh claims to come from specific search results.It also requires referenceable evidence before attributing statements to scholars.
- The system prompt: The prompt instructs Ansari to handle disagreement objectively, foreground consensus, and avoid presenting one legitimate position as definitively correct.This policy addresses long-standing differences within Islamic law and theology.
- The system prompt: Pastoral-safety rules require warmth during mental-health crises, referral to a local imam or community, and helpline information before anything else.This is the prompt’s explicit crisis-response policy.
- The system prompt: The openly published prompt makes the policy shaping the assistant inspectable as an accountability feature.The project describes the prompt as selecting which model behavior is expressed in deployment.
- Models and cost: Ansari decouples the base model from the agent loop, retrieval tools, and system prompt, allowing model changes through configuration.Production models changed across GPT-class, Claude, and Gemini configurations as evaluations exposed weaknesses.
- Models and cost: Model changes were evidence-driven: IslamicLegalBench exposed Claude’s weaker deep reasoning, while Gemini 3.5 Flash improved synthesis speed and cost.The project reports that Flash was better at synthesizing retrieved passages into cohesive answers, faster, and cheaper.
- Models and cost: A typical conversation uses about 100,000 tokens of context, with estimated cost falling from roughly US$0.20 on Claude to US$0.10 on Gemini.The token load results from grounding substantive answers in retrieved passages.
4.6 Data and privacy
Ansari combines guest access and persistent account features with a PostgreSQL-backed data model, while its retrieval architecture evolved toward native tool use and modular workflows. It is also distributed through external assistant interfaces.
- Data and privacy: Guest access lets anyone ask questions without creating an account, while signing in enables history, personalization, sharing, and feedback.Conversation history is saved across devices for signed-in users.
- Data and privacy: User accounts, threads, messages, tokens, feedback, shares, and preferences are stored in PostgreSQL through the Drizzle ORM.Message logging supports quality analysis while avoiding identifying metadata, and users may request deletion.
- Architecture evolution: Ansari’s successive generations moved from prompt-tuned chat without retrieval to RAG over the Qur’an, hadith, and an approximately 18,000-page fiqh encyclopedia.The current generation added tafsir retrieval, native tool use, and more precise citations.
- Architecture evolution: The current architecture decomposes the workflow into query generation, search, and answer generation.This modular workflow replaced the earlier generic LLM-plus-tools loop.
- Distribution: Ansari exposes an answer_islamic_question tool through an MCP server that MCP-capable clients can call.The MCP surface extends the system beyond its own applications.
- Distribution: An Agent Skill packages Ansari inside Claude and similar assistants without requiring a separate app or API key.Together with MCP, this distributes Ansari as an invoked capability rather than only a destination.
5 Deployment and usage
Ansari had reached substantial multilingual usage, and its conversation sample shows demand extending beyond information retrieval. Users seek practical guidance, pastoral support, and help preparing religious content.
- Scale: More than 140,000 conversations were served in 25+ languages by March 2026, with about 65% of answers citing primary sources and roughly 84% positive satisfaction.The service averaged roughly 290 conversations per day and peaked around 450 per day during the busy late-winter period.
- What people ask: The usage taxonomy hand-categorized a random sample of 10,000 conversations into understanding, practice, content preparation, support, and miscellaneous intents.The categories cover informational, action-oriented, drafting, pastoral, and residual uses.
- What people ask: Practice rivals understanding in the conversation sample, indicating that users seek guidance on how to act as well as information about Islam.Examples include fasting with anesthesia, shared food, and motivating a teenager who stopped praying.
- What people ask: A smaller support segment addresses distress and directly drove the crisis-response policy in the system prompt.These conversations are few but described as high in human stakes.
- What people ask: Imams and teachers are power users of content preparation, using Ansari as a drafting partner for sermons and lessons.This places the assistant in professional religious preparation as well as individual questioning.
6 Evaluation
Ansari is evaluated through complementary institutional, human-rated, behavioral, and independent external benchmarks. Results show strong Islamic knowledge and resistance to false premises, while exposing sensitivity to model choice and prompt-level policy.
- IslamicMMLU: Ansari ranks first on IslamicMMLU with 94.2% average accuracy, ahead of the strongest frontier systems.The live leaderboard covers Qur’an, hadith, and fiqh, with per-track scores of 99.5%, 95.8%, and 87.2%, respectively.
- IslamicLegalBench: Ansari scores 64.5% on IslamicLegalBench, close to GPT-5 at 67.7% and Claude Sonnet 4.5 at 65.6%.The benchmark’s authors evaluated the production agent few-shot with pass@1 over 718 questions.
- IslamicLegalBench: 96.1% of Ansari’s responses challenge flawed premises, versus 82.4% for GPT-5, with sycophancy at 3.9%.By complexity, scores are 78.4% for low, 43.6% for moderate, and 59.4% for high tasks; moderate reasoning is hardest across systems.
- Institutional exams: 80% on an Introduction to Qur’an exam and 78% on a theology exam were achieved zero-shot on accredited human-student assessments.These exams probe reasoning and synthesis beyond multiple-choice testing.
- Human-rated validation: 4.41/5 was Ansari’s Ramadan human rating across 34 questions, with 100% helpfulness, zero hallucinations, and references in 82% of answers.The zero-hallucination result is promising but limited by the small sample.
- Behavioral robustness: A one-page steadfastness instruction lifted Ansari’s JaleesBench score from +0.48 to +0.84 across 20,160 runs.The result indicates that behavior is sensitive to prompt-level policy, creating both an improvement opportunity and a warning.
7 Lessons learned
Ansari’s lessons show that authenticity and user trust require more than retrieval: policy, transparency, and ongoing review remain central. Grounding reduces fabrication but leaves practical failures and does not make the system infallible.
- The system prompt is a normative artifact that makes interpretive and authority choices explicit and publishing it supports accountability.It addresses whose interpretation to privilege, how to handle disagreement, where authority ends, and how to prevent fabricated sacred sources.
- 1,914 feedback entries were 85.5% positive, 11.8% negative, and 2.7% serious, with loops, incorrectness, and unanswered questions among common complaints.A separate Ramadan-date error illustrates why time-sensitive facts should be externalized rather than trusted to the model.
- Grounding lowers fabrication risk but does not eliminate mundane failures such as temporal mistakes, loops, and plain incorrectness.
8 Limitations and ethical considerations
Ansari’s ethical boundaries include refusing to replace qualified scholars, acknowledging residual errors, and recognizing limits in model formation, evaluation, and data stewardship. These constraints define where a grounded assistant remains insufficient on its own.
- Ansari assists understanding rather than replacing qualified scholars or community, because treating a fluent machine as authority is a social risk that engineering cannot fully remove.
- The zero-hallucination result comes from a small validation set and is not claimed as a universal guarantee.Continuous human review of flagged cases remains part of the system.
- The base model inherits a formation process with broadly Western values, which domain policy can mitigate but not replace.The paper identifies the absence of the faith community’s communal formation as a gap that the system prompt only partly fills.
- The benchmarks emphasize knowledge and short validation sets, under-measuring long-conversation behavior and cross-lingual fidelity.The paper calls broader evaluation its most important methodological debt.
- Faith conversations are sensitive: Ansari logs content for quality without identifying metadata and supports deletion, but logging still carries ongoing responsibility.
9 Conclusion
Ansari demonstrates that an Islamic assistant can prioritize authenticity through retrieval and policy while remaining inspectable and useful in real practice. Its deepest unresolved challenge is the absence of community in the underlying model’s formation.
- Ansari’s architecture makes the assistant assert only retrieved content and uses an openly published system prompt to make its judgments inspectable.
- Real usage is heavily practical and partly pastoral rather than merely encyclopedic.
- The underlying model’s lack of community formation remains the deepest gap, making faithful assistance a problem about people as well as models.The conclusion presents this lesson as applicable beyond Islam to groups serving faith communities.