Source-linked AI summary
Evidence-Gated Bilingual RAG for Reliable Consumer Banking Search
Ahmed Mohamed Mahmoud Rayyan, Rana Mahmoud Mohammed, Aya Abdallah Abdelmonem, Mohamed Khaled Goda Othman, Rahatara Ferdousi
TL;DR
Consumers may express a banking need in Arabic or English using terms that differ from product pages, and raw webpages can obscure retrieval-relevant information. The paper proposes a local evidence-gated bilingual RAG framework; on 210 URL-labeled queries it improves Top-1 hit rate over a hybrid baseline, while a separate small end-to-end evaluation reports 69.2% strict success.
Problem
Natural-language banking needs in Arabic or English may not share the terminology used on product pages, and raw webpages can contain content that obscures relevant information.
Source: §I · p. 1Method
The local framework refines web content and combines hybrid retrieval, intent-aware evidence selection, cross-encoder reranking, and an Answer / Related Mismatch / No Answer decision layer.
Source: §I · p. 1, D. Hybrid Retrieval and Reranking · p. 3, F. Answer / Related Mismatch / No… · p. 4Results
Top-1 hit rate rose from 50.48% to 79.05% over the hybrid baseline on 210 URL-labeled queries, and the separate 26-case evaluation achieved 69.2% strict end-to-end success (18/26).
Source: A. Retrieval Performance · p. 5, C. End-to-End Evaluation · p. 5Takeaways & Limitations
The authors conclude that intent-aware retrieval with explicit evidence eligibility can improve bilingual consumer banking search over hybrid retrieval alone.
Source: D. Discussion · p. 6The 22 answerable end-to-end cases overlap with the reference set informing domain rules; the automated judge is not human adjudication, and URL matching does not verify individual factual claims.
Source: C. End-to-End Evaluation · p. 6
Abstract
from arXiv · showhide
Consumers often describe what they need rather than use the terminology found on a banking website. For example, a natural-language request ("a card for travel") may not share the keywords of the relevant product page, and the problem is harder in bilingual services, where the same intent may be expressed in Arabic or English. This paper presents an evidence-gated bilingual retrieval-augmented generation (RAG) framework for consumer banking search that operates entirely on local infrastructure. The framework combines structured web-content refinement, hybrid dense-lexical retrieval, intent-aware evidence selection, cross-encoder reranking, and a three-outcome Answer / Related Mismatch / No Answer decision mechanism. On 210 URL-labeled Arabic and English queries, the proposed approach improves Top-1 hit rate from 50.48% to 79.05% and MRR@5 from 0.571 to 0.797 over a hybrid baseline. A separate 26-case end-to-end evaluation achieved 69.2% strict success, with 72.7% of generated answers classified as supported by an automated faithfulness judge. These results are preliminary: the retrieval queries are paraphrases of a small reference set, and the end-to-end test is small and uses an automated rather than human judge, so they do not establish human-verified answer correctness or generalizable performance.
I. INTRODUCTION
Consumer banking searches can use natural-language needs in Arabic or English that do not match product-page terminology, while raw webpages contain material that obscures relevant information. The paper proposes a local, evidence-gated bilingual RAG framework that refines web content and distinguishes relevant retrieval from evidence sufficient for an answer.
I. INTRODUCTION
- Raw banking webpages contain navigation, repeated boilerplate, scripts, and embedded media that can obscure retrieval-relevant information.The studied NBE website also has dynamically rendered content, motivating structured extraction and refinement.Source: §I · p. 1
- Semantic relevance alone does not establish that retrieved content belongs to the intended banking service or provides sufficient evidence for an answer.The paper identifies plausible but mismatched evidence as a concern in consumer financial applications.Source: §I · p. 1
- The framework refines raw web content into structured evidence and combines Arabic–English hybrid retrieval, intent-aware evidence selection, and an Answer / Related Mismatch / No Answer decision layer.It runs on local infrastructure rather than sending consumer queries to an external cloud AI service.Source: §I · p. 1
- 50.48% to 79.05%: the framework improves Top-1 hit rate over the hybrid baseline on 210 URL-labeled queries.The introduction also reports a separate, small end-to-end evaluation of generation, citation, and abstention behavior.Source: §I · p. 2
II. RELATED WORK
Prior work combines lexical and semantic retrieval and applies reranking, while multilingual search continues to face linguistic variation and generation challenges. These issues are relevant when users express an information need differently from the language used in product documents.
A. Hybrid and Multilingual Retrieval
- BM25 can match shared terminology, dense retrieval captures semantic similarity beyond exact overlap, and hybrid retrieval combines these signals; cross-encoders can refine candidate ordering, [3],, [6].Financial RAG studies also report benefits from full-text and semantic retrieval and neural reranking over domain-specific documents [7].Source: A. Hybrid and Multilingual Retrieval · p. 2
- Multilingual retrieval faces differing query and document forms, including Arabic dialect, colloquial and code-switched language, and orthographic variation [9], [10], [11],.The passage notes persistent multilingual RAG issues including irrelevant retrieval and language-specific generation errors [9].Source: A. Hybrid and Multilingual Retrieval · p. 2
B. Reliability and Evidence-Aware RAG
Evidence-aware RAG work treats retrieval quality as part of the generation decision, while consumer banking search additionally requires evidence to match the intended service. The proposed framework processes content and queries locally, then releases evidence for grounded generation or returns an explicit outcome without an answer.
B. Reliability and Evidence-Aware RAG
- Self-RAG assesses retrieved passages and generated responses, while Corrective RAG evaluates documents and may trigger corrective retrieval; both reflect the need to evaluate evidence rather than automatically use retrieved passages, [14].Abstention remains difficult, and a semantically related result may belong to the wrong product or service category [15],.Source: B. Reliability and Evidence-Aware RAG · p. 2
C. Consumer Banking RAG and Research Gap
- Consumer banking search involves short natural-language needs and evidence spread across product and service pages, so generation must be gated on evidence sufficiency rather than retrieval relevance alone.The passage identifies this as an evidence-eligibility gap not addressed by the cited financial RAG or local RAG-toolkit work.Source: C. Consumer Banking RAG and Research Gap · p. 3
III. PROPOSED FRAMEWORK
- The framework’s offline phase refines and indexes web content, while its online phase applies query understanding, hybrid retrieval, intent-aware evidence selection, and a decision layer on local infrastructure.The decision layer either releases evidence for grounded generation or returns an explicit outcome without an answer.Source: §III · p. 3
A. Data Refinery and Knowledge Base Construction
The pipeline converts noisy bilingual banking webpages into structured, provenance-preserving evidence and processes queries through multilingual indexing, query understanding, retrieval, and reranking. Its refinery produces a corpus of 555 effective documents and 962 chunks.
A. Data Refinery and Knowledge Base Construction
- 555 effective documents and 962 Arabic and English chunks result from a refinery that handles dynamic rendering, removes boilerplate, normalizes Arabic, preserves tables, filters near-duplicates, and retains provenance and metadata.Section-aware parent sections use an approximate 800-token budget; indexed child windows use up to 280 tokens with 50-token overlap.Source: A. Data Refinery and Knowledge Base… · p. 3
B. Dense and Lexical Indexing
- BGE-M3 produces dense vectors and sparse lexical weights, while parallel cosine-distance vector and BM25 indexes store chunk content and metadata.Tokenization is selected according to the document language.Source: B. Dense and Lexical Indexing · p. 3
C. Query Understanding
- Queries pass through language detection, Arabic normalization, intent classification, entity extraction, expansion, and rewriting, with deterministic contact and authentication rules preceding semantic classification.A planner can split distinct intents into at most two subqueries; the retrieval evaluation measures retrieval alone rather than this complete path.Source: C. Query Understanding · p. 3
D. Hybrid Retrieval and Reranking
- Dense and BM25 ranked lists are fused, filtered and adjusted, then a multilingual cross-encoder reranks up to 20 candidates and retains five.The dense channel retrieves up to 50 candidates with a default dense weight of 0.7, while the lexical channel retrieves up to 50 positive-scoring matches; RRF uses k = 60.Source: D. Hybrid Retrieval and Reranking · p. 3
E. Intent-Aware Evidence Selection
Intent-aware selection narrows and, when needed, broadens retrieval before the decision layer checks evidence eligibility. The layer distinguishes Answer, Related Mismatch, and No Answer; generation and faithfulness checking are evaluated separately from retrieval.
E. Intent-Aware Evidence Selection
- Intent-conditioned metadata filtering narrows candidates; retrieval can broaden to related document types and then language-restricted results, while ranking adjustments reorder retrieved evidence.These adjustments do not add documents that retrieval did not find.Source: E. Intent-Aware Evidence Selection · p. 3
F. Answer / Related Mismatch / No Answer Decision Layer
- The retrieval-only experiment evaluates eligibility without generation; eligible evidence can proceed to grounded generation in the deployed end-to-end path.The 220-query retrieval experiment does not evaluate generation or faithfulness checking.Source: F. Answer / Related Mismatch / No… · p. 3, Figure 2 · p. 4
- Answer requires present evidence, sufficient intent confidence, and a leading result matching the classified banking family; production generation uses up to three cited sources.The compressed context is limited to 2,000 tokens and is passed to a locally hosted Qwen3 model.Source: F. Answer / Related Mismatch / No… · p. 3
- At intent confidence of at least 0.90, a leading result outside the expected document family produces Related Mismatch, a classification outcome rather than a second retrieval.Any candidate broadening occurs earlier in retrieval.Source: F. Answer / Related Mismatch / No… · pp. 3–4
- No Answer is returned when evidence is absent, confidence is below 0.42, or remaining checks find a family mismatch; unsupported generated answers trigger one stricter regeneration, then abstention if still unsupported.The generation-and-checking path is assessed separately in the 26-case end-to-end experiment.Source: F. Answer / Related Mismatch / No… · p. 4, Figure 2 · p. 4
G. Confidence Estimation
Confidence is an engineered retrieval-evidence score rather than a calibrated probability of answer correctness. The full framework includes a language-model sufficiency component, but it was inactive in the retrieval experiment.
G. Confidence Estimation
- Confidence combines six weighted retrieval and intent signals, scales the core for high-confidence intent–metadata disagreement, then blends it with keyword relevance.The six weights are intent confidence 0.15, leading ranking score 0.25, top-five mean 0.15, category consistency 0.20, score agreement 0.15, and source diversity 0.10; the disagreement scale is 0.55 and the blend weights are 0.75 and 0.25.Source: G. Confidence Estimation · p. 4
- Confidence is not a calibrated probability of answer correctness, and the full framework’s language-model sufficiency component was inactive in the 220-query retrieval experiment.Source: G. Confidence Estimation · p. 5
IV. EXPERIMENTAL SETUP
The study builds a bilingual knowledge base from National Bank of Egypt webpages and evaluates retrieval on Arabic and English paraphrase queries. The evaluation set includes 210 URL-labeled queries for ranking metrics and 220 queries for eligibility decisions.
A. Dataset and Knowledge Base
- The refined National Bank of Egypt corpus contains 555 documents and 962 Arabic and English chunks.Browser rendering captures dynamically loaded content before refinement, boilerplate removal, and near-duplicate filtering.Source: A. Dataset and Knowledge Base · p. 5
B. Evaluation Set
- The evaluation uses 220 queries—127 Arabic and 93 English—paraphrased from 22 references; 210 have expected URLs for ranking evaluation.All 220 queries assess eligibility and forbidden-URL behavior; 43 specify forbidden URLs, and Arabic–English code-switched queries are excluded. Exchange-rate and personal-loan queries account for 142 queries (64.55%); the reference set also informed some domain rules, so results characterize this test setting rather than a fully independent benchmark.Source: B. Evaluation Set · p. 5
C. Retrieval Configurations
The retrieval comparison tests the complete Evidence-Gated configuration against a hybrid baseline, with several components changing together. The setup uses multilingual embedding and cross-encoder reranking, and reports ranking metrics on 210 URL-labeled queries.
C. Retrieval Configurations
- Evidence-Gated adds intent-aware evidence selection, domain-aware ranking, and confidence-based decisions to the Hybrid Baseline, so the comparison does not isolate individual component effects.Source: C. Retrieval Configurations · p. 5
D. Models and Infrastructure
- BGE-M3 provides multilingual embeddings and BGE-Reranker-v2-M3 reranks candidates; Qwen3-8B is used only in the separate 26-case end-to-end evaluation.Retrieval ran on a single 16 GB GPU; Qwen3-8B also performs automated faithfulness assessment in the end-to-end evaluation.Source: D. Models and Infrastructure · p. 5
E. Evaluation Metrics
- Ranking metrics on 210 URL-labeled queries include Top-1, Top-3, Hit@5, MRR@5, and nDCG@5; evidence eligibility and forbidden-URL violations use all 220 queries.End-to-end strict success requires expected answer or abstain behavior, supported answers when expected, and correct source citation.Source: E. Evaluation Metrics · p. 5, Table I · p. 5
V. RESULTS AND DISCUSSION
The framework improves retrieval ranking over the hybrid baseline and adds evidence-eligibility decisions. Its separate 26-case end-to-end evaluation reports strict success and answer-support outcomes, with preliminary results limited by overlap with the reference set and automated judging.
A. Retrieval Performance
- Top-1 hit rate rose from 50.48% to 79.05% over the hybrid baseline on 210 URL-labeled queries, with improvements also reported for Top-3, Hit@5, MRR@5, and nDCG@5.The comparison is a single run without significance testing, and multiple components differ between configurations.Source: A. Retrieval Performance · p. 5
B. Evidence Eligibility and Efficiency
- The framework classified 214 of 220 queries (97.3%) as answerable and abstained on six (2.7%); it had no forbidden-URL violations, versus one for the baseline.Related Mismatch was not triggered, and abstention correctness was not independently assessed.Source: B. Evidence Eligibility and Efficiency · p. 5
C. End-to-End Evaluation
- 69.2% of 26 cases achieved strict end-to-end success (18/26), and the automated judge classified 72.7% of 22 generated answers as supported.The evaluation also correctly abstained on all four designated abstention cases; the judge was Qwen3, the model family used for generation.Source: C. End-to-End Evaluation · pp. 5–6, Table II · p. 6
- The end-to-end findings are preliminary because answerable cases overlap with the reference set used to inform domain rules, and judging was automated rather than human-adjudicated.URL matching does not verify individual factual claims.Source: C. End-to-End Evaluation · p. 6
D. Discussion
The discussion presents improved Top-1 retrieval as promising but preliminary within the evaluated setting. The conclusion summarizes the local evidence-gated bilingual framework and calls for independently collected queries and human evaluation.
D. Discussion
- The combined approach improved Top-1 hit rate from 50.48% to 79.05% over hybrid retrieval alone, but evaluation remains limited to one bank, paraphrase-based queries, and a small automatically judged end-to-end test.The authors identify independent queries, human evaluation, and component-level ablations as future work.Source: D. Discussion · p. 6
VI. CONCLUSION
- The local framework combines web-content refinement, hybrid dense–lexical retrieval, intent-aware evidence selection, and explicit answer eligibility.On 210 URL-labeled queries it improved Top-1 hit rate from 50.48% to 79.05% and MRR@5 from 0.571 to 0.797; a small end-to-end evaluation achieved 69.2% strict success.Source: §VI · p. 6
- Future work will add independently collected queries and human evaluation.The authors characterize the reported results as preliminary.Source: §VI · p. 6