UAE GovTrust Scope Filtering Gains and Routing Failures in Bilingual Public Service Retrieval

• A location mention alone is insufficient: retrieval must identify which location relation controls the requested service.A toll question about Abu Dhabi should retrieve the Abu Dhabi toll card even if the vehicle is registered in Dubai.Source: §1 · p. 1
• UAE GovTrust is a controlled diagnostic of retrieval and routing, using authored service summaries and synthetic questions rather than an expert-validated benchmark of UAE law.The experiments isolate these stages without a generator or automated answer judge.Source: §1 · p. 2
• The study compares twelve methods to disentangle scope selection, ranking, sparse-branch eligibility, and direct service lookup.Its contribution also documents how a correct hard filter can improve ranking while the mechanism selecting that filter fails.Source: §1 · p. 2
• The engineering conclusion is to evaluate scope inference independently from retrieval gains obtained when the scope is correct.The paper does not claim a new general filtering algorithm or production readiness.Source: §1 · p. 2

• Prior regulatory retrieval includes ObliQA and the RIRAG-2025 shared task [1], as well as deployed version- and scope-aware question answering over normative documents [3].The paper therefore does not present jurisdiction or version filtering itself as new.Source: §2 · p. 2
• Multilingual and bilingual evaluation already spans French-Dutch legal QA, multilingual RAG [7], retrieval and answer quality [6], Arabic-English legal QA [15], and jurisdiction assumptions [14].GovTrust instead tests competing locations and evaluates ranked cards and gate decisions rather than legal citations or generated explanations.Source: §2 · p. 2
• CheckList, FollowIR, and NevIR motivate targeted capability, instruction-following, and negation tests; GovTrust applies these behavioral principles to service-location roles.The report does not claim a new general testing paradigm, and its Arabic is Modern Standard Arabic rather than an Emirati-dialect benchmark [5].Source: §2 · pp. 2–3
• The comparison uses BM25, multilingual E5 [9], reciprocal rank fusion, and a compact multilingual cross-encoder [16].The cross-encoder is a practical open-model comparison, while direct lookup tests whether retrieval complexity is needed for a 22-card catalogue.Source: §2 · p. 3

3.1 Task and catalogue annotations

• The catalogue contains 22 English-Arabic service cards, with five applicability labels that are routing annotations rather than a complete legal ontology.Publisher level and service applicability are separate fields; six cards remain supported only by indexed excerpts, and the summaries and labels have not been independently validated.Source: §3.1 · p. 3
• The primary retrieval analysis uses 72 English-Arabic variants of 36 answerable scenarios; the full set has 50 scenarios, and mixed-language variants are exploratory.Questions and labels were co-developed without an untouched test split, and the Table 1 caption cautions that translations and perturbations are dependent observations.Source: §3.1 · p. 3, Table 1 · p. 4

3.2 Scope filtering and ranking

• Routed variants filter by allowed applicability scopes before ranking, retaining federal cards alongside the selected local scope under this catalogue’s policy.If the gate suppresses retrieval or excludes every relevant card, a later reranker cannot recover the evidence; RoleRRF uses the original router, while OracleRRF substitutes annotated routing as an upper-bound control.Source: §3.2 · p. 4
• The comparisons include BM25 and dense retrieval, reciprocal-rank fusion, positive-score fusion, cross-encoding, and catalogue-specific direct lookup.PositiveRRF excludes zero-score BM25 cards from its lexical branch; DirectLookup does not search card text and can miss a service even when scope selection is correct.Source: §3.2 · pp. 4–5

3.3 Behavioral diagnostics after the audit

• A frozen post-audit diagnostic set applies seven transformations to 24 controls across six services, yielding 168 perturbations.The transformations target known weaknesses, so the set is not representative user traffic or a held-out robustness benchmark.Source: §3.3 · p. 5

3.4 Measures and statistical interpretation

• Evaluation separates Hit@1 and other ranking measures from routing agreement, evidence survival, gate errors, wrong-scope results, and empty rankings.The exploratory bootstrap varies over authored scenario families; new comparisons are descriptive, not confirmatory significance tests or population generalizations.Source: §3.4 · pp. 5–6

3.5 Execution and latency protocol

• Latency measurements cover 72 bilingual answerable queries across three warm-up requests and three randomized passes per method on one host.They include routing and request-time encoding or scoring, but exclude caching, network requests, generation, concurrency, and cold starts; the measurements are not service-level guarantees.Source: §3.5 · p. 6

4.1 Original matched retrieval comparison

• 69/72 queries yield a relevant first result with RoleRRF, versus 47/72 with RRF, a 30.6-percentage-point difference.Both methods can often retrieve relevant material within the first five results, so the first-rank difference is not an equal difference in complete evidence coverage.Source: §4.1 · p. 6, Table 2 · p. 7
• RoleDense and DirectLookup each achieve 68/72 correct first hits; the tested cross-encoder achieves 31/72 unfiltered and 62/72 filtered.The result does not establish that cross-encoders generally underperform dense retrieval; it applies to this model, corpus, and input format.Source: §4.1 · p. 6

4.2 Evidence language and sparse branch behavior

• With English-only cards, RoleRRF achieves 16/36 Arabic first hits despite correct scope predictions; excluding zero-score lexical candidates raises RolePositiveRRF from 52/72 to 64/72.The ablation identifies document-ID ordering as a hazard when zero-overlap sparse candidates enter fusion; filtering alone does not fix it.Source: §4.2 · p. 7, Table 3 · p. 8

4.3 Routing under disclosed perturbations

• The unchanged router selects the correct action and exact scope on only 2/168 perturbations, and all required evidence survives its candidate filter in only 2/168 cases.RoleRRF retrieves a relevant first result for 1/168 perturbations, versus 76/168 for RRF, 82/168 for E5, and 59/168 for the unfiltered cross-encoder; the authored stress set is not an unbiased estimate of natural-query frequency.Source: §4.3 · p. 8, Figure 2 · p. 9, Figure 1 · p. 9
• Router failures include irrelevant-location triggers, historical-year suppression, distracting locations, and unresolved negation or corrections, which can leave no candidates or exclude needed cards.Correct scope selection can still help under the perturbations: OracleRRF retrieves relevant first results for 161/168 queries.Source: §4.3 · p. 8

4.4 Latency and reproduction

• Fresh-environment execution reproduces all 1,320 original ranking orders and actions with 0 mismatches; the latency measurements make no claim about large corpora, concurrency, or complete RAG requests.The reproduction is a clean-install run on the same host, not an external replication or independent human validation.Source: §4.4 · p. 10, Table 4 · p. 10

5 Discussion and limitations

• Correct scope metadata and correct inference are distinct: perturbations expose failures when irrelevant facts, syntax changes, negation, or corrections enter the input.The authors argue that evaluation should count non-answer errors and evidence exclusion alongside ranking errors.Source: §5 · p. 10
• Direct lookup performs strongly on this small catalogue, but does not establish a production alternative or show that a complex hybrid architecture is necessary.The lookup detector and router share development constraints and stress failures; full-policy retrieval would involve substantially more documents, overlapping services, chunk boundaries, duplicated or superseded provisions, and broader relevance judgments.Source: §5 · pp. 10–11
• A multilingual dense model can retrieve across languages, but zero-overlap documents promoted by the sparse branch can dilute that ability.The positive-score ablation is simple and reproducible, but does not establish an optimal fusion constant, weighting scheme, or sparse cutoff; one compact cross-encoder checkpoint also cannot represent the full range of rerankers.Source: §5 · p. 11
• The corpus, translations, prompts, parser, and labels were co-developed with AI assistance, without independent Arabic or domain validation or an untouched test set.The post hoc stress set is intentionally adversarial, covers only six services, and contains dependent translations and transformations.Source: §5 · p. 11
• The service catalogue is not a collection of official source passages, and the study does not evaluate generated-answer faithfulness, citation correctness, legal eligibility, or production effectiveness.Six cards lack an inspected full page body; observing a current webpage does not establish an authentic historical version or legally complete description.Source: §5 · p. 11
• The proposed next study separates construction from evaluation, using qualified reviewers and independently authored unseen questions before testing retrieval, routing, and then a fixed generator.The authors propose comparing learned or schema-constrained role extraction, query rewriting, and clarification strategies against the disclosed rules; these experiments were not conducted in this report.Source: §5 · p. 11

6 Conclusion

• UAE GovTrust is a reproducible diagnostic of scope-filtering gains and routing failures, not an expert-validated UAE benchmark or end-to-end RAG system.Role-filtered fusion improves first-result retrieval on the original templates; filtered dense retrieval and simple lookup achieve similar results, while excluding zero-score lexical candidates improves English-only evidence retrieval.Source: §6 · p. 11
• Under disclosed post-audit perturbations, the unchanged router often blocks retrieval or selects an inapplicable scope, reversing filtering's practical benefit.The conclusion recommends testing scope inference on independently authored questions before attributing trustworthiness to a metadata filter.Source: §6 · pp. 11–12

Author affiliation and funding

• Aleksandar Mihajlovic is the sole author, and the study was self-funded without external funding, a research grant, or commercial sponsorship.The work is associated with his PhD studies at Singidunum University; he lives and works in Abu Dhabi.Source: Author affiliation and funding · p. 12

Contributions and use of AI

• The human author directed and takes responsibility for the study, with ChatGPT/Codex assisting across materials, code, analysis, and manuscript preparation.AI assistance is not presented as independent Arabic or domain validation, and generated answers or LLM-as-judge scores are not experimental outcomes.Source: Contributions and use of AI · p. 12

Employment and competing interests

• The author discloses employment at Modon Holding; experiments used public-source-derived summaries and synthetic questions, not employer data.The findings do not represent an official position or endorsement of Modon Holding, Singidunum University, or any UAE public authority.Source: Employment and competing interests · p. 12

Ethics and data availability

• No human participants or private administrative records were used, and ancillary materials include frozen inputs, rankings, timings, scripts, manifests, and reproduction instructions.Source code is provided under the MIT license; government webpage bodies and external model weights are not redistributed.Source: Ethics and data availability · p. 12

Appendix A Complete stress set results

• Empty counts perturbations with no returned card, while wrong scope counts returned inapplicable first cards.OracleRRF is an annotation-assisted reference.Source: Appendix A · p. 14
• The full per-query action, scope, reason, ranking, and metric records accompany the paper.Source: Appendix A · p. 14
• Table 5 reports complete stress-set first-result counts for 24 controls and 168 perturbations.The controls repeat original queries; the perturbations vary the same service-location-language combinations, and all use bilingual cards.Source: Appendix A · p. 13, Table 5 · p. 14

Appendix B Reproduction and provenance

• The ancillary directory preserves the original frozen data and version 0.1.1 outputs alongside version 0.2.0 revision data and results.Source: Appendix B · p. 14
• The unchanged implementation is src/benchmark.py, while the revision includes benchmark, analysis, and audit-probe reproduction entry points.Source: Appendix B · p. 14
• E5 and the cross-encoder are pinned to specific revisions, and reproduction requires separately downloading model files and matching their hashes.The environment file records package versions, token limits, CPU, thread settings, and index construction times; timing may vary by host and load.Source: Appendix B · p. 14