PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Does Retrieval Fail? Evaluating RAG Architectures for Agricultural Advisory

Khan Raiyan Ibne Reza, Sanjana Aktar Maria, Sumaiya Tabassum Nimi

arXiv:2608.14886v1cs.CL

TL;DR

Retrieval evaluation in low-resource RAG can obscure differences across query types and language conditions, especially when colloquial Bengali farmer queries must match formal agricultural documents. This paper benchmarks five retrieval architectures and six embedding models under controlled conditions, finding that no single method is consistently best and that retrieval depends jointly on language boundary and query register.

  • Problem

    Evidence is limited on how retrieval quality varies by query type and language condition in low-resource, domain-specific RAG, where Bengali farmer language differs from formal agricultural terminology.

  • Method

    The paper builds a provenance-grounded Bengali agricultural benchmark from 284 authoritative publications and evaluates five architectures and six embedding models across three language conditions.

  • Results

    No single retrieval method is consistently best: Hybrid RRF reaches overall R@10=0.539, while dense R@10 ranges from 0.093 on farmer queries to 0.970 on safety queries.

  • Takeaways & Limitations

    Low-resource RAG evaluation should report retrieval performance by language condition and query type rather than relying on aggregate scores alone.

  • Takeaways & Limitations

    Evaluation is limited to agricultural advisory and first-stage zero-shot retrievers, leaving learned sparse models and cross-encoder rerankers for future work.

Abstract

from arXiv · show

Retrieval quality in RAG systems is commonly reported as a single aggregate score, which can hide large differences across query types and language conditions. We study this problem in Bengali agricultural advisory, where farmer queries are often colloquial while official advisory documents use formal scientific terminology. We construct a test collection of 1,000 queries and 2,882 knowledge nodes extracted from 284 official Bangladeshi agricultural publications, and use it to evaluate five retrieval architectures and six embedding models under three controlled language conditions. The results show that no single retrieval method is consistently best. For native Bengali queries, BM25 is the strongest single retriever (R@10 = 0.506) while Hybrid RRF reaches the highest overall R@10 of 0.539. However, dense retrieval performance varies sharply by query type: R@10 is 0.093 on colloquial farmer queries and 0.970 on formal safety queries. Across language conditions, BM25 R@10 drops from 0.506 on Bengali queries to 0.004 when English queries are matched against the Bengali corpus, while dense retrieval falls only from 0.464 to 0.425. We also find that embedding task configuration and passage length can each change reported R@10 by a factor of seven, independent of architecture. These results show why low-resource RAG evaluation should report performance by language condition and query type rather than relying on aggregate scores alone. The dataset and evaluation scripts are available at https://huggingface.co/datasets/RaiyanKhaan/AgriTrust-RAG.

1 Introduction · 2 Related Work

This paper addresses the lack of retrieval-only, provenance-grounded evaluation for Bengali agricultural advisory, where colloquial farmer queries diverge from formal scientific documents. It positions a benchmark designed to isolate architecture, language, and register effects in a low-resource domain.

  • 1.1 Motivation and Gap: 237 million speakers and a register gap between colloquial farmer queries and formal scientific Bengali motivate Bengali agricultural advisory as a retrieval test case.The gap can challenge lexical and dense retrieval differently, while advisory errors have concrete stakes.
  • 1.1 Motivation and Gap: Existing benchmarks are predominantly English or lack agricultural, provenance-level, retrieval-only, and controlled cross-lingual evaluation of query-register variation.Prior Bengali systems translate queries before retrieval or do not isolate retrieval, while the closest agricultural resource is not a retrieval-only benchmark.
  • 1.2 The Benchmark Collection: Six design properties combine traceable retrieval units, canonical structured nodes, document traceability, controlled multilingual evaluation, agricultural grounding, and register-aware queries.The evaluation covers both farmer queries and formal queries, and reports four key findings developed later in the paper.
  • 1.2 The Benchmark Collection: 284 source PDFs, 2,882 knowledge nodes, 19,768 entities, 17,501 triples, and 1,000 annotated queries form the benchmark collection.The collection is available at https://huggingface.co/datasets/RaiyanKhaan/AgriTrust-RAG.
  • 1.3 Contributions: The benchmark contributes canonical retrieval units that unite natural-language content, structured facts, and deterministic source metadata through bounded, schema-guided extraction.This contribution accompanies the rigorous low-resource Bengali agricultural retrieval test collection.
  • 2.1 IR and RAG Evaluation Benchmarks: MIRACL and Mr. TyDi cover diverse languages, but existing multilingual and RAG benchmarks do not isolate retrieval architectures under controlled intra-language register shifts.The cited benchmark landscape is also largely English and does not target provenance-grounded, low-resource agricultural advisory.
  • 2.2 Retrieval in Low-Resource Languages: Dense retrieval can degrade in low-resource languages through tokenization fragmentation and language bias, while vocabulary mismatch separates lay agricultural queries from formal documents.Prior work reports modest cross-lingual DPR gains, finer token matching with ColBERT, and KinyaColBERT errors attributed to vocabulary coverage rather than architecture.
  • 2.2 Retrieval in Low-Resource Languages: The benchmark is among the first to combine low-resource language evaluation, architecture isolation, retrieval-only measurement, controlled cross-lingual conditions, and provenance-grounded gold labels.This design distinguishes the work from generation-focused Bengali advisory systems and prior query-translation approaches.

3 The Benchmark Collection

The benchmark combines 284 provenance-preserving Bengali agricultural publications into 2,882 structured knowledge nodes and evaluates retrieval with 1,000 queries spanning farmer-anchored, KG-grounded, and safety categories. Its construction emphasizes source grounding, independent verification, expert auditing, and explicit limitations around image-linked content.

  • Corpus construction: 284 Bengali agricultural PDFs from five organizations span 25 years, five document genres, 915 crops, 704 disease variants, and 2,729 chemical or pesticide entities.MoA and DAE contribute approximately 65% of the corpus, while BRRI, IRRI, and SRDI supply targeted diagnostic manuals.
  • Knowledge nodes: 2,882 topic-coherent knowledge nodes preserve natural-language content, structured agricultural facts, and provenance fields linking each unit to its source, page, and organization.Nodes average ≈1,180 characters and span a 13-category agricultural taxonomy.
  • Verification and audit: 9.8% of nodes, or 283 of 2,882, initially fell below the verification threshold and were regenerated with verifier error traces until passing without discarding agricultural content.Automated verification compared generated nodes against ground-truth source passages, while three expert annotators audited 200 nodes; 12 of 200 nodes (6.0%) were corrected.
  • Multimodal extension and limitation: 1,022 of 2,882 nodes (35.5%) carry linked visual assets, but image references and bilingual descriptions are not separately indexed or evaluated in the retrieval experiments.The linkage is intended to address farmer descriptions of observable symptoms rather than formal entity names, including the reported R@10=0.093 on farmer queries.
  • Query benchmark: 1,000 queries are organized into 400 farmer-anchored, 400 KG-grounded, and 200 safety queries, with 900 answerable queries included in retrieval evaluation.Farmer-anchored queries retain 300 of 400 after majority-vote mapping, whereas all 400 KG-grounded and 200 safety queries are included.

4 Experimental Design

The experiments isolate architecture, embedding model, and language-condition effects in Bengali agricultural retrieval, using multiple architectures, embeddings, query conditions, and a configuration audit. Evaluation uses strict provenance-matched retrieval metrics with bootstrap intervals and corrected paired significance tests.

  • Research questions: The study evaluates five retrieval architectures to identify Bengali leaders, language-boundary effects, query-type moderation, and embedding-choice effects, with a configuration audit for implementation artifacts.These four research questions organize the primary experiments and test whether aggregate comparisons conceal retrieval failure modes.
  • Retrieval architectures: The five systems span BM25, Dense (Gemini), Dense (BGE-M3 Native), ColBERT, and Hybrid RRF retrieval families with architecture-specific configurations.BM25 uses Okapi weighting and character-bigram plus full-word tokenization, while Dense (Gemini) uses normalized 3,072-dimensional vectors and exact inner-product search.
  • Embedding isolation: Six embedding models spanning 384–4,096 dimensions are tested on the same Bengali queries under identical retrieval code, varying embedding choice alone.This design separates embedding quality from architectural effects.
  • Evaluation protocol: R@1, R@5, R@10, MRR, and nDCG@10 are reported for 900 answerable queries, counting hits only when retrieved node identifiers exactly match provenance-verified gold nodes.The random R@10 baseline for 2,882 corpus nodes is 10/2,882 ≈0.003.
  • Statistical analysis: BCa bootstrap confidence intervals use 10,000 resamples, while paired Wilcoxon tests apply Holm-Bonferroni correction across 10 architecture pairs.Per-category and per-embedding results are reported descriptively, with significance levels and 95% confidence intervals reported inline.

5 Results

Results show that retrieval performance depends strongly on architecture, query register, language alignment, embedding choice, and configuration. No single method is consistently best, while implementation settings can substantially alter reported rankings.

  • Overall retrieval performance: Hybrid RRF achieves the highest overall R@10 (0.539), while BM25 leads single-method retrievers with R@10=0.506.BM25 outperforms dense Gemini-001 (0.464) by 9% (p<0.001).
  • Language-condition effects: BM25’s cross-lingual R@10 falls from 0.506 to 0.004 for English queries over the Bengali corpus, a 99% drop.Within-language results instead favor BM25 in Bengali (0.506 vs. 0.464) and Dense in English (0.442 vs. 0.384).
  • Query-category performance: Dense Gemini retrieval reaches R@10=0.093 on colloquial farmer queries but 0.970 on formal safety queries.The disparity reflects the register gap between symptom-based farmer language and formal advisory terminology.
  • Embedding-model comparison: Gemini-001 achieves R@10=0.464, outperforming larger Qwen3-8B (0.241), while BGE-M3 reaches higher mean cosine separation (+0.133) but lower Recall@10 (.408).Retrieval-specific training matters more than embedding dimensionality, and mean separation does not necessarily predict top-k recall.
  • Configuration sensitivity: Using a sentence-similarity task type instead of RETRIEVAL_QUERY/RETRIEVAL_DOCUMENT reduces Gemini Dense R@10 from 0.464 to 0.063, a 7× drop.The audit shows that embedding task configuration independently changes reported retrieval performance.
  • Configuration sensitivity: At 128 tokens, ColBERT reaches R@10=0.376 versus BGE-M3’s 0.408, but increasing context to 512 tokens lifts ColBERT to R@10=0.487.The default truncates Bengali knowledge nodes by 95–99%, reversing the ranking between these methods.

6 Analysis of Retrieval Failures

Retrieval failures reflect a near-universal lexical gap between farmer queries and formal agricultural documents, with dense retrieval especially weak on colloquial queries. BM25 and dense retrieval fail on different query registers, explaining why hybrid fusion can recover complementary cases.

  • Query-document register gap: 96.4% of queries have Jaccard < 0.10 with their gold document, with mean 0.044 and maximum 0.172.The lexical gap shows that benchmark evaluation requires semantic matching rather than direct surface overlap.
  • Query-document register gap: Safety queries achieve mean Jaccard 0.055 versus 0.034 for farmer queries, a 1.6× higher alignment.Farmer queries describe observable symptoms, while authoritative documents use formal scientific terminology.
  • Query-document register gap: 3.5% of gold document entity names appear verbatim in farmer queries, while 92% are entirely absent, coinciding with Dense R@10=0.093 on colloquial queries.This entity mismatch helps explain dense retrieval’s collapse on colloquial farmer queries.
  • Complementary retrieval failures: BM25 remains comparatively stable across registers—farmer 0.523, safety 0.539, KG-grounded 0.478—whereas Dense is sharply bimodal: farmer 0.093 and safety 0.970.Their strongest and weakest performance occurs on largely different query registers, producing complementary failure patterns that support Hybrid RRF.

7 Discussion

Aggregate retrieval scores conceal major failure modes in low-resource agricultural advisory, while the benchmark’s scope, annotation artifacts, and safety-critical use require cautious interpretation and human oversight.

  • Evaluation findings: Dense retrieval R@10 drops from 0.970 on formal safety queries to 0.093 on colloquial farmer queries, exposing substantial query-type variation hidden by aggregate scores.The discussion corroborates concurrent findings on stratified retrieval evaluation.
  • Evaluation findings: BM25 leads within Bengali, whereas dense retrieval leads under cross-lingual conditions.
  • Limitations: The evaluation covers agricultural advisory and first-stage zero-shot retrievers, leaving learned sparse models and cross-encoder rerankers for future work.Machine-translated English queries achieved 97% back-translation equivalence on a 100-query random sample, but validation was not expanded to the full corpus.
  • Limitations: A human audit found 12 nodes requiring correction among 200, so some unresolved failure queries may reflect benchmark annotation artifacts.
  • Limitations: The benchmark spans source documents from 1999–2024 and tests retrieval of authoritative sources, not whether recommendations remain current.Image-linked nodes are not separately embedded or evaluated as a distinct retrieval modality.
  • Ethics and broader impact: Retrieval accuracy does not certify advisory correctness for unsupervised deployment, so high-stakes systems should retain human oversight.Annotations were completed by three compensated domain experts with informed consent, and the benchmark contains no personally identifiable information.

8 Conclusion

The paper presents a provenance-grounded Bengali agricultural retrieval benchmark built from authoritative government publications. Its experiments show that retrieval behavior depends jointly on linguistic register, language boundary, and implementation configuration, not retrieval architecture alone.

  • Benchmark: The benchmark uses canonical knowledge nodes and image-linked resources extracted from authoritative government publications.It is designed as a provenance-grounded resource for Bengali agricultural retrieval.
  • Evaluation: Evaluation is stratified by register and language to support reproducible analysis of Bengali agricultural retrieval.The benchmark combines provenance grounding with register- and language-stratified evaluation.
  • Findings: Retrieval behavior depends jointly on linguistic register, language boundary, and implementation configuration rather than retrieval architecture alone.This conclusion identifies multiple interacting determinants of retrieval performance.

A Reproducibility Checklist

The reproducibility checklist specifies the released dataset, retrieval configurations, audit variants, and statistical procedures used in the evaluation.

  • Data and release: The release contains 284 source PDFs, 2,882 knowledge nodes, and 1,000 queries, including 900 verified gold-node mappings and 100 low-agreement queries.The dataset is available at https://huggingface.co/datasets/RaiyanKhaan/AgriTrust-RAG.
  • Retriever configurations: BM25 uses Okapi parameters k_1=1.5 and b=0.75 with char-bigram and full-word tokenization.
  • Retriever configurations: Gemini dense retrieval uses gemini-embedding-001 with 3,072 dimensions and asymmetric RETRIEVAL_QUERY/RETRIEVAL_DOCUMENT task types.
  • Retriever configurations: BGE-M3 uses its native 1,024-dimensional L2-normalized projection head, while ColBERT uses 512-token sequences and MaxSim scoring.
  • Audits and statistics: Hybrid RRF uses k=60 and top-100 candidates per method; audits vary sentence-similarity task type and 128-token truncation, with BCa bootstrap and corrected paired Wilcoxon tests.The bootstrap uses 10,000 resamples, and Holm-Bonferroni correction is applied to the paired Wilcoxon signed-rank tests.

B Node Representation

Each knowledge node uses a three-layer representation combining natural-language content, structured facts, and deterministic provenance. The canonical example stores Bengali advisory content alongside structured symptoms, management, prevention, and entity fields, with bracketed values showing English translations.

  • Three-layer structure: The node schema has three layers: natural-language content, structured facts, and deterministic provenance.A representative canonical node is shown in trimmed form, with the full schema released with the benchmark code.
  • Canonical node fields: The canonical example records a Bengali title, summary, key points, symptoms, management actions, prevention guidance, and chemical entities.Its example concerns chickpea wilt control, including seed treatment, resistant varieties, fungicide spraying, and infected-plant management.
  • Language representation: Bracketed field values are English translations of the stored UTF-8 Bengali text.The underlying node content remains stored in Bengali, while the displayed bracketed values provide translations.

C Image-Linked Node Structure

Image-linked nodes retain the standard node schema while adding an image_refs field containing descriptive metadata such as captions and image types. Of 2,882 nodes, 1,022 (35.5%) include image references, which are treated as structural resources rather than independently verified evaluation assets.

  • Schema design: Image-linked nodes extend the standard schema with an image_refs field rather than introducing a separate node type.The remaining standard fields, including node_id, category, title_bn, and _provenance_layer, remain unchanged.
  • Reference metadata: image_refs entries include caption_en and image_type metadata, such as diagnostic_photo.The example caption describes a chickpea leaf showing early wilt symptoms.
  • Coverage and validation: 1,022 of 2,882 nodes (35.5%) carry one or more image references.These references have not undergone the three-annotator human audit applied to node text.
  • Coverage and validation: Image references are reported as a structural resource, not an independently verified evaluation asset.This limitation distinguishes image references from the audited node text evaluation resource.

D Node Extraction Protocol

Canonical benchmark nodes are created through a two-stage generative pipeline bounded by deterministic checks. The process extracts explicitly mentioned entities, generates semantically bounded JSON nodes, verifies entity fidelity, and injects provenance deterministically.

  • D Node Extraction Protocol: Stage 1 extracts crops, diseases_and_pests, and chemicals into a temporary registry using exact surface-form matching and no hallucination.Missing categories produce empty lists, and the output must be valid JSON only.
  • D Node Extraction Protocol: The extraction prompt restricts entities to those explicitly mentioned in the Bengali source text and forbids inferred or external knowledge.Entities must be copied verbatim as they appear in the text.
  • D Node Extraction Protocol: Stage 2 converts each Markdown passage into structured JSON node(s) with Gemini-3.1-Flash-Lite, using 1-to-N dynamic chunking and semantic bounding.One coherent topic maps to one node, while distinct sub-topics can be separated.
  • D Node Extraction Protocol: A deterministic script verifies that Stage 1 entities exactly match those in generated nodes, after which provenance metadata is injected deterministically.Generated content must remain grounded in the specific provided document, with unsupported fields left blank.
Loading 2608.14886v1…