PaperlayerView exact revision on arXiv

Source-linked AI summary

Tool-calling retrieval versus vector RAG for a small Greek--English knowledge base: accuracy and robustness to how users type Greek

Nikolaos D. Tantaroudas, Ilias Karachalios, Andrew J. McCracken

arXiv:2610.08205v1cs.CEcs.AI

TL;DR

For remote-island farmers, useful LLM advice needs authoritative local information, and trust in an assistant’s answers without human verification was limited. The paper compares tool retrieval and vector RAG on KyGround, a Greek–English agricultural benchmark with varied query forms. Vector RAG answered canonical Greek questions more accurately, while tool losses arose in retrieval.

  • Problem

    For remote-island farmers, LLM advice needs to be anchored in authoritative local information, and fewer than a quarter of surveyed assistant users trusted its answers without human verification.

    Source: §1 · pp. 1–2

  • Method

    The study compares tool retrieval and vector RAG on KyGround, a benchmark of 198 questions with automatically verified answers posed in up to nine forms.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    95% of canonical Greek questions were answered correctly by vector RAG, versus 72% by tool retrieval; the tool agent’s losses arose in literal search when router arguments did not match records.

    Source: §5 · p. 9, §6 · p. 10

  • Takeaways & Limitations

    Tool interfaces for small community knowledge bases should tolerate users’ spellings, including case and accent variation and inflected tokens; Greeklish remains difficult for both designs.

    Source: §5 · p. 9

    The accuracies estimate the designs with Claude Haiku 4.5 rather than the deployed system, and C2 is a reconstruction of that system.

    Source: §5 · p. 9

Abstract

from arXiv · show

Assistants grounded in a small, frequently edited knowledge base can retrieve through tool calls to a live data interface or through vector retrieval-augmented generation (RAG). We compare the two on KyGround, a benchmark of 198 questions drawn from the published records of a Greek--English agricultural platform on Kythera, Greece, with answers verified automatically against the records and each question posed in up to nine forms, including Greek without accents, in capitals and in three Latin-script (Greeklish) schemes. With Claude Haiku 4.5 as router and answer model, a reconstruction of the platform's tool agent answered 71.6\% of canonical Greek questions correctly and vector RAG 95.3\% (difference $-23.6$ percentage points, 95\% CI $-33.1$ to $-15.1$). Letting the router write the vector query changed nothing, and placing the whole knowledge base of about 26,000 tokens in the prompt reached 99.3\%. The tool agent's losses arose in retrieval. Its literal searches returned nothing when the router's arguments did not occur verbatim in a record, for example when it transliterated Greek into Latin script or combined words that occur in a record but not as one phrase, and the agent then abstained. Unaccented and capitalised questions cost the tool agent about 20 points and vector RAG at most 2; accent-insensitive search removed this loss, and matching stemmed tokens raised the tool agent to 83.8\% on canonical Greek. Greeklish cost both designs about 21 to 32 points. Tool interfaces for community knowledge bases need search that tolerates how users type.

1 Introduction

The paper tests the accuracy claim for tool-calling retrieval against vector RAG, focusing on whether retrieval remains dependable when users type Greek in varied ways. It introduces a benchmark, controlled comparisons, and a search-normalisation ablation.

1 Introduction

  • KyGround provides 198 record-based questions with automatically verified gold answers, each posed in up to nine surface forms and scored deterministically.The benchmark is designed to compare retrieval approaches while testing variation in how users type Greek.Source: §1 · p. 2
  • The six-condition comparison shares one router and answer model, using router-written vector queries and whole-knowledge-base prompting as controls.These controls are intended to locate where the difference between retrieval designs arises.Source: §1 · p. 2
  • An ablation of free-text search normalisation measures how much of the tool agent’s deficit can be engineered away.The paper tests changes to how the data interface matches search terms.Source: §1 · p. 2

2 Related work

Prior work describes vector retrieval and tool calling as distinct ways to ground language models in external information. The paper situates its comparison in the absence of a head-to-head Greek evaluation that accounts for users’ orthographic variation.

2 Related work

  • RAG couples a generator with a retriever over non-parametric memory, usually dense passage retrieval [Lewis et al., 2020, Karpukhin et al., 2020]; ReAct interleaves reasoning with API-call actions, and Toolformer learns when and how to call tools.Source: §2 · p. 2
  • Tool parameters can be written in the user’s language, which may break interfaces expecting a fixed one.Tool choice also depends on tool descriptions [Hasan et al., 2026].Source: §2 · p. 2
  • Greek users type without accents, in capitals, and in Greeklish, which has no single mapping and can be difficult even for native speakers.The paper notes that informal phonetic and visual schemes dominate over formal transliteration [ISO, 1997].Source: §2 · p. 2
  • The authors know of no prior head-to-head comparison of tool-calling and vector retrieval in Greek that includes users’ orthographic variation.Source: §2 · p. 3

3 Methods

The methods pair a frozen Kythera knowledge-base snapshot with a verified benchmark, then compare retrieval conditions using a shared model and scoring protocol. The protocol also specifies how generation, accuracy, and robustness are assessed.

3.1 Knowledge base and benchmark

  • 3.1 Knowledge base and benchmark: the frozen bilingual snapshot contains 103 published records, and KyGround provides 198 questions with automatically verified answers across varied forms.The primary analysis uses 148 answerable test items in 79 clusters; the release contains 1,469 queries. Table 1 gives surface forms of an olive-harvest question and its gold answer, November to January.Source: §3.1 · p. 3, Table 1 · p. 3

3.2 Conditions

  • 3.2 Conditions: all six conditions use Claude Haiku 4.5 as router and answer model, with tool retrieval, vector RAG, and controls that vary retrieved evidence.C1q uses the same router, prompt, and function-calling mechanism as C2, but retrieves through vector search; C4 places the whole knowledge base in the prompt. Table 2 identifies the shared model, prompt, history policy, and snapshot.Source: §3.2 · pp. 3–4, Table 2 · p. 4

3.3 Generation

  • 3.3 Generation: batched replies were audited for cross-request facts, and flagged replies were regenerated in isolation.The protocol, hypotheses, and analysis plan were fixed before runs with a real model, but the study was pre-specified rather than pre-registered.Source: §3.3 · p. 4

3.4 Scoring and statistics

  • 3.4 Scoring and statistics: a deterministic scorer normalises and matches answers, while accuracy intervals use cluster bootstrapping and pre-specified hypotheses use cluster-level sign-flip tests with Holm correction.A second model served as a scorer, and controlled answers were used to test scorer performance.Source: §3.4 · p. 5

4 Results

On canonical Greek questions, vector RAG outperformed the tool agent, whose shortfall was mainly retrieval-related. Surface-form losses were partly reduced through search normalization, while Greeklish remained difficult for both systems.

4.1 Accuracy on canonical questions

  • 71.6% for C2 versus 95.3% for C1 on canonical Greek questions; the paired difference was −23.6 percentage points (95% CI −33.1 to −15.1).Letting the router write the vector query did not change accuracy, while adding vector-RAG chunks to tool results reached 96.6% and placing the whole knowledge base in the prompt reached 99.3%.Source: Table 3 · p. 5, §4.1 · p. 5

4.2 Where the tool agent fails

  • 75.7% of C2 answers retrieved the gold record, versus 98.6% for C1; C2’s shortfall consisted mainly of abstentions after searches returned no relevant record.For canonical Greek questions, 28.7% of router arguments were written in Latin script, and 46.9% of those returned no record, versus 26.2% of arguments written in Greek.Source: §4.2 · pp. 5, 7

4.3 Robustness to Greek surface forms

  • Unaccented questions lowered C2 accuracy by 23.6 pp and C1 accuracy by 0.7 pp; Greeklish lowered both systems’ accuracy by 31.1–31.8 pp for C2 and 20.9–27.0 pp for C1.Colloquial and regional wording changed C2 accuracy by +5.2 pp and C1 by −6.9 pp; the seven other-inflection items were too few for inference.Source: §4.3 · pp. 7–8, Table 4 · p. 7

4.4 Search normalisation

  • Accent folding removed C2’s losses on unaccented and capitalised Greek and raised canonical Greek accuracy from 71.6% to 73.6%; matching all stemmed tokens raised it to 83.8%.Stemmed-token matching raised accuracy on questions that did not quote a record title from 46.5% to 72.1%, but neither mode helped much with Greeklish; even with stemmed tokens, C2 remained 11.5 pp below C1 on canonical Greek.Source: §4.4 · p. 8, Table 5 · p. 8

4.5 Unanswerable questions, prices and context size

  • 0 of 20 answers in every condition gave an amount for a price shown only on request; the deterministic scorer counted at most 2 of 20 unanswerable answers per condition as hallucinations.All nine such answers were abstentions in wordings not covered by the scorer’s cue lists; per-question input averaged 1,612 tokens for C1 and 26,294 for C4.Source: §4.5 · p. 8

4.6 Validation of scoring and of batched generation

  • 97.4% of 1,248 judged test answers agreed between the deterministic scorer and the LLM judge (κ = 0.92), and judge grades led to the same conclusions.The contamination correction changed primary contrasts by less than one point, but a residual inflation of a few points among unflagged replies could not be excluded.Source: §4.6 · pp. 8–9

5 Discussion

Vector retrieval outperformed tool retrieval in the tested setup, with the controls locating the gap in retrieval rather than answer generation. Literal matching and Greek typing variation explain key tool-agent failures, while interface changes can recover some accuracy.

5 Discussion

  • Vector RAG answered the small, community-edited knowledge base more accurately than tool retrieval, and the controls locate the difference in retrieval.C1q changed nothing when given the same router, prompt, and function-calling mechanism as C2; C4 showed the answer model answered almost every question when evidence was present.Source: §5 · p. 9
  • The tool agent abstained when router arguments did not occur verbatim in records, including transliterated keywords or words not present as one phrase.The router transliterated Greek into Latin script even though the questions and records were Greek.Source: §5 · p. 9
  • Case- and accent-insensitive search, final-sigma handling, and stemmed-token matching removed losses on unaccented and capitalised questions and recovered about half the canonical-Greek gap.The router wrote 28.7% of its free-text arguments in Latin script for canonical Greek questions despite a tool description asking for terms preferably in Greek; Greeklish remained hard for both designs.Source: §5 · p. 9
  • The study uses a reconstructed tool agent and one model, and its record-derived questions may favor literal matching over users’ own wording.The questions were derived from records, and 105 of 148 primary questions quote an exact record title; the authors note the tool agent’s accuracy on users’ wording may therefore be lower than reported.Source: §5 · p. 9

6 Conclusions

For the tested small Greek–English knowledge base, vector RAG was more accurate than tool retrieval. Literal search and Greeklish remain challenges, while typing-tolerant search is a practical concern.

6 Conclusions

  • 95% of canonical Greek questions were answered correctly by vector RAG, compared with 72% by tool retrieval.Source: §6 · p. 10
  • The tool-retrieval gap arose in literal search, widened for unaccented and capitalised Greek, and was partly closed by accent-insensitive, stemmed matching.Source: §6 · p. 10
  • Greeklish remained difficult for both designs, and tool interfaces should be evaluated using the spellings users actually type.Source: §6 · p. 10

Data and code availability

The material is available on request for specified research purposes, with restrictions on redistribution and use.

Data and code availability

Related papers

Loading 2610.08205v1…