Source-linked AI summary
Keyword search is all you need: Achieving RAG-Level Performance without vector databases using agentic tool use
Shreyas Subramanian, Adewale Akinfaderin, Yanyan Zhang, Ishan Singh, Mani Khanuja, Sandeep Singh, Maira Ladeira Tanke
TL;DR
The paper asks whether vector databases and semantic search add substantial value over agentic keyword search for document question answering. It systematically compares traditional RAG with tool-augmented keyword-search agents and finds that the agentic approach can perform competitively, while remaining subject to document-size, multimedia, and context-window constraints.
Problem
The paper examines whether vector databases and semantic search provide substantial additional value over simple agentic keyword search for document question answering.
Method
The study compares a vector-database RAG baseline with tool-augmented agents using keyword search, evaluated through LLM-as-a-Judge metrics across varied datasets.
Results
Agentic keyword search achieved comparable performance to traditional vector-database RAG, with 94.52% average faithfulness attainment across all datasets.
Takeaways & Limitations
The results support using agentic keyword search as a competitive alternative to traditional RAG without maintaining a standing vector database.
Takeaways & Limitations
The approach degrades with large documents, has restricted multimedia handling, faces context-window constraints, and may miss contextual nuances.
Abstract
from arXiv · showhide
While Retrieval-Augmented Generation (RAG) has proven effective for generating accurate, context-based responses based on existing knowledge bases, it presents several challenges including retrieval quality dependencies, integration complexity and cost. Recent advances in agentic-RAG and tool-augmented LLM architectures have introduced alternative approaches to information retrieval and processing. We question how much additional value vector databases and semantic search bring to RAG over simple, agentic keyword search in documents for question-answering. In this study, we conducted a systematic comparison between RAG-based systems and tool-augmented LLM agents, specifically evaluating their retrieval mechanisms and response quality when the agent only has access to basic keyword search tools. Our empirical analysis demonstrates that tool-based keyword search implementations within an agentic framework can attain over $90\%$ of the performance metrics compared to traditional RAG systems without using a standing vector database. Our approach is simple to implement, cost effective, and is particularly useful in scenarios requiring frequent updates to knowledge bases.
Introduction
LLMs have driven widespread adoption of language-based applications, but their standalone responses can lack factual accuracy, motivating systems that retrieve external knowledge. This study compares traditional RAG with tool-augmented agents to assess their retrieval and response-quality trade-offs.
- Introduction: RAG combines LLMs with external knowledge sources to improve factual accuracy and relevance while mitigating hallucinations.The approach also introduces challenges involving retrieval integration and knowledge-base maintenance.
- Introduction: Tool-augmented LLM agents offer an alternative by using external tools to retrieve information during question answering.The study frames this comparison around retrieval mechanisms and response quality.
- Introduction: The study systematically compares traditional RAG systems with tool-augmented LLM agents across their retrieval mechanisms and response quality.Its methodology includes RAG and agentic-framework implementations, LLM-as-a-Judge evaluation, and datasets from varied domains.
- Introduction: The paper concludes that agentic systems without a standing vector database can perform competitively with traditional RAG approaches.This conclusion motivates examining simpler retrieval alternatives for document question answering.
Related Work
Prior work presents RAG and tool-augmented agents as complementary retrieval paradigms with different capabilities, maintenance demands, and operational risks. The literature also lacks standardized benchmarks that adequately reflect diverse document formats and practical deployment conditions.
- Related Work: RAG research reports benefits from grounding, dense retrieval, and hybrid dense-sparse methods, including improved factual consistency and retrieval quality.These approaches are associated with reduced hallucinations and stronger document retrieval performance.
- Related Work: Tool-augmented LLMs dynamically access search engines, APIs, and databases, supporting current information and multi-turn tool interactions.Their flexibility is especially relevant when information changes frequently.
- Related Work: Tool-augmented systems can chain tools for complex reasoning, but their practical use may involve service dependencies and latency concerns.These considerations qualify their advantages over static retrieval implementations.
- Related Work: Existing evaluations lack comprehensive standardized datasets covering varied document formats and real-world retrieval scenarios.Current benchmarks often emphasize retrieval accuracy, response quality, latency, and maintenance costs without fully capturing diverse document-handling challenges.
- Related Work: RAG systems require substantial knowledge-base maintenance, whereas tool-augmented agents may reduce maintenance overhead but depend on external-service availability.The approaches therefore involve different infrastructure and reliability trade-offs.
Methodology
The study builds a reference vector-database RAG baseline and compares it with an agentic keyword-search approach using LLM-as-a-Judge metrics across datasets from various domains.
- Methodology: The evaluation compares a reference baseline vector-database RAG system with an agentic keyword-search approach.A high-level comparison of both pipelines is provided in Figure 1.
- Methodology: LLM-as-a-Judge metrics are computed with the RAGAS Python library on datasets spanning various domains.The implementation and evaluation code is associated with a GitHub repository.
appendix).
The appendix presents two document-question-answering pipelines and an agentic search procedure that iteratively uses shell commands to gather context and produce a final answer.
- appendix).: The comparison contrasts a red RAG pipeline with a blue agent-based pipeline for document question answering.Figure 1 provides the high-level pipeline comparison.
- appendix).: The procedure returns a final answer after successive observation, command execution, context updates, and stopping decisions.Its inputs are a user query, a source-file folder, and a maximum iteration parameter.
- appendix).: The agent begins with document metadata analysis, then iteratively searches files with commands such as rga and pdfgrep until it can update or finalize the answer.The loop continues while the iteration limit has not been reached and stops when a final answer is found.
Datasets
The study uses diverse, reproducible document datasets spanning technical complexity, domain specificity, and linguistic structure to compare vector-based RAG with agentic search.
- Datasets: The corpus comprises standard RAG benchmarks sourced from Llamahub, with source PDFs, questions, reference contexts, and answers.The datasets were selected to support reproducibility.
- Datasets: The datasets cover essays, technical papers, historical accounts, blockchain documentation, an LLM survey, and public-company filings.Examples include Paul Graham’s essays, Llama 2, AlexNet, Solana, an LLM survey, and FinanceBench.
Experiment 1: Baseline RAG Implementation
The baseline is a managed Amazon Bedrock RAG pipeline that embeds fixed-size document chunks into an OpenSearch Serverless index for retrieval.
- Experiment 1: Baseline RAG Implementation: The baseline uses an Amazon Bedrock Knowledge Base with Titan Text Embedding Model V2 and 1024-dimensional floating-point embeddings.Source documents are ingested into an associated OpenSearch Serverless index.
- Experiment 1: Baseline RAG Implementation: Documents are processed with fixed 300-token chunks and 20% overlap before retrieval.
Experiment 2: Agentic Search Framework
The proposed agentic framework uses an LLM with ReAct reasoning to plan and refine keyword and regex searches across documents instead of relying on a standing vector database.
- Experiment 2: Agentic Search Framework: The agent uses Anthropic Claude 3 Sonnet on Amazon Bedrock through LangChain, with a 200K context window and ReAct reasoning.All experiments use a temperature of 0.001.
- Experiment 2: Agentic Search Framework: Search begins with metadata analysis, then dynamically selects broad keyword searches or targeted regex patterns across one or more documents.The workflow supports successive context expansion and error handling.
- Experiment 2: Agentic Search Framework: The implementation provides PDF metadata, regex and multi-keyword search, and PDF-specific page-range and recursive search tools.
- Experiment 2: Agentic Search Framework: For each dataset, the agent answers a subset of questions, and its selected source segments are compared with ground-truth contexts.
Evaluation Methodology
The comparison evaluates baseline RAG and agentic search with RAGAS metrics measuring answer support, retrieval coverage, and answer accuracy, while accounting for repeated-run variance and FinanceBench’s structural complexity.
- Evaluation Methodology: The study uses RAGAS to compare traditional and LLM-as-a-judge metrics across the baseline RAG and keyword-search agent.
- Evaluation Methodology: Faithfulness measures factual consistency between generated answers and their supporting contexts.
- Evaluation Methodology: Context Recall measures how extensively the system retrieves chunks relevant to answering each query.
- Evaluation Methodology: Answer Correctness measures the generated answer’s factual accuracy against the ground-truth answer.
- Evaluation Methodology: Metrics are computed over multiple runs, but FinanceBench evaluates only answer correctness because complex tables and structures challenge context-chunk metrics.
Results
Across the evaluated datasets, the keyword-search agent generally approached the vector-based RAG baseline, while outperforming it on FinanceBench answer correctness.
- 94.52% average faithfulness attainment, 88.05% context-recall attainment, and 91.48% answer-correctness attainment were achieved relative to RAG across datasets.Attainment scores represent the agent’s percentage achievement compared with the RAG baseline.
- Context recall reached 99.62% and 98.71% of RAG performance on BlockchainSolana and the LLM Survey dataset, respectively, but fell to 70.56% on Llama2Paper.Performance therefore varied substantially by dataset and metric.
- The agent generally performed slightly below RAG but exceeded 88% average attainment on all three metrics without semantic search through a vector database.BlockchainSolana produced over 99% of baseline performance for context recall and answer correctness.
- PaulGrahamEssay produced lower attainment scores, indicating that interdisciplinary writing remains challenging for keyword-based contextual comprehension.The authors identify this variation as motivation for further improvement beyond keyword matching.
- 30.40% versus 24.24%: the agent improved FinanceBench answer correctness by approximately 6 percentage points over traditional RAG.The comparison covered a subset of FinanceBench documents, with consistency across multiple runs.
Agentic keyword search vs. Claude computer use
The study also compared the keyword-search agent with Claude’s Computer Use capability for document retrieval without a vector database.
- On seven FinanceBench questions, Computer Use consistently produced accurate and comprehensive answers through direct PDF interaction.The agent typically opened PDFs in Firefox, searched with ctrl + F, and captured relevant screenshots.
- Side-by-side comparisons found effectively similar results between the keyword-search agent and Computer Use.The authors nevertheless describe their agent as simpler to set up and reproduce.
- Computer Use introduced occasional failures between API calls, adding an operational challenge to the approach.
Conclusion
Agentic keyword search can approach vector-database RAG performance for document question answering without maintaining a standing vector database, while remaining useful for evolving or resource-constrained settings. The approach still faces limitations involving document scale, multimedia, context windows, ambiguity, and long-term knowledge retention.
- Conclusion: Over 90% of vector-based RAG performance metrics were attained by tool-augmented agents using simple keyword search, without maintaining standing vector databases.The approach uses command-line search tools and LLM reasoning rather than semantic search through a vector database.
- Conclusion: The method is particularly suited to rapidly changing information and settings where maintaining extensive vector databases is impractical.Its stated advantages include leveraging existing command-line tools and reducing maintenance requirements.
- Conclusion: Performance degrades with large documents and is constrained by restricted multimedia handling and limited context windows.The keyword-search approach also has difficulty capturing contextual nuances, though iterative refinement and semantic reasoning partly mitigate this issue.
- Conclusion: The implementation struggles with ambiguous queries, lacks long-term knowledge retention, and requires further privacy and ethical safeguards.Future work includes more automated and generalizable search strategies across document types.
- Conclusion: The agent uses terminal-based metadata inspection and keyword-search tools to locate and progressively expand relevant document context.The documented workflow includes metadata discovery, searches across files, and successive context expansion before answering.