Source-linked AI summary
A-RAG: Scaling Agentic Retrieval-Augmented Generation via Hierarchical Retrieval Interfaces
Mingxuan Du, Benfeng Xu, Chiwei Zhu, Shaohan Wang, Pengyu Wang, Xiaorui Wang, Zhendong Mao
TL;DR
Existing RAG systems rely on one-shot retrieval or predefined workflows, limiting models’ participation in retrieval decisions. A-RAG exposes hierarchical retrieval interfaces through keyword search, semantic search, and chunk read, and experiments show consistent improvements across benchmarks with efficient scaling under stronger models and more test-time computation.
Problem
Existing RAG paradigms do not allow models to adapt retrieval workflows, choose interaction strategies, or decide when sufficient evidence has been gathered.
Method
A-RAG exposes hierarchical retrieval interfaces that let agents access corpus information at keyword, sentence, and chunk levels.
Results
A-RAG consistently outperforms existing Graph-RAG and Workflow RAG methods across diverse benchmarks and improves steadily with increased test-time computation.
Takeaways & Limitations
The findings support designing agent-friendly retrieval interfaces and exploring new interaction paradigms between language models and external knowledge sources.
Takeaways & Limitations
The study does not exhaustively compare tool designs or tool subsets, and it lacks validation on larger models such as GPT-5 and Gemini-3.
Abstract
from arXiv · showhide
Frontier language models have demonstrated strong reasoning and long-horizon tool-use capabilities. However, existing RAG systems fail to leverage these capabilities. They still rely on two paradigms: (1) designing an algorithm that retrieves passages in a single shot and concatenates them into the model's input, or (2) predefining a workflow and prompting the model to execute it step-by-step. Neither paradigm allows the model to participate in retrieval decisions, preventing efficient scaling with model improvements. In this paper, we introduce A-RAG, an Agentic RAG framework that exposes hierarchical retrieval interfaces directly to the model. A-RAG provides three retrieval tools: keyword search, semantic search, and chunk read, enabling the agent to adaptively search and retrieve information across multiple granularities. Experiments on multiple open-domain QA benchmarks show that A-RAG consistently outperforms existing approaches with comparable or lower retrieved tokens, demonstrating that A-RAG effectively leverages model capabilities and dynamically adapts to different RAG tasks. We further systematically study how A-RAG scales with model size and test-time compute. We will release our code and evaluation suite to facilitate future research. Code and evaluation suite are available at https://github.com/Ayanami0730/arag.
1 Introduction
Existing RAG systems use static retrieval or predefined workflows that limit model autonomy. A-RAG addresses this gap with hierarchical retrieval interfaces and shows consistent gains across benchmarks and scaling conditions.
- Existing RAG methods either retrieve passages in one shot or prompt models to execute predefined workflows step by step.
- These paradigms prevent models from adapting workflows, choosing interaction strategies, or deciding when sufficient evidence has been gathered.
- Even Naive Agentic RAG with a single embedding-based retrieval tool consistently outperforms Naive RAG and previous baselines in preliminary experiments.
- A-RAG exposes hierarchical retrieval interfaces that let agents access corpus information across multiple granularities and spontaneously generalize to task-specific workflows.
- A-RAG substantially surpasses prior methods across multiple benchmarks, while its performance improves steadily with increased test-time computation.
- The paper introduces A-RAG and argues that multi-granularity tools are essential for unlocking stronger model performance.
- The framework further analyzes scaling across model capabilities and test-time computation.
2 Related Work
Related work spans graph-based, workflow-based, training-based, and multi-agent approaches to retrieval-augmented generation. A-RAG is positioned as an agentic paradigm defined by autonomous strategy, iterative execution, and interleaved tool use.
- A-RAG compares Graph RAG, Workflow RAG, and Agentic RAG using autonomous strategy, iterative execution, and interleaved tool use as principles of agentic autonomy.
- Graph RAG constructs entity-relation graphs to support holistic understanding of large-scale knowledge bases.
- Workflow RAG methods prompt models to execute predefined retrieval procedures step by step, sometimes using supervised fine-tuning or reinforcement learning.
- Training-free workflow methods include confidence-triggered retrieval, interleaved reasoning and retrieval, and iterative self-feedback for query decomposition.
- Multi-agent approaches coordinate specialized agents or combine hybrid retrieval with citation tracking for question answering.
3 Methodology
A-RAG exposes hierarchical retrieval interfaces through which an agent can iteratively search at keyword and sentence levels, read full chunks, and autonomously control retrieval. Its lightweight index and simple agent loop support incremental, on-demand evidence gathering while reducing redundant context.
- Framework overview: A-RAG combines a hierarchical index, retrieval tools, and a simple agent loop to expose corpus information at multiple granularities.The index organizes information through chunking and sentence-level embeddings, while keyword matching remains implicit at query time.
- Retrieval interfaces: The agent uses keyword search, semantic search, and chunk read to choose retrieval strategies suited to each question.Keyword search performs exact lexical matching, semantic search finds dense-retrieval matches, and chunk read returns complete or adjacent chunk content.
- Retrieval interfaces: Keyword search ranks chunks using keyword frequency and character length, then returns top-k chunk IDs with snippets containing matched terms.Longer keywords receive greater weight, and snippets extract sentences containing at least one query keyword.
- Retrieval interfaces: Semantic search embeds a natural-language query, ranks sentence matches by cosine similarity, aggregates them by parent chunk, and returns matched snippets.Each chunk’s relevance is determined by its highest-scoring sentence, allowing the agent to decide whether further reading is needed.
- Agent loop: The agent incrementally retrieves information on demand and reads full chunks only after inspecting search snippets, preserving flexibility while minimizing context overhead.The agent may also read adjacent chunks when additional context is needed.
- Agent loop: A ReAct-like loop lets the model alternate reasoning and tool calls, while a context tracker prevents repeated chunk reads from consuming additional tokens.The design intentionally avoids parallel tool calling and other sophisticated orchestration to isolate hierarchical-interface effects.
4 Experiments
Experiments evaluate A-RAG on four multi-hop QA benchmarks against vanilla, graph-based, and workflow-based baselines. Naive A-RAG is a strong baseline, while the full hierarchical version performs best more consistently, especially with GPT-5-mini.
- Experimental setting: A-RAG is evaluated on HotpotQA, 2WikiMultiHopQA, MuSiQue, and GraphRAG-Bench using the same corpus and questions as LinearRAG.The study reports LLM-Acc and Contain-Acc for short-form datasets, but LLM-Acc only for GraphRAG-Bench’s long-form answers.
- Experimental setting: The comparison includes direct zero-shot inference, Naive RAG, graph-enhanced methods, workflow-based methods, and A-RAG variants.Backbones include GPT-4o-mini and GPT-5-mini, with Qwen3-Embedding-0.6B used for dense retrieval except in LinearRAG.
- Main results: Vanilla retrieval baselines remain robust, while existing Graph-RAG and Workflow RAG methods do not consistently outperform them across datasets.This pattern holds under the unified evaluation setting for both GPT-4o-mini and GPT-5-mini backbones.
- Main results: Naive A-RAG surpasses existing Graph-RAG and Workflow RAG methods on multiple datasets using only a single embedding-based retrieval tool.Its advantage becomes more pronounced with GPT-5-mini as the backbone.
- Main results: A-RAG (Full) achieves the best performance on 3 out of 5 datasets with GPT-4o-mini and superior results across all benchmarks with GPT-5-mini.The full configuration improves over both baseline methods and Naive A-RAG.
- Ablations: A-RAG (Full) achieves optimal overall performance in ablations, while removing keyword search, semantic search, or chunk read degrades results.The findings support the contribution of multi-granularity retrieval and progressive information acquisition.
5 Analysis and Discussion
A-RAG is analyzed through test-time scaling, context efficiency, and failure modes. Its performance improves with additional computation, while hierarchical retrieval supports efficient context use and shifts remaining errors toward reasoning chains.
- Test-Time Scaling Analysis: Increasing max steps and reasoning effort improves A-RAG performance on MuSiQue-300.From 5 to 20 steps, GPT-5-mini improves by approximately 8% and GPT-4o-mini by about 4%; increasing reasoning effort from minimal to high improves GPT-5-mini and GPT-5 by approximately 25%.
- Test-Time Scaling Analysis: GPT-5-mini benefits more from longer-horizon exploration than GPT-4o-mini when max steps increase from 5 to 20.The reported improvements are approximately 8% for GPT-5-mini and about 4% for GPT-4o-mini.
- Context Efficiency Analysis: A-RAG achieves superior accuracy while retrieving comparable or fewer tokens than traditional RAG methods.Retrieved-token counts are evaluated as a measure of context efficiency, with lower values indicating higher efficiency.
- Context Efficiency Analysis: A-RAG (Full) retrieves fewer tokens than A-RAG (Naive) while achieving higher performance, supporting hierarchical interface design.Progressive information disclosure gives the model greater autonomy while avoiding irrelevant content.
- Failure Mode Analysis: Most A-RAG failures arise from reasoning chain errors, with entity confusion the most common error within that category.Wrong retrieval strategies and question misunderstanding also account for substantial portions of reasoning-chain errors.
6 Conclusion
The paper presents A-RAG as an agentic RAG framework with hierarchical retrieval interfaces that let language models access corpus information at keyword, sentence, and chunk levels. Across diverse benchmarks, it reports consistent improvements over existing RAG methods and efficient test-time scaling.
- Conclusion: A-RAG exposes hierarchical retrieval interfaces that enable autonomous access to corpus information at keyword, sentence, and chunk levels.The conclusion frames agentic RAG as a paradigm shift and emphasizes agent-friendly interfaces.
- Conclusion: A-RAG consistently outperforms existing Graph-RAG and Workflow RAG methods across diverse benchmarks.The paper’s analysis also validates efficient test-time scaling behavior.
Limitations
The paper leaves several scope and evaluation gaps: tool configurations are not comprehensively ablated, larger frontier models are not fully validated, and generalization beyond multi-hop QA remains open.
- Limitations: A-RAG does not exhaustively compare tool subsets or their effects on agent behavior.A comprehensive ablation across diverse tool configurations is left for future work.
- Limitations: The framework has not been validated on larger models such as GPT-5 and Gemini-3 because of computational resource constraints.The authors anticipate stronger gains on frontier models, but empirical verification remains outstanding.
- Limitations: Generalization beyond multi-hop question answering remains to be investigated for tasks such as fact verification, dialogue systems, and long-form generation.The paper reports strong results on multi-hop QA benchmarks but does not establish performance on these other knowledge-intensive tasks.
Ethical Considerations
The study uses publicly available benchmarks previously curated and processed by prior research, does not collect new data or involve human subjects, and reports no additional ethical risks beyond those inherent to RAG systems.
- Ethical Considerations: The study uses publicly available benchmarks and involves neither new data collection nor human subjects.The datasets were previously curated and processed with appropriate ethical considerations.
- Ethical Considerations: The authors state that A-RAG introduces no additional ethical risks beyond those inherent to the underlying language-model RAG systems.
A Comparison of RAG Method Autonomy
A-RAG is the only compared method that fully satisfies all three criteria for agentic retrieval: autonomous strategy, iterative execution, and interleaved tool use.
- Autonomous strategy requires the LLM to dynamically choose and organize retrieval, decomposition, verification, or replanning without a fixed workflow or external decision rules.
- A-RAG is the only method that fully satisfies autonomous strategy, iterative execution, and interleaved tool use.
- Table 4 compares existing RAG methods across the three agentic dimensions using ✓, ✗, and ∆ to indicate satisfaction, non-satisfaction, or boundary cases.
B Baseline Reproduction Details
The evaluation reproduces baselines under a unified configuration and analyzes both the A-RAG agent loop and failure modes across Naive RAG and A-RAG settings.
- Baseline reproduction: All baseline methods use top-k=5 retrieval and max_tokens≥16384 to prevent reasoning truncation.
- Baseline reproduction: The reproduction section documents baseline configurations and identifies the analyzed incorrect cases as GPT-4o-mini Naive RAG and GPT-5-mini A-RAG settings.
- A-RAG agent loop: The A-RAG loop repeatedly gives the LLM access to tools, appends tool results to message history, and stops when it answers or reaches the iteration limit.
- Failure analysis: Naive RAG failures are categorized as model understanding, multi-hop retrieval, judge error, or insufficient top-k coverage.
- Failure analysis: A-RAG failures are categorized into reasoning-chain errors, judge errors, giving up, and missing corpus evidence.
- Failure analysis: Reasoning-chain errors are further divided into entity confusion, wrong strategy, question misunderstanding, and exceeding the retrieval budget.
D.3 Analysis
The analysis finds that agentic retrieval shifts errors from finding documents toward reasoning over retrieved evidence, while dataset characteristics produce different failure patterns and optimization priorities.
- Failure patterns: Approximately 50% of Naive RAG failures arise from multi-hop retrieval and insufficient top-k coverage, making document finding the central bottleneck.
- Failure patterns: 82% of A-RAG failures on MuSiQue are reasoning-chain errors, shifting the bottleneck to incorrect reasoning after documents are found.
- Failure patterns: Entity confusion is the largest secondary failure mode, occurring in 40% of MuSiQue cases and 71% of 2Wiki cases.
- Dataset differences: MuSiQue has 22% question-misunderstanding errors, whereas 2Wiki has 33% model-gave-up cases.
- Prompt and tool configurations: Naive RAG uses a direct-answer prompt without tool calling, while A-RAG configurations differ through their available retrieval tools and strategy descriptions.
- Prompt and tool configurations: The retrieval interfaces include embedding search, keyword search for exact matching, semantic search for meaning-based retrieval, and chunk reading for full document content.