Source-linked AI summary
Quantifying the Accuracy and Cost Impact of Design Decisions in Budget-Constrained Agentic LLM Search
Kyle McCleary, James Ghawaly
TL;DR
Agentic RAG systems face explicit tool-call and completion-token budgets, while evidence on how design choices jointly affect accuracy and cost under fixed budgets remains limited. This paper uses BCAS to compare search depth, retrieval configurations, and completion budgets across models and QA benchmarks. Accuracy generally improves with searches up to a small cap, while retrieval and completion-budget gains vary with task demands.
Problem
Fewer studies quantify how agentic RAG design choices jointly affect quality and cost under fixed search and token budgets across models and datasets.
Source: §1 · p. 1Method
Results
Across experiments, accuracy improves with search depth up to a small cap in most settings, hybrid retrieval with lightweight re-ranking yields the largest average gains in the HotpotQA ablation grid, and larger completion budgets matter most for synthesis-heavy HotpotQA.
Source: §6 · p. 8Takeaways & Limitations
The observed configuration order is to expand search depth first, improve retrieval evidence next, and raise completion limits when synthesis demands justify the added cost.
Source: §6 · p. 8The benchmarks use short final answers with known references, leaving citation fidelity, extended synthesis, interactive clarification, and user-specific constraints outside their scope.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Agentic Retrieval-Augmented Generation (RAG) systems combine iterative search, planning prompts, and retrieval backends, but deployed settings impose explicit budgets on tool calls and completion tokens. We present a controlled measurement study of how search depth, retrieval strategy, and completion budget affect accuracy and cost under fixed constraints. Using Budget-Constrained Agentic Search (BCAS), a model-agnostic evaluation harness that surfaces remaining budget and gates tool use, we run comparisons across six LLMs and three question-answering benchmarks. Across models and datasets, accuracy improves with additional searches up to a small cap, hybrid lexical and dense retrieval with lightweight re-ranking produces the largest average gains in our ablation grid, and larger completion budgets are most helpful on HotpotQA-style synthesis. These results provide practical guidance for configuring budgeted agentic retrieval pipelines and are accompanied by reproducible prompts and evaluation settings.
1. Introduction
The introduction identifies a measurement gap: agentic RAG design choices are less often evaluated jointly for accuracy and cost under fixed search and token budgets. The study uses BCAS to compare these choices across models and benchmarks and frames questions about model capacity, component tuning, and budget trade-offs.
1. Introduction
- Existing studies less often quantify how agentic design choices jointly affect quality and cost under fixed search and token budgets across multiple models and datasets.The introduction notes that many prior studies report accuracy gains for individual methods, often with permissive tool use.Source: §1 · p. 1
- The study compares hybrid search, lightweight re-ranking, pre-planning, reflection, and completion limits while holding budgets constant across six LLMs and three multi-hop QA benchmarks.The comparisons toggle one design decision at a time.Source: §1 · p. 1
- BCAS surfaces remaining search and token allowances and gates tool calls, while logging per-question resource consumption for cost reinterpretation.Source: §1 · p. 1
1.1. Research Questions
- The research questions ask how model size relates to retrieval performance and how agentic components can be tuned under a fixed search budget.Source: §1.1 · p. 1
- The study also asks how accuracy changes as search steps and generated-token budgets are tightened together in agentic RAG.The introduction motivates this question with the inference and training overheads of recent reinforcement-learning-driven systems.Source: §1.1 · pp. 1–2
2. Related Work
The paper situates BCAS amid the shift from static retrieval to prompted and increasingly autonomous search agents. Its contribution is a controlled measurement scaffold, not a new trained agent or benchmark state-of-the-art claim.
2. Related Work
- Prior work spans static retrieve-then-generate systems, prompted iterative reasoning and reflection, and newer agents trained for planning and query scheduling.Source: §2 · p. 2
- Many recent approaches assume generous tool use or require bespoke training, while BCAS makes inference-time constraints explicit and logs cost telemetry for shared-budget comparisons.Source: §2 · p. 2
- BCAS isolates design choices under capped searches and completion length rather than introducing a trained search agent or claiming benchmark state of the art.The paper characterizes its contribution as controlled systems measurement rather than leaderboard optimization.Source: §2 · p. 2
3. Methodology
BCAS is a model-agnostic evaluation scaffold that explicitly tracks search and token budgets while allowing controlled component comparisons. Its loop updates remaining resources as the model reasons, uses tools, and decides when to answer.
3. Methodology
- BCAS is used as a controlled evaluation scaffold to measure how search depth, retrieval components, and token limits affect accuracy and cost under explicit budgets.Source: §3 · p. 2
3.1. Architectural Principles
- BCAS exposes remaining search and token allowances, supports configurable retrieval, planning, and reflection, and uses plain-language tool prompts without fine-tuning.Source: §3.1 · p. 2
3.2. Prompt Policy
- A shared prompt template is used across model families without model-specific tuning, so cross-model differences reflect sensitivity to the common scaffold rather than best-case tuned performance.Source: §3.2 · p. 2
3.3. Budget Accounting
- The search cap is enforced as a hard tool constraint, and the loop stops when cumulative completion usage reaches the configured token limit.Token accounting uses provider-reported usage; the scheme does not model queueing delays, batching, or provider-specific pricing tiers.Source: §3.3 · pp. 2–3
3.4. The BCAS Execution Loop
- Each loop turn reasons, selects an available tool, observes its result, and updates the budget until the model answers or a budget is exhausted.Optional pre-planning builds a stepwise plan around available budgets, while reflection prompts progress review and strategic adjustment.Source: §3.4 · p. 3
3.5. Information Retrieval Components
The retrieval experiments compare BM25, hybrid lexical-dense search, and re-ranking while keeping the model context to five chunks. Evaluation spans six LLMs, and correctness is assessed with a binary LLM judge that the authors audit manually.
3.5. Information Retrieval Components
- The component ablations compare BM25 with phrase boosting, hybrid search, and re-ranking applied over BM25 or hybrid retrieval.The backend uses ParadeDB, a PostgreSQL extension.Source: §3.5 · p. 3
Re-ranking
- For HS+RR, the system retrieves 100 hybrid candidates, re-scores them with a cross-encoder, and keeps the top five chunks for context.The final context window stays at five chunks to keep search policy, rather than context expansion, as the focus of improvement.Source: §3.5 · pp. 3–4
Vector Embedding
- Hybrid search combines BM25 with dense vector search, using BGE-M3 embeddings and cosine similarity for document chunks and queries.Source: §3.5 · p. 3, Vector Embedding · p. 4
3.6. Evaluation
- The evaluation measures search-depth and completion-token scaling across six LLMs and also ablates retrieval components on unconstrained BCAS search.Source: §3.6 · p. 4, LLM Selection · p. 4
LLM Selection
- The six evaluated models span different size categories, capabilities, costs, and architectures, and all support tool use in BCAS tasks.Source: LLM Selection · p. 4
Grading
- GPT-4o-mini provides binary correctness labels, and a manual audit found 200/200 correct TriviaQA judgments, 198/200 HotpotQA judgments, and 197/200 2WikiMultihopQA judgments.The five discrepancies were false negatives; the authors treat judge labels as a proxy and describe reported accuracies as slightly conservative.Source: Grading · p. 4
3.7. Datasets
The evaluation spans three QA benchmarks and compares ablations, search depth, and context limits under controlled budget settings. The study also excludes o4-mini from context-scaling comparisons because its reasoning-token behavior could confound low-limit tests.
3.7. Datasets
- Three benchmarks range from single-fact retrieval to complex multi-hop reasoning, with each dataset’s documents kept in a separate retrieval collection.The datasets are TriviaQA, HotpotQA, and 2WikiMultihopQA; the same samples were used for each model within a dataset.Source: §3.7 · p. 4
3.8. Ablation Study
- The HotpotQA ablation compares five feature configurations against a BM25 baseline with unlimited searches and a 16,000-token limit.The full factorial ablation is reported only on HotpotQA, where multi-hop synthesis makes component interactions most visible; the authors do not claim its gain ordering is universal.Source: §3.8 · pp. 4–5
3.9. Search Scaling Study
- Search scaling tests 1, 2, 3, and unlimited searches with hybrid retrieval and 16,000 tokens, then adds planning and planning plus reflection at unlimited searches.Pre-planning and reflection are disabled in the core search-scaling configurations, isolating search-budget variation.Source: §3.9 · p. 5
3.10. Context Scaling Study
- Context scaling tests limits of 500, 1K, 2K, 4K, and 16K tokens with hybrid search and unlimited searches, excluding o4-mini.The authors exclude o4-mini because its completion-token limit includes internal reasoning and low limits could truncate responses mid-reasoning.Source: §3.10 · p. 5
4. Results
Across models and datasets, accuracy improves with additional searches up to about three, hybrid retrieval with re-ranking gives the most consistent component gains, and larger token budgets chiefly help multi-hop synthesis. The results compare model capacities and budget choices, with ablation effects reported specifically for HotpotQA.
4. Results
- Accuracy improves reliably up to about three searches, hybrid retrieval with re-ranking yields the most consistent component gains, and larger token budgets primarily help difficult multi-hop synthesis.Figure 2 presents search and context scaling; Table 1 gives accuracy across models, datasets, and budget configurations, while Table 2 reports HotpotQA component ablations.Source: §4 · p. 5, Table 1 · p. 6, Table 2 · p. 6
4.1. Performance Across Model Capacities (RQ1)
- On HotpotQA, Qwen 3 14B with unlimited searches and planning reaches 75.33%, above o4-mini’s single-search 70.17% but below its two-search 86.05%.The result illustrates how additional search can narrow the capacity gap; similar patterns are reported on TriviaQA and 2WikiMultihopQA.Source: §4.1 · p. 5
4.2. Budget-Aware Component Tuning (RQ2)
- +9.29 points is the average HotpotQA gain from re-ranking on top of hybrid search, compared with +6.36 for hybrid search alone.Planning and reflection improve smaller models by 4 to 12 points and have limited effect on o4-mini; Table 2 reports the component ablations.Source: §4.2 · p. 5, Table 2 · p. 6
4.3. Accuracy-Budget Trade-off (RQ3)
- Accuracy gains from additional searches plateau after roughly three; context scaling is nearly flat for TriviaQA, lifts HotpotQA from 4K to 16K tokens, and adds little for 2WikiMultihopQA.Figure 2 shows the search and context scaling trends, and Table 1 contains the per-configuration results.Source: §4.3 · p. 5, Figure 2 · p. 6, Table 1 · p. 6
5. Discussion
The discussion recommends allocating budget first to search depth, then retrieval components, and then generation limits, while emphasizing that the most useful allocation depends on task and model. The study supports configuration choices within its controlled setup, not a universal ranking of agents.
5. Discussion
- The recommended order is additional searches, hybrid retrieval with re-ranking, then larger generation limits mainly for multi-hop reasoning tasks.The authors say this ordering captures most observed gains while bounding compute.Source: §5 · p. 6
- The results support configuration decisions within a shared prompt scaffold, retriever stack, and budget rules, rather than a universal ranking of agents.The discussion frames the comparisons as design leverage under control, not a leaderboard across heterogeneous systems.Source: §5 · p. 6
5.1. Dataset-dependent Budget Leverage
- HotpotQA accuracy rises sharply from 4K to 16K tokens, while TriviaQA and 2WikiMultihopQA remain largely flat; the authors hypothesize that HotpotQA is more synthesis-limited and 2WikiMultihopQA more retrieval-limited.The discussion links these proposed difficulty differences to the datasets’ differing responses to search count and token allowance.Source: §5.1 · p. 7
- On 2WikiMultihopQA, 500 completion tokens yield 55.84% for GPT-4.1-mini and 27.68% for DeepSeek V3, versus 30.62% and 14.07% respectively with one search at 16K tokens.The authors attribute the pattern to succinct behavior under tight token limits enabling multiple searches, while generous single-search contexts can be consumed by verbose responses.Source: §5.1 · p. 7
5.2. The o4-mini Anomaly
- On HotpotQA, o4-mini gains less than 1.10 points from enhancement strategies, while combining planning and reflection raises its 2WikiMultihopQA accuracy from 38.12% to 63.20%.The authors hypothesize external planning is often redundant for this reasoning model, while reflection can support mid-search strategy revision on 2WikiMultihopQA.Source: §5.2 · p. 7
5.3. Component Hierarchy and Model Variety
- Hybrid retrieval with re-ranking gains 6 to 18 points for most models, while smaller models gain more from retrieval enhancements and planning than larger models.Combining pre-planning with reflection slightly degrades performance for some conventional models, suggesting adaptive re-planning can be counter-productive for them.Source: §5.3 · p. 7
5.4. Deployment Recommendations
- LLaMA 3.1 8B with three BM25 searches surpasses the single-search baselines of larger systems except o4-mini and GPT-4.1-mini; with planning and reflection it matches or exceeds every one-search model.For maximum accuracy, o4-mini with both planning strategies reaches 63.20% on 2WikiMultihopQA; the discussion advises matching enhancements to model capability.Source: §5.4 · p. 8
6. Conclusion
BCAS measures how search depth, retrieval configuration, and completion budgets trade off accuracy and cost across six LLMs and three benchmarks. The reported patterns suggest a configuration order for budgeted deployments, while future work would broaden the evaluation.
6. Conclusion
- BCAS quantifies accuracy-cost trade-offs for search depth, retrieval configuration, and completion budgets under explicit search and token limits.The study covers six LLMs and TriviaQA, HotpotQA, and 2WikiMultihopQA.Source: §6 · p. 8
- Accuracy improves with additional searches up to a small cap, hybrid retrieval with lightweight re-ranking yields the largest average gains in HotpotQA ablations, and larger completion budgets matter most for synthesis-heavy HotpotQA.Retrieval-easier settings show faster saturation.Source: §6 · p. 8
- For budgeted deployments, the suggested configuration order is to expand search depth, improve retrieval and re-ranking, then raise completion budgets when synthesis demands justify the added cost.Making budgets explicit and logging usage supports reproducible accuracy-cost analysis.Source: §6 · p. 8
6.1. Future Directions
- Future work should add direct non-agentic baselines, broaden component ablations beyond HotpotQA, and evaluate open-web and multimodal environments.It should also report queueing and throughput under production concurrency constraints.Source: §6.1 · p. 8
7. Limitations
The evaluation is bounded by its factual-QA benchmark setting, HotpotQA-only full component ablations, shared prompts across models, and approximate cost comparisons.
7. Limitations
- The study uses factual QA over static benchmark corpora, so its trends may not transfer directly to rapid knowledge drift, multilingual or multimodal evidence, or long-horizon interactive tasks.Benchmarks emphasize short answers with known references, omitting citation fidelity, extended synthesis, interactive clarification, and user-specific constraints.Source: §7 · p. 8
- The full component ablation is reported only on HotpotQA, and the exact ordering of component gains is not claimed to be universal across datasets.TriviaQA and 2WikiMultihopQA may shift the relative value of retrieval quality, search depth, and completion budget.Source: §7 · p. 8
- A single prompt template is shared across model families rather than tuned per model, which may under-optimize some models.Cross-model differences therefore indicate sensitivity to a shared scaffold rather than each model’s best-case tuned performance.Source: §7 · p. 8
- Binary LLM judging can miss partial correctness and nuances in reasoning quality, despite high agreement in the manual audit.Source: §7 · p. 8
- Cost and latency estimates depend on provider pricing, rate limits, and runtime settings, making cross-provider cost comparisons approximate.Token and search usage are reported as stable budget proxies alongside cost estimates.Source: §7 · p. 8
- The study omits both a pure single-pass retrieve-then-read baseline and head-to-head results against RL-trained open-web search agents.It therefore does not directly estimate multi-turn agency’s incremental value over one-shot retrieval; direct comparison with RL-trained agents would require aligned tools, corpora, and training conditions.Source: §7 · p. 8
8. Data and Code Availability
The study provides code, configurations, prompts, scripts, dataset split hashes, and evaluation outputs for reproducing its plots and tables under the same budget settings.
8. Data and Code Availability
- The BCAS repository makes the study’s code, evaluation harness configurations, prompts, and analysis scripts available.Source: §8 · p. 9
- Dataset split hashes and evaluation outputs are provided to support reproduction of the reported plots and tables.Source: §8 · p. 9
- Reproduction materials correspond to the study’s same budget settings.Source: §8 · p. 9