PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic RAG Evaluation: Budget Allocation Across Questions, Trajectories, and Reads

Jingjie Ning, Xueqi Li, Yibo Kong

arXiv:2610.05034v1cs.IR

TL;DR

Agentic RAG evaluation divides budgets among questions, search trajectories, and repeated answers, while each level affects comparison precision and cost. This study measures these tradeoffs in retrieval-feedback comparisons and tests audit-based forecasts across allocation subsets. Broader question coverage lowers standard error at similar model-token costs, while some fee rankings and deeper trajectory forecasts remain uncertain.

  • Problem

    Evaluation budgets span questions, search trajectories, and final answers, and applying established coverage and variance-based design principles requires measuring complete costs and testing whether audits predict precision as allocation counts change.

    Source: §1 · p. 2

  • Method

    The study measures realized precision and complete call costs for retrieval-feedback comparisons on HotpotQA and MuSiQue across near-equal-token allocations, using saved subsets to test forecasts.

    Source: §3.1 · p. 3, §4.1 · p. 4

  • Results

    A reduces measured standard error by 33.3% relative to B and 12.6% relative to C; at search prices of $0–1 per 1,000 requests, more questions beat more trajectories in fee efficiency, while question-versus-read fee rankings remain unresolved.

    Source: §5.2 · pp. 6–7, §6 · p. 9

  • Takeaways & Limitations

    For the tested Flash workflow at temperature 0.7, results support single-read evaluation and broader question coverage when allocating model tokens; trajectory extrapolation calls for an audit with more than two trajectories.

    Source: §6 · p. 9

    Evidence covers one policy comparison, one model family, one FineWeb endpoint, and two QA datasets; the 96-question Pro panel provides insufficient information for a reliable read-allocation default.

    Source: §6 · p. 9

Abstract

from arXiv · show

Evaluation budgets in agentic retrieval-augmented generation span questions, search trajectories, and repeated answers. We measure allocation precision, reading efficiency, and cost boundaries using a retrieval-feedback comparison on HotpotQA and MuSiQue. At 34.14--34.39M model tokens, broader question coverage lowers standard error by 33\% versus five reads and 12.6\% versus three trajectories. Archived nested and Q-only forecasts predict these allocations within 4.0\% and 3.5\%, respectively. Depth subsets establish no clear forecasting advantage beyond the two-trajectory audit. One-read variance penalties relative to the fitted optimum at the same token budget are 0--9.9\%, with substantial Pro uncertainty. Under recorded model fees, more questions beat more trajectories at search prices of \$0--1 per 1,000 requests; question-versus-read fee rankings remain unresolved. Temperature zero cuts answer disagreement from 14.3\% to 3.4\% while comparison precision stays similar. \par\medskip\noindent\textbf{Keywords:} Agentic RAG; Evaluation budget; Generalizability theory; Repeated sampling.

1 Introduction

Agentic RAG evaluations must divide resources among questions, search trajectories, and repeated answers, but this creates distinct precision and cost tradeoffs. This study measures those tradeoffs and tests whether prior audits predict precision as allocations change.

1 Introduction

  • Agentic RAG evaluation budgets span questions, search trajectories, and final answers, with variation at each level affecting policy-comparison precision.Evaluators can spend more on question coverage or repeated generations, with different consequences for uncertainty and cost.Source: §1 · p. 2
  • Applying established coverage and variance-based design principles requires measuring complete costs and checking whether an audit predicts precision as trajectory and read counts change.Agentic RAG adds distinct planning, retrieval, and reading costs.Source: §1 · p. 2
  • The study compares retrieval-feedback policies on HotpotQA and MuSiQue, measuring realized precision and complete call costs across near-equal-token allocations and saved subsets.Model and temperature controls assess reading efficiency.Source: §1 · p. 2
  • Broader question coverage yields 33% lower standard error than five reads at matched model-token cost, while fee comparisons depend on caching and search charges.The study reports a measured budget comparison and empirical guidance on sampling and cost boundaries.Source: §1 · p. 2

2 Related work

The related work supplies statistical foundations for allocation and repeated evaluation, alongside retrieval methods and diagnostics that motivate the paper’s policy comparison. The paper focuses on precision and cost for repeated paired policy evaluation.

2 Related work

  • Information-retrieval coverage studies and generalizability theory motivate variance-based allocation, cost rules, and reporting prediction error and uncertainty.Prior work also documents sensitivity of D-study predictions to the fitting sample.Source: §2 · p. 2
  • Repeated-evaluation research separates question and answer variation and supports paired inference, while this study targets precision for a paired four-query policy comparison [7] [8] [11] [12] [14] [15].The cited work also examines temperature effects, repeated runs, and answer compatibility under fixed single-turn policies.Source: §2 · p. 2
  • Retrieval and resource-allocation work motivates comparing relevance-feedback policies and explicit final-packet selection, while distinguishing retrieval quality from downstream benefit [1] [16] [17] [18] [20] [21] [22] [23] [24] [25] [26] [28] [29].The paper’s focus is the cost and precision of repeated policy evaluation.Source: §2 · p. 2

3 Study design

The study compares paired Open and Adaptive search policies using a shared first query and a five-document final packet. It evaluates nested question, trajectory, and read allocations on a separate panel while retaining a prior audit and model controls.

3.1 Paired search policies

  • The outcome is the Adaptive-minus-Open answer-F1 contrast, and the evaluation target is its standard error.Each policy pair shares its first query and search result; the final reader answers from five documents.Source: §3.1 · p. 3
  • Figure 2 presents paired policies and nested question panels, with repeated reads holding each packet fixed and identical packets sharing calls.RRF denotes reciprocal rank fusion and RR denotes round robin.Source: Figure 2 · p. 3
  • RRF and RR each select five records from a common pool; RRF sums rank-based contributions across distinct query lists, while RR cycles by rank and skips duplicates.The study uses FineWeb through DeepResearchGym [16] [30].Source: §3.1 · p. 3
  • Planner and reader use non-thinking requests, temperature 0.7, and a 256-token output cap; the exact model or index snapshot is unavailable.The routes are DeepSeek-V4.1-Flash and DeepSeek-V4-Pro-0813; prompts and recorded serving settings are in Appendix A.Source: §3.1 · p. 3

3.2 Evaluation panels and sampling allocations

  • A separate 446-question panel supplies A/B/C allocations of 446/2/1, 174/2/5, and 298/3/1 questions/trajectories/reads, respectively.The prior audit has 96 questions, two trajectories, and five reads; its questions and MuSiQue components are excluded from the new panel.Source: §3.2 · p. 4
  • Configurations, IDs, the primary selector-average contrast, and a 34.35M model-token target were fixed before calls; shared cells execute once across allocation analyses.The separate model control uses 96 questions with T = 2, R = 2, and the depth control uses 48 with T = 5, R = 2.Source: §3.2 · p. 4

4 Estimation and budget accounting

The paper estimates variance and forecasts precision from nested question, trajectory, and read samples, then evaluates one-read efficiency and complete model-plus-search costs. Subset analyses and bootstrap procedures quantify forecast error and uncertainty.

4.1 Variance estimation and precision forecasting

  • Standard errors combine within-dataset variances with equal dataset weights, and paired 95% intervals resample complete questions within datasets 10,000 times [7].Outcomes use answer-normalized token F1 and exact match for HotpotQA [31] and MuSiQue [32].Source: §4.1 · p. 4
  • Balanced nested moments estimate variance components from separate dataset means and within-dataset question mean squares using Q −2 degrees of freedom.Second-moment estimation accommodates bounded, non-Gaussian F1 scores; negative component estimates are truncated at zero.Source: §4.1 · p. 4
  • Archived-audit forecasts vary Q, T, and R in the nested model, while the Q-only baseline retains per-question variance and scales standard error by 1/√Q.The archived audit parameters and variance equation were fixed before the extension; no new outcomes informed fitting or selection.Source: §4.1 · p. 4
  • The study enumerates trajectory and read subsets in the depth panel and read subsets in allocation B, comparing empirical standard errors with forecast MAE across configurations and T > 2.Subset weights are equal; intervals condition on recorded trajectories and omit prior-audit parameter uncertainty.Source: §4.1 · p. 4

4.2 Reader calls and one-read efficiency loss

  • One-read efficiency loss is excess estimator variance at the same model-token budget, evaluated across read counts with fixed T and bootstrap-refitted variance and cost.The reported loss uses 100[JT(1)/minR JT(R) −1]; the fixed-Q calculation retains the unchanged question term and at least one trajectory.Source: §4.2 · pp. 4–5

4.3 Cost accounting and reader temperature

  • Cost accounting includes planning and reading inputs and outputs, counts the shared first planner call once, and varies search charges per unique request.Model fees use recorded cache discounts and tariffs, held fixed along search-price curves; temperature summaries keep one-read precision distinct from the three-read mean.Source: §4.3 · p. 5

5 Results

Across the tested allocations, broader question coverage improved comparison precision at nearly matched model-token cost. Forecasting depth, the value of repeated reads, and fee-based rankings retain important qualifications.

5.1 Precision forecasts across sampling configurations

  • Measured A/B/C SEs were 1.03/1.54/1.18 F1 points; archived nested and Q-only forecasts were within 4.0% and 3.5%, respectively.For T > 2, nested and Q-only MAEs were 0.16 and 0.13, with an advantage interval of [−0.25,0.25], showing no reliable deeper-trajectory forecasting advantage.Source: §5.1 · pp. 5–6, Figure 3 · p. 6

5.2 Question coverage and evaluation cost

  • At 34.20/34.14/34.39M model tokens, allocation A reduced measured SE by 33.3% versus B and 12.6% versus C.Under recorded model fees, more questions beat more trajectories at search prices of $0–1 per 1,000 requests; the question-versus-read fee ranking remained unresolved.Source: §5.2 · pp. 6–7, Figure 4 · p. 7

5.3 Reader repetition and component stability

  • Across 15 model/panel/selector settings, fitted one-read penalties were 0–9.9% in estimator variance at a common model-token budget.Pro/RRF’s fixed-T penalty was 9.9%, with a central 95% bootstrap range of 0–57.1%.Source: §5.3 · p. 7, Figure 5 · p. 8
  • On 48 Flash questions, raising T from two to five shifted question/trajectory shares from 69.5/16.5% to 31.6/55.0%.The direct one-read/two-read variance-times-token ratio was 0.736 [0.727,0.761]; averaging paired comparisons yielded a narrow interval despite wider individual-batch intervals.Source: Figure 5 · p. 8, §5.3 · p. 8

5.4 Answer consistency and comparison precision

  • With fixed packets, temperature zero reduced answer disagreement from 14.3% to 3.4%, while its typical one-read SE was 2.38 versus 2.36 F1 points at temperature 0.7.The policy-comparison SE ratio was 1.01 [0.91,1.14], indicating little precision gain.Source: §5.4 · p. 8, Figure 6 · p. 9

6 Conclusion

The measured allocations favor broader question coverage at matched model-token cost, while fee efficiency favors A over C and leaves A versus B unresolved. The conclusions are bounded to the tested workflow and study scope.

6 Conclusion

  • The measured allocations favor broader question coverage at matched model-token cost; fee efficiency favors A over C and leaves A versus B unresolved.Source: §6 · p. 9
  • For the tested Flash workflow at temperature 0.7, results support single-read evaluation and broader question coverage when allocating model tokens.Monetary comparisons require complete model and retrieval fees.Source: §6 · p. 9
  • Audit-calibrated Q−1/2 scaling offers a practical precision estimate with trajectory and read counts fixed.Source: §6 · p. 9
  • Extrapolating trajectory counts calls for an audit with more than two trajectories and a check of the empirical precision curve.Source: §6 · p. 9
  • Evidence covers one policy comparison, one model family, one FineWeb endpoint, and two QA datasets; independent equal-dollar allocations were not executed.Source: §6 · p. 9

A Exact message templates

The appendix records the request settings and message formats used for planning and reading, along with the document fields shown to the reader.

A Exact message templates

  • FineWeb requests fix the search endpoint, seven shards, k = 10, complexity 64, and returned distances; model requests follow Sec. 3.1 settings and JSON format.Observations and final packets contain at most five 5,000-character document prefixes.Source: A Exact message templates · p. 10
  • The planner prompt requests one concise next query, using query history and observed evidence to address missing entities or relations while treating guesses as tentative.Open supplies empty observations; Adaptive supplies earlier observations in query order.Source: A Exact message templates · p. 10
  • The reader prompt requires the shortest complete answer from supplied evidence, treats retrieved instructions as text, and returns UNKNOWN when evidence is insufficient.Source: A Exact message templates · p. 10
  • Each reader call contains the question, prior observations, and new documents, whose records include rank, ID, URL, title, text, and character counts.Displayed text is truncated to 5,000 characters; excluded benchmark URLs occupy the same slot, and the first query uses empty history and observations.Source: A Exact message templates · p. 10

B Selector definitions

The selectors build final document packets from the top-ten results for each distinct query, deduplicate document identities, and retain the first five records in each selector’s ranking.

B Selector definitions

  • Both selectors take the union of the first top-ten result list for each distinct query, preserving query order.Source: B Selector definitions · p. 10
  • RRF sums 1/(60 + r) using each document’s first occurrence per list; ties follow a query-major scan that retains that encountered record.Source: B Selector definitions · p. 10
  • RR scans ranks before queries, skips previously encountered identities, and retains the encountered record; each selector displays its first five ranked records.The complete membership/order crossing protocol and implementation are preserved in the artifact.Source: B Selector definitions · p. 10

C Additional measurement checks

The appendix documents reproducibility materials and checks evidence availability and benchmark-URL masking, while noting that lexical matches do not establish evidence sufficiency.

C Additional measurement checks

  • The artifact preserves frozen manifests, disjointness checks, execution timestamps, scheduled failures, and supporting controls.Source: C Additional measurement checks · p. 11
  • Eligible normalized aliases appear in 54.95% of valid Open candidate pools and 55.29% of Adaptive pools; reader-packet rates range from 36.56–40.12%.Alias presence measures lexical availability; evidence sufficiency and an oracle F1 ceiling remain unmeasured.Source: C Additional measurement checks · p. 11
  • The benchmark-URL mask matches zero returned candidate slots and zero final-reader slots across the two selectors, but other forms of contamination remain possible.Planning failures are 2/892 Open and 1/892 Adaptive trajectories.Source: C Additional measurement checks · p. 11

D Efficiency-loss definitions

The efficiency-loss comparison fixes Q and anchors the budget to the measured panel cost. It retains the question term in the loss ratio while continuously allocating the free sampling level.

D Efficiency-loss definitions

  • At fixed Q, the budget is anchored to measured panel cost B = QT(cT + RobscR) for comparing allocations.Candidate reads must permit at least one trajectory; integer deployment choices and changing search charges can alter exact costs.Source: D Efficiency-loss definitions · p. 11
  • The question term remains in the loss ratio, and candidate reads require B/[Q(cT + rcR)] ≥ 1 trajectory.The calculations continuously allocate the free sampling level, while integer choices and changing search charges can alter exact costs.Source: D Efficiency-loss definitions · p. 11
  • Bootstrap loss ranges use central 95% percentiles, while the direct one-read check averages empirical variance-times-cost ratios against the two-read mean.A paired question bootstrap retains both realizations, and this check uses no ANOVA components.Source: D Efficiency-loss definitions · p. 11

E Dataset-adjusted variance audit

The revised variance audit fits a separate fixed mean for each dataset while retaining nested trajectory and reader terms. It checks balanced-panel reconstruction and refits dataset means within stratified question bootstraps.

E Dataset-adjusted variance audit

  • The revised model uses a separate fixed mean for each dataset, with trajectory and reader mean squares retaining their nested definitions.For balanced T and R, the question mean square is used in the dataset-adjusted variance audit.Source: E Dataset-adjusted variance audit · p. 11
  • Dataset-specific means are refitted within every stratified question bootstrap.The reader mean square is retained before truncation.Source: E Dataset-adjusted variance audit · p. 11
  • On balanced panels, raw components equal the mean of separate-dataset fits and reconstruct the stratified SE; additive dataset offsets leave components unchanged.The dataset-adjustment checks are recorded in dataset_adjustment_audit.json.Source: E Dataset-adjusted variance audit · p. 11

Related papers

Loading 2610.05034v1…