PaperlayerView exact revision on arXiv

Source-linked AI summary

RAGCap-Bench: Benchmarking Capabilities of LLMs in Agentic Retrieval Augmented Generation Systems

Jingru Lin, Chen Zhang, Stephen Y. Liu, Haizhou Li

arXiv:2510.13910v2cs.CL

TL;DR

End-to-end evaluations of agentic RAG provide limited feedback about intermediate processes such as planning, retrieval, and reasoning. RAGCap-Bench assesses those capabilities through targeted questions derived from agentic-system trajectories. Its scores correlate with downstream multi-hop QA performance, though the benchmark is relatively small.

  • Problem

    Existing agentic RAG evaluations focus on end-to-end multi-hop QA and provide limited fine-grained feedback on intermediate processes.

    Source: §1 · p. 1

  • Method

    RAGCap-Bench uses questions based on a taxonomy of intermediate-stage errors to assess Planning, Evidence Extraction, Grounded Reasoning, and Noise Robustness.

    Source: §1 · p. 1

  • Results

    RAGCap-Bench scores strongly correlate with end-to-end results on complex multi-hop QA tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark enables decoupled, interpretable capability assessment and is substantially faster than full agentic evaluations.

    Source: §1 · p. 2

    With approximately 60 questions per category, the benchmark may have reduced statistical power and may not support strong conclusions from small model-performance differences.

    Source: Limitation · p. 9

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) mitigates key limitations of Large Language Models (LLMs)-such as factual errors, outdated knowledge, and hallucinations-by dynamically retrieving external information. Recent work extends this paradigm through agentic RAG systems, where LLMs act as agents to iteratively plan, retrieve, and reason over complex queries. However, these systems still struggle with challenging multi-hop questions, and their intermediate reasoning capabilities remain underexplored. To address this, we propose RAGCap-Bench, a capability-oriented benchmark for fine-grained evaluation of intermediate tasks in agentic RAG workflows. We analyze outputs from state-of-the-art systems to identify common tasks and the core capabilities required for their execution, then construct a taxonomy of typical LLM errors to design targeted evaluation questions. Experiments show that "slow-thinking" models with stronger RAGCap performance achieve better end-to-end results, underscoring the benchmark's validity and the importance of enhancing these intermediate capabilities.

1 Introduction

The introduction motivates RAGCap-Bench as a way to assess intermediate capabilities that end-to-end evaluations of agentic RAG do not isolate, and reports that its scores correlate with downstream multi-hop QA performance.

1 Introduction

  • LLMs’ reliance on internal knowledge can produce factual errors, outdated information, and hallucinations, while traditional RAG can have static sources and shallow reasoning or context integration.Agentic RAG instead supports dynamic information gathering, structured reasoning, and adaptive planning.Source: §1 · p. 1
  • Existing agentic RAG evaluations focus on end-to-end multi-hop QA, offering limited feedback for improving system components.Errors in planning, retrieval, or reasoning can compound across stages.Source: §1 · p. 1
  • RAGCap-Bench evaluates four capabilities—Planning, Evidence Extraction, Grounded Reasoning, and Noise Robustness—using questions based on a taxonomy of common intermediate-stage errors.The questions undergo filtering, manual inspection, and annotation.Source: §1 · pp. 1–2
  • RAGCap-Bench scores strongly correlate with end-to-end results on complex multi-hop QA, and DeepSeek-R1 evaluation is roughly 50× faster than a comparable full WebThinker evaluation.The speed comparison is for BrowseComp-ZH and a dataset comparable in size to RAGCap-Bench.Source: §1 · p. 2

2 Problem Formulation

The paper frames agentic RAG as an iterative process in which an LLM plans, retrieves evidence, reasons, and revises its plan before answering. RAGCap-Bench evaluates these intermediate processes because errors can propagate through the workflow.

2 Problem Formulation

  • An agent begins with a plan for the query, then iteratively searches, analyzes evidence, reasons, and updates its plan for up to T steps.It stops searching when it judges the current evidence and reasoning sufficient, then generates an answer.Source: §2 · p. 2
  • At each step, the agent retrieves top-K web evidence and identifies the useful subset before reasoning from it.Some systems instead aggregate relevant evidence before a final reasoning step.Source: §2 · p. 2
  • Because an incorrect evidence extraction can lead to faulty reasoning and later errors, the benchmark assesses capabilities at the process level rather than only the final answer.It targets planning, useful-evidence extraction, and faithful reasoning, as well as handling low-quality, misleading, or less credible sources.Source: §2 · p. 2

3 RAGCap-Bench

RAGCap-Bench builds multiple-choice questions from agentic RAG trajectories to assess four capabilities and their component tasks. It filters and annotates the questions, then aggregates Exact Match and instancewise macro F1 scores across task types.

3 RAGCap-Bench

  • The benchmark contains 255 curated questions derived from open-source deep-research queries and execution trajectories collected from agentic RAG systems.Intermediate information from the logs is used to generate task-specific questions.Source: §3 · p. 2

3.1 MCQ Format

  • Questions use a query, intermediate outputs, and an instruction to select correct or incorrect options; Figure 1 depicts the data-construction pipeline.The pipeline draws on open-source QA queries and system outputs, generates questions using Vanilla Generation or Error-Guided Generation, filters them, and recruits experts to provide answers.Source: §3.1 · pp. 2–3, Figure 1 · p. 3

3.2 Dataset Construction

  • Planning questions assess initial and updated plans, including convergent search narrowing and divergent search broadening.The questions draw on planning trajectories that lead to correct answers, including agentic-system traces and human-annotated strategies; error-guided prompts generate distractors from common mistakes.Source: §3.2 · pp. 3–4
  • Evidence Extraction questions turn retrieved webpages into options, requiring models to rule out evidence irrelevant to the query.Grounded Reasoning questions ask models to identify incorrect statements, using both injected errors and flawed reasoning traces from real workflows.Source: §3.2 · p. 4
  • Noise Robustness tests whether models abstain when search results provide little useful information and whether they identify less credible sources.The benchmark treats admitting that a question is unanswerable from the supplied results as noise-abstain, and separately assesses noise reliability.Source: §3.2 · pp. 4–5

3.3 Dataset Post Processing

  • Difficulty filtering removes items that multiple models all answer the same way, and format filtering removes poorly formatted or ambiguous questions before expert annotation.Difficulty filtering removes approximately 70–85% of generated items per dataset; format filtering removes another 10–15%.Source: §3.3 · p. 5

3.4 Evaluation Metrics

  • The benchmark reports Exact Match and instancewise macro F1, averaging subcategories within groups before averaging the four question-type scores.Planning has converging and diverging subcategories; Noise Robustness has abstain and reliability subcategories.Source: §3.4 · p. 5

4 Evaluations

The evaluations test fast- and slow-thinking models on intermediate RAG capabilities and find that RAGCap-Bench scores align with downstream QA performance. Results also expose weaknesses in evidence extraction, grounded reasoning, and source reliability, while informative prompts improve overall scores.

4.1 Setups

  • Fast- and slow-thinking LLMs are evaluated across task types with bare and informative prompts; informative prompts include examples of common errors.The evaluation includes the models listed for each group and reports Table 3 results averaged over three runs.Source: §4.1 · p. 5, §4.2 · p. 5

4.2 Benchmark Results

  • Gemini-2.5-Flash leads slow-thinking convergent planning (EMc 58.82%; F1c 78.41%), while evidence-extraction EM remains below 40% for most models.The strongest extraction EM scores are 42.02% for Gemini-3.1-Pro and 40.10% for Qwen3-235B-A22B w/ think.Source: §4.2 · p. 6
  • Grounded-reasoning EM peaks at 52.83% for Gemini-3.1-Pro and 57.23% for Qwen3-235B-A22B; many models score below 50% on noise-reliability EMr, with some at 10%.The reported analysis also describes planning, extraction, grounded reasoning, and noise robustness as interdependent capabilities.Source: §4.2 · pp. 6–7

4.3 With vs. Without Error-Guided Prompts

  • Informative prompts consistently improve overall EM for both fast- and slow-thinking models compared with bare prompts.Figure 3 compares overall EM under the two prompting settings.Source: §4.3 · p. 7, Figure 3 · p. 7

4.4 Correlation with Downstream

  • RAGCap-Bench scores show a clear positive correlation with downstream QA performance across four benchmarks, and size-matched Qwen comparisons indicate the alignment is not solely determined by model scale.Downstream accuracy is measured as pass@1 in WebThinker; the scale-controlled comparison uses four comparably sized Qwen models.Source: §4.4 · p. 7

4.5 Relationship With Direct Scoring

  • LLM-as-a-judge correlations with downstream outcomes align with RAGCap-Bench evidence-extraction and grounded-reasoning scores; Qwen3-235B has the strongest correlation.The analysis scores grounded reasoning and evidence extraction across sampled WebThinker trajectories and compares those scores with binary downstream outcomes.Source: §4.5 · p. 8

4.6 Ablations

  • Grounded reasoning and planning correlate most strongly with downstream QA, while shuffled MCQ options have limited impact on overall performance.The sub-capability comparison is shown in Figure 5; the shuffle comparison is reported in Table 6.Source: §4.6 · pp. 8–9, Figure 5 · p. 8, Table 6 · p. 9

5 Related Work

Prior work presents agentic RAG as an active, adaptive extension of traditional RAG, while existing benchmarks largely assess end-task QA rather than intermediate processes. This section situates RAGCap-Bench in that benchmarking gap.

5 Related Work

  • Agentic RAG makes the LLM an active agent for adaptive planning, dynamic information seeking, and iterative reasoning, extending traditional RAG.Traditional RAG provides access to external knowledge but still struggles with complex tasks (Singh et al., 2025).Source: §5 · p. 9
  • OpenAI Deep Research, Gemini Deep Research, and Perplexity Deep Research are cited as practical applications of the agentic RAG paradigm.The passage describes these applications as leveraging LLMs as autonomous agents.Source: §5 · p. 9
  • Existing agentic RAG benchmarks focus mainly on multi-hop QA, offering limited insight into intermediate planning, retrieval, and reasoning tasks.These include benchmarks cited as Xi et al. (2025), Zhou et al. (2025), and Wei et al. (2025).Source: §5 · p. 9

6 Conclusion

The conclusion introduces RAGCap-Bench for component-wise evaluation of agentic RAG systems. Its scores correlate with downstream performance, and exploratory experiments examine LLM assessment of intermediate outputs.

6 Conclusion

  • RAGCap-Bench provides fine-grained, component-wise evaluation of agentic RAG systems’ intermediate processes.The benchmark addresses the gap in existing benchmarks that lack evaluation of those processes.Source: §6 · p. 9
  • RAGCap-Bench scores correlate with downstream task performance, supporting the benchmark’s practical relevance.Source: §6 · p. 9
  • Exploratory experiments examine LLM assessment of agentic RAG intermediate outputs as a direction for future research.The passage describes this as paving the way for research into integrating LLMs to improve agentic RAG systems.Source: §6 · p. 9

Limitation

RAGCap-Bench has approximately 60 questions per category, constraining the strength of conclusions about small between-model differences. Expanding it is labor-intensive, motivating exploration of automated construction pipelines.

Limitation

  • Approximately 60 questions per category may reduce statistical power and limit strong conclusions about small performance differences across models.Source: Limitation · p. 9
  • Constructing each MCQ requires costly agentic RAG runs, manual inspection of multi-step trajectories, and careful annotation of options.The process can also require full review of source webpages to assess evidence quality and credibility.Source: Limitation · p. 9
  • The authors plan to explore automated pipelines to enable construction of larger-scale MCQ datasets.Source: Limitation · p. 9

A Datasets

The dataset construction uses outputs from named agentic RAG systems and several datasets; the capability discussion distinguishes narrowing search from exploring diverse perspectives.

A Datasets

  • WebThinker, WebSailor, WebDancer, and HiRA are run on InfoDeepSeek, BrowseComp-Zh, and Frames to obtain intermediate outputs.Source: A Datasets · p. 12
  • Glaveai/RAG-v11 supplies retrieved documents, while other datasets provide human-annotated stepwise strategies used to construct planning questions.Source: A Datasets · p. 12

B Convergent and Divergent Capabilities

C Examples of Errors

Figure 7 presents examples of errors collected from agentic RAG pipelines, with the errors highlighted in red.

D Generation of MCQs

The section describes adapting MCQ-generation prompts to pipeline and dataset formats, using error-guided and vanilla generation, and provides examples from RAGCap-Bench.

D Generation of MCQs

  • Prompts are customized to fit the varying structures and formats of each pipeline’s and dataset’s intermediate outputs.The examples illustrate the core idea and can be adapted for specific pipelines and datasets.Source: D Generation of MCQs · pp. 12–13
  • Figure 6 depicts convergent and divergent planning capabilities required for different user queries.Source: Figure 6 · p. 13
  • For most vanilla-generation questions, LLM-based generation is unnecessary; grounded-reasoning questions use an LLM, while other prompts follow Figure 1 formats.The section shows error-guided prompts for planning and grounded reasoning, plus a vanilla-generation example for grounded reasoning and an evidence-extraction example.Source: D Generation of MCQs · p. 13

E Examples of MCQs

F Results with Bare Prompt

The appendix shows bare-prompt evaluation materials, including a table comparing fast- and slow-thinking LLMs and examples of prompts and benchmark questions.

G Direct Evaluation Prompts

H Bare and Informative Prompts

F Results with Bare Prompt

  • Table 8 reports fast- and slow-thinking LLM performance with bare prompts, using EMc and F1c for converging ability, EMd for diverging ability, EMa for noise-abstain, and EMr and F1r for noise-reliability.The scores are presented as percentages; the supplied passage does not give table-cell results.Source: F Results with Bare Prompt · p. 13, Table 8 · p. 18

H Bare and Informative Prompts

Related papers

Loading 2510.13910v2…