Adaptive Retrieval, Selective Answers: Evaluating Conformal Factuality in Agentic RAG

1 Introduction

• Retrieval can ground responses without ensuring every generated claim is supported; conformal filtering retains claims using a calibration-derived threshold after claim decomposition.The study builds on prior single-stage and multi-hop RAG work.Source: §1 · p. 1
• In agentic RAG, the policy chooses follow-up queries or termination as evidence accumulates, changing evidence acquisition while leaving conformal post-processing conceptually unchanged.The retrieval trajectory and stopping time depend on the question and evidence observed so far.Source: §1 · p. 1
• The study asks whether higher conformal targets improve observed response-level support and measures surviving claims and answers across six configurations, with target sweeps and unfiltered baselines.It is an experimental extension, not a ranking of agentic against multi-hop retrieval.Source: §1 · pp. 1–2

RAG and adaptive retrieval

• An agentic policy can continue retrieval or finish, but a finish decision does not certify that all relevant evidence has been retrieved.This distinguishes bounded decision-making from an oracle for evidence sufficiency.Source: §2 · p. 2

Conformal factuality

• Conformal factuality uses held-out calibration to select a nested refinement or filtering operation for language-model outputs.The SIGIR 2025 Conformal-RAG study introduced retrieval-informed claim scores with product scoring followed by max aggregation [1].Source: §2 · p. 2
• The study retains the retrieval-based relevance-scoring formulation and split-conformal protocol from prior work while replacing multi-hop evidence construction with an agentic retriever.The prior multi-hop work examined the framework when evidence came from multiple retrieval hops [7].Source: §2 · p. 2

3.1 Policy-driven retrieval with an adaptive stopping time

• The agent starts with the question, then chooses follow-up retrieval or termination based on a bounded evidence view, with configured limits also able to end retrieval.The process allows question-specific retrieval trajectories; experiments use a three-operation retrieval budget and three policy decisions.Source: §3.1 · pp. 2–3

3.2 Generation, claim decomposition, and relevance scoring

• After termination, responses are decomposed into variable numbers of atomic claims and scored using claim-passage similarity and retrieval similarity tied to each document’s retrieval query.Follow-up documents are scored against their agent-generated query, while initially retrieved documents are scored against the original question.Source: §3.2 · pp. 3–4

3.3 Calibration and response-level factuality

• Calibration thresholds use the highest-scoring unsupported claim; retained responses count as fully supported when every retained claim is supported, with empty sets included by convention.The five targets are 0.80, 0.85, 0.90, 0.925, and 0.95, using order-statistic indices 25, 27, 28, 29, and 30 for 30 calibration cases.Source: §3.3 · p. 4

4.1 Datasets and models

• Each of six model–dataset configurations uses 60 eligible questions, with Natural Questions and TriviaQA selections matched across model experiments.The evaluated population follows dataset-specific answer, provenance, and extractable-Wikipedia-evidence eligibility rules.Source: §4.1 · p. 4

4.2 Calibration and evaluation protocol

• For each configuration, five conformal targets are evaluated over 100 seeded 30/30 calibration–test partitions of the same 60 examples, alongside a no-CP baseline.The 100 runs repartition fixed generated outputs; they are not independent retrieval-and-generation runs.Source: §4.2 · p. 5

4.3 Reported metrics

• The reported metrics distinguish response-level support and non-empty coverage from retained-claim fractions; no-CP retains all claims and has 100% non-empty coverage.Percentages average over the 100 partitions and should not be treated as independent sample replicates.Source: §4.3 · p. 5

5.1 Reliability–coverage trade-off across six configurations

• Across six settings, increasing the conformal target raises response-level full support while reducing the fraction of responses retaining any claim.The no-CP baselines have 100% non-empty coverage, and NQ coverage is much lower than TriviaQA coverage at stringent targets.Source: §5.1 · p. 5, Figure 1 · p. 6

5.2 Results at the 95% target

• At 95%, non-empty coverage spans 17.23%–60.17% and full support among non-empty outputs spans 89.14%–95.56%, as stricter targets raise observed support and reduce response coverage.The full-support range among non-empty outputs separates substantive-answer reliability from vacuously supported empty filtered outputs.Source: §5.1 · p. 5, §5.2 · p. 6, Figure 1 · p. 6

5.3 Conditional reliability and answer availability

• Response-level full support counts empty filtered outputs as supported, whereas conditional full support concerns non-empty answers and complements answer-coverage and retained-content measures.Table 2 reports the 95% target and defines “Full support | non-empty” by retaining at least one claim.Source: §5.3 · pp. 6–7, Table 2 · p. 7

6 Discussion

• Across both model families, tighter targets produce progressively more selective outputs, extending observations from conventional and multi-hop RAG to model-controlled retrieval.Conformal filtering is applied after the agent terminates and generates a response.Source: §6 · p. 7
• At the 95% target, non-empty coverage exceeds 50% for both TriviaQA settings but falls below 22% for both NQ settings.The authors do not interpret variation across settings as evidence that one retrieval architecture universally outperforms another.Source: §6 · p. 7
• Answer availability and retained-claim fraction should accompany response-level factuality in deployment-facing reports.Source: §6 · p. 7

• 60-question subsets from HotpotQA, Natural Questions, and TriviaQA may limit generalizability to larger, more diverse question sets.Source: §7 · p. 7
• Retrieval is capped at three operations and policy decisions, leaving longer trajectories and alternative agent policies unexplored.Source: §7 · p. 7
• The evaluation covers GPT-6 Luna and Llama 3.1 8B; other model families remain untested for robustness and generalizability.Source: §7 · p. 7

• The study formulates bounded Agentic RAG as an evidence-dependent sequence of retrieval-policy decisions and applies retrieval-based conformal claim filtering after the agent stops.Source: §8 · p. 8
• Across six configurations, higher conformal targets yielded higher observed response-level support alongside lower claim retention and non-empty answer coverage.Source: §8 · p. 8
• The authors motivate broader evaluations with larger datasets, longer retrieval trajectories, and additional language-model families to study the reliability–informativeness trade-off.Source: §8 · p. 8

• The GPT-6 Luna and Llama 3.1 8B tables report no-CP baselines and five conformal targets: 80%, 85%, 90%, 92.5%, and 95%.Source: A Complete Target Sweeps and No-CP… · p. 8, Table 3 · p. 8, Table 4 · p. 9
• Table 3 presents the complete target sweep and no-CP baseline for Agentic RAG with GPT-6 Luna using the relevance-scoring formulation.Source: Table 3 · p. 8
• Table 4 presents the complete target sweep and no-CP baseline for Agentic RAG with Llama 3.1 8B using the relevance-scoring formulation.Source: Table 4 · p. 9

• Table 5 is identified as a summary of shared conditions across the six prioritized Agentic RAG experiments.Source: Table 5 · p. 9