PaperlayerView exact revision on arXiv

Source-linked AI summary

Conformal Factuality Control for Multi-Hop Retrieval-Augmented Generation

Muhammad Aimal Rehman, Chi-Kuang Yeh

arXiv:2609.38222v1cs.CLcs.LGstat.ML

TL;DR

Retrieved evidence does not guarantee that generated claims are supported, and this gap is especially relevant when multi-hop retrieval and reasoning depend on one another. The paper applies claim-level conformal filtering to multi-hop RAG and evaluates it across benchmarks and models. More stringent targets increase full-support rates, but the improvement comes with lower claim retention and more empty responses.

  • Problem

    Retrieved context does not guarantee supported claims, and the reliability of claims in multi-hop RAG remains less characterized when retrieval and reasoning proceed through dependent stages.

    Source: §1 · p. 1

  • Method

    The study applies a common conformal filtering framework to multi-stage retrieval responses, evaluating three QA benchmarks and two LLM families with a single-hop reference experiment.

    Source: §1 · p. 2

  • Results

    Across all six multi-hop model–dataset configurations, increasingly stringent conformal targets raise the response-level rate at which all retained claims are supported.

    Source: §7 · p. 10

  • Takeaways & Limitations

    Conformal factuality applies to multi-hop RAG, but reliability targets should be interpreted jointly with claim retention and response coverage.

    Source: §9 · p. 11

    The conformal interpretation depends on exchangeability between calibration and test examples and annotation quality; distribution shift or systematic verifier errors can weaken the practical meaning of the nominal target.

    Source: §8 · p. 10

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) can ground large language models in external evidence, but retrieved context does not guarantee that generated claims are factually supported. This problem is especially relevant in multi-hop RAG, where retrieval and reasoning proceed through multiple dependent stages. We study whether claim-level conformal factuality control, previously developed for RAG, remains effective in this setting. We apply split-conformal claim filtering to multi-hop RAG and evaluate it on HotpotQA, Natural Questions, and TriviaQA using Llama 3.1 8B and GPT-4o-mini, together with a single-hop reference experiment. Across all six multi-hop model-dataset configurations, increasingly stringent conformal targets consistently increase the fraction of responses whose retained claims are fully supported. At the 95% target, this rate ranges from 95.80% to 97.20%, compared with 55.60%-76.03% without filtering. However, the improvement is strongly selective: only 4.41%-31.09% of generated claims are retained and 9.70%-51.40% of responses remain non-empty at the 95% target. These results show that conformal factuality extends to multi-hop RAG, while demonstrating that nominal reliability must be interpreted jointly with claim retention and abstention.

1 Introduction

Retrieved evidence does not guarantee that generated claims are supported, a concern sharpened when multi-hop retrieval and reasoning depend on one another. This study tests claim-level conformal factuality control in multi-hop RAG while tracking reliability, retained information, and abstention.

1 Introduction

  • Multi-hop retrieval and reasoning create dependencies between stages, so retrieved evidence alone does not provide a principled measure of final-claim reliability.Information obtained at one step may determine what is retrieved next.Source: §1 · p. 1
  • Prior conformal RAG work leaves claim-level factuality behavior under dependent multi-hop retrieval less characterized.Earlier work includes conformal generation-risk methods and filtering of supported sub-claims under prescribed factuality targets (Kang et al., 2024; Feng et al., 2025).Source: §1 · p. 2
  • The study evaluates multi-hop RAG on HotpotQA, Natural Questions, and TriviaQA with Llama 3.1 8B and GPT-4o-mini, alongside a single-hop HotpotQA reference.The benchmarks connect to Wikipedia evidence in different ways (Yang et al., 2018; Joshi et al., 2017).Source: §1 · p. 2
  • Increasingly stringent conformal targets raise the fraction of responses whose retained claims are fully supported, while reducing claim retention and increasing empty responses.The trade-off varies across datasets and model families; the authors conclude that practical reliability should be assessed alongside useful content preserved.Source: §1 · p. 2
  • The paper characterizes the reliability–informativeness trade-off by jointly analyzing factual support, claim retention, and abstention across increasingly stringent conformal targets.Its evaluation also examines whether behavior persists across datasets and generation models.Source: §1 · p. 2

2 Related Work

Related work covers iterative retrieval and reasoning for compositional questions, RAG evaluation, and conformal methods for language-model reliability. This study focuses on claim-level conformal factuality when retrieval proceeds across dependent hops.

Retrieval-augmented and multi-hop generation

  • IRCoT interleaves retrieval with chain-of-thought reasoning so intermediate steps can guide later retrieval (Trivedi et al., 2023), while MultiHop-RAG highlights continuing retrieval and generation challenges (Tang and Yang, 2024).These works motivate reliability evaluation when retrieval and reasoning are sequentially dependent.Source: §2 · pp. 2–3

Factuality and evaluation in RAG

Conformal prediction for language-model reliability

  • Prior conformal RAG methods address generation risk or sub-claim filtering, while this work studies claim-level factuality under multi-hop retrieval (Kang et al., 2024; Feng et al., 2025).Prior analyses also note informativeness costs, vacuous outputs, and calibration sensitivity to distribution shift (Chen et al., 2026); MiCP instead allocates error budgets across reasoning turns (Zhou et al., 2026).Source: §2 · p. 3

3 Background and Problem Formulation

Multi-hop RAG builds answers from evidence gathered through dependent retrieval and reasoning stages, while claim-level conformal filtering controls which generated claims are retained. The paper frames this as a joint reliability–informativeness problem: stricter targets can improve support but reduce retained content or leave responses empty.

3.1 Multi-Hop Retrieval-Augmented Generation

  • Multi-hop RAG retrieves evidence sequentially, with later queries potentially depending on earlier results, and uses accumulated evidence to generate an initial response.Errors in intermediate retrieval or reasoning can affect subsequent stages and the final response.Source: §3.1 · p. 3

3.2 Conformal Factuality for RAG

  • The framework decomposes generated responses into atomic claims and scores each claim using the query and retrieved evidence; scores are high when documents match both the question and claim.The filtering operation retains claims meeting a threshold, and increasing that threshold can only remove additional claims.Source: §3.2 · p. 4
  • A calibration example’s conformal score is the smallest threshold for which all remaining claims are factual; a held-out calibration set determines the inference threshold.The stated response-level guarantee relies on exchangeability and annotation-correctness assumptions.Source: §3.2 · p. 4
  • The target 1−α is a probability target for filtered-response factuality, not a required fraction of factual claims within each response.For example, the passage equates a 95% target with α = 0.05.Source: §3.2 · p. 4

3.3 Problem Formulation

  • The central question is whether claim filtering remains effective when the generated response comes from multi-hop rather than single-hop retrieval.The study considers accumulated evidence from multiple retrieval stages.Source: §3.3 · p. 4
  • The evaluation jointly considers whether retained claims are supported and how much of the original response survives, since stricter targets can reduce retention or produce empty responses.The trade-off is characterized across retrieval settings, datasets, and language models.Source: §3.3 · p. 4
  • Figure 1 separates the multi-hop test-time path—retrieval, claim decomposition, scoring, and filtering—from the calibration-only branch that determines the threshold.The test-time response is filtered using the calibrated split-conformal threshold.Source: Figure 1 · p. 5

4 Method

The method combines iterative retrieval and query refinement with claim-level conformal filtering, keeping the filtering rule fixed while changing the evidence provenance to multi-hop retrieval. It calibrates a threshold on held-out examples and evaluates reliability alongside claim retention and response coverage.

4.1 Overview

  • The pipeline adds iterative multi-hop retrieval to claim-level conformal factuality, while keeping the conformal filtering rule fixed to examine behavior with multi-stage evidence.Retrieved documents from dependent hops supply evidence for generation and claim scoring.Source: §4.1 · p. 5

4.2 Multi-Hop Retrieval and Response Generation

  • Retrieval runs for at most three hops, with an LLM rewriting later queries from earlier-hop information and accumulated documents supplied to the response generator.Each hop returns up to 10 documents subject to a 0.3 similarity threshold, and retrieval stops early if no unseen document is found.Source: §4.2 · p. 5

4.3 Atomic Claim Decomposition and Verification

  • Generated answers are decomposed into independently verifiable atomic claims, and an LLM verifier labels claims as supported, irrelevant, unverifiable, or non-factual.Only supported claims count as supported; verifier annotations are used for calibration and evaluation, not test-time filtering.Source: §4.3 · p. 6

4.4 Retrieval-Based Claim Scoring

  • Claim relevance scoring uses maximum aggregation across retrieved documents and a product scoring rule, with a deterministic perturbation to break ties.Documents from all multi-hop stages contribute to the accumulated evidence used for scoring.Source: §4.4 · p. 6

4.5 Split-Conformal Calibration

  • Split-conformal calibration uses equally sized calibration and test sets, repeated over 100 deterministic seeded splits, while target levels change only the filtering threshold.For 60-question experiments, each split contains 30 calibration and 30 test examples; Table 1 identifies Plain-RAG as a reference and six Multi-Hop RAG configurations as the primary evaluation.Source: §4.5 · p. 6, Table 1 · p. 7

4.6 Reliability and Selectivity

  • Fully supported responses require every retained claim to be supported, but an empty retained set qualifies vacuously, so reliability is assessed alongside claim retention and non-empty response coverage.The study also reports fully-supported rates conditional on responses being non-empty.Source: §4.6 · p. 7

5 Experimental Setup

The evaluation covers three QA datasets and two generation backbones, with a HotpotQA single-hop reference condition. The protocol reuses fixed generated responses across conformal targets and averages results over repeated calibration–test splits.

5.1 Datasets

  • HotpotQA, Natural Questions, and TriviaQA provide the three evaluation benchmarks; HotpotQA targets multi-document reasoning, while the other two add complementary open-domain QA settings.Source: §5.1 · p. 7

5.2 Models and Experimental Conditions

  • The primary evaluation combines multi-hop retrieval with both model families across all three datasets, alongside a Llama 3.1 8B Plain-RAG reference on HotpotQA.Llama 3.1 8B and GPT-4o-mini are the two generation backbones.Source: §5.2 · p. 7

5.3 Conformal Evaluation Protocol

  • Each 60-example set is split into 30 calibration and 30 test examples, with results averaged over 100 seeded splits; the same generated responses are reused across targets.The unfiltered condition retains all generated claims, and emphasis is on 90% and 95% targets.Source: §5.3 · p. 8

6 Results

Across six multi-hop configurations, more stringent conformal targets raise response-level full-support rates, while substantially reducing retained content and non-empty responses. Empty outputs can contribute to high full-support rates, so reliability must be read alongside coverage and retention.

6.1 Reference Single-Hop Experiment

  • On HotpotQA with Llama 3.1 8B, the Plain-RAG baseline yields 61.67% fully supported responses; at 95%, full support reaches 96.27%, with 11.10% claim retention.At that target, 20.97% of responses remain non-empty.Source: §6.1 · p. 8

6.2 Conformal Filtering Raises Response-Level Support in Multi-Hop RAG

  • At 95%, multi-hop full-support rates range from 95.80% to 97.20%, versus 55.60% to 76.03% without filtering, across both models and all three datasets.Source: §6.2 · p. 8

6.3 Reliability Comes at a Substantial Selectivity Cost

  • At 95%, claim retention ranges from 4.41% to 31.09% and non-empty response coverage from 9.70% to 51.40%; stricter targets generally trade coverage for reliability.GPT-4o-mini claim retention is 5.16% on HotpotQA and 4.41% on NQ; Figure 2 depicts the reliability–coverage trajectories.Source: §6.3 · pp. 8–9, Figure 2 · p. 9

6.4 Response-Level Reliability Can Be Dominated by Abstention

  • At 95%, GPT-4o-mini’s 97.20% full-support rate on HotpotQA and 96.43% on NQ coexist with only 10.80% and 9.70% non-empty responses, respectively.Among non-empty outputs, fully supported rates are 78.48% and 71.72%, respectively.Source: §6.4 · p. 9

7 Discussion

The results support claim-level conformal factuality control in multi-hop RAG, but its reliability gains come through selective output. The practical cost varies across model–dataset combinations, and the single-hop reference does not establish a general comparison with multi-hop RAG.

7 Discussion

  • Across all six multi-hop configurations, stricter conformal targets raise the rate of responses whose retained claims are all supported, for both models and all three datasets.Source: §7 · p. 10
  • The improvement is selective factuality control rather than correction of the underlying generation, and empty responses can satisfy the full-support event vacuously.High overall factuality can therefore coexist with low non-empty coverage; the paper recommends interpreting it alongside retention and abstention.Source: §7 · p. 10
  • The same nominal target can have substantially different operational costs across model–dataset combinations; the Plain-RAG result is only a HotpotQA/Llama 3.1 8B reference.The study does not draw a general conclusion about relative costs in single-hop versus multi-hop RAG.Source: §7 · p. 10

8 Limitations

The study’s limitations concern evaluation scale and variability, annotation and decomposition reliability, and the scope of the tested models and retrieval design.

8 Limitations

  • The experiments use controlled 60-question subsets rather than complete benchmark sets, limiting the precision of percentages as estimates of full-benchmark performance.Source: §8 · p. 10
  • The 100 runs repartition fixed generated responses, so reported variation reflects calibration-sample sensitivity rather than retrieval or generation stochasticity.Source: §8 · p. 10
  • LLM-based factuality verification may introduce annotation errors and model-dependent support judgments; exchangeability and annotation quality underpin the conformal interpretation.Distribution shift or systematic verifier errors can weaken the practical meaning of the nominal target.Source: §8 · p. 10
  • Automated, model-dependent atomic-claim decomposition can still introduce errors that affect downstream scoring, verification, and response-level metrics.Source: §8 · p. 10
  • The evaluation covers two model families and one multi-hop design with at most three hops; other models, retrieval strategies, domains, or longer chains may differ.The Plain-RAG reference is limited to HotpotQA with Llama 3.1 8B and does not support a general single-hop/multi-hop cost comparison.Source: §8 · p. 10

9 Conclusion

Across six multi-hop configurations, stricter conformal targets increase full-support rates while reducing claim retention and sometimes producing empty responses. The conclusion recommends interpreting reliability alongside preserved content and response coverage.

9 Conclusion

  • Across HotpotQA, Natural Questions, and TriviaQA with Llama 3.1 8B and GPT-4o-mini, stricter targets increased full-support rates while reducing claim retention.Some settings also produced many empty responses.Source: §9 · p. 10
  • Reliability targets should be assessed jointly with claim retention and response coverage, since practical value depends on useful content preserved.Source: §9 · p. 11

Reproducibility statement

  • The reproducibility statement reports publicly available implementation, fixed generated-response artifacts, and repeated seeded calibration–test splits.It points to the paper’s methods, experimental protocol, and appendix for implementation details.Source: Reproducibility statement · p. 11

A Additional Experimental Results

The appendix provides complete conformal target sweeps, expanding the reported metrics and covering a single-hop reference and multi-hop experiments with two model families.

A Additional Experimental Results

  • The extended sweeps report claim retention, non-empty response coverage, supported-claim factuality, and full support conditional on non-empty responses.The conditional metric excludes vacuous empty responses and complements the main-text response-level conformal event.Source: A Additional Experimental Results · p. 12

A.1 Plain-RAG Reference Sweep

  • Table 3 presents percentage-valued conformal target sweeps for the Plain-RAG HotpotQA reference experiment with Llama 3.1 8B.Source: Table 3 · p. 13

A.2 Multi-Hop RAG with Llama 3.1 8B

  • Table 4 presents percentage-valued conformal target sweeps for multi-hop RAG with Llama 3.1 8B.Source: Table 4 · p. 13

A.3 Multi-Hop RAG with GPT-4o-mini

  • Table 5 presents percentage-valued conformal target sweeps for multi-hop RAG with GPT-4o-mini.Source: Table 5 · p. 14

B Implementation Details

The implementation specifies multi-hop retrieval and processing choices, split-conformal settings, and the execution environment used for a representative run.

B.1 Retrieval and Indexing Configuration

  • Retrieval uses up to 10 documents per hop, a 0.3 similarity threshold, and at most three hops, accumulating documents and stopping when no unseen document appears.For GPT-4o-mini, the embedding and retrieval configuration is aligned with the Llama 3.1 8B experiments.Source: §B.1 · p. 14

B.2 Claim Processing

  • Generated responses are decomposed into atomic claims with up to three retries for malformed outputs, and claims receive supported, irrelevant, unverifiable, or non-factual labels.Only supported claims count as factual in the described evaluation.Source: §B.2 · pp. 14–15

B.3 Conformal Configuration

  • The study uses split-conformal calibration, fixes the factuality parameter at a = 1, and evaluates target levels from 80% through 95%.Each configuration reuses generated responses, claim scores, and verifier annotations across targets; calibration–test splits use seed 42 and are repeated 100 times.Source: §B.3 · p. 15

B.4 Execution Environment

  • A representative Llama 3.1 8B HotpotQA run used Google Colab, Python 3.13.15, PyTorch 2.11.0, and an NVIDIA A100-SXM4-40GB GPU.The run used 60 controlled questions, seed 42, and 100 conformal evaluation splits.Source: §B.4 · p. 15

Related papers

Loading 2609.38222v1…