PaperlayerView exact revision on arXiv

Source-linked AI summary

BELIEFRAG: Making Adaptive RAG State-Aware under Evolving Evidence

Hongji Pu

arXiv:2609.39139v1cs.AI

TL;DR

Multi-step retrieval needs a persistent view of what evidence supports and what remains missing, since local signals can imply different next actions. BeliefRAG updates an explicit evidence state to coordinate those actions. Across six QA benchmarks, it reports higher mean token F1 than fixed iterative retrieval while using fewer tokens.

  • Problem

    Multi-step retrieval must combine local signals into a persistent view of current evidence because missing facts, conflicting passages, and weak support call for different decisions.

    Source: §1 · p. 1

  • Method

    BeliefRAG updates an explicit state over sufficiency, reliability, conflict, uncertainty, evidence gaps, and cost to coordinate retrieval and other actions.

    Source: §1 · p. 2

  • Results

    Across six QA benchmarks, BeliefRAG improves quality while using fewer tokens than fixed Iterative RAG; on gpt-oss-120b, mean token F1 is 0.572 versus 0.555 with 39% fewer tokens.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported gains mainly come from evidence replacement and answerability-guided stopping, while calibrated answerability and transferable thresholds are highlighted as operational findings.

    Source: §1 · p. 2, §6 · p. 10

    Filtering irrelevant or misleading passages does not solve source shift, where the diagnostic itself can lose ranking quality.

    Source: §5.5 · p. 10

Abstract

from arXiv · show

Adaptive RAG uses signals such as confidence, relevance, support, and retrieval quality to decide when to search or correct evidence. In multi-step retrieval, however, these local signals must be combined into a persistent view of what the current evidence supports, what remains missing, and which action should follow. Existing methods often use such signals as separate triggers, making it difficult to preserve a coherent evidence state across a trajectory; we call this problem evidence-state fragmentation. We introduce BELIEFRAG, a closed-loop controller that updates an explicit state over sufficiency, reliability, conflict, uncertainty, evidence gaps, and acquisition cost, then chooses among retrieval, query rewriting, verification, answering, stopping, and abstention. Across six QA benchmarks with gpt-oss-120b, BELIEFRAG reaches mean token F1 0.572 with 3.89k tokens per question, outperforming fixed iterative retrieval (0.555 F1) while using 39% fewer tokens. The same quality-cost pattern transfers to Qwen3-32B, where BELIEFRAG reaches 0.552 F1 versus 0.523 for iterative retrieval while using 35% fewer tokens. Analysis shows that the main gains come from corrective re-retrieval rather than pruning alone, while several belief dimensions are redundant and calibrated answerability plays the strongest operational role. Calibration improves threshold stability across related evidence sources, although source shift can still invalidate the same decision signal.

1 Introduction

Multi-step retrieval requires a persistent view of evolving evidence because local signals can reflect different problems that call for different actions. BeliefRAG maintains that state to coordinate retrieval and other decisions, and its evaluations report a quality–cost advantage over iterative retrieval.

1 Introduction

  • Multi-hop questions require repeated decisions about missing information, whether another search is useful, and when evidence suffices for an answer.Source: §1 · p. 1
  • Adaptive RAG uses confidence, relevance, support, information need, and retrieval quality, but multi-step control must combine these local signals into a persistent evidence view.Source: §1 · p. 1
  • Low confidence can reflect missing facts, conflicting passages, or weak support, which call for different actions such as retrieval, verification, or stopping.Source: §1 · p. 1
  • BeliefRAG updates six operational dimensions—sufficiency, reliability, conflict, uncertainty, evidence gap, and cost—to coordinate retrieval, rewriting, verification, answering, stopping, and abstention.Source: §1 · p. 2
  • 0.572 mean token F1 with 3.89k tokens per question beats fixed Iterative RAG’s 0.555 F1 using 39% fewer tokens; Qwen3-32B reaches 0.552 F1 versus 0.523 with 35% fewer tokens.Source: §1 · p. 2
  • The reported analyses identify corrective re-retrieval, calibrated answerability, and transferable decision thresholds as mechanisms associated with the quality–cost trade-off.Source: §1 · p. 2

2 Related Work

Prior adaptive RAG work uses varied signals and strategies to govern retrieval. The literature also notes that confidence can be unreliable and that relevant evidence may still be misleading or conflicting.

2 Related Work

  • Self-RAG learns reflection signals for retrieval, relevance, support, and utility (Asai et al., 2024); CRAG triggers correction for poor evidence (Yan et al., 2024); Adaptive-RAG routes among retrieval depths (Jeong et al., 2024).Source: §2 · p. 2
  • Uncertainty can guide adaptive retrieval, but its reliability varies across tasks and estimators, and no single confidence signal is uniformly reliable.Source: §2 · pp. 2–3
  • Topically relevant evidence can mislead, while conflicts between retrieved evidence and internal knowledge can degrade RAG performance (Jin et al., 2024; Pham et al., 2024).Source: §2 · p. 3

3 Methodology

BELIEFRAG maintains a persistent state of what the current evidence supports and uses it to coordinate retrieval, rewriting, verification, answering, stopping, and abstention. Its decision process combines evidence diagnostics into beliefs and applies them through a sequence of controller rules.

3.2 Measuring the Current Evidence

  • Seven evidence diagnostics are computed before decisions, with four semantic quantities produced by one structured verifier call and retriever-related quantities computed locally.The locally computed quantities include calibrated retrieval relevance, novelty, and cost.Source: §3.2 · pp. 3–4, Table 1 · p. 4

3.3 From Diagnostics to Belief

  • The diagnostics update six operational beliefs—sufficiency, reliability, conflict, uncertainty, evidence gap, and cost—that summarize distinct aspects of the retained evidence.Cost is observed directly; the other dimensions are inferred from the diagnostic vector and blended with previous beliefs.Source: §3.3 · p. 4
  • The answerability estimate is an operational prediction of whether the frozen generator answers correctly, so it can reflect parametric knowledge rather than evidence-only entailment.The calibrator is fit on HotpotQA training data, selected on development data, and frozen across the six benchmarks.Source: §3.3 · p. 4

3.4 How Beliefs Produce Actions

  • BELIEFRAG evaluates correction, answerability, and retrieval value in sequence, then chooses among retrieval, rewriting, verification, answering, stopping, and abstention.Correction removes problematic passages before reformulating the query and retrieving replacements; another search depends on its estimated value and remaining budget.Source: §3.4 · p. 5

3.5 Controlled Evaluation

  • The controlled evaluation holds the retriever, generator, verifier, answer prompt, evaluator, and budget fixed across compared controllers.The main answerability calibrator is fitted on HotpotQA training data, selected on development data, then reused unchanged across all six benchmarks.Source: §3.5 · p. 5

4 Experiments

BELIEFRAG is evaluated in a shared harness across six QA benchmarks and two backbones, with diagnostic benchmarks reported separately. The experiments compare retrieval controllers using answer-quality, efficiency, and supporting-fact measures under a specified calibration and evaluation protocol.

4 Experiments

  • The shared harness executes controller actions and records evidence, belief state, and cost while keeping the generator, retriever, verifier, answer prompt, evaluator, and budget common across methods.The model writes search queries and final answers but cannot retrieve documents or execute actions itself.Source: §4 · pp. 5–6

4.1 Tasks and Benchmarks

  • The six main QA benchmarks combine three multi-hop tasks, which test whether more evidence is needed, with three factoid tasks that serve as controls for cases where one retrieval or parametric knowledge may suffice.RGB and HoloBench are additional diagnostic benchmarks, reported separately because their metrics are not directly comparable with QA accuracy.Source: §4.1 · p. 6

4.2 Baselines and Backbones

  • The primary comparison uses gpt-oss-120b and repeats the six-dataset evaluation with Qwen3-32B against retrieval baselines ranging from closed-book and static retrieval to adaptive and iterative controllers.BELIEFRAG uses the evolving evidence state to choose whether to correct, retrieve again, rewrite, answer, stop, or abstain.Source: §4.2 · p. 6

4.3 Metrics and Evaluation Protocol

  • Headline QA comparisons use deterministic normalized token-level answer F1, while efficiency is mean total tokens per question and supporting-fact recovery is reported as evidence recall when annotations exist.The protocol also reports exact match and supplementary LLM-judged semantic accuracy; calibrator fitting and selection are separated from evaluation.Source: §4.3 · pp. 6–7

5 Results

Across two backbones, BELIEFRAG improves mean F1 over fixed Iterative RAG while using fewer tokens. Analyses attribute the trade-off to corrective evidence replacement and calibrated answerability, while showing that calibration does not ensure transfer across different evidence sources.

5.1 Main Result: Quality and Cost

  • BELIEFRAG reaches mean F1 0.572 versus 0.555 for fixed Iterative RAG, using 3.89k rather than 6.35k tokens per question, about 39% fewer.With Qwen3-32B, it reaches 0.552 mean F1 versus 0.523 for Iterative RAG while using 35% fewer tokens.Source: §5.1 · p. 7

5.2 Analysis 1: Which Parts Create the Gain?

  • The full controller reaches 0.680 multi-hop ACC; removing correction, removing verification, or pruning weak evidence without replacement each reduces ACC to 0.630, showing gains require replacement rather than detection alone.Live answerability reaches 0.685 ACC across the three multi-hop datasets; permuting its score across matched states yields 0.650, while freezing it throughout the trajectory yields 0.655. It averages 1.48 retrieval rounds versus 1.81 for both controls.Source: §5.2 · pp. 7–8

5.3 Analysis 2: Do Thresholds Keep the Same Meaning?

  • Calibration reduces the raw answerability score’s cross-dataset median spread from 0.352 to 0.048, placing a fixed threshold in a more consistent operating region.The calibrated answer gate fires on 67.7% of audited main-table states; the conflict gate fires on 72% of counterfactual-evidence cases and 8% of ordinary QA.Source: §5.3 · p. 8

5.4 Analysis 3: What Information Is in the Belief State?

  • The joint belief state reaches AUC 0.816 versus 0.761 for the strongest single signal, uncertainty, although several dimensions are correlated and unevenly useful.Sufficiency and reliability have ECE values of 0.039 and 0.045; higher values of these signals are associated with answering, while higher uncertainty, gap, and cost are associated with stopping or further retrieval.Source: §5.4 · pp. 8–9

5.5 Analysis 4: What Happens When Evidence Is Imperfect?

  • At 80% irrelevant-noise injection, BELIEFRAG retains 0.87 ACC; under counterfactual evidence, it follows the false answer on 25% of questions and recovers the true answer on 30%.Under evidence-source transfer, average AUC drops from 0.775 to 0.651, so calibration does not guarantee transfer across qualitatively different sources.Source: §5.5 · p. 10

6 Conclusion and Limitations

BELIEFRAG treats retrieval as sequential evidence control and reports a quality–cost trade-off across two backbones. Its gains are attributed mainly to evidence replacement and answerability-guided stopping, with several implementation and transfer limitations.

6 Conclusion and Limitations

  • BELIEFRAG reaches 0.572 mean F1 with 39% fewer tokens than Iterative RAG on gpt-oss-120b, and 0.552 with 35% fewer tokens on Qwen3-32B.The conclusion also lists lexical retrieval, 100 questions per dataset, backbone-specific calibration, self-judged auxiliary metrics, and degraded transfer under evidence-source shift as limitations.Source: §6 · p. 10
  • Gains mainly come from evidence replacement and answerability-guided stopping.Source: §6 · p. 10
  • The reported scope is limited by lexical retrieval, only 100 questions per dataset, backbone-specific calibration, self-judged auxiliary metrics, and degraded transfer under evidence-source shift.Source: §6 · p. 10

A Category 1: Implementation Details

The implementation maintains a persistent estimate of the evidence condition and uses diagnostics to guide a rule-based controller. Its evidence workspace can be revised, while the answerability calibrator and ordered decision branches determine when to answer, retrieve, correct, stop, or abstain.

A Category 1: Implementation Details

  • The operational belief state is a persistent estimate of the evidence condition computed from observable diagnostics and the previous state, not an exact Bayesian update over a latent environment variable.Source: A Category 1: Implementation Details · p. 14

A.1 Evidence Workspace and Diagnostics

  • The evidence set is a mutable workspace: retrieval enlarges it, rewriting changes the query, and verification can remove unhelpful passages.Dropped passage identifiers are retained to prevent later retrieval from silently reintroducing them.Source: §A.1 · p. 14
  • Relevance, novelty, and normalized cost are computed locally, while support, conflict, uncertainty, and evidence gap come from a structured verifier call.The verifier observes evidence but does not mutate it; removal occurs only when the controller selects VERIFY.Source: §A.1 · p. 14

A.2 Belief Mapping and Default Parameters

  • The state update uses a weighted diagnostic vector in logit space, while cost is observed directly and default coefficients are unfitted priors.The main stopping probability instead uses a separately fitted answerability calibrator.Source: §A.2 · p. 14

A.3 Main Answerability Calibrator

  • The main answerability calibrator is a logistic model over seven diagnostics, fitted to generator success judged on 300 HotpotQA training examples and selected on 156 development examples.It is frozen and reused on five other QA datasets; its target is operational generator success, not entailment by retained evidence alone.Source: §A.3 · p. 14

A.4 Terminal Controller

  • The rule-based controller prioritizes finishing correction, correcting problematic evidence, answering above threshold, retrieving when value is sufficient, rewriting after low novelty, then stopping or abstaining.Defaults include an answerability threshold of 0.50, conflict threshold 0.50, reliability threshold 0.30, minimum retrieval value 0.10, and low-novelty threshold 0.20.Source: §A.4 · p. 15

B Category 1: Worked HotpotQA Trajectory

The worked example illustrates how verification and evidence-aware querying alter a retrieval trajectory, while the accompanying prompts specify the model-facing roles. The example reaches the correct answer without retrieving a complete two-hop proof.

B Category 1: Worked HotpotQA Trajectory

  • The HotpotQA trace shrinks the retained evidence from five passages to one, then searches using the bridge entity Ringo Starr found in retained evidence.The second retrieval does not recover a Ringo-Starr-to-Beatles passage, so the correct final answer combines the retained clue with frozen parametric knowledge; evidence-recall is annotation coverage, not proof of distinct retrieval of every supporting passage.Source: B Category 1: Worked HotpotQA Trajectory · p. 15, Table 5 · p. 16

C.1 Backend System Prompt

  • The backend prompt requires valid JSON and forbids external tools, retrieval, or hidden assumptions.Source: §C.1 · p. 16

C.2 Verifier Prompt (verifier_v1)

  • The verifier returns support, conflict, gap, and uncertainty judgments for the evidence and draft answer, and identifies passages to drop.Source: §C.2 · p. 16

C.3 Evidence-Aware Query Writer (query_refiner_v2)

  • The evidence-aware query writer targets a missing fact, avoids searching for established information, and names a newly identified entity explicitly.It can return SUFFICIENT when nothing further is needed.Source: §C.3 · p. 17

C.4 Query Rewriter (query_rewrite_v1)

  • The query rewriter changes wording or targets a different required fact rather than repeating the current query.Source: §C.4 · p. 17

C.5 Answer Generator (answer_generator_v1)

  • The answer generator is instructed to return a short answer from the documents, but to give its best short answer even when the documents lack it.That fallback permits parametric knowledge in the operational answerability target.Source: §C.5 · p. 17

D Category 1: Controlled Harness and Reproducibility

The controlled harness standardizes action execution and cost accounting while comparing mechanism-focused baseline variants. Backend behavior and configuration impose reproducibility and measurement qualifications.

D Category 1: Controlled Harness and Reproducibility

D.1 Execution Invariants

  • Each episode observes evidence, computes diagnostics, updates belief, proposes one legal action, and executes it through the harness.The harness enforces feasible actions; only the evidence manager may mutate the evidence set, and mutations are logged.Source: §D.1 · pp. 17–18

D.2 Shared Budgets and Cost Accounting

  • Token cost includes retrieval context each time it is model-facing, rewriting, verification, answer generation, and controller prompting; retrieval calls are logged separately.Source: §D.2 · p. 18, Table 6 · p. 18

D.3 Baseline Implementations

  • The shared harness isolates different mechanisms across baselines, which are generally common-harness variants rather than exact end-to-end reproductions.Source: §D.3 · p. 18

D.4 Model Backend

  • The Qwen checkpoint details and serving stack are undisclosed, and temperature-zero calls are not fully deterministic across repeats.Uncached reruns may differ; within a backbone, the shared backend also makes the semantic accuracy judge self-judged and unsuitable as a cross-backbone scale.Source: §D.4 · p. 18

D.5 Data Partitions

The appendix specifies separate data partitions and calibration procedures for each backbone, with dedicated holdout splits for causal analyses.

D.5 Data Partitions

  • 300 HotpotQA training examples fit each backbone’s answerability calibrator, while 156 development examples select it by Brier score.The retrieval-value model is also refit for each backbone.Source: §D.5 · p. 18
  • Qwen evaluation stops if either Qwen-specific artifact is missing rather than reusing gpt-oss artifacts.Source: §D.5 · p. 18
  • Causal analyses use deterministic 40/30/30% xfit/xsel/xhold splits, with holdout labels excluded from fitting.Splits are assigned by salted question-id hash.Source: §D.5 · p. 18
  • Within each dataset, methods share the corpus, retrieval and generation components, prompts, evaluator, and budget; configurations are recorded in run metadata.Source: §D.5 · p. 18

D.6 Metrics and Aggregation

The evaluation records answer quality, evidence, actions, and resource use per episode, then aggregates repeated seeds and estimates uncertainty through resampling.

D.6 Metrics and Aggregation

  • Normalized token F1 is the primary metric, while acc_cover and acc_judge are supplementary measures.Headline cross-dataset results use deterministic token F1 rather than the earlier LLM-judge interval.Source: §D.6 · p. 18
  • Abstentions score zero on answer metrics, and each episode logs answer or abstain status, evidence recall when annotated, retrieval rounds, actions, calls, tokens, and latency.The harness also stores EM and normalized token F1.Source: §D.6 · p. 18
  • Repeated seeds are averaged within question before bootstrap; mean intervals use 2,000 resamples and paired within-dataset comparisons use 10,000.Source: §D.6 · p. 18

E Category 2: Complementary Results

Complementary results cover HoloBench aggregation retrieval, Qwen3-32B answer quality and costs, and secondary gpt-oss-120b metrics, with table captions specifying the reported scopes.

E Category 2: Complementary Results

  • Table 7 reports Qwen3-32B F1, EM, and supplementary Judge scores by dataset, with mean F1 over six datasets and n = 100 per dataset cell.Source: Table 7 · p. 19
  • Table 8 reports Qwen3-32B token costs and logged retrieval-recall fields; BeliefRAG open-domain recall values are unavailable, and supporting-fact recall is used only where annotated.Source: Table 8 · p. 19
  • Table 10 covers gpt-oss-120b cost and multi-hop evidence recall, while Table 9 reports secondary EM and Judge metrics with simple six-dataset means.Table 10 reports evidence recall only for the three multi-hop benchmarks with supporting-fact annotations.Source: Table 9 · p. 19, Table 10 · p. 20
  • 0.145 row recall after reading 9.73 rows on average for BeliefRAG compares with 0.152 after 22.98 rows for fixed Iterative retrieval; BeliefRAG has higher recall per row read.The figure frames this as selection efficiency, not a solution to aggregation retrieval.Source: Figure 7 · p. 20
  • The HoloBench structural ceiling is 0.985 at 50 rows, while absolute recall remains far below it; these numbers are not directly comparable with QA F1 scores in Table 3.Source: Figure 7 · p. 20

Related papers

Loading 2609.39139v1…