Source-linked AI summary
Predicting the Benefit of Retrieval Augmentation in Open-Domain Question Answering
Or Dado, David Carmel, Oren Kurland
TL;DR
Because retrieved information does not always help answer quality, the paper studies how to predict RAG’s benefit for individual questions. It evaluates gain predictors, introduces an internal-state probe, and reports that selective RAG outperforms applying retrieval to every question.
Problem
Retrieved content can be unhelpful or distracting, so the paper asks whether RAG’s benefit over no retrieval can be predicted for each question.
Source: §1 · p. 1Method
The paper evaluates retrieval- and answer-based predictors, models semantic relations among questions, passages, and generated answers, and probes the LLM’s state before answer generation.
Source: §1 · p. 1, §8 · pp. 13–14Results
Selective RAG consistently outperforms applying RAG uniformly, while Internal Prober substantially narrows the performance gap between post-generation and post-retrieval methods.
Source: §8 · p. 14Takeaways & Limitations
Per-question gain prediction offers a route to selectively apply retrieval, with the stated aim of improving system effectiveness and computational efficiency.
Source: §1 · p. 1, §8 · p. 14Answer-quality evaluation is constrained because EM and token-level F1 miss semantic aspects, while LLM judges are model-sensitive and computationally costly.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
While retrieval augmented generation has become a common approach for enhancing question answering systems, retrieval is not universally advantageous. We study the problem of predicting whether incorporating external retrieved information is likely to improve response quality for a given question. To this end, we evaluate a range of prediction methods that are based on retrieval signals, answer characteristics, and semantic consistency between generated responses and retrieved passages. We further devise a predictor that probes the LLM's internal state. Its prediction performance significantly narrows the performance gap between post-generation methods which are computationally demanding and pre-generation (post-retrieval) methods. We use the prediction methods to devise a selective retrieval framework that dynamically chooses between retrieval and non-retrieval generation modes per question. Experimental results demonstrate that selectively applying retrieval augmentation yields answer quality that transcends that of using retrieval for all queries.
1 Introduction
The introduction asks whether RAG’s benefit can be predicted per question, since retrieved content can be unhelpful or distracting. It evaluates predictors and proposes an internal-state probe, then applies gain prediction to selective RAG.
1 Introduction
- The motivating problem is that retrieved content may be unhelpful or distracting, potentially degrading answer quality.The paper asks whether RAG’s benefit relative to not applying RAG can be predicted for a given question.Source: §1 · p. 1
- Pre-retrieval predictors are ineffective for RAG gain prediction, while some post-retrieval predictors have limited but non-negligible capability.Supervised post-retrieval methods substantially outperform unsupervised methods.Source: §1 · p. 1
- The proposed post-generation predictor models semantic relationships among the question, retrieved passages, and generated answer, achieving the highest prediction performance among methods considered.It incorporates the LLM-generated response in addition to pre-generation information.Source: §1 · p. 1
- Internal Prober examines the LLM’s state before answer generation and substantially narrows the performance gap between post-generation and post-retrieval methods.It offers a practical, efficient alternative to post-generation gain prediction.Source: §1 · pp. 1–2
- Selective RAG consistently outperforms applying retrieval uniformly across all questions.The framework applies RAG per question only when improvement in answer quality is predicted.Source: §1 · p. 2
- The study investigates prediction strategies across two LLMs, two retrieval systems, and three question-answering datasets.The authors describe supervised post-retrieval and post-generation predictors as promising candidates for real-world RAG systems.Source: §1 · p. 2
2 Related Work
Related work examines signals for predicting RAG utility, adaptive retrieval decisions, and prior direct RAG-gain prediction. The paper distinguishes its approach by directly predicting gain and probing the LLM state before answer generation.
2 Related Work
- Huly et al. formalized RAG gain for text completion, and this work modifies the definition for open-domain question answering.The paper characterizes question answering as a setting with complex interaction between retrieval relevance and answer quality.Source: §2 · p. 2
- Prior work indicates retrieval utility depends on context length, relevant-passage position, and conflicts between retrieved evidence and LLM knowledge.Dai et al. [13] predicted RAG effects using LLM uncertainty during answer generation; the paper reports its predictors outperform that approach.Source: §2 · p. 2
- Lee et al. [14] used attention-head activations during answer generation to predict mathematical-problem complexity, whereas Internal Prober probes before answer generation for relative RAG answer quality.Source: §2 · p. 2
- Self-RAG [15] and Active-RAG [16] dynamically decide whether to retrieve, while prior adaptive systems also choose retrieval strategies or turns without using retrieval quality or LLM internal knowledge.The paper contrasts these policy and control approaches with predictors that estimate RAG gain as a quantifiable outcome.Source: §2 · p. 2
- Tian et al. [11] studied pre- and post-retrieval predictors and a post-generation uncertainty predictor; this work reports its novel predictors outperform that predictor.It also reports post-generation predictors substantially outperforming post-retrieval predictors, and uses supervision to train novel methods directly on RAG gain.Source: §2 · p. 2
3 Task Definition
The paper predicts the relative answer-quality gain from adding retrieved context, with prediction made without access to the reference answer. It compares answer-quality metrics and finds that RAG gains vary substantially by question, motivating selective retrieval.
3 Task Definition
- RAG gain is the relative log answer-quality change from adding retrieved context, and the prediction task uses no reference answer.The reference answer is used to measure generated-answer quality, but is unavailable to the predictor.Source: §3 · pp. 2–3
3.1 Quality Metrics
- Exact Match and token-level F1 miss semantic aspects such as misinformation and whether an answer satisfies the information need.LLM-as-a-judge evaluation can be biased by the judging model and incurs substantial computational cost; BERT-based shallow models are cited as an alternative.Source: §3.1 · p. 3
- Quality metrics correlate from 0.61 to 0.84 but are not interchangeable, and gain estimates also show substantial agreement across metrics.QE5 and QCE cluster together, while QNLI correlates with QEM and QLLM.Source: §3.1 · p. 4, Figure 1 · p. 4
3.2 RAG-gain Analysis
- More than 30% of HotpotQA questions have negative gain, despite positive average gains across datasets and quality metrics.Gain distributions have a pronounced peak near zero and heavy tails skewed positive.Source: §3.2 · p. 4
4 Gain Prediction Methods
The section surveys predictors for estimating retrieval effectiveness before retrieval and introduces the broader timing-based categories used to study RAG-gain prediction. Pre-retrieval predictors use the question and corpus statistics.
4 Gain Prediction Methods
- Prediction methods are grouped by when they operate: pre-retrieval uses the question and corpus statistics, post-retrieval adds retrieved passages, and post-generation also uses the generated answer.Pre-retrieval and post-retrieval methods are adapted from ad hoc retrieval prediction.Source: §4 · pp. 4–5
4.1 Pre-Retrieval Predictors
- Pre-retrieval predictors estimate retrieval effectiveness before retrieval by relating query terms to their statistical properties in the corpus.These are unsupervised measures originally developed for ad hoc retrieval.Source: §4.1 · p. 5
- SCQ sums query-term tf-idf weights, IDF averages term idf values, and VAR averages the across-document variances of query-term tf-idf weights.IDF and VAR use term rarity and discriminative variation as retrieval-effectiveness signals.Source: §4.1 · p. 5
4.2 Post-Retrieval Predictors
Post-retrieval predictors use the retrieved list alongside the query to estimate whether the top passages will improve answer quality. The section covers unsupervised retrieval-score signals and a supervised query-and-passage predictor.
4.2 Post-Retrieval Predictors
- Post-retrieval predictors test whether properties of the retrieved list correlate with gain from augmenting answers with the highest-ranked passages.They use the retrieved list in addition to query and corpus statistics.Source: §4.2 · p. 5
4.2.1 Unsupervised predictors
- WIG contrasts the top-k retrieval-score average with the corpus score, while NQC normalizes top-k score variation by the corpus score.WIG also has an unnormalized variant; NQC assumes higher score variation indicates reduced query drift and better retrieval effectiveness.Source: §4.2.1 · pp. 5–6
- SMV combines the mean and variance of top-k retrieval scores with corpus-score normalization, while REF compares the original list with a reranked reference list.SMV has an unnormalized variant; REF uses Rank Biased Overlap after reranking the top 100 passages.Source: §4.2.1 · p. 6
4.2.2 Supervised predictors
- Bert-Ret feeds the query and top passages to ModernBERT-large, then trains a linear regression layer end to end with MSE loss to predict RAG gain.The model is inspired by BERT-QPP.Source: §4.2.2 · p. 6
4.3 Post-Generation Predictors
Post-generation predictors use generated answers, their uncertainty, or their agreement with retrieved context to estimate RAG gain. Supervised variants use answer inputs to predict gain, with different costs depending on whether one or both answers are generated.
4.3.1 Unsupervised predictors
- Uncertainty estimates gain from the reduction in maximum token-level entropy between answers generated with and without RAG.It assumes that a larger reduction reflects a larger quality gain.Source: §4.3.1 · p. 6
- Entailment predicts gain as the difference between NLI entailment scores for RAG and no-RAG answers against the question and top-five retrieved passages.The generated answer is concatenated with the question as the hypothesis.Source: §4.3.1 · p. 6
4.3.2 Supervised predictors
- Bert-Gen variants use one or both generated answers with the question and retrieved passages; Bert-Gen-2A compares both answers at the cost of generating both.Single-answer variants can make an early decision about whether to generate the alternative answer.Source: §4.3.2 · pp. 6–7
- Figure 3 depicts two independent LoRA adapters on a shared frozen LLM, with separate branches for the question alone and the question with retrieved passages.The branches produce paired quality estimates that are combined into gain predictions for three metrics.Source: Figure 3 · p. 7, §4.3.2 · p. 7
4.4 Semi-Generation (Internal Prober)
Internal Prober predicts RAG gain from LLM hidden states without generating answer tokens. Its paired input branches estimate the marginal value of retrieved context, while trained LoRA adapters reshape the shared model for the prediction task.
4.4 Semi-Generation (Internal Prober)
- Internal Prober estimates RAG gain from final hidden states during prompt processing, avoiding full autoregressive answer generation.It probes the model before generating any output tokens.Source: §4.4 · p. 7
- A shared frozen LLM processes the question alone and the question with the top-five retrieved passages through separate LoRA adapters.Each branch has a linear regression head; paired quality scores are combined to approximate gain and trained with MSE.Source: §4.4 · p. 7
- Unlike a lightweight probe on fixed representations, Internal Prober trains LoRA adapters to reshape hidden states for RAG gain prediction.At inference, it replaces the sequential forward passes needed to generate both candidate answers with two parallel forward passes.Source: §4.4 · p. 7
5 Experimental Setup
The experiments evaluate gain prediction across two instruction-tuned LLMs, two retrieval methods, and three open-domain QA datasets. Predictors are trained or tuned against gain correlations, with separate training setups for cross-encoders and Internal Prober.
5 Experimental Setup
- The evaluation uses Falcon3-10B-Instruct and Llama-3.1-8B-Instruct, BM25 and E5 retrieval, and NQ-Open, HotpotQA, and TriviaQA.Prediction quality is measured by Pearson correlation between predicted and actual gain; the study also uses Williams’ test and Bonferroni correction for significance testing.Source: §5 · p. 8
5.1.1 Cross-Encoder-based Predictors:
- Supervised gain prediction is framed as regression; cross-encoders fine-tune ModernBERT-large on concatenated questions, passages, and generated answers.They use a pooled [CLS] representation and a linear regression layer, training separate configurations for each LLM, retriever, dataset, and metric.Source: §5.1 · p. 8, §5.1.1 · p. 8
5.1.2 Internal Prober:
- Internal Prober uses a shared frozen LLM with separate LoRA adapters and linear heads for question-only and question-plus-passages inputs.The heads predict all three metrics jointly; gains are normalized before MSE training.Source: §5.1.2 · p. 9
5.2 Hyper-Parameters Tuning
- Hyperparameters are selected by maximizing validation-set gain correlation, defined as the sum of correlations across the three quality measures.Post-retrieval and post-generation predictors use five passages for augmentation; max pooling is strongest for Uncertainty across evaluated configurations.Source: §5.2 · p. 9
- The best Internal Prober configuration applies LoRA across all layers with rank 16, scaling factor 32, and learning rate 5 × 10^-5.The tuning compared adapter placement across all layers with placement limited to final layers.Source: §5.2 · p. 9
6 Results
Across the evaluated configurations, Bert-Gen-2A achieves the strongest gain-prediction correlations, while Internal Prober narrows the gap to post-generation methods with lower generated-token computation. The results also compare predictor families, probe-generation length, and consistency across models, retrieval methods, and quality metrics.
6.0.1 Pre-retrieval Predictors.
- Pre-retrieval predictors have approximately 0–0.1 correlation with actual gain; post-retrieval methods improve on them, with score-distribution methods around 0.1–0.2.REF outperforms score-distribution predictors in most configurations.Source: §6.0.1 · p. 9, §6.0.2 · p. 9
6.0.2 Post-retrieval Predictors.
- REF reaches 0.25–0.30 on NQ-Open with BM25, while Bert-Ret reaches 0.45–0.49 and significantly beats other pre- and post-retrieval predictors in 34 of 36 configurations.Table 1 reports Pearson correlations across datasets, LLMs, retrieval methods, and quality metrics.Source: §6.0.2 · p. 10, Table 1 · p. 10
6.0.3 Post-Generation Predictors.
- Uncertainty reaches correlations of 0.35–0.55 across most configurations, while Entailment varies; Bert-Gen-1A variants outperform Bert-Ret but trail Bert-Gen-2A.The two Bert-Gen-1A variants have no consistent winner across configurations.Source: §6.0.3 · pp. 10–11
- 0.87–0.89 on TriviaQA and 0.71–0.79 on NQ-Open are Bert-Gen-2A’s peak Pearson correlations, and it significantly outperforms all other predictors across configurations.The authors identify it as state of the art for RAG gain prediction to their knowledge.Source: §6.0.3 · p. 11
6.1 Internal Prober performance
- 82–95% fewer generated-token computations are needed by Internal Prober on average, while its prediction significantly exceeds Bert-Ret and often exceeds Uncertainty, Entailment, and Bert-Gen-1A.It remains significantly below Bert-Gen-2A.Source: §6.1 · p. 11
- For Falcon with BM25 on TriviaQA, Internal Prober’s prediction correlation rises with generated-token count K, with the largest increase early and saturation around K = 8.Figure 4 compares the three metrics against Bert-Gen-2A reference lines.Source: §6.1 · p. 11, Figure 4 · p. 11
6.2 Cross-Architecture and Cross-Metric Consistency
- Predictor rankings are largely consistent across BM25 and E5, both LLMs, and all three answer-quality metrics, though QE5 correlations are usually higher.The authors relate this difference to the smoother distribution of QE5 values compared with QCE and QNLI.Source: §6.2 · p. 12
7 Selective RAG
Selective RAG uses predicted gain to choose retrieval or no-retrieval answers per question. Across nearly all configurations, prediction-based selectors outperform Always RAG, though results vary by selector, retrieval system, and evaluation metric.
7 Selective RAG
- When predicted gain is positive, the selector uses RAG; otherwise, it generates an answer without retrieval.The evaluation measures selected-answer quality and the accuracy of each retrieval decision.Source: §7 · p. 12
- Answer-quality scores from selectors trained on different metrics are not directly comparable: the E5 metric’s 0.7–1.0 score range can yield higher scores without higher accuracy.The section therefore interprets performance relative to baselines.Source: §7 · p. 12
- Always RAG significantly outperforms No RAG, but a substantial gap to Oracle remains for prediction-based selectors to reduce.Table 2 reports answer quality, accuracy, and the portion of the Always RAG–Oracle gap achieved.Source: §7 · p. 12, Table 2 · p. 13
- Across nearly all configurations, prediction-based selectors outperform Always RAG in answer quality and accuracy; Bert-Gen-2A has the strongest, most consistent gains and often recovers Oracle’s performance gap.Its gains are statistically significant over baselines in almost all settings.Source: §7 · p. 12, Table 2 · p. 13
- Internal Prober performs competitively and often matches or exceeds Bert-Gen-2A under QNLI, but is less consistent across retrievers and more sensitive to retrieval noise.For Falcon/BM25 evaluated with QLLM, Internal Prober outperforms Bert-Gen-2A; it jointly predicts multiple quality metrics, which may contribute to robustness under a metric it was not optimized for.Source: §7 · p. 13
- Selective RAG generally yields larger relative improvements with BM25 than with E5-based retrieval, consistent with greater noise in lexical retrieval and the value of filtering distracting context.Source: §7 · p. 13
8 Conclusions and Future Work
The paper concludes that supervised gain prediction, including Internal Prober, can guide selective RAG. The authors report that selective RAG consistently outperforms applying retrieval uniformly, while noting access, model-size, and computational limitations.
8 Conclusions and Future Work
- The most effective approach is a supervised predictor modeling semantic relationships among the question, retrieved passages, and generated answers.Source: §8 · p. 13
- Internal Prober probes LLM internal representations for gain prediction, providing a computationally efficient alternative to costly post-generation methods.Source: §8 · p. 14
- Selective RAG guided by leading gain predictors consistently outperforms applying RAG uniformly across questions.The authors plan to explore additional signals, combine predictors, and evaluate the methods in real-world QA scenarios.Source: §8 · p. 14
- Internal Prober requires white-box access, evaluation is limited to 8–10B-parameter LLMs, and Bert-Gen-2A is computationally expensive.The authors include Bert-Gen-2A primarily as a strong prediction-performance reference, not a practical deployment solution.Source: §8 · p. 14
A LLM-as-a-Judge prompt
The appendix prompt asks an LLM judge to assess an actual answer against expected answers and return reasoning plus a score. Its rubric rewards matches and penalizes irrelevant or mixed correct-and-incorrect content.
A LLM-as-a-Judge prompt
- The judge evaluates the actual answer with respect to expected answers, which may list equally valid alternatives separated by “|”.Source: A LLM-as-a-Judge prompt · p. 17
- A normalized match to any expected answer earns a high score, while correct answers buried in irrelevant content or mixed with incorrect claims are penalized.Source: A LLM-as-a-Judge prompt · p. 17
- The rubric assigns 0–1 to completely incorrect answers, 2–3 to loosely related wrong answers, 4–6 to partial matches or buried answers, 7–8 to mostly correct answers with minor redundancy, and 9–10 to precise matches.Source: A LLM-as-a-Judge prompt · p. 17
- The output is JSON containing reasoning and a 0–10 score, with the judge instructed to explain briefly before scoring.Source: A LLM-as-a-Judge prompt · p. 17