Source-linked AI summary
Predicting Retrieval Utility and Answer Quality in Retrieval-Augmented Generation
Fangzheng Tian, Debasis Ganguly, Craig Macdonald
TL;DR
RAG answer quality depends on retrieved context, motivating prediction of both context utility and final answer quality. The paper combines QPP-based relevance estimates with context perplexity and document-quality signals in linear-regression models. Across rankers and context sizes, ensembles outperform individual predictors, although utility prediction remains harder than answer-quality prediction.
Problem
RAG lacks systematic prediction of both concatenated-context utility and answer quality, despite their relevance to evaluating retrieved context and generated responses.
Source: §1 · p. 2Method
The paper adapts QPP and combines relevance estimates with context and answer perplexity, readability, and document-quality signals in linear-regression predictors for RPP and GPP.
Source: §1 · pp. 2–3Results
Ensembles consistently outperform individual predictors across rankers and context sizes, while RPP is harder to predict than GPP.
Source: §1 · p. 3, §6 · p. 13Takeaways & Limitations
Per-query context optimization requires improving predictor precision before these estimates can be effectively leveraged in adaptive RAG systems.
Source: §6 · p. 13Per-query context optimization still requires more precise predictors.
Source: §6 · p. 13
Abstract
from arXiv · showhide
The quality of answers generated by large language models (LLMs) in retrieval-augmented generation (RAG) is largely influenced by the contextual information contained in the retrieved documents. A key challenge for improving RAG is to predict both the utility of retrieved documents -- quantified as the performance gain from using context over generation without context -- and the quality of the final answers in terms of correctness and relevance. In this paper, we define two prediction tasks within RAG. The first is retrieval performance prediction (RPP), which estimates the utility of retrieved documents. The second is generation performance prediction (GPP), which estimates the final answer quality. We hypothesise that in RAG, the topical relevance of retrieved documents correlates with their utility, suggesting that query performance prediction (QPP) approaches can be adapted for RPP and GPP. Beyond these retriever-centric signals, we argue that reader-centric features, such as the LLM's perplexity of the retrieved context conditioned on the input query, can further enhance prediction accuracy for both RPP and GPP. Finally, we propose that features reflecting query-agnostic document quality and readability can also provide useful signals to the predictions. We train linear regression models with the above categories of predictors for both RPP and GPP. Experiments on the Natural Questions (NQ) dataset show that combining predictors from multiple feature categories yields the most accurate estimates of RAG performance.
1 Introduction
RAG uses retrieved documents to supplement an LLM’s parametric knowledge during k-shot generation [38], [32], [24]. This work predicts both the utility of retrieved context and the quality of generated answers, using relevance, LLM-centered signals, and document-quality features.
1 Introduction
- RPP predicts the answer-quality gain from k-shot over zero-shot generation, while GPP predicts whether context supports correct, relevant answers.The paper adapts query performance prediction (QPP) approaches to these two tasks.Source: §1 · p. 2
- Predicting context utility and answer quality could inform query-specific RAG workflows, but integrating these estimators into adaptive systems is left for future research.Such workflows may vary retrieved-document counts or ranking models by query characteristics.Source: §1 · pp. 1–2
- Context perplexity can signal when a topically relevant document is misaligned with the LLM’s knowledge and fails to support a correct answer.In the NBA MVP example, the retrieved passage omits the player’s name even though zero-shot generation gives the correct answer.Source: §1 · p. 3
- Readability and query-agnostic document-quality characteristics may also inform predictions of context utility and generated-answer quality.The paper proposes these signals alongside relevance and perplexity.Source: §1 · p. 3
- PreGen uses retriever-only information for greater efficiency, while PostGen can use generated-answer signals for improved effectiveness at modestly lower efficiency.Generated-answer perplexity is one such PostGen signal.Source: §1 · p. 3
- A linear-regression ensemble consistently outperforms individual predictors across retrieval models and context sizes.The paper combines predictors from multiple feature categories for RPP and GPP.Source: §1 · p. 3
2 Related Work
Prior work studies retrieval prediction, document readability and quality, and several approaches to RAG utility or answer-quality estimation. The paper identifies a gap in systematic prediction of concatenated-context utility and answer quality using existing retrieval-analysis signals.
2 Related Work
- Earlier RAG prediction work includes answer uncertainty, token uncertainty, retrieval-need prediction, and LLM assessment of individual documents, but not systematic context-level prediction.Answer-level semantic uncertainty approaches typically require multiple sampled generations; one context-utility definition based on uncertainty reduction is disconnected from factuality and relevance.Source: §2 · p. 4
- QPP estimates retrieval effectiveness using retrieval-score distributions, coherence, dense query–document representations, or supervised query-and-document text models.These methods provide retriever-centric signals from top-retrieved documents and the query.Source: §2 · p. 4
- Human-oriented document evaluation uses readability features and query-agnostic quality estimates, but their effects on LLM readers remain underexplored.Readability measures can include sentence length, word difficulty, and syntactic complexity.Source: §2 · p. 4
- Prior work had not systematically predicted RAG context utility and answer quality; the closest cited effort assessed individual documents rather than concatenated contexts.That effort used LLM-based utility estimation without exploring QPP or document-quality signals.Source: §2 · p. 4
3 Predicting Retrieval Utility and Answer Quality
The paper defines RPP to estimate retrieved-context utility relative to zero-shot generation and GPP to estimate final answer quality. It combines retriever-centric relevance, reader-centric perplexity, and query-agnostic document-quality predictors using linear regression.
3.1 Prediction Task Description
- RPP estimates whether retrieved documents improve answer quality over zero-shot generation, while GPP predicts the quality of the final RAG answer.RPP targets context utility; GPP targets answer quality under the current RAG configuration.Source: §3.1 · p. 5
3.2 Information Sources for RPP and GPP
- Predictors cover retrieved-context relevance, LLM-based context or answer perplexity, and query-agnostic document quality and readability.The NQ prompt places the opening of an answer unit at the end to elicit an immediate answer.Source: §3.2 · pp. 6–7, Figure 2 · p. 7
- QPP methods provide retriever-centric relevance signals, but treating documents independently makes them less able to capture interactions in the concatenated LLM context.Context perplexity (PerpC) measures how well retrieved context aligns with the LLM’s expectations given the query; answer perplexity (PerpA) reflects confidence in the generated answer.Source: §3.2 · p. 6
3.3 Combining Multiple Predictions
- Linear regression combines predictor outputs on training queries to estimate either context utility for RPP or answer quality for GPP.The learned weights minimize squared error between predictions and ground-truth targets.Source: §3.3 · p. 7
4 Experimental Setup
The experiments evaluate RPP and GPP on Natural Questions using three retrieval configurations and measure prediction accuracy with Spearman’s ρ. They test individual QPP predictors and linear-regression combinations of relevance, perplexity, and document-quality signals.
4 Experimental Setup
- The experiments compare single QPP predictors with regression combinations, testing context perplexity, query-agnostic readability and quality, and post-generation answer perplexity.The research questions progressively add predictor groups to assess their contributions to RPP and GPP accuracy.Source: §4 · pp. 7–8, §4.1 · p. 9
- On NQ, prediction accuracy is Spearman’s ρ against context utility for RPP and answer F1 for GPP, evaluated on 3,610 test queries.Contexts are retrieved from a 2018 English Wikipedia snapshot; answer quality is F1 against the golden answer.Source: §4 · p. 8
- The three configurations are BM25, BM25 reranked with MonoT5, and E5; their average answer F1 scores are 0.3284, 0.4019, and 0.4798, respectively.These scores use the top two retrieved documents as context.Source: §4 · p. 8
4.1 Investigated Predictors
- QPP baselines include retrieval-score, embedding, coherence, and query-document-model signals; document measures include readability metrics and query-agnostic QualT5 scores.PerpC is used before generation and PerpA after generation; the combined predictors are trained on 8,757 NQ development queries.Source: §4.1 · pp. 8–9
- Readability and quality measures summarize individual documents with maximum, minimum, and average scores, and also score the concatenated context directly.The measures include Dale-Chall, Spache, Flesch–Kincaid, Gunning Fog, Coleman–Liau, and QualT5.Source: §4.1 · p. 9
5 Results
The results assess QPP and combined predictors for RPP and GPP, then examine how prediction accuracy changes with context size. Combining predictor signals improves accuracy, with benefits varying by task, retriever, and context size.
5.1 Main Observations
- QPP estimates are positively correlated with context utility and answer quality, but their accuracy varies across retrieval configurations.QPP methods generally achieve higher correlations for GPP than RPP, and no method performs best across all rankers.Source: §5.1 · p. 10
- Aggregating multiple QPP predictors generally outperforms the best individual QPP method, especially for BM25 and E5.The section reports results for combinations using linear regression.Source: §5.1 · p. 11
- Adding PerpC improves GPP prediction accuracy across all retrievers, but improves RPP accuracy only with BM25.For BM25 GPP, the reported correlation rises from 0.2454 to 0.2821; the passage suggests PerpC is more discriminative for weaker retrievers.Source: §5.1 · p. 11
- Readability and document-quality predictors provide small but consistent gains, mostly for BM25 and MonoT5, and add only limited value for RAG prediction.For BM25, readability alone achieves the best pre-generation results.Source: §5.1 · p. 11
- Combining PerpA with pre-generation signals improves both RPP and GPP over either signal alone, but requires waiting for the full answer.For E5 GPP, the combined correlation is 0.3915, versus 0.2873 for PerpA and 0.2952 for QPP; the improvements are significant under Fisher’s z-test.Source: §5.1 · p. 11
5.2 Prediction Accuracy for Varying Context Size (k)
- PostGen-Ensmbl yields the best overall performance across context sizes, while pre-generation prediction accuracy declines as k grows for both RPP and GPP.The contribution of pre-generation signals diminishes as k increases, as the PerpA and PostGen-Ensmbl curves converge.Source: §5.2 · p. 12, Figure 3 · p. 12
6 Conclusions
The paper defines RPP for context utility and GPP for answer quality. Its experiments find that combining diverse predictors with linear regression outperforms individual predictors, while broader evaluation and greater predictor precision remain future needs.
6 Conclusions
- RPP targets context utility, while GPP targets answer quality.The conclusion describes these as two prediction tasks for RAG.Source: §6 · p. 12
- Relevance estimates, retrieved- and generated-content perplexity, readability, and relevance priors are useful information sources for RPP and GPP.An ensemble learned via linear regression consistently outperforms individual predictors across rankers and context sizes.Source: §6 · p. 13
- Predicting utility is harder than predicting answer quality because utility also depends on how retrieved context interacts with the LLM’s inherent knowledge.The paper states that predictor precision must improve for effective per-query context optimization and calls for evaluation across more datasets and LLMs.Source: §6 · p. 13