PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Retrieval Without Self-Knowledge? Bringing Uncertainty Back Home

Viktor Moskvoretskii, Maria Lysyuk, Mikhail Salnikov, Nikolay Ivanov, Sergey Pletenev, Daria Galimzianova, Nikita Krayko, Vasily Konovalov, Irina Nikishina, Alexander Panchenko

arXiv:2501.12835v2cs.CLcs.LG

TL;DR

Adaptive retrieval aims to balance LLM knowledge with external information, but existing evaluations often omit LLM-call costs, self-knowledge, or comparisons with established uncertainty methods. The paper evaluates 35 methods across six datasets and finds that established uncertainty methods can match QA performance while improving efficiency and self-knowledge relative to recent approaches.

  • Problem

    Recent adaptive-retrieval studies often overlook LLM-call costs, self-knowledge assessment, and comparisons with established uncertainty-estimation methods.

    Source: §1 · p. 1

  • Method

    The paper compares 35 adaptive retrieval methods, including 8 recent approaches and 27 established uncertainty methods, across six QA datasets using 10 metrics.

    Source: §1 · p. 2

  • Results

    Established uncertainty methods achieve QA performance comparable to recent adaptive-retrieval approaches while being more efficient and showing stronger self-knowledge capabilities.

    Source: §8 · p. 8

  • Takeaways & Limitations

    No single method dominates all evaluated axes, and uncertainty-estimation methods are often more useful than recent, more complex pipelines.

    Source: §1 · p. 2

    The evaluation covers one open-source model within its parameter range and six standard QA datasets; additional architectures and domain-specific datasets could test consistency and generalizability.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Retrieval Augmented Generation (RAG) improves correctness of Question Answering (QA) and addresses hallucinations in Large Language Models (LLMs), yet greatly increase computational costs. Besides, RAG is not always needed as may introduce irrelevant information. Recent adaptive retrieval methods integrate LLMs' intrinsic knowledge with external information appealing to LLM self-knowledge, but they often neglect efficiency evaluations and comparisons with uncertainty estimation techniques. We bridge this gap by conducting a comprehensive analysis of 35 adaptive retrieval methods, including 8 recent approaches and 27 uncertainty estimation techniques, across 6 datasets using 10 metrics for QA performance, self-knowledge, and efficiency. Our findings show that uncertainty estimation techniques often outperform complex pipelines in terms of efficiency and self-knowledge, while maintaining comparable QA performance.

1 Introduction

RAG can help address gaps in LLM knowledge, but irrelevant retrieved information can also contribute to errors. This paper compares adaptive retrieval methods and uncertainty estimation techniques across QA, self-knowledge, and efficiency measures.

1 Introduction

  • RAG can fill gaps in parametric knowledge, but irrelevant external information may accumulate errors or contribute to external hallucinations.Using only parametric knowledge is also limited by LLM knowledge boundaries and may lead to internal hallucinations.Source: §1 · p. 1
  • Recent studies often overlook LLM-call costs, self-knowledge assessment, and comparisons with established uncertainty estimation methods.LLM calls can be especially costly with proprietary models.Source: §1 · p. 1
  • The study evaluates 35 adaptive retrieval methods, including 8 recent approaches and 27 established uncertainty estimation methods, on six datasets using 10 metrics.The datasets include simple one-hop and complex multi-hop questions.Source: §1 · pp. 1–2
  • Established uncertainty estimation methods are often more useful than recent complex pipelines, although no single method dominates every evaluated axis.The comparison covers QA quality, self-knowledge, and efficiency across six datasets.Source: Figure 1 · p. 1, §1 · p. 2
  • The paper also assesses uncertainty methods’ out-of-distribution performance and analyzes the complexity of their functional classes.The authors state that they publish the code and data.Source: §1 · p. 2

2 Related Work

The related work contrasts single-round RAG with adaptive approaches that decide whether external knowledge is needed. It also distinguishes adaptive retrieval strategies and uncertainty-estimation methods.

2 Related Work

  • Single-round retrieval augmentation adds retrieved context before generation, while adaptive RAG may retrieve per query to avoid unnecessary or potentially misleading retrieval.Single-round examples include retrieval for up-to-date information or rare entities (Jiang et al., 2024; Allen-Zhu and Li, 2024).Source: §2 · p. 2
  • Adaptive retrieval uses models’ self-knowledge to decide whether external information is needed, with decisions based on outputs, consistency, internal uncertainty, or a trained classifier.Examples include text outputs (Trivedi et al., 2023), consistency, internal states (Jiang et al., 2023; Su et al., 2024b; Yao et al., 2024), and classifiers (Jeong et al., 2024).Source: §2 · p. 2
  • Uncertainty estimation measures confidence in LLM predictions; white-box methods use internal model details, whereas black-box methods require only model outputs.White-box methods include information-, ensemble-, and density-based approaches.Source: §2 · p. 2

3 Methods

The methods section describes end-to-end adaptive retrieval systems and uncertainty-estimation approaches used to determine when retrieval is needed. The end-to-end systems trigger retrieval using reasoning progress, predicted retrieval choices, uncertainty, or inconsistency.

3.1 End-to-End Methods

  • IRCoT retrieves additional passages when a chain-of-thought step has not produced an answer, using the last generated reasoning sentence as its query (Trivedi et al., 2023).Adaptive RAG instead uses a T5-large classifier to select no retrieval, one retrieval, or multiple IRCoT retrievals (Jeong et al., 2024).Source: §3.1 · p. 2
  • FLARE triggers retrieval when token probability falls below a threshold and regenerates through the next uncertain token or completion (Jiang et al., 2023).DRAGIN filters stopwords to identify uncertainty tokens and uses attention weights and reasoning to reformulate retrieval queries (Su et al., 2024b).Source: §3.1 · pp. 2–3
  • Rowen triggers retrieval from inconsistency scores based on cross-language answer consistency or semantic coherence across models.SeaKR monitors internal states and reranks retrieved snippets to reduce uncertainty and improve factual accuracy (Yao et al., 2024).Source: §3.1 · p. 3

3.2 Uncertainty Estimation Methods

  • The study uses 27 uncertainty-estimation methods and focuses its main-paper discussion on five top-performing methods from different method families.These are Lexical Similarity, Max Entropy, Mean Entropy, SAR, and EigValLaplacian.Source: §3.2 · p. 3
  • Lexical Similarity scores average similarity among sampled responses, while EigValLaplacian sums eigenvalues of a graph built from response consistency (Fomicheva et al., 2020; Lin et al., 2023).Max Entropy takes the maximum token entropy; Mean Entropy averages token entropy (Fomicheva et al., 2020).Source: §3.2 · p. 3
  • SAR reweights token entropy by token relevance and sums the resulting values over the sequence (Duan et al., 2023).Source: §3.2 · p. 3

4 Experimental Setup

The experimental setup uses a shared LLM, retrieval corpus, and protocol for comparing adaptive retrieval and uncertainty-estimation methods. It evaluates QA quality, efficiency, and self-knowledge across single-hop and multi-hop datasets.

4.1 Implementation Details

  • Experiments use LLaMA 3.1-8b-instruct with default generation settings; uncertainty methods follow the AdaptiveRAG protocol, and all methods use BM25 with a Wikipedia corpus.Baseline methods retain their original protocols, prompts, and parameter settings.Source: §4.1 · p. 3

4.2 Datasets

  • The evaluation uses six datasets: SQuAD v1.1, Natural Questions, TriviaQA, MuSiQue, HotpotQA, and 2WikiMultiHopQA.The test subsets contain 500 questions per dataset and cover single-hop and multi-hop queries of varying difficulty.Source: §4.2 · p. 3

4.3 Evaluation

  • The study evaluates downstream QA, efficiency, and self-knowledge, then averages per-dataset method ranks so gains across datasets are compared on a balanced basis.Source: §4.3 · p. 3

4.3.1 Downstream QA Metrics

  • In-Accuracy, Exact Match, and F1 measure downstream answers, with In-Accuracy treated as the primary correctness metric.The authors describe In-Accuracy as more robust to answer variations than Exact Match and a better correctness measure than F1.Source: §4.3.1 · pp. 3–4

4.3.2 Efficiency Metrics

  • Efficiency is measured by average retriever calls and average language-model calls per question.Language-model calls can include repeated calls for uncertainty calculations, question rephrasing, or additional rationales.Source: §4.3.2 · p. 4

4.3.3 Self-Knowledge Metrics

  • Self-knowledge identification is framed as binary classification, with labels derived from the model’s answer accuracy without external knowledge.Each method maps the input to a self-knowledge score, which a classifier uses to predict the label.Source: §4.3.3 · p. 4
  • The self-knowledge metrics include ROC-AUC, score-to-label Spearman correlation, and classification accuracy.They assess identification performance, score alignment, and correctness, respectively.Source: §4.3.3 · p. 4
  • Overconfidence measures incorrect answers treated as known, while underconfidence measures correct answers treated as unknown.Source: §4.3.3 · p. 4

5 Results

The section compares baseline and uncertainty-estimation methods on downstream performance, efficiency, and self-knowledge, using dataset-specific “Best UE” and an “Ideal” retrieval-need oracle. Uncertainty methods are more compute-efficient, while performance varies across tasks and self-knowledge measures.

5.1 Downstream and Efficiency Performance

  • Uncertainty estimation methods outperform baselines on single-hop datasets, perform comparably on multi-hop datasets, and are significantly more compute-efficient.They consistently use fewer than one retriever call and two or fewer LM calls per question.Source: §5.1 · p. 4, Table 1 · p. 5

5.2 Self-Knowledge Performance

  • SeaKR performs strongly on single-hop self-knowledge but declines on multi-hop tasks, where AdaptiveRAG performs better.The results associate internal-state uncertainty with simple questions and reflexive methods with complex reasoning.Source: §5.2 · p. 5
  • Uncertainty methods balance overconfidence and underconfidence better than nearly all baselines, with AdaptiveRAG the exception.Both overconfidence and underconfidence are undesirable deviations from zero.Source: §5.2 · p. 5, Figure 2 · p. 6
  • Uncertainty methods rank first or second for self-knowledge identification overall, while many adaptive methods show near-zero correlation or random predictions.DRAGIN typically dominates downstream tasks but performs poorly on self-knowledge metrics.Source: §5.2 · p. 5

5.3 Uncertainty Estimation

  • EigValLaplacian and Lex-Similarity rank highest for In-Accuracy, while SAR variants and Mean Entropy lead ROC-AUC; the rank correlation is 0.65.The differing rankings indicate that downstream performance and self-knowledge evaluation do not align consistently.Source: §5.3 · pp. 5–6
  • Internal-state methods generally rank lower on In-Accuracy and ROC-AUC but higher in efficiency, while consistency-based methods excel in QA and lag in self-knowledge.Logit-based methods perform better than consistency-based methods on self-knowledge and show better stability to distribution shifts.Source: §5.3 · p. 6
  • The Hybrid method ranks in the top five for In-Accuracy and ROC-AUC and first for efficiency, but calculating all uncertainty estimates adds computational overhead.The overhead may still be justified in retrieval-limited scenarios.Source: §5.3 · p. 6

6 Out-of-Domain Transfer

The OOD evaluation trains on each dataset and tests on every other. QA performance transfers relatively well, while self-knowledge and retrieval-call behavior show less stable transfer.

6 Out-of-Domain Transfer

  • Robustness is assessed by training on each dataset, testing on every other dataset, and comparing relative performance changes with in-domain results.Source: §6 · p. 6
  • OOD QA-performance losses typically remain under 4%, although EigValLaplacian significantly lags behind most methods.Most methods are centered around no change, with occasional positive improvement.Source: §6 · pp. 6–7
  • Self-knowledge transfer has a heavier loss tail and no positive-transfer cases; EigValLaplacian is weakest, while other methods show comparable results.The other methods do not differ significantly from one another.Source: §6 · p. 7
  • After transfer, methods tend to make more retriever calls, and efficiency shows the largest percentage changes without significant differences between methods.Source: §6 · p. 7

7 Uncertainty Estimation Complexity

The section assesses uncertainty-method complexity using Rademacher Complexity and loss-landscape sharpness. Internal-based features are most complex, while moderate complexity can accompany better QA performance.

7 Uncertainty Estimation Complexity

  • Rademacher Complexity measures a hypothesis class’s capacity to fit random noise, with higher values indicating greater complexity.The hypothesis class H_f is induced by uncertainty method f; its predictions are evaluated with Rademacher random variables.Source: §7 · pp. 7–8
  • Internal-based features have the sharpest, most complex functions; consistency-based methods are more complex than logit-based ones, while EigValLaplacian achieves better QA performance.The section reports these patterns from Figures 5 and 6.Source: Figure 5 · p. 8, Figure 6 · p. 8, §7 · p. 8
  • Loss-landscape sharpness measures curvature through the largest Hessian eigenvalue at optimized parameters; higher values indicate more complex, harder-to-generalize functions.The sharpness is defined at w* and uses λ_max, the largest eigenvalue of the Hessian there.Source: §7 · p. 8

8 Conclusion

The paper compares adaptive retrieval approaches across QA performance, efficiency, and self-knowledge, including an in-depth analysis of uncertainty methods. Established uncertainty methods are more efficient and stronger on self-knowledge while achieving comparable performance, but OOD transfer affects the metrics differently.

8 Conclusion

  • Across 10 metrics, 27 established uncertainty methods are compared with 8 recent task-tailored methods; established methods achieve comparable performance to recent adaptive retrieval approaches, with greater efficiency and stronger self-knowledge.Source: §8 · p. 8
  • Established uncertainty methods achieve QA performance comparable to recent adaptive retrieval approaches while being more efficient and exhibiting stronger self-knowledge capabilities.The study evaluates 27 established uncertainty methods alongside 8 recently published methods across 10 metrics.Source: §8 · p. 8
  • OOD transfer produces minimal downstream-performance deviations but a significant self-knowledge decline, with no substantial differences between uncertainty methods.The analysis also identifies higher functional complexity for internal-based methods.Source: §8 · p. 8

Limitations

The evaluation is bounded by its use of one language model and six QA datasets. The authors identify broader model and dataset coverage as useful for testing consistency and generalizability.

Limitations

  • The study uses LLaMA3.1-8b-instruct, so consistency across additional model architectures remains to be validated.The authors describe it as among the best open-source models within its parameter range.Source: Limitations · pp. 8–9
  • The evaluation covers 6 QA datasets, which the authors describe as standard for this task.Source: Limitations · p. 9
  • Adding more QA datasets, particularly domain-specific ones, could provide further insights into method generalizability.Source: Limitations · p. 9

Ethical Considerations

The section discusses retrieval-system bias, classifier sensitivity in uncertainty estimation, and rank-based tests for OOD comparisons. Classifier sensitivity varies by method family, while the statistical procedure uses Friedman and Nemenyi tests.

Ethical Considerations

  • Retrieved documents can be biased and may bias generation in an undesirable direction, so real-world RAG deployments should consider this risk.Source: Ethical Considerations · p. 9

A Classifier for UE

  • Except for NumSemSets, consistency-based methods are more classifier-sensitive than logit-based methods; internal-state and hybrid approaches show the greatest sensitivity.The authors suggest this may reflect less explicit features that capture subtle internal-state changes and are harder to fit and explain.Source: Figure 7 · p. 13, A Classifier for UE · p. 13

B Statistical Tests for OOD Testing

  • OOD comparisons use the Friedman test followed by Nemenyi pairwise tests, with asterisks marking significance.The tests are rank-based and account for repeated measurements and multiple comparisons.Source: B Statistical Tests for OOD Testing · p. 13

C Performance Analysis Across Datasets

The section describes comparisons of QA performance, self-knowledge, efficiency, and transfer across adaptive retrieval methods, then details OOD behavior, hybrid feature importance, experimental settings, methods, and metric correlations.

C Performance Analysis Across Datasets

  • Figure 8 plots Bootstrap Mean In-Accuracy against LLM calls, with point size encoding retrieval calls; normalized metrics show low correlation between downstream and self-knowledge measures.The table caption identifies InAcc, EM, and F1 as downstream metrics and Acc, AUC, and Corr as self-knowledge metrics.Source: C Performance Analysis Across Datasets · p. 13, Figure 8 · p. 14, Table 13 · p. 27
  • For NQ, standard deviations of Acc, EM, and F1 are ≈0.022 ± 0.001 across methods; for SQUAD and Trivia they are ≈0.018 ± 0.006, and methods exhibit similar deviations overall.For one-hop and multihop baselines, Rowen shows the lowest deviation, typically ≤0.01.Source: Table 7 · p. 17, Table 8 · p. 17

D Performance Analysis Across OOD Datasets

  • OOD heatmaps show Accuracy and In-Accuracy changes for Lex-Similarity, MaxEntropy, and EigValLaplacian, and In-Accuracy changes for SeaKR, DRAGIN, FLARE, and AdaptiveRAG; AdaptiveRAG is described as most stable.Table 9 defines transfer values as percentage differences in In-Accuracy, with negative values indicating a loss and positive values a gain versus in-domain testing.Source: Table 9 · p. 18, Figure 9 · p. 21, Figure 10 · p. 21

E Feature Importance Analysis for Hybrid Method of Uncertainty Estimation

F Technical Details

  • Experiments use LLaMA 3.1-8b-instruct, fixed hyperparameters, and bootstrap sampling; uncertainty estimations together take less than one hour, versus 2–36 hours for the listed systems.The listed runtimes include approximately 6 hours for SeaKR, 18 for DRAGIN and FLARE, and 36 for IRCoT.Source: F Technical Details · p. 25

G Methods

  • IRCoT interleaves retrieval with chain-of-thought steps, while AdaptiveRAG classifies questions into no retrieval, one retrieval, or iterative retrieval.AdaptiveRAG trains its classifier on development data from six datasets and labels the most efficient correct strategy.Source: G Methods · pp. 25–26
  • Rowen CL uses cross-language query perturbations and response consistency to detect uncertainty, while Rowen CM compares responses from primary and verification models.The Rowen Hybrid uses the sum of the CL and CM consistency scores to determine whether to retrieve.Source: G Methods · pp. 26–27
  • SeaKR triggers retrieval from internal-state uncertainty and reranks snippets by uncertainty reduction; FLARE and DRAGIN monitor token probabilities, with DRAGIN also reformulating retrieval queries.FLARE retrieves when a generated token probability falls below a threshold; DRAGIN filters stopwords and uses attention and reasoning to form keywords.Source: G Methods · pp. 26–27
Loading 2501.12835v2…