PaperlayerView exact revision on arXiv

Source-linked AI summary

Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation

Ruiyang Ren, Yuhao Wang, Yingqi Qu, Wayne Xin Zhao, Jing Liu, Hao Tian, Hua Wu, Ji-Rong Wen, Haifeng Wang

arXiv:2307.11019v3cs.CLcs.IR

TL;DR

The paper asks how accurately LLMs perceive their factual knowledge boundaries, particularly when retrieval augmentation provides external resources. It evaluates QA and priori/posteriori judgements across models and document conditions, finding that retrieval improves boundary awareness while document relevance strongly affects behavior.

  • Problem

    Existing research lacks a deep understanding of LLMs’ factual-boundary perception when external resources are available.

  • Method

    The paper evaluates QA, priori judgement, and posteriori judgement under retrieval augmentation, testing model, question, and supporting-document factors.

  • Results

    LLMs are often overconfident and handle internal-external conflicts poorly, while retrieval augmentation improves factual-boundary perception and document relevance affects QA and judgement capabilities.

  • Takeaways & Limitations

    Retrieval augmentation can supplement LLMs’ internal knowledge, especially for smaller models, but its usefulness depends substantially on supporting-document relevance.

  • Takeaways & Limitations

    Results involving closed-source LLMs may have limited long-term reproducibility because their APIs can change or become inaccessible.

Abstract

from arXiv · show

Large language models (LLMs) have shown impressive prowess in solving a wide range of tasks with world knowledge. However, it remains unclear how well LLMs are able to perceive their factual knowledge boundaries, particularly under retrieval augmentation settings. In this study, we present the first analysis on the factual knowledge boundaries of LLMs and how retrieval augmentation affects LLMs on open-domain question answering (QA), with a bunch of important findings. Specifically, we focus on three research questions and analyze them by examining QA, priori judgement and posteriori judgement capabilities of LLMs. We show evidence that LLMs possess unwavering confidence in their knowledge and cannot handle the conflict between internal and external knowledge well. Furthermore, retrieval augmentation proves to be an effective approach in enhancing LLMs' awareness of knowledge boundaries. We further conduct thorough experiments to examine how different factors affect LLMs and propose a simple method to dynamically utilize supporting documents with our judgement strategy. Additionally, we find that the relevance between the supporting documents and the questions significantly impacts LLMs' QA and judgemental capabilities. The code to reproduce this work is available at https://github.com/RUCAIBox/LLM-Knowledge-Boundary.

1 Introduction

The paper studies how LLMs perceive their factual knowledge boundaries in open-domain QA, especially when retrieval augmentation supplies external information. It evaluates QA, priori judgement, and posteriori judgement to analyze retrieval’s effects and supporting-document factors.

  • Existing work lacks a deep understanding of whether LLMs can assess their answering ability, reference sufficiency, and answer accuracy when external resources are available.
  • The experiments cover closed-source and publicly available LLMs using priori judgement before answering and posteriori judgement after answering.
  • The study asks how accurately LLMs perceive factual boundaries, how retrieval augmentation affects them, and how supporting-document characteristics influence them.
  • LLMs often overestimate their factual knowledge and handle conflicts between internal and external knowledge poorly.
  • Retrieval augmentation supplements LLMs’ internal knowledge, particularly for smaller models, and improves their perception of factual knowledge boundaries.
  • High-quality supporting documents improve performance and confidence, while reliance and confidence depend on document-question relevance.

2 Background and Methodology

The methodology evaluates open-domain QA under normal and retrieval-augmented prompting, alongside prompting strategies that measure LLMs’ factual-boundary judgements. It combines answer-quality metrics with prior and posterior self-assessment metrics and tests multiple retrieval sources.

  • 2.1 Task Formulation: Open-domain QA asks an LLM to generate an answer to question q from internal knowledge or a document collection D.
  • 2.1 Task Formulation: Retrieval-augmented prompting supplies retrieved supporting documents to help the LLM answer the question.
  • 2.2 Prompting Strategies: The study uses QA prompting and judgemental prompting in both normal and retrieval-augmented settings.
  • 2.2 Prompting Strategies: Priori judgement measures whether models give up before answering, whereas posteriori judgement evaluates the correctness of their generated answers.
  • 2.3 Evaluation Metrics: Exact match and F1 evaluate QA answers, while Give-up, Right/G, Right/¬G, Eval-Right, and Eval-Acc evaluate judgement abilities.
  • 2.3 Retrieval Sources: Supporting documents are obtained using dense retrieval, sparse retrieval, or ChatGPT-based retrieval sources.

3 Experimental Analysis and Findings

Experiments show that LLMs are often overconfident about their factual knowledge boundaries, while retrieval augmentation improves both QA and boundary perception but remains sensitive to document quality, quantity, and relevance.

  • 3.1 To What Extent Can LLMs Perceive Their Factual Knowledge Boundaries?: LLMs frequently overestimate their knowledge before answering and their correctness after answering, despite substantial QA performance without external documents.Prior overconfidence appears in high incorrect-response rates and low give-up rates, while posterior judgement produces high Eval-Right but relatively low Eval-Acc.
  • 3.2.1 Main Findings: Retrieval augmentation supplements insufficiently utilized internal knowledge and usually improves QA, with larger gains for publicly available and smaller models.Dense-retrieved documents often yield the best performance, while Wikipedia documents can still improve QA despite training exposure to Wikipedia.
  • 3.2.1 Main Findings: Retrieval augmentation improves priori and posteriori knowledge-boundary judgements, increasing priori accuracy and posterior Eval-Acc while making Eval-Right more consistent with EM.These changes indicate improved self-assessment after adding sparse- or dense-retrieved supporting documents.
  • 3.2.1 Main Findings: QA performance rises as supporting-document count increases until a model-specific threshold, after which it may decline; LLaMA2 reaches lower thresholds than GPT-series models.Document ordering has little observed effect, and gains from more documents are not fully attributable to recall improvements.
  • 3.2.1 Main Findings: A priori judgement alone can reduce answering accuracy versus always retrieving, but retrieval-informed judgement surpasses that always-retrieval baseline on NQ.The authors use this judgement strategy to dynamically introduce supporting documents.
  • 3.3 Supporting Documents with Different Characteristics: Higher-quality supporting documents improve QA and knowledge-boundary perception, whereas incorrect documents can reduce performance below the no-retrieval setting.Golden documents improve performance; higher quality also lowers give-up rates and raises Eval-Acc.
  • 3.3 Supporting Documents with Different Characteristics: LLMs handle conflicts between internal and external knowledge poorly and rely heavily on supporting documents, making resilience to irrelevant documents a key performance constraint.The gap between models narrows with golden documents, while document relevance affects confidence and random incorrect documents can outperform more relevant incorrect ones.

4 Conclusion

The paper investigates LLMs’ factual knowledge boundaries under retrieval augmentation using QA, priori judgement, and posteriori judgement. It finds weaknesses in boundary perception and internal–external knowledge conflict handling, while retrieval augmentation improves boundary awareness and supports dynamic document use.

  • LLMs cannot accurately perceive their factual knowledge boundaries or handle conflicts between internal and external knowledge well.
  • Retrieval augmentation helps LLMs perceive their factual knowledge boundaries, although they cannot sufficiently utilize internal knowledge alone.
  • Boundary perception is affected by retrieval model, supporting-document number, question type, and LLM scale.
  • Supporting-document relevance significantly influences LLMs’ reliance on those documents.
  • The paper proposes dynamically using retrieval augmentation according to an LLM’s priori judgement.

Limitations

The paper’s analysis includes publicly available and closed-source LLMs, but closed-source API updates may limit long-term reproducibility. Retaining generated responses and applying the methodology to newer models are proposed responses.

  • Closed-source API updates may make earlier interfaces inaccessible, creating a risk to long-term reproducibility.
  • Retaining responses from models with reproducibility risks can support future use of the reported results.
  • The methodology can be applied to any latest LLM, so the reproducibility risk does not affect the contribution’s significance.

A.1 Datasets

The study uses three established open-domain QA benchmarks and samples 500 examples from each. Retrieval augmentation uses a 21M-passage Wikipedia collection from DPR.

  • The datasets are Natural Questions, TriviaQA, and HotpotQA, covering user-search questions, trivia with evidence, and multi-hop reasoning.
  • 500 data points are sampled from each dataset for the experiments.
  • QA evaluation uses the datasets’ short answers as labels.
  • Retrieval augmentation uses Wikipedia from DPR, consisting of 21M split passages.

A.2 Evaluation Models

The evaluation spans closed-source and publicly available LLMs, using multiple GPT, LLaMA, and Mistral models. Retrieval experiments compare several retrievers, while judgement and QA outputs are parsed with heuristic rules.

  • The experiments use three GPT models: Davinci003, ChatGPT, and GPT-4.
  • Publicly available evaluation models include LLaMA-2-Chat-7B and Mistral-7B-Instruct-v0.1.
  • Retrieval results are reported for sparse retrieval, dense retrieval, and ChatGPT across NQ, TriviaQA, and HotpotQA.
  • Judgement outputs use heuristic phrases such as “unknown,” “true,” and “false” to identify decisions.
  • Each question receives supporting documents, with ChatGPT outputs accepted when the generated count fluctuates around ten.

A.4 Retrieval Results

The paper documents the retrieval and prompting configurations used in its experiments, including retrieval performance and instructions for QA and judgemental prompting.

  • Table 4 reports retrieval performance for sparse retrieval, dense retrieval, and ChatGPT.
  • Supporting documents use a numbered passage format with titles and content, while ChatGPT-generated documents contain only passage numbers and content.
  • The experiments include separate prompt designs for QA and judgemental prompting under varied settings.

B.1 Analysis on Retrieval Sources

Retrieval-source quality depends not only on document recall but also on how much relevant information documents contain per token. Concise ChatGPT-generated documents can therefore support competitive or stronger QA despite lower recall.

  • Retrieval-document efficiency is evaluated through recall rate, positive-document percentage, and answers per 1K token.
  • ChatGPT-generated documents have lower recall than dense retrieval but produce retrieval-augmented QA performance that does not lag far behind it.
  • LLaMA2 achieves higher retrieval-augmented QA performance with ChatGPT as retriever than with dense retrieval.
  • ChatGPT-generated documents are more concise and closely related to the questions, which may contribute to higher LLM confidence and lower Give-up rates.
  • Dense-retrieved Wikipedia documents have higher recall and positive-document percentages but are longer and contain more irrelevant information.

B.2 Analysis on Query Types

Retrieval augmentation changes QA and knowledge-boundary perception unevenly across question types. Improvements are concentrated in many categories, while some categories remain resistant.

  • The analysis separately measures answer accuracy across question categories using documents retrieved by a dense retriever.
  • ChatGPT and LLaMA2 achieve their highest EM on “who” and “which” questions, while “how” questions remain difficult in knowledge-intensive settings.
  • Overall EM increases do not extend to every category, including “how” and “yesno” for ChatGPT and “which” for LLaMA2.
  • Most question types achieve higher posteriori judgement accuracy, except “how” and “yesno” for LLaMA2 and “yesno” for ChatGPT.
  • Retrieval augmentation does not uniformly improve factual knowledge-boundary perception across question types.

B.3 Analysis on Parameter Scale

Retrieval augmentation benefits smaller LLaMA2 models more strongly than larger ones in both QA and knowledge-boundary perception. The experiments compare parameter scales using EM and Eval-Acc.

  • Retrieval augmentation more strongly improves smaller LLaMA2 models in both QA performance and knowledge-boundary perception accuracy.
  • Retrieval augmentation reduces QA-performance gaps across LLaMA2 parameter scales, with LLaMA2-7B showing the largest improvement.
  • The comparison uses EM for QA performance and Eval-Acc for knowledge-boundary perception across LLaMA2-7B, 13B, and 70B versions.
  • The authors speculate that smaller models possess less knowledge in knowledge-intensive tasks and therefore gain more from retrieval augmentation.
Loading 2307.11019v3…