Source-linked AI summary
Do Large Language Models Know What They Don't Know?
Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, Xuanjing Huang
TL;DR
LLMs can perform many NLP tasks but have limited capacity to retain and comprehend information, making recognition of their own knowledge limits important. The paper evaluates this self-knowledge with an automated uncertainty measure and the SelfAware dataset, finding intrinsic self-knowledge alongside a substantial human-model gap.
Problem
The paper asks whether LLMs can recognize their limitations when answering unanswerable or unknowable questions, despite limits in retaining and comprehending information.
Method
The paper introduces SelfAware and an automated text-similarity method for detecting uncertainty, evaluating LLM self-knowledge across answerable and unanswerable questions.
Results
75.47% GPT-4 self-knowledge versus 84.93% human self-knowledge, while in-context learning and instruction tuning enhance LLM self-knowledge.
Takeaways & Limitations
LLMs possess a degree of self-knowledge, but their ability to recognize knowledge limits remains below human self-knowledge.
Takeaways & Limitations
Reference sentences were selected only from GPT-3 and InstructGPT outputs, potentially overlooking uncertainty expressed by other LLMs.
Abstract
from arXiv · showhide
Large language models (LLMs) have a wealth of knowledge that allows them to excel in various Natural Language Processing (NLP) tasks. Current research focuses on enhancing their performance within their existing knowledge. Despite their vast knowledge, LLMs are still limited by the amount of information they can accommodate and comprehend. Therefore, the ability to understand their own limitations on the unknows, referred to as self-knowledge, is of paramount importance. This study aims to evaluate LLMs' self-knowledge by assessing their ability to identify unanswerable or unknowable questions. We introduce an automated methodology to detect uncertainty in the responses of these models, providing a novel measure of their self-knowledge. We further introduce a unique dataset, SelfAware, consisting of unanswerable questions from five diverse categories and their answerable counterparts. Our extensive analysis, involving 20 LLMs including GPT-3, InstructGPT, and LLaMA, discovering an intrinsic capacity for self-knowledge within these models. Moreover, we demonstrate that in-context learning and instruction tuning can further enhance this self-knowledge. Despite this promising insight, our findings also highlight a considerable gap between the capabilities of these models and human proficiency in recognizing the limits of their knowledge.
1 Introduction
This study frames self-knowledge as LLMs’ ability to recognize limitations and uncertainty when facing unanswerable or unknowable questions. It introduces the SelfAware dataset and an automated uncertainty measure, finding that LLMs retain self-knowledge but remain behind humans.
- The Know-Unknow quadrant relates knowledge capacity to the ability to comprehend and use knowledge, while self-knowledge concerns understanding limitations in the unknows.
- LLMs’ practical applicability depends partly on recognizing their limitations and conveying uncertainty for unanswerable or unknowable questions.The paper calls this ability model self-knowledge.
- SelfAware provides diverse commonly posed unanswerable questions, paired with answerable questions for evaluation.
- The paper proposes text-similarity-based uncertainty detection to quantify uncertainty in model outputs.
- 20 LLMs show a significant disparity from humans in self-knowledge, while in-context learning and instruction tuning can enhance it.
2 Dataset Construction
SelfAware was constructed to provide a larger and more diverse evaluation set of unanswerable questions, together with semantically matched answerable questions. The dataset spans five categories and contains 1,032 unanswerable and 2,337 answerable questions.
- 2,858 unanswerable questions were collected from Quora and HowStuffWorks and independently evaluated by three annotation analysts.The analysts could use external resources such as search engines.
- The dataset is designed to improve on Know-Unknowns by increasing both the number and diversity of unanswerable questions.
- 1,487 SQuAD, 182 HotpotQA, and 668 TriviaQA questions supplied 2,337 answerable questions selected as semantically closest to the unanswerable questions.SimCSE guided the matching process.
- SelfAware contains 1,032 unanswerable and 2,337 answerable questions, with answerable questions intentionally capped for testing feasibility.
- Manual analysis organized unanswerable questions into five categories, including no scientific consensus, imagination, and completely subjective questions.The categories capture questions that resist concrete answers or involve speculative scenarios.
3 Evaluation Method
The evaluation method detects uncertainty by comparing target text with reference sentences carrying uncertain meanings. It classifies sufficiently similar responses as uncertain and uses F1 to measure self-knowledge on unanswerable-question detection.
- The method defines a similarity function to compute similarity S_i between target sentence t and each uncertain reference sentence u_i.
- When any S_i exceeds threshold T, the target text is classified as expressing uncertain meaning without manual evaluation.
- F1 measures self-knowledge because unanswerable questions are designated positive cases and answerable questions negative cases.
4 Experiment
Experiments evaluate LLM self-knowledge across model size, instruction tuning, and input form, using GPT-3, InstructGPT, LLaMA, and derivative models. Self-knowledge generally improves with larger models, instruction tuning, and richer contextual inputs, but GPT-4 remains below the human benchmark.
- Experimental Setup: The experiments assess GPT-3, InstructGPT, LLaMA, Alpaca, and Vicuna using direct, instruction, and in-context learning input forms.The analysis centers on model size, instruction tuning, and input-form effects.
- Model Size: Larger model parameter size is associated with higher self-knowledge F1 scores across all three input forms, especially in-context learning.The reported trend is consistent with the scaling law.
- Instruction Tuning: Instruction-tuned models show greater self-knowledge than their base counterparts, with Vicuna-13B outperforming LLaMA-65B.The comparison includes InstructGPT versus GPT-3, text-davinci versus davinci, and Alpaca and Vicuna versus LLaMA.
- Input Forms: 27.96% improvement over direct input is reported for davinci with in-context learning, which provides richer contextual information.Instructions and examples also reduce the performance disparity between davinci and text-davinci models.
- Compared with Human: 75.47% F1 is achieved by GPT-4, compared with the human self-knowledge benchmark of 84.93%.The comparison indicates a remaining gap between the best tested model and human performance.
- Answerable Questions: Answerable-question accuracy in the InstructGPT series increases with model parameter size and continued learning.The experiments use a closed-book question-answering setting in which accuracy depends on producing the correct answer.
5 Conclusion
The study evaluates whether LLMs can identify unanswerable questions using a new dataset and automated uncertainty detection. Models show some self-knowledge, and in-context learning and instruction tuning improve it, but performance remains below human self-knowledge.
- Conclusion: The study measures LLM self-knowledge by evaluating their ability to identify unanswerable questions.It introduces a novel dataset and an automated method for detecting uncertainty in model responses.
- Conclusion: In-context learning and instruction tuning effectively enhance LLM self-knowledge.The reported experiments include GPT-3, InstructGPT, LLaMA, and other LLMs.
- Conclusion: LLMs possess some self-knowledge but remain considerably less proficient than humans at recognizing the limits of their knowledge.The authors identify improving this capability as a direction for further research.
Limitations
The study identifies limitations in reference-sentence generalization and input-form coverage. It proposes broader automated reference acquisition and additional cognitive and decision-making methods as future directions.
- Generalization of Reference Sentences: Reference sentences were selected only from GPT-3 and InstructGPT, potentially overlooking uncertainty in other LLMs’ responses.The authors propose automatically acquiring more accurate reference sentences.
- Limitations of Input Forms: The examination was confined to direct, instruction, and in-context learning input forms.Future work will incorporate additional cognitive and decision-making methods.
Ethics Statement
The dataset was reviewed for relevance, harmlessness, privacy, security, and licensing compliance. Annotation analysts conducted content review, while anonymization and filtration mechanisms protected data integrity.
- Dataset Review: Three annotation analysts independently reviewed dataset content for relevance and harmlessness during regular working hours.They were compensated according to local wage standards.
- Data Protection: The researchers implemented anonymization and content filtration mechanisms to protect data integrity.They also emphasized privacy, data security, and compliance with dataset licenses.
- Dataset Release: The publicly released dataset is reserved exclusively for research under the CC-BY-SA-4.0 protocol.The authors also commit to addressing concerns about the dataset and expect high ethical standards in its use.
A.1 Uncertainty Text
The study constructs reference sentences expressing uncertainty from model responses, then normalizes them for similarity-based uncertainty detection. The reference set includes multiple formulations of unknown, unclear, unavailable, debated, and difficult-to-predict answers.
- Reference sentence construction: 16 manually curated sentences from GPT-3 and InstructGPT responses served as reference sentences expressing uncertainty.The sentences were selected after a preliminary direct-input test and normalized by removing punctuation and converting text to lowercase.
- Uncertainty categories: The reference expressions include statements that answers are unknown, uncertain, unclear, or lack a definitive or correct resolution.
- Uncertainty categories: Other reference expressions indicate absent scientific evidence, public information, known cases, or concrete answers.
- Uncertainty categories: The reference set also captures uncertainty through statements that much debate exists around the question.
- Uncertainty categories: Several expressions explicitly state that the answer is impossible to know or answer, difficult to predict, not known, or unknown to the speaker.
A.2 Threshold ablation
The study evaluates automated uncertainty filtering by comparing similarity thresholds against manually annotated responses. A threshold of 0.75 achieved the highest F1 score and was selected for later experiments, while volunteer responses also received high F1 scores.
- Threshold ablation: 100 new responses were generated with text-davinci-002 using the direct input form and manually filtered for uncertainty.
- Threshold ablation: Similarity between generated sentences and reference sentences was computed with SimCSE across multiple filtering thresholds.
- Threshold ablation: 0.75 produced the highest F1 score, balancing precision with inclusion of other uncertain sentences, so it became the threshold for subsequent experiments.
- Volunteer evaluation: The volunteer evaluation yielded high F1 scores for both participants, indicating strong self-knowledge in their responses.
- Input templates: The experiments used Direct, Instruction, and ICL input templates, with ICL providing three answerable and three unanswerable examples with manually supplied answers.