Source-linked AI summary
Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Method
Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, Dawei Yin
TL;DR
LLMs intermittently generate non-factual responses because they may not recognize what they do not know, limiting their reliability. The paper proposes self-detection by diversifying question wording, comparing generated answers, and using input atypicality, achieving effective detection across multiple LLMs and tasks. Its main boundary is that consistently incorrect answers can produce false negatives.
Problem
LLMs intermittently generate falsehoods while being unaware of what they do not know, which impedes their reliability.
Source: §1 · p. 1Method
The method diversifies questions into equivalent verbalizations, examines answer divergence and input atypicality, and relies only on the LLM itself.
Source: §1 · p. 2Results
Takeaways & Limitations
The approach can help LLMs detect and improve specific weaknesses while supporting continual model upgrading.
Source: §6 · pp. 8–9Consistently incorrect answers can evade detection, producing false negatives because the method relies on divergence among generated answers.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Large Language Models (LLMs) have shown great potential in Natural Language Processing (NLP) tasks. However, recent literature reveals that LLMs generate nonfactual responses intermittently, which impedes the LLMs' reliability for further utilization. In this paper, we propose a novel self-detection method to detect which questions that a LLM does not know that are prone to generate nonfactual results. Specifically, we first diversify the textual expressions for a given question and collect the corresponding answers. Then we examine the divergencies between the generated answers to identify the questions that the model may generate falsehoods. All of the above steps can be accomplished by prompting the LLMs themselves without referring to any other external resources. We conduct comprehensive experiments and demonstrate the effectiveness of our method on recently released LLMs, e.g., Vicuna, ChatGPT, and GPT-4.
1 Introduction
LLMs can produce intermittent falsehoods because they do not reliably recognize what they do not know, motivating methods that detect non-factuality without external resources. This paper proposes self-detection through divergent answers to equivalent verbalizations and reports effectiveness across four LLMs and three task types.
Motivation
- LLMs intermittently generate falsehoods while lacking awareness of what they do not know, undermining their reliability in practice.The paper frames detecting such unknown questions as an important research problem.Source: §1 · p. 1
Limitations of prior work
- Existing self-detection methods can be limited because LLMs are overconfident and become less effective after further alignment training.Prior approaches include confidence estimates and self-consistency among sampled answers.Source: §1 · pp. 1–2
Proposed method
- The method diversifies a question into semantically equivalent verbalizations, compares the resulting answers, and measures input atypicality using negative log-likelihood.It combines response divergence with whether the verbalized question is typical for the model.Source: §1 · p. 2
Evaluation
- The paper evaluates GPT-4, ChatGPT, Vicuna, and Llama 2 on factoid question answering, commonsense reasoning, and arithmetic reasoning, finding superior self-detection performance.The experiments are described as extensive across three task types.Source: §1 · p. 2
Contributions
- The contributions include identifying verbalization-sensitive contradictions, introducing an LLM-only self-detection suite, and relating unknown questions to popularity, reasoning steps, and formulations.The approach requires no further fine-tuning or external detection resource.Source: §1 · pp. 1–2
2 Related Work
Prior work detects hallucinations through calibration, confidence scores, or consistency among sampled answers, but these approaches depend on model calibration and can weaken after alignment. The paper instead examines behavioral divergence across diversified questions together with input atypicality, without external models or resources.
Model Calibration
- Calibration methods estimate confidence aligned with correctness, but task-specific calibration can be difficult to generalize to out-of-distribution data.Prior work includes post-hoc scaling, feature fitting, ensembling, and fine-tuning.Source: §2 · pp. 2–3
Hallucination Detection
- Hallucination detection matters because LLMs can generate non-factual responses that hinder reliability across language tasks.The related work situates hallucination detection as a response to unreliable generations.Source: §2 · p. 3
Self-detection methods
- Self-evaluation methods use verbalized confidence, token probabilities, or consistency among high-temperature samples, but model overconfidence and alignment can limit them.These signals rely heavily on the model’s current training state.Source: §2 · p. 3
This work
- The proposed method combines divergence across diversified questions with consistency and atypicality analysis, while requiring neither additional LLMs nor external resources.This distinguishes it from verifier-based and rephrasing-based concurrent approaches.Source: §2 · p. 3
3 Inconsistency and Atypicality in LLMs
The paper treats non-factuality as arising when an LLM lacks relevant knowledge or fails to understand a question yet still produces plausible text. It therefore detects unknown questions through contradictory answers to equivalent verbalizations and atypicality of the input wording.
Inconsistency and Atypicality
- Sequential token generation can produce plausible responses even when an LLM lacks the relevant knowledge or does not understand the question.This generative behavior is presented as a source of non-factuality.Source: §3 · p. 3
Inconsistency
- Drastically different answers to semantically equivalent verbalizations indicate that the model may not know the question.The method assumes correct answers should remain consistent across wording changes.Source: §3 · p. 3
Atypicality
- Atypical wording may be harder for the model to understand and can lead to lower-quality responses.The paper illustrates this with equivalent questions receiving completely different answers.Source: §3 · p. 3
Detection signals
- The framework measures response divergence across equivalent question sets and input atypicality A(q) for the verbalized question.These are the two signals used to identify potentially unknown questions.Source: §3 · p. 3
4 Self-Detecting What LLMs Do Not Know
The framework detects questions an LLM may not know by comparing answers to semantically equivalent verbalizations and measuring how atypical the input wording is. It combines consistency-based divergence with verbalization-based atypicality into a final confidence score.
4.1.1 Diversifying Question Verbalizations
- Questions are diversified with model-generated paraphrases for open QA, while rule-based substitutions or reorderings handle commonsense and arithmetic reasoning.Generated paraphrases are filtered for semantic equivalence; rules alter choices or person names while preserving the task structure.Source: §4.1.1 · p. 4
4.1.2 Calculating Consistency Score
- Consistency is assessed by clustering responses according to pairwise agreement, using exact final-answer matches for fixed formats and LLM judgments for free-form answers.Greedy decoding is used during response generation, and consistency judgments classify answers as “Contradicted” or “Same.”Source: §4.1.2 · p. 4
- Higher entropy in the clustered answer distribution indicates greater generation divergence and a lower likelihood that the LLM knows the question.Entropy is calculated after clustering the responses and measures divergence among answers to the same question.Source: §4.1.2 · p. 5
4.2 Verbalization-based Detection
- Verbalization-based detection uses the question’s negative log-likelihood, with token normalization, to quantify how atypical the wording is for the language model.A higher atypicality score indicates that the verbalization is less representative in the model.Source: §4.2 · p. 5
4 Self-Detecting What LLMs Do Not Know
- The method diversifies each question into semantically equivalent verbalizations, gathers corresponding responses, and combines response divergence with input atypicality.The final confidence score estimates whether the LLM does not know the question.Source: §4.1.1 · p. 4, §4.2 · p. 5
5 Experiments
Experiments evaluate self-detection across four LLMs and three task types, finding mostly superior performance and identifying why answer divergence, atypical inputs, and question characteristics matter.
5.2 Overall Performance
- Mostly best performance across six datasets shows self-detection is effective across ChatGPT, GPT-4, Vicuna-13B, and Llama2-13B.PR-AUC values are compared within each model because their ground-truth unknown ratios differ.Source: §5.2 · pp. 6–7, Table 3 · p. 6
5.3 Ablation Study
- Removing either atypicality or consistency lowers performance, with consistency removal usually causing the larger drop.This indicates divergence among answers to diversified questions is generally more important than atypicality.Source: §5.3 · p. 7
- Combining the method with TokenProbs, Perplexity, ConsistAnswers, and SelfCheckGPT continuously improves performance, while simple and XGBoost combinations are comparable.The authors therefore do not regard the exact combination mechanism as vital.Source: §5.3 · p. 7, Figure 3 · p. 7
5.4 Unknown Questions Study
- Unknown questions have higher atypicality and fewer search results, indicating poorer memorization of less popular knowledge.The study uses search-result counts as a popularity indicator and negative log-likelihood to compare known and unknown questions.Source: §5.4 · pp. 7–8, Table 4 · p. 7, Table 6 · p. 8
- Longer or multi-step reasoning and distracting formulations are associated with more errors, including operation-order confusion and attention to irrelevant details.The study gives examples from arithmetic and commonsense reasoning questions, including distractors involving baseball cards and Cape Canaveral.Source: §5.4 · pp. 7–8, Table 7 · p. 8
5.5 Impact of Diversified Questions
- Increasing paraphrased questions from 10 to 30 slightly improves ChatGPT self-detection on representative datasets.More paraphrases expose coincidental correctness in unknown questions while confident questions remain consistent.Source: §5.5 · p. 8, Figure 4 · p. 8
6 Conclusion
The paper presents a simple self-detection method for nonfactual LLM responses and reports effectiveness across four LLMs and three task types. It also identifies question characteristics associated with LLM difficulty and suggests future integration with existing methods.
6 Conclusion
- Experiments on ChatGPT, GPT-4, Vicuna, and Llama 2 across three task types demonstrated the effectiveness of the self-detection method.The tasks included factoid question answering, commonsense reasoning, and arithmetic reasoning.Source: §6 · p. 8
- The study explores whether low popularity and distracting formulations are associated with questions that LLMs may struggle with.Source: §6 · p. 9
- The two proposed components can be combined with existing methods for further utilization.Source: §6 · p. 9
- The method may support continual LLM upgrading by helping models detect and improve specific weaknesses.Source: §6 · p. 9
Limitations
The method’s effectiveness is constrained by the model’s ability to generate diverse verbalizations and by its inability to detect consistently incorrect answers. The authors suggest user-generated questions, verifier LLMs, or external knowledge as future improvements.
Limitations
- The self-detection method’s verbalization diversity is constrained by the abilities of the LLM itself.The authors propose collecting more end-user questions from conversational agents or search engines to capture built-in question ambiguity.Source: Limitations · p. 9
- Divergence-based detection produces false negatives when the model generates answers that are consistently but incorrectly.Additional verifier LLMs or external knowledge could support cross-checking, but these combinations are omitted from the paper’s scope.Source: Limitations · p. 9
Ethics Statement
The authors report no explicit anonymity or privacy concerns because all models and datasets used are public. They note that possible problematic bias in publicly available LLMs was not examined.
Ethics Statement
- The paper reports no explicit anonymity or privacy considerations because its models and datasets are public.Source: Ethics Statement · p. 9
- Whether publicly available LLMs encode problematic bias remains unclear because bias is outside the paper’s focus.Source: Ethics Statement · p. 9
- The technique is intended to detect what LLMs do not know and should not be used for other applications.Source: Ethics Statement · p. 9
A.1 Prompts
The appendix documents prompts for generating and filtering paraphrased questions. It also describes an optimized diversification instruction that generates ten rephrased questions in one model call, with future evaluation planned.
A.1 Prompts
- The appendix includes instructions for the diversifying task and identifies the corresponding diversification tables.Table 8 presents the original instruction, while Table 9 presents a new instruction for diversifying.Source: §A.1 · p. 12, Table 8 · p. 12, Table 9 · p. 12
- The optimized diversification instruction asks the model to provide 10 rephrased questions at once, reducing rephrasing generations to one.The paper states that performance with this new prompt will be reported in future work.Source: §A.1 · p. 12
- The appendix provides an instruction for detecting wrong paraphrases of the question.Table 10 contains the instruction used for this filtering step.Source: §A.1 · p. 12, Table 10 · p. 12
A.2 Evaluation of the Upgrading LLMs
The experiments compare unknown-question detection across upgraded LLMs and evaluate SelfDetection’s components, verbalization strategy, linear combination, and cost. GPT-4 performs best among the compared models, while diversifying verbalizations improves detection relative to the non-diversified variant.
A.2 Evaluation of the Upgrading LLMs
- GPT-4 performs best, ChatGPT is weaker, and Vicuna-13B and Llama2-13B perform closely but below the GPT models across openQA, commonsense reasoning, and arithmetic reasoning.Unknown and known questions are determined using golden correctness labels.Source: §A.2 · p. 12, Table 11 · p. 12
A.3 Additional Experiments
- SelfDetection outperforms ConsistAns combined with Atypicality on most datasets, showing the importance of diversifying verbalizations.Source: §A.3 · p. 12, Table 12 · p. 13
- The equal-weight linear combination achieves performance close to the XGBoost-fitted version.The component features are normalized to [0, 1] before combining them.Source: §A.3 · p. 12, Table 13 · p. 13
A.4 Component Evaluation
- Paraphrase precision reaches 100% for commonsense reasoning, 99% for arithmetic reasoning, and 93%–95% for openQA across the evaluated models.The arithmetic errors arise from conflicts between animal and human names.Source: §A.4 · p. 12
- Across four LLMs, answer-clustering precision was 100% for commonsense reasoning, 81%–90% for openQA, and 88%–93% for arithmetic reasoning tasks.Clustering precision was reported for 400 manually labeled clusters, based on assignments between output clusters and ground-truth clusters.Source: §A.4 · p. 13
A.5 Costs
- The study reports per-question costs for SelfDetection and comparison methods across openQA, commonsense reasoning, and arithmetic reasoning tasks.Open-source models are deployed for inference, whereas closed-source models are accessed through APIs.Source: §A.5 · p. 13, Table 14 · p. 13