Why Does ChatGPT Fall Short in Providing Truthful Answers?

• The study examines ChatGPT’s weaknesses in complex open-domain question answering to identify common failure modes, deficient abilities, and potential improvements.The task is chosen because it reflects everyday search demands and requires extensive knowledge, understanding, and reasoning.Source: §1 · p. 1
• The authors categorize failures into comprehension, factuality, specificity, and inference errors.Source: §1 · p. 2
• Factuality is identified as the primary failure, with knowledge memorization and knowledge recall treated as critical associated abilities.Source: §1 · p. 2
• Supplying granular external knowledge and cues for knowledge recall is reported to enhance ChatGPT’s factuality in question answering.Source: §1 · p. 2

• Existing studies examine ChatGPT across general evaluation, understanding, mathematics, bug fixing, out-of-distribution behavior, translation, and question answering.Source: §2 · p. 2
• Although prior work identifies several problems, it mainly categorizes common issues rather than investigating the underlying deficiencies contributing to failures.Source: §2 · p. 2

• The evaluation uses HotpotQA [Yang et al., 2018] and BoolQ [Clark et al., 2019], both of which use Wikipedia as their knowledge source.Source: §3 · p. 2
• The authors analyze 200 HotpotQA questions and evaluate factuality on an additional 500 HotpotQA and 1000 BoolQ questions.Source: §3 · p. 2
• GPT-3.5 and GPT-4 responses are generated through the public OpenAI API.Source: §3 · p. 2

4.1 Thematic Analysis

• Thematic analysis of 200 HotpotQA responses used two independent annotators, reconciliation, and validation on 40 additional questions.The computed inter-coder agreement was 0.8394, indicating high consensus.Source: §4.1 · pp. 2–3
• The analysis defines comprehension errors as failures to understand question context or intent, especially under grammar mistakes or ambiguity.Source: §4.1 · p. 3
• Factuality errors arise when the model lacks or has incorrect supporting facts, and they account for a majority of the model’s errors.Source: §4.1 · p. 3
• Specificity errors occur when answers are too general or too specific, whereas inference errors occur when the model has relevant knowledge but fails to reason effectively.Source: §4.1 · p. 3

4.2 Results

• The experiment compares plain GPT-3.5 questions, GPT-3.5 with gold evidence, and plain GPT-4 questions using failure counts summarized in Table 2.Source: §4.2 · p. 4, Table 2 · p. 4
• Nearly half of failures are factuality errors, followed by inference, comprehension, and specificity errors.GPT-4 improves comprehension and specificity relative to GPT-3.5, but factuality improves only marginally.Source: §4.2 · p. 4, Table 2 · p. 4

• Factuality errors are framed as failures to retrieve relevant knowledge from language-model memory for a question.Source: §5 · p. 5
• Essential knowledge is defined as information indispensable for answering a question, with memorization and recall identified as the two key retrieval abilities.Source: §5 · p. 5
• Knowledge memorization means that an appropriate prompt can make the model produce the essential knowledge.Source: §5 · p. 5
• Knowledge recall means that the model can output memorized essential knowledge when prompted with the question.Source: §5 · p. 5
• The experiments rephrased essential knowledge to test memorization and repeated original questions in context to test recall.Source: §5 · p. 5
• 6 out of 46 errors stemmed from recall issues, while the remaining errors came from memorization.Source: §5 · p. 5

6.1 Settings

• The factuality experiments use GPT-3.5 with plain questions as baseline, HotpotQA and BoolQ, and partial match evaluation.Source: §6.1 · p. 5

6.1.1 Knowledge Memorization

• External knowledge is tested at sentence, passage, and section levels, with increasing context length representing coarser granularity.Source: §6.1.1 · p. 5

6.1.2 Knowledge Recall

• Knowledge recall is modeled as retrieving essential knowledge values from memory using question or entity-related keys.Source: §6.1.2 · pp. 5–6

6.2 Observations and Implications

• Finer-grained external knowledge yields better results, while whole Wikipedia sections substantially reduce the performance gain.Source: §6.2 · p. 6
• Providing complete entity names or definition sentences improves knowledge recall, even when the supplied descriptions omit essential knowledge.Source: §6.2 · p. 6
• Random sentences from an entity’s Wikipedia page decrease performance rather than improving it.Source: §6.2 · p. 6
• The proposed strategies are to retrieve fine-grained external evidence for memorization and provide entity descriptions as keys for recall.Source: §6.2 · p. 6

• The study identifies comprehension, factuality, specificity, and inference errors, pinpointing factuality as the most critical error.Source: §7 · p. 7
• It examines essential knowledge and the roles of knowledge memorization and knowledge recall in factual question answering.Source: §7 · p. 7
• The study investigates how evidence granularity affects knowledge memorization and how relevant keys affect knowledge recall.Source: §7 · p. 7
• It proposes methods to improve factuality in question answering and offers insights for developing more reliable language models.Source: §7 · p. 7

A.1 Knowledge Memorization

• The knowledge memorization prompts use entity-evidence pairs before asking the model to answer a question.Source: §A.1 · p. 10
• The memorization setting explicitly frames answering as using supplied knowledge consisting of entity-specific evidence.Source: §A.1 · p. 10

A.2 Knowledge Recall

• The complete-entity-name knowledge recall prompt supplies a list of entities and then asks the model to answer the question.Source: §A.2 · p. 10
• The recall settings vary the contextual support by using entity names alone or entity names augmented with background knowledge.Source: §A.2 · p. 10
• One additional knowledge recall prompt combines entity names with background evidence tied to those entities.Source: §A.2 · p. 10
• The appendix identifies a figure file for the knowledge recall section in PNG format.Source: §A.2 · p. 11