Towards Mitigating Hallucination in Large Language Models via Self-Reflection

1 Introduction

• Figure 1 illustrates a plausible-sounding answer as hallucinated because PTEN mutation is not typically associated with Noonan syndrome.The figure marks hallucinated parts in pink and underlines them.Source: Figure 1 · p. 1, §1 · p. 1
• The paper identifies a gap: the extent of hallucination in LLM-generated medical answers remains unclear, despite prior work on knowledge-intensive tasks.The introduction describes hallucinations as plausible-sounding but unfaithful or nonsensical, and notes that misleading medical information can have severe patient-care consequences.Source: §1 · pp. 1–2
• The authors examine hallucinations across five LLMs and five medical GQA datasets, evaluating their incidence and exploring potential causes.The models include general and medical LLMs, and the datasets include PubMedQA, MedQuAD, MEDIQA2019, LiveMedQA2017, and MASH-QA.Source: §1 · p. 2
• The self-reflection method iteratively generates, scores, and refines knowledge and answers to improve accuracy and reliability.The process evaluates knowledge factuality, answer consistency with that knowledge, and entailment with the question.Source: §1 · p. 2

Medical Question Answering

• Medical QA spans yes/no, multiple-choice, extractive, and generative formats, with pretrained language models supporting fluent responses to medical queries.The related work describes medical QA systems as improving information accessibility and understanding in healthcare.Source: §2 · pp. 2–3

Hallucination in Generative Question Answering

• Faithfulness is framed as grounding answers in source text or valid external knowledge, with less hallucinated content in more faithful answers.Prior approaches include REAG's rationale extraction, Read-before-Generate's use of salient information, and a benchmark measuring truthfulness across domains.Source: §2 · p. 3

Large Language Models

• LLMs demonstrate fluent, contextually relevant generation and emergent abilities, while practical deployment raises control, bias, reliability, and hallucination concerns.Source: §2 · p. 3

3.1 Models

• The analysis evaluates five models: three general LLMs and two models fine-tuned for medical use.The set includes Vicuna, Alpaca-LoRA, ChatGPT, MedAlpaca, and Robin-medical.Source: §3.1 · p. 3

3.2 Dataset

• The study evaluates generated answers on five datasets: PubMedQA, MedQuAD, MEDIQA2019, LiveMedQA2017, and MASH-QA.For MEDIQA2019, answers with scores 3 and 4 serve as golden answers.Source: §3.2 · pp. 3–4

3.3 Evaluation Protocols

• Evaluation combines unigram F1 and ROUGE-L with sample- and sentence-level Med-NLI and consistency-focused CTRLEval.Med-NLI scores entailment, neutrality, or contradiction against context or reference answers; CTRLEval is unsupervised and reference-free.Source: §3.3 · p. 4

3.4 Results and Discussion

• Table 2 reports automatic evaluation results over test sets from five medical QA datasets.The supplied passage does not provide the table’s metric values or identify an overall winner.Source: §3.4 · p. 4, Table 2 · p. 5
• The authors classify problematic answers as fact inconsistency, query inconsistency, or tangentiality, counting the first two as hallucination.The analysis is based on 250 directly generated examples from five models.Source: §3.4 · p. 4, Figure 2 · p. 4
• The authors argue that addressing these challenges requires factual recall, contextual understanding, and reasoning abilities.They call for further exploration and development of these capabilities to improve generation-system reliability and trustworthiness.Source: §3.4 · p. 4
• Robin-medical has the lowest F1 and ROUGE-L scores, and the authors exclude it from further experiments after noting its relatively poor generation performance.The authors also describe unrelated content, grammar issues, unwarranted templates, non-existent references, and lack of explanatory reasoning in medical-domain fine-tuned models’ generative QA.Source: §3.4 · p. 5
• Problematic answers in 100 sampled general-model outputs had lower Google Ngrams frequencies than fine answers, suggesting low frequency as a possible hallucination cause that needs further investigation.The authors used average keyword frequencies from 1950–2019 as a proxy for natural-world and pre-training-corpus text distribution.Source: §3.4 · p. 5, Figure 3 · p. 5

4.1 Methodology

• The Factual Knowledge Acquiring Loop generates question-relevant background knowledge and uses a reference-free factuality scorer to guide iterative refinement until a satisfactory level is reached.Source: §4.1 · p. 6
• After knowledge reaches the required quality, the Knowledge-Consistent Answering Loop generates an answer from it and iteratively revises answers whose consistency score is below threshold.Source: §4.1 · p. 6
• The Question-Entailment Answering Loop evaluates answer entailment using sentence-BERT embedding similarity and restarts the cycle when entailment or answerability is unsatisfactory.Source: §4.1 · p. 7

4.2.1 Evaluation

• Human evaluation labels query consistency and tangentiality per answer, and fact consistency per sentence, distinguishing inconsistent, tangential, entailed, fact-consistent, and generic outputs.Source: §4.2.1 · p. 7

4.2.2 Results

• Across five datasets, self-reflection outperformed direct-generation baselines on overlap and hallucination metrics, while human evaluation found reduced query inconsistency, tangentiality, and fact inconsistency in Vicuna and ChatGPT.Source: §4.2.2 · p. 7

4.3.1 Ablation Study

• Removing refinement, explicit aspect descriptions, or explicit score numbers yielded lower MedNLI and CtrlEval than the full approach, supporting their contribution to refinement.Source: §4.3.1 · p. 8

4.3.2 Case Study

• In two examples, the self-reflective answers corrected a factual error about EtCO2 and avoided irrelevant content, addressing fact and query inconsistency.Source: §4.3.2 · p. 8

• Hallucinations challenge AI accountability and trustworthiness; the paper addresses them with iterative self-reflection that generates, scores, and refines background knowledge and answers.Source: §5 · p. 8
• The authors report that the method is effective, generalizable, and scalable in reducing hallucinations.Source: §5 · p. 9
• Future work will investigate hallucination causes, study the phenomenon in other generation tasks, and extend the method to address those tasks’ challenges.Source: §5 · p. 9

Limitations

• The method does not eliminate hallucinations; ungrounded information may persist, especially in complex or ambiguous scenarios.The authors say it is not yet ready for direct real-world deployment and should complement approaches such as retrieval.Source: Limitations · p. 9
• The study focuses primarily on English medical queries, limiting generalizability to other languages, domains, and modalities.The authors identify language-specific, domain-adaptation, and multimodality challenges for future research.Source: Limitations · p. 9
• The authors identify empowering LLMs with high-level ability as a remaining challenge.Source: Limitations · p. 9

Ethics Statement

• Experiments used publicly available or synthetic datasets, which the authors say avoided potential harm to individuals or groups.Source: Ethics Statement · p. 9
• The data were selected and processed for privacy and confidentiality, anonymized, and contained no personally identifiable information.Source: Ethics Statement · p. 9
• The work aims to improve medical QA reliability and safety, not substitute for professional medical judgment.The authors acknowledge risks to patient care and clinical decision-making from hallucinations in AI-assisted healthcare.Source: Ethics Statement · p. 9

A Dataset

• PubMedQA contains 1k expert-labeled, 61.2k unlabeled, and 211.3k artificially generated instances; MedQuAD has 47,457 QA pairs from 12 NIH websites.PubMedQA uses research-article titles and abstracts, with long answers from abstract conclusions and concise yes/no/maybe answers; MedQuAD spans 37 question categories.Source: A Dataset · p. 13
• MEDIQA2019 includes three medical tasks, but this study uses Task 3; LiveMedQA2017 has expert-vetted reference answers for annotated medical questions.For MEDIQA2019 Task 3, the study treats golden answers scored 3 or 4 as correct; LiveMedQA2017 questions span 26 types in five categories.Source: A Dataset · p. 13
• MASH-QA contains 34k consumer-health QA pairs whose answers may draw on nonsequential sections of lengthy documents.Source: A Dataset · p. 13
• Except for PubMedQA, dataset answers were manually extracted from authentic web content and were pertinent and verifiable, but had not been refined.The authors say improvements are needed to harmonize contextual coherence and linkage to the question; the LiveMedQA2017 example illustrates this issue.Source: A Dataset · p. 13

B Implementation Details

• Generation used two GeForce RTX 2080 GPUs, temperature 1.0, and max new tokens 512; stated loop limits were generally three, with ChatGPT’s knowledge loop set to one for MedQuAD.Other settings were defaults; Vicuna and Alpaca-LoRA also used three as the maximum for the main, knowledge, and response loops.Source: B Implementation Details · pp. 13–14
• Factuality-scorer demonstration counts varied by model and dataset, ranging from one to three examples.Vicuna used 1 for PubMedQA and 3 for MedQuAD, MEDIQA2019, MASH-QA, and LiveMedQA2017; Alpaca-LoRA used 2 for LiveMedQA2017 and 1 for the others. ChatGPT used 3 for LiveMedQA2017 and 1 for the others; MedAlpaca used 2 for MASH-QA and MedQuAD and 1 for the others.Source: B Implementation Details · pp. 13–14

C Factuality Scorer

• The factuality-scorer demonstrations pair medical questions with knowledge about heart-disease risks, smoking’s effects on lungs, and daily aspirin use.Source: C Factuality Scorer · p. 14

D Human Evaluation

• Three annotators evaluated 50 Vicuna answers and 50 self-reflection-loop answers, with HIT approval rates of at least 95% and at least 5,000 approved HITs.Annotators were located in Australia, Canada, the United Kingdom, and the United States.Source: D Human Evaluation · p. 14
• For fact-inconsistency judgments, annotators used dataset context, retrieved WikiMedical-Terms paragraphs, and references in the Wiki-Doc medical encyclopedia.The WikiMedical-Terms dataset contains over 6,000 medical terms and their Wikipedia text.Source: D Human Evaluation · pp. 14, 16
• The method reduced query inconsistency, tangentiality, and fact inconsistency for both Vicuna and ChatGPT, while keeping generic responses relatively low and improving fact consistency.Table 6 reports the human-evaluation results on PubMedQA.Source: D Human Evaluation · p. 16, Table 6 · p. 16

E Case Study

• For the carotid-thrombus CTA question, ChatGPT denied shape analysis could distinguish thrombus from plaque, while the loop and GPT-4 gave positive answers closer to the golden answer.The loop answer mentioned machine-learning algorithms and radiomic features as promising for classification.Source: E Case Study · p. 16, Table 7 · p. 17
• GPT-4’s answer later included content unrelated to shape analysis, whereas the loop answer discussed classification approaches relevant to the question.Table 7 compares ChatGPT, the loop, and GPT-4 answers; it marks conclusions aligning or conflicting with the golden answer using different text formatting.Source: E Case Study · p. 16, Table 7 · p. 17
• Table 7 presents example answers from ChatGPT, ChatGPT with the self-reflection loop, and GPT-4, distinguishing conclusions that align with or conflict with the golden answer.Aligning conclusions are underlined and bolded; conflicting conclusions are underlined and italicized.Source: Table 7 · p. 17