PaperlayerView exact revision on arXiv

Source-linked AI summary

MedHalu: Hallucinations in Responses to Healthcare Queries by Large Language Models

Vibhor Agarwal, Yiqiao Jin, Mohit Chandra, Munmun De Choudhury, Srijan Kumar, Nishanth Sastry

arXiv:2409.19492v3cs.CLcs.AI

TL;DR

Medical hallucination evaluations based on standardized exams may not reflect ambiguous, incomplete, and diverse real-world healthcare interactions. The paper introduces MedHalu and evaluates hallucination detection by LLMs, medical experts, and laypeople, finding that LLMs underperform experts while expert-in-the-loop prompting improves detection, including a 6.3% GPT-4 macro-F1 increase.

  • Problem

    Existing medical hallucination studies mainly use standardized exam questions, leaving LLM reliability in real-world healthcare interactions insufficiently assessed.

    Source: §1 · p. 1

  • Method

    The paper introduces MedHalu, a benchmark of real-world healthcare queries and hallucinated LLM responses annotated with hallucination types and text spans, and evaluates detection across LLMs, experts, and laypeople.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    LLMs perform no better than laypeople and underperform medical experts, while expert-in-the-loop prompting improves all models, including a 6.3% macro-F1 increase for GPT-4.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Expert reasoning can improve LLM-based medical hallucination detection, while current LLMs remain limited in identifying hallucinations in complex healthcare scenarios.

    Source: §1 · p. 2, §7 · p. 7

    MedHalu contains English healthcare queries only, so hallucination behavior for non-English healthcare queries remains unknown.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Large language models (LLMs) are starting to complement traditional information seeking mechanisms such as web search. LLM-powered chatbots like ChatGPT are gaining prominence among the general public. AI chatbots are also increasingly producing content on social media platforms. However, LLMs are also prone to hallucinations, generating plausible yet factually incorrect or fabricated information. This becomes a critical problem when laypeople start seeking information about sensitive issues such as healthcare. Existing works in LLM hallucinations in the medical domain mainly focus on testing the medical knowledge of LLMs through standardized medical exam questions which are often well-defined and clear-cut with definitive answers. However, these approaches may not fully capture how these LLMs perform during real-world interactions with patients. This work conducts a pioneering study on hallucinations in LLM-generated responses to real-world healthcare queries from patients.We introduce MedHalu, a novel medical hallucination benchmark featuring diverse health-related topics and hallucinated responses from LLMs, with detailed annotation of the hallucination types and text spans. We also propose MedHaluDetect, a comprehensive framework for evaluating LLMs' abilities to detect hallucinations. Furthermore, we study the vulnerability to medical hallucinations among three groups -- medical experts, LLMs, and laypeople. Notably, LLMs significantly underperform human experts and, in some cases, even laypeople in detecting medical hallucinations. To improve hallucination detection, we propose an expert-in-the-loop approach that integrates expert reasoning into LLM inputs, significantly improving hallucination detection for all LLMs, including a 6.3% macro-F1 improvement for GPT-4. Our code and dataset are available at https://netsys.surrey.ac.uk/datasets/medhalu/.

1 Introduction

Medical hallucination research often relies on standardized exam questions, leaving uncertainty about LLM reliability in ambiguous, incomplete, and diverse real-world healthcare interactions. This work introduces MedHalu and shows that expert reasoning improves hallucination detection.

Motivation

  • LLMs’ medical hallucination risks remain difficult to assess because standardized exams omit the ambiguity, incomplete information, and user diversity of real-world healthcare queries.Laypeople increasingly seek health advice from chatbots, while hallucinated information can be factually incorrect and create safety concerns.Source: §1 · p. 1

Empirical Findings

  • LLMs perform no better than laypeople and underperform medical experts when detecting hallucinations in healthcare responses.GPT-4 achieves macro-F1 0.56, compared with macro-F1 0.57 for laypeople and 0.70 for experts.Source: §1 · p. 2

Mitigation Strategy

  • Expert-in-the-loop prompting integrates medical experts’ reasoning and improves hallucination detection across evaluated models.Experts cross-validate answers using domain knowledge and trusted health resources; the approach yields an average macro-F1 increase of 6.3% for GPT-4.Source: §1 · pp. 2–3

Dataset

  • MedHalu introduces a benchmark of real-world healthcare questions and hallucinated LLM answers annotated with hallucination types and text spans.The benchmark draws on queries spanning users with different levels of medical expertise and includes input-conflicting, context-conflicting, and fact-conflicting cases.Source: §1 · p. 2

2 The MedHalu Benchmark

The MedHalu benchmark targets hallucinations in patient- and layperson-oriented healthcare queries. It defines three hallucination types, constructs diverse examples from expert-curated datasets, and validates them through structured medical-expert annotation.

2 The MedHalu Benchmark

  • MedHalu is designed to study LLM hallucinations in real-world healthcare queries commonly asked by patients and laypeople on web forums.The benchmark covers hallucination types, dataset generation, and human evaluation.Source: §2 · p. 3

2.1 Hallucination Types

  • The benchmark distinguishes fact-conflicting, input-conflicting, and context-conflicting hallucinations.These represent contradictions of facts, deviations from the input query, and internal self-contradictions, respectively.Source: §2.1 · p. 3

2.2 Dataset Generation

  • MedHalu combines HealthQA, LiveQA, and MedicationQA to cover diverse healthcare topics, query sources, lengths, and user contexts, then prompts GPT-3.5 to generate hallucinations for each query.The source datasets contain expert-curated or consumer healthcare questions and answers; tailored prompts target each hallucination type.Source: §2.2 · p. 3

2.3 Human Evaluation

  • Medical experts validate a stratified 500-pair sample, label hallucination types and spans, and achieve an average Cohen’s Kappa of 0.83.Thirty experts evaluate ten batches with attention checks and annotation guidelines, supporting the reliability of the benchmark.Source: §2.3 · p. 4

3 Hallucination Detection in Healthcare Queries

MEDHALUDETECT evaluates hallucination detection by LLMs, medical experts, and laypeople on healthcare question-answer pairs, including both labels and hallucinated spans. The study uses MEDHALU prompts, controlled evaluator groups, and classification and span-level metrics.

3 Hallucination Detection in Healthcare Queries

  • MEDHALUDETECT is designed for difficult cases because hallucinated healthcare content can appear plausible and semantically similar to correct answers.The framework includes the MEDHALU benchmark and a detection prompt for healthcare queries.Source: §3 · p. 4, Table 5 · p. 5, Table 6 · p. 5

3.1 Methodology

  • MEDHALUDETECT compares LLMs, medical experts, and laypeople on hallucination labels and highlighted hallucinated spans for healthcare question-answer pairs.LLMs receive the hallucination definitions, query, and response; human groups evaluate sampled MEDHALU pairs.Source: §3.1 · p. 4
  • 500 question-answer pairs are stratified into 10 batches of 50, with three medical experts and three laypeople evaluating each batch.The study therefore employs 30 medical experts and 30 laypeople overall.Source: §3.1 · p. 5

3.2 Experimental Setup

  • The experiments generate hallucinated healthcare responses with GPT-3.5 at temperature 0.7 and maximum generation length 512 tokens, while detection uses LLaMA-2-Instruct, GPT-3.5, and GPT-4.Source: §3.2 · p. 5
  • The models are included to contextualize performance on MEDHALU rather than claim that any one model is superior.The experimental setup explicitly frames model diversity as context for interpreting results and testing the benchmark.Source: §3.2 · p. 5

3.3 Evaluation Metrics

  • Hallucination labels are evaluated as binary classification using accuracy, macro precision, macro recall, and macro-F1.Span detection is measured by edit distance to expert-annotated hallucinated spans, where smaller values indicate greater similarity.Source: §3.3 · p. 5

4 Results

Across MEDHALU, proprietary LLMs outperform LLaMA-2 but remain comparable to or below laypeople and well below medical experts. Detection varies by hallucination type, with context-conflicting hallucinations generally easiest for GPT-3.5 and GPT-4.

4.1 Overall Results

  • GPT-3.5 and GPT-4 outperform LLaMA-2 overall, but GPT-4 shows only marginal improvement over GPT-3.5.On LiveQA, both GPT-3.5 and GPT-4 achieve an F1-score of 0.52; the highest MedicationQA F1-score is 0.55.Source: §4.1 · pp. 5–6
  • Medical experts achieve macro-F1 scores of 0.79, 0.57, and 0.71 on HealthQA, LiveQA, and MedicationQA, compared with laypeople’s 0.65, 0.47, and 0.57.LLMs generally perform no better than laypeople, while LiveQA is difficult even for trained professionals.Source: §4.1 · p. 6
  • GPT-4’s mean edit distances from expert hallucinated spans are 37.41, 84.33, and 47.8 for HealthQA, LiveQA, and MedicationQA.GPT-4 has lower edit distances than GPT-3.5 across all three subsets, while LiveQA is hardest for both models.Source: §4.1 · p. 6
  • The authors make LLM-detected hallucinated spans publicly available to support fine-grained hallucination detection research.This extends the span-level evaluation beyond the reported comparisons.Source: §4.1 · p. 6

4.2 Are some types of hallucination easier to detect than others?

  • GPT-3.5 and GPT-4 detect context-conflicting hallucinations best across all three subsets, followed by fact-conflicting and input-conflicting hallucinations.Context conflicts are easier to identify from self-inconsistency, whereas fact conflicts require medical knowledge and input conflicts require comparison with the query and system prompt.Source: §4.2 · p. 6, Table 8 · p. 6

5 Expert-in-the-loop to Improve LLM Hallucination Detection

Because LLMs underperform experts in medical hallucination detection, the study incorporates experts’ reasoning into LLM prompts. This expert-in-the-loop approach improves detection across evaluated LLMs, with GPT-4 gaining 7 percentage points on HealthQA and MedicationQA and 5 on LiveQA.

5 Expert-in-the-loop to Improve LLM Hallucination Detection

  • Experts combine prior domain knowledge with trusted health websites to cross-verify whether medical responses contain hallucinations.The named resources include UpToDate, BMJBestPractice, WebMD, and NHS.Source: §5 · pp. 6–7
  • The expert-in-the-loop method feeds expert reasoning, healthcare queries, and generated responses into LLM hallucination-detection prompts.The approach is motivated by LLMs performing no better than laypeople and sometimes worse than them.Source: §5 · pp. 6–7
  • GPT-4’s average macro-F1 rises by 7, 5, and 7 percentage points on HealthQA, LiveQA, and MedicationQA, respectively, with expert reasoning included.Its resulting average macro-F1 scores are 0.64, 0.57, and 0.62; LLaMA-2 also improves to 0.55, 0.51, and 0.56.Source: §5 · p. 7

6 Related Work

Prior work establishes LLM capabilities and the challenges of plausible but incorrect hallucinations, while existing medical benchmarks largely omit real-world layperson healthcare interactions.

6.1 Large Language Models

  • LLMs perform strongly across general language tasks and complex medical inquiries, but this proficiency can cause users to overestimate their reliability.The resulting trust can extend to outputs that are factually inaccurate.Source: §6.1 · p. 7

6.2 Hallucinations in LLMs

  • LLM hallucinations are plausible-seeming but factually incorrect or unfaithful content, categorized here as input-conflicting, context-conflicting, or fact-conflicting.These categories capture deviations from user input, internal consistency, and established facts.Source: §6.2 · p. 7

6.3 Benchmarks for LLM Hallucinations

  • Existing medical hallucination benchmarks differ from layperson healthcare use because they primarily test examination-derived questions, medical summaries, or health claims rather than patient queries.MedHalu is introduced to address this gap with real-world healthcare questions from layperson patients.Source: §6.3 · p. 7

7 Conclusions and Future Works

The paper presents MEDHALU as a benchmark for fine-grained hallucination detection in diverse healthcare queries and identifies limitations in current LLM detectors. It proposes expert-informed and knowledge-based directions for improving reliability.

Conclusions

  • MEDHALU contains diverse healthcare queries and LLM responses annotated with hallucination types and text spans, and evaluations show current LLMs struggle in complex domain-specific scenarios.The benchmark evaluates medical experts, LLMs, and laypeople.Source: §7 · p. 7

Future Works

  • Future work could adapt MEDHALU with parameter-efficient fine-tuning, rule-based systems, knowledge graphs, and medical literature containing verified expert knowledge.These directions are intended to support hallucination detection and mitigation in real-world healthcare queries.Source: §7 · p. 7
  • The expert-in-the-loop approach integrates expert reasoning into LLM prompts to improve automatic hallucination detection.Table 9 presents the corresponding expert-in-the-loop hallucination-detection prompt.Source: Table 9 · p. 8
  • Future research could extend expert feedback mechanisms and evaluate hallucinations in non-English languages and modalities such as medical videos and broadcasts.The proposed extensions follow the study’s focus on textual English medical queries.Source: §7 · p. 8

Limitations

The study’s dataset and conclusions are bounded by English-only healthcare queries and by the evolving difficulty of detecting hallucinations as models and training data change.

Limitations

  • MEDHALU contains English healthcare queries only, so hallucination behavior for non-English healthcare questions remains unknown.The authors identify multilingual benchmarks, including Hindi, Bengali, and Marathi, as future work.Source: Limitations · p. 8
  • As LLMs train on more datasets and new models are released, hallucination detection may become increasingly challenging.The authors recommend continuous evaluation to track safety and reliability over time.Source: Limitations · p. 8
  • Future detectors may need domain-specific knowledge from knowledge graphs or high-quality medical literature for limited-context real-world medical queries.The paper names sources such as WebMD, Mayo Clinic, and NHS help articles as examples of relevant medical knowledge.Source: Limitations · p. 8

Ethical Considerations

The study uses open-source medical question-answering datasets and expert evaluations under institutional ethics approval, with dataset access restricted to research purposes.

Ethical Considerations

  • The study uses three open-source medical question-answering datasets and six consenting medical experts to evaluate hallucinations.The protocol received institutional ethics approval.Source: Ethical Considerations · p. 8
  • Dataset access is intended for research purposes only under the project’s ethical usage terms.The terms apply to the publicly shared dataset, expert evaluations, and LLM-generated responses.Source: Ethical Considerations · p. 8
  • The paper argues that LLMs’ weaker hallucination detection than experts, and sometimes laypeople, raises concerns about readiness for accuracy-critical applications without oversight.It recommends longitudinal studies to track changes as models encounter new data and contexts.Source: Ethical Considerations · p. 9

A Hallucination Generation Prompts

The prompts generate plausible but incorrect healthcare answers under three hallucination categories: fact-conflicting, input-conflicting, and context-conflicting. Each category uses a healthcare question, correct answer, and hallucinated answer format, with examples illustrating the targeted error.

A.1 Fact-conflicting Hallucination

  • Each fact-conflicting prompt receives a healthcare query, expert answer, and generated hallucinated-answer slot.The generator is instructed to produce a plausible but incorrect response for the supplied question and correct answer.Source: §A.1 · p. 10
  • Fact-conflicting hallucinations contradict established world knowledge while remaining plausibly written.The prompt asks the generator to fabricate information that does not exist in the correct answer.Source: §A.1 · p. 10
  • The fact-conflicting example falsely advises waiting at least 24 hours after delivery before feeding a baby.The correct answer recommends feeding as soon as possible after delivery, whereas the hallucinated answer supplies contrary medical claims.Source: §A.1 · p. 10

A.2 Input-conflicting Hallucination

  • Input-conflicting hallucinations deviate from the provided healthcare query and correct answer.The prompt again permits fabricating information absent from the right answer, but defines the error by mismatch with the supplied input.Source: §A.2 · p. 11
  • The input-conflicting example uses the same postpartum question and answer while generating the 24-hour feeding delay response.Its prompt structure likewise supplies the question, expert answer, and hallucinated-answer field for completion.Source: §A.2 · p. 11

A.3 Context-conflicting Hallucination

  • The context-conflicting prompt follows the same question–correct-answer–hallucinated-answer template while targeting internal inconsistency.The generator is asked to produce a plausible but incorrect answer for the supplied healthcare question and expert answer.Source: §A.3 · p. 11
  • Context-conflicting hallucinations contain statements that conflict with one another within the generated answer.The example says to wait 24 hours while simultaneously claiming immediate energy is needed, creating an internal contradiction.Source: §A.3 · p. 11

B Annotation Platform For Human Evaluation

Human evaluators assess hallucinated healthcare responses through a customized annotation platform. Its guidelines and example pages support the annotation workflow and evaluator questions.

B Annotation Platform For Human Evaluation

Loading 2409.19492v3…