PaperlayerView exact revision on arXiv

Source-linked AI summary

MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models

Shrey Pandit, Jiawei Xu, Junyuan Hong, Zhangyang Wang, Tianlong Chen, Kaidi Xu, Ying Ding

arXiv:2502.14302v1cs.CLcs.AIcs.LG

TL;DR

Medical hallucinations threaten reliable healthcare use of LLMs, and existing evaluation resources do not adequately capture medically specific detection difficulty. The paper introduces MedHallu, a 10,000-pair PubMedQA-derived benchmark generated and filtered through a controlled pipeline. Results show that models struggle particularly with subtle hallucinations, while knowledge provision and a “not sure” option improve detection outcomes.

  • Problem

    Medical LLMs can produce incorrect information, but existing hallucination benchmarks provide limited medically curated and structured evidence for detecting subtle medical hallucinations.

  • Method

    MedHallu constructs 10,000 PubMedQA-derived question-answer pairs with hallucinated answers generated, filtered, refined, and evaluated using multi-stage semantic and LLM-based checks.

  • Results

    General-purpose LLMs outperform fine-tuned medical LLMs, while relevant knowledge and a “not sure” category improve precision and F1 scores.

  • Takeaways & Limitations

    Hard hallucinations are closer to ground truth and remain difficult to detect, whereas knowledge provision and answer abstention improve benchmark performance.

  • Takeaways & Limitations

    The study omits advanced reasoning models and techniques, and its hallucination generation pipeline relies on the PubMedQA corpus.

Abstract

from arXiv · show

Advancements in Large Language Models (LLMs) and their increasing use in medical question-answering necessitate rigorous evaluation of their reliability. A critical challenge lies in hallucination, where models generate plausible yet factually incorrect outputs. In the medical domain, this poses serious risks to patient safety and clinical decision-making. To address this, we introduce MedHallu, the first benchmark specifically designed for medical hallucination detection. MedHallu comprises 10,000 high-quality question-answer pairs derived from PubMedQA, with hallucinated answers systematically generated through a controlled pipeline. Our experiments show that state-of-the-art LLMs, including GPT-4o, Llama-3.1, and the medically fine-tuned UltraMedical, struggle with this binary hallucination detection task, with the best model achieving an F1 score as low as 0.625 for detecting "hard" category hallucinations. Using bidirectional entailment clustering, we show that harder-to-detect hallucinations are semantically closer to ground truth. Through experiments, we also show incorporating domain-specific knowledge and introducing a "not sure" category as one of the answer categories improves the precision and F1 scores by up to 38% relative to baselines.

1 Introduction

Medical LLM hallucinations create a high-stakes reliability problem, while existing benchmarks insufficiently capture medically specific and subtle detection challenges. MedHallu addresses these gaps with a 10,000-pair benchmark stratified by hallucination difficulty and analyzes model behavior.

  • Medical hallucinations can generate plausible but incorrect information, making reliable detection especially important in healthcare.
  • Existing benchmarks primarily target general tasks, while medical resources lack domain-specific curation or structured evaluation of hallucination subtlety.
  • MedHallu contains 10,000 medical question-answer pairs derived from PubMedQA and stratifies hallucinations into easy, medium, and hard detection tiers.
  • Harder-to-detect hallucinations are semantically closer to ground truth, and bidirectional-entailment clusters show consistent easy-or-hard detection patterns.
  • General-purpose LLMs outperform fine-tuned medical LLMs, while relevant knowledge and a “not sure” class improve detection precision.

2 Related Work

Prior hallucination benchmarks and detection methods provide useful foundations but do not fully address medical specificity, structured data resources, or domain-specific knowledge. MedHallu builds on these gaps with medically tailored hallucination generation and evaluation examples.

  • Existing hallucination benchmarks cover tasks such as translation, dialogue, summarization, and question answering, but commonly use generic methodologies.
  • HaluBench includes some medical samples, but its generation process is not specifically tailored to medical contexts.
  • Med-HALT focuses on benchmarking medical hallucination performance rather than providing a structured dataset.
  • Prior semantic analyses note that hallucinated text may contain overconfident or statistically improbable tokens, while systematic comparison with corresponding ground truth remains limited.
  • The paper illustrates generation with a question, contextual knowledge, ground-truth answer, hallucinated answer, and a subsequent correctness comparison.

3 MedHallu Benchmark

MedHallu uses a multi-stage pipeline to generate, filter, refine, and validate medically specific hallucinated answers. Its dataset covers four hallucination categories and three difficulty levels, with semantic checks helping separate hallucinations from ground truth.

  • Dataset Generation Pipeline: The benchmark adapts hallucination categories to the medical domain and organizes examples by hallucination type and detection difficulty.
  • 76% of the dataset involves question misinterpretation, while evidence fabrication accounts for 0.5%.
  • Dataset Generation Pipeline: MedHallu generates hallucinated answers through sampling, quality filtering, correctness checking, TextGrad refinement, and fallback selection.
  • Dataset Generation Pipeline: Quality filtering labels answers as hard when all ensemble LLMs are deceived, medium when multiple are deceived, and easy when only one fails.
  • Dataset Generation Pipeline: Bidirectional entailment measures semantic divergence from ground truth, retaining samples whose score falls below a stringent threshold.
  • Dataset Generation Pipeline: Failed generations are refined using TextGrad feedback focused on artificial linguistic patterns and structural naturalness before further filtering or fallback selection.

4 Implementation Details

MedHallu generation combines controlled hallucinated-response sampling with quality and correctness checks, while evaluation compares zero-shot and context-aware discrimination across general and medical models.

  • Dataset generation: MedHallu uses PubMedQA ground-truth pairs and regulated sampling to generate hallucinated responses.The supplied implementation passage specifies Qwen2.5B-14B generation, temperature variation from 0.3 to 0.7, and 10,000 total instances.
  • Quality and correctness checks: Responses are retained only when at least one of three LLMs is deceived, while correctness uses bidirectional entailment with a 0.75 confidence threshold.The quality check uses GPT-4o mini, Gemma2-9B, and Qwen2.5-7B; correctness uses microsoft/deberta-large-mnli.
  • Generation fallback: TextGrad with GPT-4o mini supplies feedback for failed samples, with up to five generation attempts before selecting the most semantically similar fallback.The fallback applies when no valid response is produced within the permitted iterations.
  • Discriminator settings: Discriminators are evaluated zero-shot and with ground-truth context to compare baseline detection with context-aware discrimination.The evaluation includes both general-purpose and specialized medical language models.
  • Evaluation reporting: Table 2 compares model performance with and without knowledge on the 10,000-sample MedHallu benchmark, reporting overall precision and knowledge-related F1 change.The table caption notes that GPT-4o is excluded from model-size averages for a fair general-versus-fine-tuned comparison.

5 Results and Analysis

Results show that medical hallucination detection varies by model, difficulty, semantic proximity, hallucination category, and evaluation design. Knowledge access and a “not sure” option improve performance, while harder hallucinations remain closer to ground truth.

  • Model performance: 0.625 is GPT-4o’s F1 score on hard samples, while general LLMs outperform medical fine-tuned models on easy and medium categories without knowledge.Qwen2.5-3B also achieves 0.606 overall F1, exceeding several larger models in the zero-shot setting.
  • Knowledge augmentation: +0.251 is the average overall F1 gain for general LLMs after adding knowledge, compared with +0.138 for medically fine-tuned LLMs.General-model average F1 rises from 0.533 to 0.784, while medical-model average F1 rises from 0.522 to 0.660.
  • Knowledge augmentation: 0.852 is Qwen2.5-14B’s overall F1 with domain knowledge, whereas Qwen2.5-3B gains only +0.07 F1.The results associate larger model scale and fine-tuning status with differences in the extent of knowledge-related improvement.
  • Semantic analysis: Hard-to-detect hallucination clusters are closer to ground truth in semantic space, supported by Euclidean distance, cosine similarity, and ROUGE.Hallucinated responses within the same semantic cluster tend to show near-uniform detection outcomes.
  • Semantic analysis: Ground-truth responses remain semantically isolated from hallucination clusters, while clustered hallucinations retain contextual relevance with distinct semantics.The supplied analysis presents this separation as evidence supporting the generation pipeline’s robustness.
  • Declining to answer: A “not sure” option improves F1 and precision, with smaller models gaining 3–5% and larger models gaining about 10–15%.GPT-4o reaches up to 79.5% performance, while Qwen2.5-14B reaches 76.2%.
  • Declining to answer: Qwen2.5-14B answers only 27.9% of questions definitively when “not sure” is available, whereas medical models usually attempt nearly all questions.UltraMedical has a 69.7% response rate and OpenBioLLM reaches 99.7%.
  • Hallucination categories: Incomplete Information is hardest to detect, with 41% hard cases and 54% detection, while MPM reaches 68% detection accuracy.The analysis contrasts subtle incomplete presentation with better detection of more overt fabrication.

6 Conclusion

MedHallu is a comprehensive medical hallucination benchmark that evaluates diverse LLM configurations and highlights persistent difficulty with subtle hallucinations. It also identifies knowledge provision and a “not sure” option as ways to improve detection in high-stakes settings.

  • MedHallu comprises 10,000 rigorously curated medical question-answer pairs with hallucinated answers.
  • Hard hallucination answers are particularly difficult for existing LLMs to detect because they are closer to the ground truth.
  • Providing knowledge can allow general-purpose LLMs to outperform medically fine-tuned models in hallucination detection.
  • Allowing models to answer “not sure” improves precision in critical medical applications.

7 Limitations

The study identifies resource and corpus constraints affecting benchmark generation, evaluation methods, and domain coverage.

  • Advanced reasoning models were not used for benchmark generation because their computational costs were prohibitive.
  • Evaluation was restricted to zero-shot input-output prompting, excluding techniques such as chain-of-thought and self-consistency.
  • Relying on PubMedQA preserves biomedical relevance but limits corpus diversity, scalability, and domain coverage.

8 Ethics Statement

MedHallu uses licensed, de-identified biomedical literature and plans to release artifacts with documentation to support transparency and reproducibility.

  • The benchmark uses publicly available PubMedQA data under MIT licenses with attribution and source-term compliance.
  • Patient privacy is preserved through exclusive use of de-identified biomedical literature.
  • All artifacts will be released with detailed documentation to promote transparency and reproducibility in medical AI safety research.

Appendices

The appendices distinguish factuality from faithfulness hallucinations and situate medical hallucination detection within prior comparisons of general-purpose and fine-tuned models.

  • Fine-tuned LLMs have been reported to exhibit reduced hallucination during text generation compared with general-purpose models.
  • Factuality hallucination departs from established real-world facts, whereas faithfulness hallucination diverges from the provided context or prompt.

B Incorporating Knowledge into the Analysis of Models’ Denial Capabilities

Providing relevant knowledge and allowing models to answer “not sure” improves medical hallucination detection, especially precision. The evaluation also examines how hallucination categories and verification procedures support dataset construction.

  • Knowledge-Enabled Detection: Models given relevant knowledge show improved ability to answer the detection questions compared with the knowledge-disabled setting.The supplied passage states that knowledge improves model capability and that Qwen2.4-14B achieves the highest F1 and precision in that setting.
  • Knowledge-Enabled Detection: The “not sure” option is evaluated alongside required-answer performance using F1, precision, and response-rate measures.F1NS and PNS correspond to the option being available, while F1R and PR apply when answering is mandatory.
  • Knowledge-Enabled Detection: Precision generally improves even when F1 increases only slightly or remains nearly unchanged.This pattern is reported across the evaluated models.
  • Dataset Construction: The dataset construction uses hallucination categories adapted from KnowHallu, while verification combines correctness checks and language-model quality filtering.The broader pipeline also includes bidirectional entailment and sequential improvement procedures.
  • Dataset Construction: The project generates 10,000 samples and documents computational infrastructure separately from the discriminator models used for benchmarking.The dataset-generation setup includes Qwen2.5-14B and verifier models, with infrastructure details reported in a dedicated table.

H LLMs Used in Discriminative Tasks

The discriminative-task evaluation uses a broad set of general-purpose and medically specialized language models. The reported tables compare model performance under datasets generated by different model families and with or without external knowledge.

  • General-Purpose Models: The evaluation includes GPT-4o and GPT-4o mini as commercial models accessed through the official OpenAI API.Other listed models are implemented through Hugging Face.
  • General-Purpose Models: Llama-3.1 and Llama-3.2 are multilingual open-source model families with multiple parameter sizes used for comparative evaluation.The passage lists Llama-3.1 sizes from 8B to 405B and Llama-3.2 sizes from 1B to 90B.
  • General-Purpose Models: Qwen2.5 is evaluated in its 3B, 7B, and 14B variants for medical hallucination detection.The passage characterizes Qwen2.5 as an advanced language model applied to complex language tasks.
  • General-Purpose Models: Gemma2 is evaluated in 2B and 9B variants and is described as having robust performance in discriminative tasks.The supplied passages identify the model sizes but provide no comparative score here.
  • Evaluation Tables: Tables 8 and 9 report overall F1, overall precision, difficulty-level F1, and the F1 change associated with external knowledge.The tables use datasets generated by Gemma2-9B-it and Qwen2.5-14B, respectively.

I Additional Implementation Details

The implementation uses PubMedQA context for both hallucination generation and detection, with prompts specifying generation and judging behavior. Bidirectional-entailment clusters organize semantically equivalent responses.

  • Infrastructure: The experiments use PyTorch, CUDA, Hugging Face resources, and vLLM with tensor parallelism to support model inference.The configuration includes PyTorch 2.4.0, CUDA 12.2, vLLM 0.6.3.post1, tensor_parallel_size 4, and gpu_memory_utilization 0.80.
  • PubMedQA and Prompts: PubMedQA supplies relevant context for generating hallucinated answers and for the hallucination detection task.The dataset contains expert-annotated questions and machine-labeled questions from PubMed abstracts.
  • PubMedQA and Prompts: The generation and detection prompts use PubMedQA’s relevant context as the “knowledge” or context provided to the model.Figure 6 covers dataset generation, while Figure 7 covers hallucination detection.
  • Semantic Clustering: Bidirectional-entailment clustering groups sentences that have identical meaning despite differing syntax.The example also includes a cluster of sentences that fail to fool a discriminator.
  • PubMedQA and Prompts: The generation prompt asks for plausible but factually incorrect answers and justifications that blend with real facts.The detection prompt instead asks a model to judge whether an answer contains non-factual or hallucinated information.
Loading 2502.14302v1…