MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models

• Existing benchmarks emphasize general tasks and common-sense knowledge, while medical hallucination subtlety and comparisons between pretrained and fine-tuned medical LLMs remain underexplored.Source: §1 · pp. 1–2
• MedHallu provides 10,000 PubMedQA-derived medical question-answer pairs annotated for accuracy and stratified into easy, medium, and hard detection tiers.Source: §1 · p. 2
• Hallucinated answers that are semantically closer to ground truth are harder to detect, while bi-directional entailment clusters contain entries uniformly easy or hard to detect.Source: §1 · p. 2
• General-purpose LLMs outperform fine-tuned medical LLMs in medical hallucination detection; relevant knowledge and a “not sure” class improve performance or precision.The “not sure” class supplements the “hallucinated” and “not-hallucinated” classes.Source: §1 · p. 2

Hallucination Detection Benchmarks

• Hades (Liu et al., 2022) and HaluEval (Li et al., 2023a) provide detection benchmarks, but existing approaches leave medical-domain gaps addressed by MedHallu’s controlled dataset.HaluBench (Ravi et al., 2024) includes medical samples without medically tailored generation, while Med-HALT (Pal et al., 2023) is an evaluation tool rather than a structured dataset.Source: §2 · p. 2

Semantic Analysis of Hallucinated Text

• Prior research has not systematically compared hallucinated sentences with their ground-truth counterparts, a gap MedHallu addresses by examining their semantic relationships.Earlier studies describe overconfident hallucinations (Miao et al., 2021; Chen et al., 2022), improbable contextual tokens, and decoding adjustments intended to improve factual accuracy.Source: §2 · p. 2

Improvement Methods in Hallucination Detection

• MedHallu combines task-specific medical knowledge with a self-supervised “not sure” class, addressing gaps in domain-relevant knowledge integration and abstention approaches.Prior approaches include general-knowledge retrieval, complex supervision or thresholds for refusal, and less-adaptable refusal of out-of-domain questions.Source: §2 · p. 2

3 MedHallu Benchmark

• MedHallu adapts hallucination categories from KnowHalu to the medical domain, defining categories intended to reflect potential medical hallucinations.The revised category definitions are presented in Table 1. The paper cites KnowHalu (Zhang et al., 2024a) as the source of inspiration.Source: §3 · p. 3, Table 1 · p. 4
• 10,000 samples comprise MedHallu; misinterpretation of the question is the most common category (76%), evidence fabrication the smallest (0.5%), and hard examples are slightly more common than easy and medium ones.The difficulty distribution is described as relatively even; Figure 3 presents the category and difficulty breakdown.Source: §3 · p. 3, Figure 3 · p. 3

Dataset Generation Pipeline

• For each question, an LLM generates diverse answers using category definitions, examples, ground truth, and context, with category choice left to the model rather than manually forced.The generated answer is intended to be semantically similar to the ground truth but incorrect.Source: Dataset Generation Pipeline · p. 4
• An ensemble including Gemma2, GPT-4o-mini, and Qwen2.5 retains a candidate only when a majority judge it more accurate than ground truth; voting also defines difficulty.A candidate is hard when all ensemble models are fooled, medium when multiple but not all are fooled, and easy when only one is fooled.Source: Dataset Generation Pipeline · p. 4
• Generated answers undergo bidirectional entailment checking with RoBERTa-large-MNLI, and only samples below a semantic-divergence threshold are retained.This check is intended to keep hallucinations distinct from ground truth while maintaining coherence and minimizing false positives.Source: Dataset Generation Pipeline · pp. 4–5
• Failed candidates receive up to five TextGrad refinement iterations using GPT-4o-mini feedback; if none passes filtering, the pipeline selects the candidate with maximum cosine similarity to ground truth.Feedback targets linguistic signals of artificial content and structural features affecting naturalness.Source: Dataset Generation Pipeline · p. 5

4 Implementation Details

• Hallucinated answers were generated from 10,000 PubMedQA pairs using regulated sampling and length constraints.The source combines 1,000 expert-annotated pqa_labeled samples with 9,000 randomly selected pqa_artificial instances; Qwen2.5B-14B generation used temperature 0.3–0.7, top-p 0.95, a 512-token maximum, and answer lengths within ±10% of ground truth.Source: §4 · p. 5
• Generated responses were retained only if they deceived at least one of three LLMs and passed bidirectional entailment checking.The quality models were GPT-4o mini, Gemma2-9B, and Qwen2.5-7B; DeBERTa-large-MNLI correctness checking used a 0.75 confidence threshold.Source: §4 · p. 5
• TextGrad feedback and repeated generation were used to repair samples failing either check, with a semantic-similarity fallback after five attempts.GPT-4o mini generated the feedback; if no valid response emerged within the maximum five attempts, the most semantically similar generated answer to ground truth was selected.Source: §4 · p. 5
• Discriminators were evaluated both without additional knowledge and with ground-truth context to compare baseline detection and contextual discrimination.The prompt is described in Figure 7.Source: §4 · pp. 5–6
• Evaluation covered general-purpose and medically fine-tuned LLMs, using low-temperature inference through official OpenAI APIs or Hugging Face Pipelines.Models ran at approximately 0.2–0.3 temperature; the general group included Gemma-2, Llama-3.1, Qwen-2.5, DeepSeek-R1-Llama, GPT-4o, and GPT-4o mini, while the medical group included OpenBioLLM, Llama3-Med42, BioMistral, and UltraMedical.Source: §4 · p. 6

5.1 Which language model performs the best at medical hallucination detection task?

• In zero-shot testing, Qwen2.5-3B’s overall F1 of 0.606 exceeded Gemma-9B (0.515), Llama-3.1-8B-Instruct (0.522), and Qwen2.5-7B (0.533).Models degraded on hard samples, with GPT-4o reaching only 0.625 F1; general LLMs outperformed medical fine-tuned LLMs in precision and F1 on easy and medium categories without additional knowledge.Source: §5.1 · p. 6

5.2 How does providing knowledge impact detection performance?

• With knowledge, general LLMs’ average overall F1 rose from 0.533 to 0.784 (+0.251), versus 0.522 to 0.660 (+0.138) for medically fine-tuned LLMs.Qwen2.5-14B reached 0.852 overall F1 with domain knowledge; Gemma-2-9B gained +0.323, while Qwen2.5-3B gained +0.07. Larger models and medical fine-tuning affected the extent of improvement.Source: §5.2 · pp. 6–7

5.3 Semantic analysis of hallucinated and ground truth sentences.

• Hallucination clusters that fooled detection LLMs were closer to ground truth in semantic vector space, supported by Euclidean distance, cosine similarity, and ROUGE.The cluster similarity analysis compares clusters generated in Section 5.3 with ground-truth samples; the harder-to-detect outputs remained similar while incorporating meaningful deviations.Source: Table 3 · p. 7, §5.3 · p. 7
• Within semantic clusters, hallucinated responses tended to pass detection together or be flagged together, while ground-truth responses rarely aligned with hallucination clusters.This pattern indicates semantic content plays a pivotal role in detection behavior and that ground-truth responses remain semantically distinct from hallucinated outputs.Source: §5.3 · p. 7

5.4 Analysis of models’ ability to decline to answer

• Offering “not sure” improved many models’ F1 and precision, with gains of 3–5% for smaller models and around 10–15% for larger ones.GPT-4o achieved up to 79.5% in performance and Qwen2.5-14B 76.2%. General LLMs answered fewer questions definitively (Qwen2.5-14B: 27.9% response rate), while UltraMedical had a 69.7% response rate and OpenBioLLM 99.7%; comparing the option with knowledge-sharing showed more general-LLM questions attempted alongside increased confidence, F1, and precision.Source: §5.4 · pp. 7–8

Which hallucination category is hardest to detect?

• Incomplete Information was hardest (41% hard; 54% detection), while Methodological and Evidence Fabrication had 76.6% detection despite 37% hard cases.Mechanism/Pathway Misattribution had 68% detection accuracy, and Question Misinterpretation had 68.8%; subtle incomplete presentation was harder to detect than outright fabrication.Source: Which hallucination category is… · p. 8

Which medical category (MeSH term) hallucination is the hardest to detect?

• Among the top five MeSH categories, Psychiatry had the lowest detection rate (53.7%), compared with 57.1% for Diseases and 67.7% for Chemical/Drug queries.The five categories were Diseases (25.9% of samples), Analytical Procedures (20.1%), Chemical/Drug Queries (15.8%), Healthcare Management (9.7%), and Psychiatric Conditions (6.7%).Source: Which medical category (MeSH term)… · p. 8, Figure 5 · p. 8

6 Conclusion

• Existing LLMs struggle to detect medical hallucinations, especially hard answers that are closer to the ground truth.The benchmark also finds that general-purpose LLMs can outperform medically fine-tuned models when provided with knowledge, and that allowing a “not sure” response improves precision in critical applications.Source: §6 · p. 8
• MedHallu comprises 10,000 curated medical question-answer pairs and categorizes hallucination types while generating hallucinated answers across difficulty levels.Its generation framework uses multi-model majority voting to mitigate single-LLM bias.Source: §6 · p. 8
• As the largest open medical hallucination benchmark to date, MedHallu supports evaluation of detection capabilities and cautious use of LLMs in high-stakes medicine.Source: §6 · p. 8

7 Limitations

• Resource constraints prevented benchmark generation with advanced reasoning models such as OpenAI o1, Gemini 2.0, and DeepSeek-R1.The pipeline used multi-stage LLM quality checks and regeneration, but state-of-the-art generation models were prohibitively costly.Source: §7 · p. 9
• Evaluation was restricted to zero-shot input-output prompting, with or without knowledge provision, excluding chain-of-thought and self-consistency.Resource limitations prevented exploration of these techniques, which might better elicit model capabilities.Source: §7 · p. 9
• Hallucination generation relied solely on PubMedQA, ensuring biomedical relevance but limiting scalability and domain coverage.The authors recommend incorporating diverse high-quality corpora in future work.Source: §7 · p. 9

8 Ethics Statement

• The benchmark uses publicly available PubMedQA data under MIT licenses and preserves privacy by relying exclusively on de-identified biomedical literature.Source: §8 · p. 9
• The authors acknowledge dual-use risks, advocate responsible medical LLM deployment with human oversight, and use stratification to target dangerous hard hallucinations closest to factual content.Source: §8 · p. 9
• All artifacts will be released with detailed documentation to promote transparency and reproducibility in medical AI safety research.Source: §8 · p. 9

A Additional Related Work

• Fine-tuned LLMs have been reported to hallucinate less during text generation than general-purpose models (Azaria and Mitchell, 2023; Xiong et al., 2024; Arteaga et al., 2024).Source: A Additional Related Work · p. 12
• No prior work had systematically evaluated domain-specific fine-tuned LLMs on hallucination detection tasks.Source: A Additional Related Work · p. 12
• Lynx outperformed general-purpose LLMs across diverse datasets, but its evaluation did not include models fine-tuned for specialized domains such as medicine or finance (Ravi et al., 2024).Source: A Additional Related Work · p. 12
• MedHallu addresses this gap by comparing several fine-tuned medical LLMs on medical hallucination detection.Source: A Additional Related Work · p. 12
• Hallucinations are distinguished as factuality errors against real-world facts and faithfulness errors that diverge from the supplied context or prompt.Source: A Additional Related Work · p. 12
• Faithfulness detection has been studied both with context (Ming et al., 2024) and without context (Roller et al., 2020; Min et al., 2023; Chern et al., 2023; Wei et al., 2024).Source: A Additional Related Work · p. 12
• Unlike faithfulness detection, hallucination detection has mainly focused on LLM outputs rather than context (Li et al., 2023a; Liu et al., 2022; Hu et al., 2024).Source: A Additional Related Work · p. 12

B Incorporating Knowledge into the Analysis of Models’ Denial Capabilities

• With domain knowledge and a “not sure” option, models’ question-answering performance improved over the knowledge-disabled setting; Qwen2.4-14B had the highest F1 and precision, while precision generally improved even when F1 changed little.Table 5 distinguishes performance with the option (F1NS and PNS) from performance when models are required to answer (F1R and PR); Response% is the share of questions answered yes or no despite the option.Source: B Incorporating Knowledge into the… · p. 12, Table 5 · p. 12

C Additional Data Correctness Check

• Llama-3.1 performs a lightweight semantic comparison of each hallucinated response with its ground truth to filter outputs that may pass the initial entailment check despite being subtly similar.The authors describe this as an additional, cost-effective verification layer alongside the existing correctness check.Source: C Additional Data Correctness Check · p. 12

D Selecting Medical Hallucination Categories for MedHallu

• The authors adapted categories from KnowHallu (Zhang et al., 2024a), rejecting categories difficult to generate convincingly or focused on stylistic nuances, and designed categories for substantive medical factual errors.They aimed for category diversity and practical relevance to medical text generation.Source: D Selecting Medical Hallucination… · p. 12

E MedHallu Creation Using Other Open-weights LLMs

• Qwen2.5-14B was selected to generate the 10,000-sample dataset after faster generation, consistent quality, and fewer average quality-check failures than Gemma2-9B.The team first generated 1,000 samples and used three independent generation runs, reporting mean and standard deviation; results for Gemma2-9B-IT and Qwen2.5-14B were said to align with earlier observations.Source: E MedHallu Creation Using Other… · p. 13

F Example Data from the MedHallu Dataset

• Randomly selected MedHallu examples show the dataset’s hallucination categories and assigned difficulty levels.The examples are presented as an illustration of dataset diversity.Source: F Example Data from the MedHallu Dataset · p. 13

G Hardware Resources and Computational Costs

• Dataset generation used Qwen2.5-14B for 24 hours on an NVIDIA RTX A6000 GPU with 48,685 MiB of RAM, with vLLM and batching to speed execution.Gemma2-9B, Qwen2.5-7B, and GPT-4o mini were also used as verifiers; the dataset contained 10,000 samples.Source: G Hardware Resources and Computational… · p. 13, Table 6 · p. 13

H LLMs Used in Discriminative Tasks

• GPT-4o and GPT-4o mini were accessed through OpenAI’s official API, while the other models were implemented through Hugging Face.The paper describes the GPT models as commercial LLMs used for medical hallucination detection.Source: H LLMs Used in Discriminative Tasks · p. 13
• The roster includes Llama-3.1 and Llama-3.2, Qwen2.5 3B, 7B, and 14B variants, and Gemma2 2B and 9B variants.The listed Llama-3.1 sizes are 8B, 70B, and 405B; Llama-3.2 sizes are 1B, 3B, 11B, and 90B.Source: H LLMs Used in Discriminative Tasks · pp. 13–14
• Other evaluated models include DeepSeek-R1-Distill-Llama-8B, OpenBioLLM-Llama3-8B, BioMistral-7B, Llama-3.1-8B-UltraMedical, and Llama3-Med42-8B.The passages describe these as reasoning-trained, biomedical-specialized, or medically fine-tuned models.Source: H LLMs Used in Discriminative Tasks · p. 14

I Additional Implementation Details

• The experiments used PyTorch 2.4.0, CUDA 12.2, Hugging Face resources, and vLLM 0.6.3.post1 with tensor parallelism 4 and GPU memory utilization 0.80.The dataset resource was qiaojin/PubMedQA; model Hugging Face names are listed in Table 7.Source: I Additional Implementation Details · p. 14

J PubMedQA

• PubMedQA provides 1,000 expert-annotated questions and 211K machine-labeled questions, with relevant context used to generate hallucinated answers and in detection.The expert-annotated set is pqa_labeled_split.Source: J PubMedQA · p. 15

K System Prompts for Hallucination Generation and Detection

• Figures 6 and 7 show the system prompts for MedHallu generation and hallucination detection, respectively, with PubMedQA knowledge supplied as context.The prompts guide model behavior in the two tasks.Source: K System Prompts for Hallucination… · p. 15, Figure 6 · p. 17, Figure 7 · p. 18

L The Clusters Formed for a Question Using Bidirectional Entailment.

• The example clusters group sentences with the same meaning but different syntax, and Cluster 2 includes sentences that fail to fool a discriminator.Table 11 shows three example clusters from a larger set.Source: L The Clusters Formed for a Question… · p. 15, Table 11 · p. 16