Source-linked AI summary
Med-HALT: Medical Domain Hallucination Test for Large Language Models
Ankit Pal, Logesh Kumar Umapathi, Malaikannan Sankarasubbu
TL;DR
Medical LLM hallucinations can create serious risks, yet medical-domain evaluation resources have been limited. The paper introduces Med-HALT, a multinational benchmark with reasoning and memory-based tests, and evaluates several leading LLMs, finding substantial performance differences and room for improvement. The authors also report that results are sensitive to prompt framing and decoding parameters.
Problem
Medical LLM hallucinations can generate incorrect or unverified information with potentially serious consequences, while prior mitigation efforts had not addressed the medical field.
Method
Med-HALT combines multinational medical-examination data with reasoning and memory-based hallucination tests and evaluates multiple LLMs using their default configurations.
Results
The evaluated models showed significant performance differences, with the comparative analysis revealing considerable room for improvement.
Takeaways & Limitations
Med-HALT provides an openly shared framework, dataset, and statistics intended to support transparency, reproducibility, and safer medical-domain LLM evaluation.
Takeaways & Limitations
Model outputs were sensitive to prompt framing and decoding parameters, with slight changes sometimes turning previously correct answers into hallucinations.
Abstract
from arXiv · showhide
This research paper focuses on the challenges posed by hallucinations in large language models (LLMs), particularly in the context of the medical domain. Hallucination, wherein these models generate plausible yet unverified or incorrect information, can have serious consequences in healthcare applications. We propose a new benchmark and dataset, Med-HALT (Medical Domain Hallucination Test), designed specifically to evaluate and reduce hallucinations. Med-HALT provides a diverse multinational dataset derived from medical examinations across various countries and includes multiple innovative testing modalities. Med-HALT includes two categories of tests reasoning and memory-based hallucination tests, designed to assess LLMs's problem-solving and information retrieval abilities. Our study evaluated leading LLMs, including Text Davinci, GPT-3.5, LlaMa-2, MPT, and Falcon, revealing significant differences in their performance. The paper provides detailed insights into the dataset, promoting transparency and reproducibility. Through this work, we aim to contribute to the development of safer and more reliable language models in healthcare. Our benchmark can be found at medhalt.github.io
1 Introduction
Med-HALT addresses the risks of medical hallucinations by introducing a benchmark and dataset for evaluating LLMs. It combines multinational examination data with reasoning and memory-based tests to assess hallucination-related capabilities.
- Medical hallucinations can produce plausible but incorrect or unverified information, creating risks for healthcare decisions, diagnosis, treatment, and patient outcomes.
- Med-HALT is proposed as a benchmark and dataset specifically designed to measure and mitigate hallucinations in medical-domain LLMs.
- The framework evaluates reasoning hallucinations and memory-based hallucinations through tests of problem solving and retrieval from encoded training information.
- The dataset combines multiple-choice medical examinations from India, Spain, the United States, and Taiwan across diverse medical subdisciplines.
- The study evaluates Text-Davinci-003, GPT-3.5, LlaMA-2, MPT, and Falcon and provides dataset statistics and openly shared materials to support transparency and reproducibility.
2 Datasets Statistics
Med-HALT comprises seven datasets drawn from medical examinations and PubMed, providing geographically diverse question and biomedical-information sources for hallucination evaluation.
- Med-HALT contains 18,866 samples per reasoning task and 4,916 PubMed samples per memory task.Reasoning-task samples average 238.0 words, while PubMed samples average 37.0 words.
- MedMCQA contributes 9,515 Indian medical-exam questions, including 6,660 from AIIMS PG and 2,855 from NEET PG.
- Headqa contributes 4,068 samples from Spain’s medical residency examination.
- Medqa USMILE contributes 2,801 samples from the United States Medical Licensing Examination.
- The Taiwan Medical Licensing Examination dataset contains 2,482 samples, while the PubMed portion contains 4,916 samples.
3 Types of Hallucination Evaluated
Med-HALT uses two complementary hallucination-test categories: reasoning tests examine medical problem solving, while memory tests examine accurate biomedical information retrieval.
- Reasoning Hallucination Tests: The framework’s reasoning hallucination tests assess whether models produce logically coherent and factually accurate answers without inventing information.
- Reasoning Hallucination Tests: The False Confidence Test asks models to judge a randomly suggested answer and explain why it is correct or incorrect and why alternatives are wrong.
- Reasoning Hallucination Tests: The None of the Above Test replaces the correct option with a none-of-the-above choice, testing recognition of irrelevant or incorrect alternatives.
- Reasoning Hallucination Tests: The Fake Questions Test presents fake or nonsensical medical questions, using both expert-crafted and GPT-3.5-generated examples.
- Memory Hallucination Tests: Memory hallucination tests evaluate retrieval through abstract-to-link, link-to-title, PMID-to-title, and title-to-link tasks.
- Together, the evaluation metrics target both reasoning and recall capabilities to detect different hallucination patterns and improve model robustness.
4 Data Analysis
The dataset spans broad medical subject matter and multinational examination sources, supporting analysis of subject coverage and cross-country exam representation.
- Med-HALT covers subjects ranging from Physiology and Pharmacology to Forensic Medicine and Radio diagnosis.
- Nearly 95% of subjects contain more than 50 topics, and 70% contain more than 100 topics.
- The dataset incorporates medical entrance exams from India, Spain, the United States, and Taiwan.
- Comparing sample ratios across exams provides information about country-level representation and dataset diversity.
- A 30% random sample from exam datasets and PubMed articles was analyzed to categorize required reasoning types, including factual, diagnostic, multihop, mathematical, and comparative reasoning.
5 Experiments
The experiments evaluate selected commercial and open-source models under default configurations using accuracy and a pointwise score that rewards correct and penalizes incorrect predictions.
- The evaluation includes Text-Davinci, GPT-3.5 Turbo, Falcon, MPT, and Llama-2 models.
- Models were assessed in their default configurations without fine-tuning or hyperparameter adjustments.
- OpenAI-model evaluation used the Azure OpenAI ChatGPT API, while open-source models used PyTorch and Hugging Face inference tools.
- Accuracy measures the ratio of correct predictions to total predictions.
- Pointwise Score averages individual scores, awarding +1 for correct predictions and applying a -0.25 penalty for incorrect predictions.
- The pointwise-score equation uses true and predicted labels, an indicator function, and correct-versus-incorrect point values to compute the final score.
6 Results
Results show substantial performance differences across hallucination tasks: open-access models generally outperform commercial variants, but performance remains uneven and limited on some tasks.
- Open-access Falcon and Llama-2 models outperform GPT-3.5 and Text-Davinci across all hallucination tasks.
- 42.21% accuracy and a 52.37 score made Llama-2 70B the top model on the Reasoning FCT task.
- No model reached an acceptable accuracy level on the Reasoning FCT task.
- 99.89% accuracy and an 18.56 score made Falcon 40B the leading model on the Reasoning Fake task.
- Llama-2 70B led the Reasoning Nota task with 77.53% accuracy and a score of 188.6.
- Falcon 40B achieved the highest average accuracy across all tasks at 42.46% and the best average pointwise score across information-retrieval tasks.
- Instruction tuning and RLHF were associated with detrimental effects on hallucination control, especially in Llama-based models.
7 Exploratory Analysis
The exploratory analyses examine how sampling, decoding temperature, few-shot exemplars, prompt framing, and repeatability relate to Med-HALT performance. Accuracy improved with more exemplars but plateaued beyond three, while temperature effects were minimal and prompt design mattered.
- Exploratory setup: 30% of questions from each exam dataset and PubMed articles were randomly sampled with stratification by country, exam type, and difficulty.The sampling supported exploratory analyses across diverse subsets.
- Decoding parameters: GPT-3.5 accuracy varied minimally across decoding temperatures, suggesting temperature adjustments had negligible effects in this analysis.The study measured performance across temperature values on sampled examples.
- Few-shot examples: 7.31% accuracy at zero shot increased with more exemplars, but performance gains decreased beyond three shots.The analysis used GPT-3.5 and summarized results in Figure 6.
- Prompt framing: Accuracy improved when prompts changed from ambiguous to more specific and direct.The prompt variants and examples are reported in Appendix Tables 9–15.
- Repeatability: Repeated GPT-3.5 evaluations showed relatively stable performance with slight fluctuations, despite limited control over commercial API generation settings.Open-source generations could be controlled with seeds and other parameters, whereas commercial API outputs could differ under the same inputs and parameters.
- Brittleness: LLMs were sensitive to prompt framing and decoding parameters, and slight changes could turn previously correct outputs into hallucinated answers.The paper identifies robustness to these settings as requiring further research.
8 Conclusion
The conclusion presents Med-HALT as a comprehensive benchmark and dataset for evaluating and mitigating medical-domain hallucinations. It emphasizes transparent, reproducible research grounded in medical datasets selected for specificity, authenticity, balanced groundedness, diversity, accessibility, and difficulty.
- Conclusion: Med-HALT advances medical-domain hallucination research by providing a comprehensive dataset and benchmark for evaluation and mitigation.The framework is intended to support analysis of hallucination in large language models.
- Dataset criteria: The dataset selection criteria include medical domain specificity and authentic grounding in real-world medical literature and resources.These criteria aim to reflect genuine medical knowledge and support reliable outputs.
- Dataset criteria: Med-HALT seeks both grounded and hallucinated examples to support direct examination of hallucination detection and mitigation techniques.The criterion explicitly pairs groundedness with hallucination.
- Dataset criteria: The datasets are intended to be large and diverse enough to support robust findings across varied medical topics.The paper notes that small datasets may overfit and inadequately represent real-world medical literature.
- Dataset criteria: Public availability and documentation are required to support reproducibility and further research.Accessibility is listed as a dataset-selection criterion.
- Dataset criteria: The datasets should pose a significant challenge for state-of-the-art language models.Difficulty is included among the selection criteria.
A.1 Difficulty and Diversity of Questions
The difficulty and diversity analysis categorizes the reasoning required across sampled exam and PubMed questions. The categories span recall, diagnosis, inference, comparison, multihop synthesis, explanation, and mathematical reasoning.
- Sampling and analysis: A manually analyzed random sample of 30% of questions from each exam dataset and PubMed articles was used to categorize reasoning requirements.The analysis aimed to characterize dataset complexity and required reasoning types.
- Knowledge and diagnosis: Factual questions require direct recall of established medical knowledge, while diagnosis questions require identifying a disease or condition’s cause.Diagnosis additionally requires applying medical knowledge in a diagnostic context.
- Reasoning operations: Fact-based reasoning applies established facts to novel problems, and exclusion-of-distractors questions eliminate unsuitable options to reach an answer.Both categories require using known information to select among possibilities.
- Reasoning operations: Question-logic tasks involve complex structures and conditions, while multihop reasoning synthesizes information from multiple passages.These categories target structured and cross-passage reasoning.
- Explanation and mathematics: Explanation or description questions require defining or describing a term or phenomenon, whereas mathematical questions involve calculations or statistical reasoning.The categories cover both verbal explanation and quantitative reasoning.
- Additional categories: The dataset also includes fill-in-the-blank, comparison, and natural-language-inference questions, with their proportions illustrated in Figure 3.These categories require completion, contrasting options or scenarios, and deriving implied information or logical inferences.
- Examples: Table 8 provides examples of the different reasoning types in the dataset.
B Parsing Output and Handling Exceptions
The study relies on structured JSON outputs and uses multiple parsing procedures to handle formatting failures. Instruction-following errors are tracked separately because output-format compliance provides a signal about model reliability and consistency, rather than directly measuring hallucination.
- Output parsing: The evaluation requires LLMs to return structured, valid JSON in response to tasks and prompts.Malformed outputs occur when models do not strictly follow the expected format.
- Parsing procedures: Basic parsing uses the Promptify module to assess instruction following and directly parse a significant proportion of samples.
- Parsing procedures: Regex-based escaping handles outputs containing both single and double quotes before Promptify parsing.This addresses quote-related formatting errors in generated strings.
- Parsing procedures: Unparsable outputs are counted after the preceding parsing and escaping procedures fail.The paper notes that several prompts retained a high ratio of unparseable outputs.
- Interpretation: Instruction-following errors are not a direct hallucination measure, but violating output constraints signals model reliability and consistency.Specific error ratios per model are reported in Table 5.
Limitations & Future Scope
The study’s evaluation is limited to reasoning and information-retrieval tasks, leaving model performance across other task types and influencing factors insufficiently examined. Future work should broaden hallucination-control strategies, test temperature findings across models, and include GPT-4 where feasible.
- Study scope: The assessment covers only reasoning and information-retrieval tasks, constraining interpretation of overall model performance across task types.The authors call for research examining model structure, training-data diversity, and task nature.
- Hallucination mitigation: Instruction tuning sometimes worsened hallucination control, while other mitigation methods were not investigated.Suggested directions include adding external knowledge or setting specific training objectives.
- Generalization: Temperature effects require further study alongside model structure, training-data diversity, and task type.The authors also question whether the identified ideal temperature range generalizes beyond GPT-3.5.
- Model coverage: Financial constraints prevented inclusion of GPT-4, leaving its medical-domain capabilities and performance for future research.Including GPT-4 is proposed to enrich understanding of large language model performance in the medical domain.
- Future research: Openly sharing the Med-HALT framework, test designs, and dataset statistics is intended to encourage reproducible follow-up research.The authors frame this sharing as a future-oriented contribution to improving reliability and safety in medical-domain language models.