PaperlayerView exact revision on arXiv

Source-linked AI summary

Medical Hallucinations in Foundation Models and Their Impact on Healthcare

Yubin Kim, Hyewon Jeong, Shan Chen, Shuyue Stella Li, Chanwoo Park, Mingyu Lu, Kumail Alhamoud, Jimin Mun, Cristina Grau, Minseok Jung, Rodrigo Gameiro, Lizhou Fan, Eugene Park, Tristan Lin, Joonsik Yoon, Wonjin Yoon, Maarten Sap, Yulia Tsvetkov, Paul Liang, Xuhai Xu, Xin Liu, Chunjong Park, Hyeonhoon Lee, Hae Won Park, Daniel McDuff, Samir Tulebaev, Cynthia Breazeal

arXiv:2503.05777v2cs.CLcs.AIcs.CY

TL;DR

Medical hallucinations threaten the reliability and safety of foundation models in healthcare, especially when plausible errors influence clinical decisions. The paper evaluates 11 general-purpose and medical-specialized models across seven tasks, combining benchmark analysis, physician audits, mitigation tests, and a clinician survey. General-purpose models showed higher hallucination resistance than medical-specialized models, while reasoning scaffolds reduced errors and audits attributed most residual failures to causal or temporal reasoning.

  • Problem

    Medical hallucinations can undermine clinical reasoning and patient outcomes because plausible AI-generated misinformation is difficult to detect in healthcare.

  • Method

    The study evaluates foundation models across medical reasoning and biomedical information retrieval using benchmarks, physician audits, clinician surveys, and mitigation strategies.

  • Results

    General-purpose models achieved higher median hallucination resistance than medical-purpose models: 76.6% vs. 51.3%, average difference 25.2%.

  • Takeaways & Limitations

    Reliable medical AI depends on sophisticated reasoning and broad knowledge integration, not narrow medical specialization alone.

  • Takeaways & Limitations

    Hallucination detection is constrained by the absence or high cost of reliable ground truth for complex or novel medical queries.

Abstract

from arXiv · show

Hallucinations in foundation models arise from autoregressive training objectives that prioritize token-likelihood optimization over epistemic accuracy, fostering overconfidence and poorly calibrated uncertainty. We define medical hallucination as any model-generated output that is factually incorrect, logically inconsistent, or unsupported by authoritative clinical evidence in ways that could alter clinical decisions. We evaluated 11 foundation models (7 general-purpose, 4 medical-specialized) across seven medical hallucination tasks spanning medical reasoning and biomedical information retrieval. General-purpose models achieved significantly higher proportions of hallucination-free responses than medical-specialized models (median: 76.6% vs 51.3%, difference = 25.2%, 95% CI: 18.7-31.3%, Mann-Whitney U = 27.0, p = 0.012, rank-biserial r = -0.64). Top-performing models such as Gemini-2.5 Pro exceeded 97% accuracy when augmented with chain-of-thought prompting (base: 87.6%), while medical-specialized models like MedGemma ranged from 28.6-61.9% despite explicit training on medical corpora. Chain-of-thought reasoning significantly reduced hallucinations in 86.4% of tested comparisons after FDR correction (q < 0.05), demonstrating that explicit reasoning traces enable self-verification and error detection. Physician audits confirmed that 64-72% of residual hallucinations stemmed from causal or temporal reasoning failures rather than knowledge gaps. A global survey of clinicians (n = 70) validated real-world impact: 91.8% had encountered medical hallucinations, and 84.7% considered them capable of causing patient harm. The underperformance of medical-specialized models despite domain training indicates that safety emerges from sophisticated reasoning capabilities and broad knowledge integration developed during large-scale pre-training, not from narrow optimization.

1 Introduction

Foundation models offer important healthcare capabilities but can generate plausible, clinically dangerous misinformation. This paper defines medical hallucinations, develops a taxonomy, evaluates models and mitigation strategies, and combines benchmark, physician-audit, and clinician-survey evidence.

  • Motivation: Medical hallucinations can produce incorrect dosages, drug interactions, or diagnostic criteria with potentially life-threatening consequences.The paper emphasizes that medical applications leave minimal margin for error.
  • Definition: Medical hallucinations are incorrect, misleading, or unsupported outputs that could influence clinical judgment or patient outcomes.The paper frames these errors as systematic, context-dependent reasoning failures analogous to cognitive biases in clinicians.
  • Challenges: Medical applications face four distinctive hurdles: rapidly changing information, precision requirements, interconnected concepts, and domain-specific jargon.These factors can make models obsolete, amplify small errors, and require specialized interpretation.
  • Contributions: The study contributes a taxonomy, cross-domain model analyses, a survey of 70 clinicians, and tests of structured prompting and reasoning scaffolds.The evaluation spans general practice, oncology, cardiology, and medical education, including general-purpose and domain-specific models.
  • Contribution: Medical-specialized models remain vulnerable to hallucinations that often arise from reasoning failures rather than missing medical knowledge.The paper combines quantitative benchmarks, physician-led qualitative analysis, and clinician surveys to characterize the problem holistically.

2 LLM Hallucinations in Medicine

Medical hallucinations are especially consequential because plausible clinical language can conceal errors that affect diagnosis, treatment, and patient safety. The paper surveys model adaptation, organizes hallucination types, and connects these errors to detection challenges, cognitive-bias parallels, and trust erosion.

  • Clinical context: Medical hallucinations can undermine diagnostic reasoning, therapeutic recommendations, and patient counseling because plausible misinformation may be difficult to detect.Patients may lack the expertise needed to verify AI-generated medical advice.
  • Model adaptation: Medical LLMs are adapted through domain-specific corpora, instruction tuning, and retrieval-augmented generation to align outputs with clinical practice.These adaptations build on transformer architectures and large-scale pretraining.
  • Related models: Medical-specialized models such as Med-PaLM, Med-PaLM 2, Med-Gemini, MedGemma, Meditron, and Med42 have been developed for medical benchmarks and clinical evaluations.The supplied passage identifies biomedical-text integration and multimodal inputs as components of these approaches.
  • Definitions and distinctions: Medical hallucinations are factually incorrect, logically inconsistent, or inadequately grounded outputs that can appear clinically valid while containing critical inaccuracies.Their specialized terminology and apparently coherent logic make expert scrutiny important.
  • Clinical implications: Medical hallucinations differ from general hallucinations because they arise in specialized clinical tasks and can directly harm patients through delayed care or inappropriate interventions.Their impact and detectability depend partly on audience expertise and prompting quality.
  • Taxonomy: The paper’s taxonomy groups hallucinations into factual errors, outdated references, spurious correlations, incomplete reasoning chains, and fabricated sources or guidelines.The framework categorizes types by underlying causes and manifestations and provides examples of superficially plausible incorrect outputs.
  • Trust: Repeated hallucinations can erode trust among healthcare providers and patients, inhibiting broader integration of AI in medical decision-making.Providers may avoid error-prone systems, while patients may become apprehensive about AI reliability.

3 Causes of Hallucinations

Medical hallucinations arise from interacting data limitations, model behaviors, and the distinctive complexity of clinical information. Failures particularly reflect ambiguity, outdated or incomplete knowledge, poor calibration, and limited causal reasoning.

  • 3 Causes of Hallucinations: Medical hallucinations arise from interacting data, architectural, behavioral, and healthcare-specific factors.These factors can operate independently or together.
  • 3 Causes of Hallucinations: Incomplete, noisy, ambiguous, and outdated clinical data can propagate false patterns and obsolete recommendations into model outputs.EHRs and physician notes may contain misspellings, incomplete entries, ambiguous abbreviations, and changing guidelines.
  • 3 Causes of Hallucinations: Biased or insufficiently diverse training data limit generalization across patient populations, rare diseases, and healthcare settings.Underrepresented groups and unfamiliar conditions can produce systematic errors or hallucinations.
  • 3 Causes of Hallucinations: Poor confidence calibration can make models express incorrect information with unwarranted certainty, misleading clinicians.Suggested responses include probabilistic modeling, confidence-aware training, and improved uncertainty estimation.
  • 3 Causes of Hallucinations: Medical reasoning failures occur when statistical correlations replace causal inference and contextual interpretation of patient information.A model may recognize a symptom-disease association yet fail to weigh age, risk factors, and alternative explanations.

4 Detection and Evaluation of Medical Hallucinations

Medical hallucinations are evaluated through complementary methods that test factual support, summary fidelity, uncertainty, context preservation, and clinical reasoning consistency. Evaluation remains constrained by ambiguous definitions and the difficulty or cost of establishing reliable expert ground truth.

  • 4 Detection and Evaluation of Medical Hallucinations: Hallucination benchmarks span medical exams, radiology reports, clinical histories, and generated summaries using accuracy, confidence, fluency, coherence, and error-reduction metrics.The benchmark landscape covers diverse task types, input sources, and evaluation criteria.
  • 4 Detection and Evaluation of Medical Hallucinations: Factual verification compares generated claims with reliable medical evidence at fine-grained or claim-specific levels.Automated overlap metrics and expert verification can assess whether outputs are supported by authoritative sources.
  • 4 Detection and Evaluation of Medical Hallucinations: Summary consistency methods test whether generated summaries faithfully preserve source information through question-answering and entailment approaches.They target omitted, distorted, fabricated, or internally inconsistent clinical details.
  • 4 Detection and Evaluation of Medical Hallucinations: Uncertainty-based detection uses sequence log-probability or semantic entropy to flag outputs that may lack strong grounding.Low sequence probability or high semantic entropy indicates potential hallucination risk.
  • 4 Detection and Evaluation of Medical Hallucinations: Clinical tests assess chronological ordering, laboratory interpretation, jargon disambiguation, contradictory evidence, and differential diagnosis generation.The paper conducts chronological ordering, lab understanding, and differential diagnosis tests with physician-annotated responses.
  • 4 Detection and Evaluation of Medical Hallucinations: Reliable ground truth is often absent or expensive for complex and novel medical queries, limiting detection evaluation and supervised training.Clinician disagreement and diagnostic complexity further complicate annotation; full agreement among three oncologists occurred in only 61.9% of cases.

5 Mitigation Strategies

Mitigation strategies target training data, model behavior, external retrieval, reasoning, and uncertainty communication. Each approach addresses different reliability challenges, while retrieval and preference-based methods introduce their own data-quality or implementation constraints.

  • 5 Mitigation Strategies: Data-centric mitigation improves hallucination resistance through higher-quality, broader, more diverse, and task-specific biomedical datasets.Curation and augmentation aim to reduce propagated errors, bridge knowledge gaps, improve generalization, and mitigate bias.
  • 5 Mitigation Strategies: Model-centric methods refine internal representations, reasoning capabilities, and output generation for medically reliable and interpretable decisions.Approaches include fine-tuning, preference optimization, knowledge feedback, knowledge editing, and self-refinement.
  • 5 Mitigation Strategies: Preference tuning and knowledge-feedback methods seek more factual outputs, but preference tuning requires costly, high-quality expert labels.Medical adoption is constrained by annotation expense, limited expert annotators, and privacy concerns.
  • 5 Mitigation Strategies: RAG can introduce errors when retrieval returns irrelevant or low-quality information or conflicts with the model’s parametric knowledge.The model may default to outdated internal knowledge when retrieved documents contradict it.
  • 5 Mitigation Strategies: Retrieval-augmented generation integrates external knowledge without additional model retraining and can make medical outputs more grounded and interpretable.In medical contexts, RAG has been reported to outperform model-only CoT methods on complex reasoning tasks.
  • 5 Mitigation Strategies: Low-resource specialties require systems to communicate uncertainty so clinicians and patients can validate potentially flawed outputs.This is especially important where training data are limited and guidelines evolve rapidly.

6 Experiments on Medical Hallucination Benchmark

The benchmark evaluates medical hallucinations across model architectures and prompting strategies, finding stronger resistance in advanced general-purpose models than in medical-specialized models. Results also show that reasoning, retrieval, and conceptual understanding affect reliability, while semantic metrics alone do not capture clinical safety.

  • Med-HALT provides a comprehensive framework assessing reasoning and memory-related medical hallucinations.
  • Model comparisons: 97.9% with CoT versus 87.6% baseline hallucination resistance was achieved by gemini-2.5-pro, exceeding earlier-generation models.o3-mini and deepseek-r1 also outperformed o1, gpt-4o, and gpt-4o-mini at baseline.
  • Prompting effects: CoT produced the most consistent hallucination reductions, while system prompting provided complementary gains in tested models.For example, o3-mini improved from 80.4% baseline to 90.7% with CoT, while deepseek-r1 improved from 86.6% to 90.7%.
  • Conceptual understanding: High semantic similarity clustered with stronger hallucination resistance, suggesting that conceptual understanding mattered more than medical-domain exposure alone.The results challenge the assumption that domain specialization inherently improves medical AI reliability.
  • Model comparisons: 76.6% versus 51.3% median baseline resistance favored general-purpose over medical-specialized models, with a 25.2% average difference and significant Mann–Whitney test results.The comparison was U = 27.0, p = 0.012, with rank-biserial r = −0.64 and 95% CI [−0.86, −0.28].
  • Search augmentation: Search augmentation improved lower-performing models substantially but did not overcome their lower absolute performance relative to advanced general-purpose models.pmc-llama improved 46.1% relatively, from 40.8% baseline to 59.9% with Search; gpt-5 reached 87.6% with a +16.5% improvement.
  • Search augmentation: Search augmentation sometimes reduced performance in advanced models, including deepseek-r1’s change from 86.6% baseline to 84.3% with Search.The authors associate this pattern with possible noise or contradiction in retrieved sources for models with strong internal knowledge.

7 Annotations of Medical Hallucination with Clinical Case Records

The study used physician-annotated NEJM case records and three tests to characterize hallucination types, clinical risks, and task-specific failure patterns. Hallucinations were more frequent in factual and temporal processing than in diagnosis prediction, while annotation agreement remained moderate.

  • Annotation Method: Seven clinical annotators independently classified generated outputs by hallucination type and clinical-risk level using established frameworks.The evaluation used NEJM Case Reports and three tests targeting consistency, factual accuracy, and clinical ambiguity.
  • Task-Specific Trends: Diagnosis Prediction had the lowest hallucination rates across models, ranging from 0% to 22%.The result contrasts with higher error frequencies in tasks requiring precise factual recall and temporal integration.
  • Task-Specific Trends: Chronological Ordering and Lab Data Understanding produced higher hallucination frequencies, with ranges of 0.25–24.6% and 0.25–18.7%, respectively.Models often struggled to extract and synthesize detailed factual and temporal information from clinical text.
  • Model Comparisons: GPT-4o showed hallucination rates of 24.6% in Chronological Ordering and 18.7% in Lab Data Understanding, with many errors classified as clinically significant or considerable risk.This highlights that hallucination frequency and clinical severity can both matter in evaluation.
  • Model Comparisons: Claude-3.5 and o1 achieved 0% hallucination rates in Diagnosis Prediction, while Claude-3.5 reached 0.5% and 0.25% in Chronological Ordering and Lab Data Understanding.These models were described as top performers across the evaluated tasks and risk categories.
  • Annotation Reliability: Inter-rater agreement was moderate, with Jaccard-like indices of 0.272 for hallucination types and 0.347 for clinical-risk levels.Annotators differed over whether omissions and temporal inaccuracies constituted clinically significant hallucinations.

8 Survey on AI/LLM Adoption and Medical Hallucinations Among Healthcare Professionals and Researchers

The survey examined professionals’ use, trust, experiences, and concerns regarding AI/LLM tools in healthcare and research. Respondents reported widespread adoption alongside frequent hallucination encounters, cautious trust, and strong demand for verification and safer integration.

  • Survey Design: The survey collected 70 complete responses from 75 participants through a 31-question online instrument with a 93% completion rate.Participants represented medical, research, analytical, and biomedical-engineering backgrounds.
  • Adoption and Trust: 40 respondents used AI/LLM tools daily, while 30 expressed high trust, 25 moderate trust, and 12 low trust in their outputs.Adoption was widespread despite more cautious trust levels.
  • Hallucination Experiences: 37 respondents reported encountering hallucinations, including 38 mentions in literature reviews, 25 in data analysis, 15 in patient diagnostics, and 13 in treatment recommendations.Hallucinations appeared across both research and clinical tasks.
  • Verification Practices: 85% of respondents used cross-referencing with external sources to address hallucinations.Other responses included consulting experts, ignoring erroneous outputs, or stopping AI/LLM use.
  • Perceived Causes: Insufficient and biased training data were each cited 31 times, followed by model-architecture limitations cited 30 times.Respondents also identified lack of real-world context and overconfidence as contributing factors.
  • Limitations and Future Outlook: Respondents identified lack of domain-specific knowledge, privacy and data security, accuracy, and validation as leading AI/LLM limitations.Future priorities emphasized accuracy, explainability, ethics, manual verification, human supervision, and confidence indicators.

9 Regulatory and Legal Considerations for AI Hallucinations in Healthcare

The paper describes regulatory and legal gaps created by generative AI’s variable outputs, evolving behavior, clinical integration, and potential for plausible but incorrect information. It discusses risk-based oversight, monitoring, documentation, and distributed liability as responses.

  • Existing Frameworks: Healthcare AI oversight includes HIPAA requirements for protected health information and AMA guidance emphasizing transparency, physician oversight, and patient safety.AMA guidance treats AI as augmentative rather than a replacement for clinical judgment.
  • Existing Frameworks: FDA oversight uses a risk-based framework for AI systems functioning as Software as a Medical Device, including Premarket Approval and De Novo classification pathways.FDA guidance also addresses data quality, algorithm validation, and performance monitoring.
  • Regulatory Challenges: Generative AI systems create regulatory gaps because their stochastic outputs, continuous learning, and clinical-workflow integration differ from traditional medical technologies.These characteristics complicate existing oversight mechanisms.
  • Regulatory Challenges: Variable responses to identical inputs make validation against ground truth challenging, while systems may span regulated and non-regulated applications.Plausible but incorrect information introduces distinct safety risks.
  • Emerging Oversight: Effective generative-AI regulation requires quantifying and categorizing hallucinations, establishing clinical risk thresholds, and monitoring and reporting adverse events.FDA adaptations primarily address supervised learning systems rather than generative AI’s unique challenges.
  • Legal Considerations: Liability is difficult to assign because AI-generated errors may involve developers, providers, and institutions, while black-box systems obscure causation and negligence.A distributed liability model allocates responsibility according to stakeholder roles and control levels.

10 Conclusion

The conclusion characterizes medical hallucination as a reasoning-driven failure mode and reports that causal and temporal reasoning errors persist despite large-scale models and medical training. It also emphasizes that structured methods can reduce, but not eliminate, clinically consequential errors.

  • Conclusion: Medical hallucination is defined as a reasoning-driven failure mode distinct from general hallucinations in origin and clinical consequence.The paper supports this definition with a taxonomy and physician-audited benchmark.
  • Conclusion: Most medical hallucinations stem from causal and temporal reasoning failures rather than missing medical knowledge.Examples include misordered symptom progression, flawed diagnostic logic, and misplaced causal inference.
  • Conclusion: Greater parameter count and data coverage alone do not translate into safer clinical reasoning.The conclusion states that these reasoning errors persist even in large-scale models.
  • Conclusion: Structured prompting and retrieval-augmented generation reduced hallucinations by over 10%, but high-risk reasoning errors remained.The finding underscores limits in current architectures’ approximation of human clinical judgment.
  • Conclusion: Over 90% of surveyed clinicians had encountered AI-generated medical hallucinations, and most recognized their potential to cause patient harm.The survey complemented benchmark and physician-audit findings with reported real-world experience.

A.1.3 Perceived Causes and Limitations

The survey asks respondents about perceived causes of AI hallucinations, limitations and usefulness of AI/LLM tools, desired improvements, future optimism, development priorities, safeguards, and additional experiences.

  • Respondents selected the main perceived causes of AI hallucinations from options including insufficient or biased training data, architectural limitations, missing real-world context, and overconfidence.
  • The survey assessed limitations including accuracy, domain-specific knowledge, explainability, privacy, and data-related concerns.
  • Respondents reported which AI/LLM tools they use, how helpful they are, and the medical tasks for which they are most useful.
  • Additional questions addressed desired improvements, optimism about future tools, development priorities, safeguards against hallucinations, and further workplace experiences.

Appendix C Constructing NEJM Medical Case Records Dataset

The NEJM case-record dataset was constructed through automated retrieval, text extraction, and structured representation, combining rule-based and machine-learning techniques while addressing acquisition compliance.

  • The dataset comprises case records retrieved from the New England Journal of Medicine.
  • Its construction used multiple steps for automated data retrieval, text extraction, and structured data representation.
  • The approach combined rule-based and machine-learning techniques to improve extraction accuracy and completeness while complying with website terms of service.

C.1 Data Collection

The data-collection pipeline automatically retrieved NEJM case-record PDFs, converted them into structured content using complementary extraction tools, and added verification for missing tables.

  • Data Collection: Case-record PDF URLs were extracted programmatically from the NEJM website and downloaded automatically using Selenium WebDriver with Chrome.
  • Data Collection: Collected PDFs were converted into JSON containing extracted text, images, and tables through complementary image-recognition and text-extraction methods.
  • Data Collection: pdfminer provided digital-PDF text extraction but lacked OCR, image recognition, and preservation of original document structure.
  • Data Collection: Marker was used to compensate for pdfminer’s limitations in extracting images and document structure.
  • Data Collection: GPT-4o reviewed Marker-extracted text for missing tables and triggered additional Marker parsing when omissions were detected.

C.4 Final Data Representation

Each case record was stored in a dedicated directory containing extracted text, images, descriptions, and structured tables, with tables represented in multiple formats and summarized.

  • C.4 Final Data Representation: Each case-record directory contains image files, description.json, text.txt, and tables.json.
  • C.4 Final Data Representation: The extracted case text is stored in text.txt while retaining Markdown formatting.
  • C.4 Final Data Representation: Each table entry includes a structured dictionary, a Markdown rendering, and a concise GPT-4o-generated summary.
  • C.4 Final Data Representation: Example table representations include laboratory variables, reference ranges, and Markdown-formatted parameters with values and units.
  • C.4 Final Data Representation: Table summaries describe the clinical content represented, such as patient vital signs including blood pressure and heart rate.

C.4.3 Images and Descriptions

The paper describes a workflow for extracting and annotating multimodal NEJM case records, including image descriptions and physician assessment of model outputs.

  • Images and Descriptions: The workflow stores extracted images as PNG files alongside captions and summaries in JSON format.The description.json structure includes image captions and summaries.
  • Images and Descriptions: The methodology extracted NEJM medical case records into a multimodal dataset containing text, images, and tables.The extraction process is described as efficient, ethical, and high quality.
  • Reasoning Prompts: The framework uses stepwise prompts to support reasoning, including chronological ordering of clinical events and explicit temporal relationships.The Chain-of-Knowledge examples illustrate decomposing a question into ordered reasoning steps before producing an answer.
  • Hallucination Mitigation: The benchmark compares hallucination resistance with similarity scores across models and mitigation approaches such as retrieval and chain-of-thought prompting.The figure highlights differences between recent advanced models and earlier or medical-purpose models.
  • Physician Annotation: Physicians annotate LLM responses for chronological ordering, laboratory-data understanding, and diagnosis prediction.The web-based tool presents clinical cases and provides task-specific annotation interfaces.
  • Hallucination and Risk Annotation: Annotators classify highlighted segments as hallucinations, assign hallucination types, and rate associated risk levels.The annotation interface includes controls for selecting hallucination categories and risk levels before confirmation.
Loading 2503.05777v2…