PaperlayerView exact revision on arXiv

Source-linked AI summary

Creating Trustworthy LLMs: Dealing with Hallucinations in Healthcare AI

Muhammad Aurangzeb Ahmad, Ilker Yaramis, Taposh Dutta Roy

arXiv:2311.01463v1cs.CLcs.AIcs.CVcs.LGcs.NE

TL;DR

Healthcare adoption of LLMs is limited by factuality, coherence, and hallucination risks in a high-stakes setting. The paper organizes trustworthy deployment around quantifying, validating, and mitigating hallucinations, while emphasizing guardrails and human oversight. It concludes that responsible use requires trustworthy, transparent, unbiased, accurate, and interpretable systems.

  • Problem

    Healthcare LLMs can generate hallucinated or unreliable information about diagnosis, treatment, and medical procedures, creating high cost and liability risks.

    Source: §I · p. 1, §II · p. 2

  • Method

    The paper examines hallucination sources, evaluation approaches, mitigation strategies, model choice, and benchmark auditing for healthcare applications.

    Source: A. Evaluating Hallucinations · p. 2, C. Mitigating Hallucinations · p. 3, G. Model Choice · p. 3

  • Results

    The paper identifies a multi-pronged pathway for trustworthy healthcare LLMs based on quantification, validation, mitigation, and appropriate guardrails.

    Source: §I · p. 1, C. Mitigating Hallucinations · p. 3

  • Takeaways & Limitations

    Healthcare LLM deployment should proceed cautiously with transparency, responsible AI principles, and human-in-the-loop validation for high-stakes tasks.

    Source: §I · p. 1, §V · p. 4

    Human-in-the-loop systems may remain necessary for high-stakes healthcare tasks, while automation is more feasible for low-stakes tasks with guardrails.

    Source: §V · p. 4

Abstract

from arXiv · show

Large language models have proliferated across multiple domains in as short period of time. There is however hesitation in the medical and healthcare domain towards their adoption because of issues like factuality, coherence, and hallucinations. Give the high stakes nature of healthcare, many researchers have even cautioned against its usage until these issues are resolved. The key to the implementation and deployment of LLMs in healthcare is to make these models trustworthy, transparent (as much possible) and explainable. In this paper we describe the key elements in creating reliable, trustworthy, and unbiased models as a necessary condition for their adoption in healthcare. Specifically we focus on the quantification, validation, and mitigation of hallucinations in the context in healthcare. Lastly, we discuss how the future of LLMs in healthcare may look like.

I. INTRODUCTION

LLMs offer potential in healthcare but raise serious concerns about hallucinations, bias, privacy, security, and unreliable medical information. The paper argues for cautious adoption with appropriate guardrails and trustworthy, transparent, unbiased, accurate, and interpretable systems.

I. INTRODUCTION

  • The paper advocates caution rather than abandonment, arguing that guardrails are needed before LLMs can realize their potential in healthcare.This position responds to recommendations against healthcare use and to more cautious adoption proposals.Source: §I · p. 1
  • LLM hallucinations are contextually implausible, inconsistent with the real world, or unfaithful to the input, making them a central Responsible AI concern.The paper notes that some researchers prefer calling these outputs fabrications.Source: §I · p. 1

II. LLMS IN HEALTHCARE

  • Healthcare adoption is constrained by hallucinations and other risks because incorrect outputs may concern diagnosis, treatment, or medical procedures.Hospitals and healthcare organizations remain hesitant because wrong answers carry high cost and liability.Source: §I · p. 1, §II · p. 2
  • Healthcare LLMs support clinical workflow, translation, triage, research, and education, but performance varies across tasks and specialized applications can produce incorrect information.Domain-specific training using EHR or medical datasets is presented as one way to address specialized-task limitations.Source: §II · pp. 1–2
  • LLMs have shown strong performance on medical licensing examinations, yet multiple-choice benchmarks can create false certainty and poorly represent clinical practice.Real-world medicine may involve multiple plausible diagnoses rather than one fixed answer.Source: §II · p. 2

III. HALLUCINATIONS IN HEALTHCARE AI

Healthcare hallucinations can arise from unreliable or biased data, probabilistic generation, insufficient context, and weak sequential reasoning. These sources motivate systematic measurement, evaluation, and mitigation.

III. HALLUCINATIONS IN HEALTHCARE AI

  • Unreliable or biased training sources can perpetuate misconceptions and generate hallucinated healthcare information.The passages identify commonly held misconceptions and biased sources as contributing factors.Source: §III · p. 2
  • Because hallucination sources are diverse, healthcare systems must address them as a multi-factor problem rather than a single failure mode.The listed mechanisms span data quality, generation, prompting, and reasoning.Source: §III · p. 2
  • Probabilistic recombination can produce false statements even when the underlying texts are reliable.The source identifies probabilistic text generation as an independent hallucination mechanism.Source: §III · p. 2
  • Insufficient prompt context can produce text with little or no correspondence to the user’s intended need.The passage links missing context to mismatched generated content.Source: §III · p. 2
  • Weak sequential reasoning can cause LLMs to contradict themselves across generated statements.Self-contradiction is listed as a distinct source of hallucination.Source: §III · p. 2

IV. ADDRESSING THE PROBLEM OF HALLUCINATIONS

Healthcare hallucinations are serious obstacles to deploying LLMs, but the paper treats them as addressable through a staged process of measurement, evaluation, and mitigation.

IV. ADDRESSING THE PROBLEM OF HALLUCINATIONS

  • Model hallucinations create substantial impediments to using LLMs in healthcare.Source: §IV · p. 2
  • The paper frames these obstacles as insurmountable only if they are not systematically addressed.It explicitly states that the problems can be overcome.Source: §IV · p. 2
  • The proposed response is to measure, evaluate, and then mitigate hallucinations.Source: §IV · p. 2

A. Evaluating Hallucinations

The paper distinguishes evaluation with model access from evaluation based only on model outputs, and it combines self-consistency checks with interpretability and human or automated measurement. It also warns that standard multiple-choice benchmarks may not reflect clinical reasoning.

A. Evaluating Hallucinations

  • When model access is unavailable, sampling an LLM repeatedly for the same concept can reveal hallucinations through divergent or contradictory responses.With model access, organizations can instead examine generation likelihood against the training-corpus distribution.Source: A. Evaluating Hallucinations · p. 2
  • Healthcare evaluation should include interpretability and transparency because users may need to understand how an LLM reached an output.The paper presents asking the LLM how it reached a conclusion as one possible aid to judging correctness and interpretation.Source: A. Evaluating Hallucinations · p. 2
  • Multiple-choice benchmarks can create false certainty because clinical cases may support multiple plausible diagnoses rather than one answer.The paper uses Hickam’s Dictum to challenge the realism of single-diagnosis benchmark framing.Source: A. Evaluating Hallucinations · p. 2

B. Measuring Hallucinations

  • Hallucination measurement uses human evaluation and automated evaluation because open-ended LLM text does not fit traditional machine-learning metrics directly.Source: B. Measuring Hallucinations · p. 2

1) Human Evaluation:

Human evaluation assesses whether generated healthcare answers are truthful through annotation, while automatic evaluation uses annotation-derived methods or external models. The section also identifies evaluation approaches for comparing generated outputs.

1) Human Evaluation:

  • Human annotators evaluate the truth claims of generated answers, including through majority voting and fact-level assessment.Examples include healthcare answers for myopia care, maternity, diabetes, cancer, and infant care, while FActScores breaks generations into atomic facts for evaluation.Source: 1) Human Evaluation · p. 2

2) Automatic Evaluation:

  • Automatic evaluation can use human annotations or external models to score truthfulness and information alignment.Examples include determining truth claims, computing information alignment between texts, and using an external model in FactScore.Source: 2) Automatic Evaluation · p. 2
  • The listed evaluation approaches are intended to organize applications of LLMs in healthcare.Source: Table I · p. 3

3) Evaluation Metrics:

Healthcare hallucination mitigation is multi-pronged, while evaluation metrics provide indirect signals about output reliability. Human oversight is one complementary mitigation strategy for high-stakes use cases.

3) Evaluation Metrics:

  • Evaluation metrics vary by task, including precision, recall, F-Score, perplexity, cross-entropy loss, ROUGE, BLEU, and METEOR.These metrics do not directly measure hallucinations, but low scores can indicate that an output is likely hallucinated.Source: 3) Evaluation Metrics · p. 3

C. Mitigating Hallucinations

1) Human-In-The-Loop (HITL):

  • Domain experts can annotate data, classify inputs, audit outputs, correct responses, and supervise models through human-in-the-loop processes.For high-stakes healthcare uses, real-time supervision may be necessary but is not scalable for checking thousands of outputs.Source: 1) Human-In-The-Loop (HITL) · p. 3

2) Algorithmic Corrections:

Algorithmic corrections and fine-tuning are proposed ways to reduce healthcare hallucinations, but fine-tuning is costly and does not guarantee better performance.

2) Algorithmic Corrections:

3) Fine Tuning:

  • Fine-tuning adapts an LLM to specific healthcare tasks or domains, potentially reducing invented health information.The approach can be computationally and financially expensive.Source: 3) Fine Tuning · p. 3
  • Fine-tuning does not guarantee improved performance, with contrary examples reported in the literature [39].Source: 3) Fine Tuning · p. 3

4) Improving Prompts:

Prompt-based uncertainty handling, adversarial training, and input validation are presented as complementary ways to reduce hallucination and attack risks.

4) Improving Prompts:

D. Adversarial Training

E. Input Validation

F. Memory Augmentation

The paper discusses memory augmentation, model choice, and benchmark auditing as factors affecting trustworthy healthcare LLM evaluation. It emphasizes that benchmark validity must be scrutinized because flawed data can amplify hallucinations.

F. Memory Augmentation

  • External knowledge can be encoded into key-value memory for integration with an LLM, but this approach has mainly been demonstrated on smaller models.Its applicability to larger LLMs remains unexplored because of substantial memory requirements.Source: F. Memory Augmentation · p. 3

G. Model Choice

  • ChatGPT and GPT-4 are better than Vicuna-13B at detecting self-contradictions, illustrating that hallucination-related performance varies across models.Source: G. Model Choice · p. 3

H. Benchmark Audits

  • Benchmark-based evaluation assumes benchmark veracity, yet incorrect information can itself induce hallucinations and distort the data distribution.Healthcare benchmarks should therefore be verified by human domain experts, and model-generated training content should receive scrutiny from multiple experts.Source: H. Benchmark Audits · pp. 3–4

V. CONCLUSION

Healthcare LLMs can support diagnosis and other clinical tasks, but hallucinations make cautious validation essential. The paper favors human oversight for high-stakes uses, guarded automation for lower-stakes tasks, and attention to regulatory issues.

V. CONCLUSION

  • Healthcare LLMs can analyze symptoms, predict diseases, suggest treatments, and support personalized medicine, but their plausible-sounding outputs may be incorrect or misleading.Source: §V · p. 4
  • Human-in-the-loop development and validation may remain necessary for high-risk healthcare tasks, while guarded automation may be achievable for low-stakes tasks.Source: §V · p. 4
  • Narrow multiple-choice medical benchmarks can create false certainty, and widespread healthcare adoption may also face regulatory issues.Source: §V · p. 4
Loading 2311.01463v1…