PaperlayerView exact revision on arXiv

Source-linked AI summary

ClinHallu: A Benchmark for Diagnosing Stage-Wise Hallucinations in Medical MLLM Reasoning

Sicheng Yang, Hangjie Yuan, Wenjun Zhang, Jinwang Wang, Yichen Qian, Weihua Chen, Fan Wang, Lei Zhu

arXiv:2606.14697v1cs.CVcs.AIcs.CL

TL;DR

Medical hallucination benchmarks often identify whether an answer is wrong without locating where the reasoning failed. ClinHallu addresses this gap with stage-wise traces and interventions, showing that bottlenecks vary across datasets and that trace-supervised fine-tuning improves accuracy while reducing stage-wise hallucinations.

  • Problem

    Final-answer accuracy cannot localize whether medical MLLM failures arise from visual recognition, knowledge recall, or reasoning integration.

  • Method

    ClinHallu is a benchmark of 7,031 validated medical VQA instances with structured traces and stage-replacement interventions across three reasoning stages.

  • Results

    Hallucination bottlenecks vary across datasets, while full trace-supervised fine-tuning improves answer accuracy and reduces stage-wise hallucinations.

  • Takeaways & Limitations

    ClinHallu offers a fine-grained testbed for understanding and mitigating hallucinations in medical MLLMs.

  • Takeaways & Limitations

    ClinHallu currently covers medical VQA-style tasks rather than long-form report generation or real-world clinical decision support.

Abstract

from arXiv · show

Building trustworthy medical multimodal large language models (MLLMs) is critical for reliable clinical decision support. Existing medical hallucination benchmarks mainly focus on data collection, but often ignore where hallucinations originate within the reasoning process. We find that hallucination sources vary across samples: errors may arise from visual misrecognition, incorrect medical knowledge recall, or flawed reasoning integration. To enable source-level hallucination diagnosis, we introduce ClinHallu, a benchmark for stage-wise hallucination diagnosis in medical MLLM reasoning. ClinHallu contains 7,031 validated instances, where each instance is augmented with a structured reasoning trace decomposed into Visual Recognition, Knowledge Recall, and Reasoning Integration. We also use stage-replacement interventions to measure how correcting specific stages affects the final answer. Beyond evaluation, we show that trace-supervised fine-tuning reduces stage-wise hallucinations. ClinHallu provides a fine-grained hallucination testbed for diagnosing and mitigating reasoning failures in medical MLLMs. The benchmark is publicly available at https://github.com/alibaba-damo-academy/ClinHallu.

1 Introduction

Medical MLLMs support VQA, report generation, and clinical decision support, but plausible confident errors can arise from distinct trace-level failures that final-answer benchmarks cannot localize. ClinHallu addresses this gap with 7,031 validated medical VQA instances, structured stage traces, replacement interventions, and analyses of stage-wise hallucinations and mitigation.

  • Motivation: Medical MLLMs are increasingly used in medical VQA, report generation, and clinical decision support, creating high demands on reliability.These applications include medical visual question answering, report generation, and clinical decision support.
  • Motivation: A seemingly plausible but incorrect response may describe a non-existent lesion, attach an incorrect clinical implication, and remain confident.This illustrates why reliable medical use requires more than judging surface plausibility.
  • Motivation: The same wrong answer can result from visual misrecognition, incorrect medical knowledge recall, or failed evidence–knowledge integration, which final-answer judgments collapse together.Consequently, existing evaluations have limited ability to diagnose where hallucinations originate and how they propagate.
  • ClinHallu: ClinHallu contains 7,031 validated medical VQA instances with reference traces decomposed into Visual Recognition, Knowledge Recall, and Reasoning Integration.The benchmark is constructed from four medical VQA datasets and augments each instance with structured, validated traces.
  • ClinHallu: Stage-replacement interventions test how correcting specific reasoning stages affects final-answer correctness and reveal stage-dependent failure patterns and error propagation.The evaluation identifies which reasoning stage limits final-answer correctness and evaluates 11 representative closed-source and open-source MLLMs.
  • ClinHallu: ClinHallu supports fine-grained bottleneck analysis across datasets and models, while annotated structured traces can supervise reductions in stage-wise hallucinations.The benchmark is presented as a diagnostic testbed and a basis for mitigating reasoning failures.

2 Related Work

Medical MLLMs adapt multimodal reasoning to specialized clinical scenarios, while existing hallucination benchmarks mainly evaluate whether outputs are wrong rather than where errors originate. ClinHallu addresses this gap with structured reasoning traces for stage-wise hallucination diagnosis.

  • Reasoning in medical MLLMs: Medical MLLMs build on general-purpose systems to support visual question answering, report understanding, and clinical decision support in specialized medical scenarios.Examples of general-purpose foundations include GPT-4V, Gemini, LLaVA, and Qwen-VL; Med-Gemma is cited as a medical variant.
  • Reasoning in medical MLLMs: Hallucinations may arise from incorrect visual evidence, inaccurate medical knowledge, or unsupported conclusions, making evaluation essential for trustworthy medical MLLMs.These distinct failure sources can produce the same incorrect final answer.
  • Medical hallucination benchmarks: Text-only and multimodal benchmarks have rapidly expanded, primarily targeting hallucination detection in medical question answering, healthcare queries, and clinical knowledge assessment.Representative text-only benchmarks include Med-HALT, MedHalu, and MedHallu; multimodal examples include CARES, Med-HallMark, MedVH, and MedHallBench.
  • Medical hallucination benchmarks: Existing medical hallucination benchmarks remain largely answer-centric, identifying hallucinated outputs while providing limited insight into their underlying sources.ClinHallu is introduced to diagnose both whether hallucination occurs and where it originates using structured reasoning traces.
  • Medical hallucination benchmarks: ClinHallu uniquely supports structured CoT annotations, stage-wise traces, source localization, and hallucination-rate evaluation for fine-grained medical MLLM diagnosis.These capabilities are presented as differentiators from representative medical hallucination benchmarks.

3 ClinHallu Benchmark

ClinHallu is a benchmark that augments medical VQA samples with validated three-stage reasoning traces to localize hallucinations in multimodal language-model reasoning. It integrates four medical VQA datasets and retains 7,031 validated instances with structured annotations for stage-wise analysis.

  • Benchmark design: ClinHallu augments each medical VQA sample with a validated structured reasoning trace and evaluates generated traces alongside final answers.Comparing generated and reference traces enables hallucination localization beyond conventional final-answer correctness.
  • Source data: ClinHallu integrates VQA-RAD, PathVQA, MedFrameQA, and MedXpertQA, covering complementary medical domains, imaging modalities, and task formulations.These four datasets provide the benchmark’s source data.
  • Structured reasoning traces: Each trace decomposes reasoning into Visual Recognition, Knowledge Recall, and Reasoning Integration.The decomposition separates visual evidence, medical knowledge, and their integration for stage-wise hallucination analysis.
  • Trace validation and filtering: Retained traces must satisfy format validity and answer consistency, ensuring complete, non-empty, and ground-truth-consistent three-stage references.An LLM-as-judge evaluates both criteria, and traces are retained only when both are satisfied.
  • Released benchmark instances: 7,031 validated VQA instances comprise DClinHallu, each containing the original multimodal sample and a three-stage reference trace.The benchmark supports stage-wise hallucination analysis and combines multimodal inputs, structured CoT annotations, and source localization.

4 Evaluation

ClinHallu evaluates both final-answer correctness and stage-specific hallucinations in Visual Recognition, Knowledge Recall, and Reasoning Integration. Stage-replacement interventions isolate upstream errors and quantify their effects on hallucination rates and answer accuracy.

  • Answer-level evaluation: ClinHallu compares each predicted answer with the ground truth to assign binary correctness and compute answer-level accuracy.A correctness label c_i=1 denotes a correct answer, while c_i=0 denotes an incorrect answer.
  • Stage-wise diagnosis: Stage-wise evaluation localizes failures across visual recognition, knowledge recall, and reasoning integration, which final-answer accuracy alone cannot identify.The benchmark reports hallucination rates for each reasoning stage under controlled intervention contexts.
  • Stage replacement intervention: Stage replacement fixes selected generated stages with reference stages before the model completes remaining reasoning and produces a new answer.ClinHallu evaluates visual-stage, knowledge-stage, and joint visual-and-knowledge replacements.
  • Stage-wise diagnosis: Upstream-controlled hallucination rates directly measure each stage’s hallucination tendency, while replacement-induced accuracy gains diagnose bottleneck stages.A larger Δ_sAcc indicates that correcting the corresponding stage produces a greater improvement in final-answer correctness.
  • Trace-supervised fine-tuning: Trace-supervised fine-tuning uses golden traces to train Qwen3.5-9B on VQA-RAD and PathVQA, whose test sets are evaluated in ClinHallu.MedFrameQA and MedXpertQA are excluded because they lack training sets.

5 Experiments

Experiments across diverse medical MLLMs show that hallucination bottlenecks are dataset-dependent and usually originate in visual grounding or knowledge recall rather than reasoning integration. Stage replacement improves performance, trace-supervised fine-tuning mitigates stage-wise hallucinations, and the automatic judge closely matches human evaluations.

  • Finding 1: Visual hallucination exceeds 40% on average across subsets; VQA-RAD is visual-bottlenecked, MedXpertQA is knowledge-bottlenecked, while PathVQA and MedFrameQA are more balanced.VQA-RAD has 42.9% visual versus 13.7% knowledge hallucination, whereas MedXpertQA reaches 43.1% knowledge hallucination.
  • Finding 2: Replacing visual and/or knowledge stages improves accuracy, but the gains depend on the dataset.Average gains alone cannot distinguish corrected wrong answers from originally correct answers that become incorrect, motivating Fix and Break rates.
  • Finding 2: 73% versus 29% of wrong VQA-RAD answers are fixed by Rep-V versus Rep-K, while MedXpertQA shows 64.0% versus 42% Fix Rates and 10% versus 22% Break Rates for Rep-K versus Rep-V.PathVQA and MedFrameQA exhibit more balanced Fix/Break patterns, consistent with mixed visual and knowledge failure sources.
  • Finding 3: Reasoning-stage hallucination is generally lower than visual- and knowledge-stage hallucinations, and Rep-VK usually produces the largest accuracy gains.MedGemma-4B is an exception, partly because of frequent failed or incomplete chain-of-thought generation.
  • Finding 4: Trace-supervised fine-tuning is evaluated against no fine-tuning and answer-only fine-tuning using visual, knowledge, combined, or full-trace supervision.The ablation uses Qwen3.5-9B, constructs training sets from VQA-RAD and PathVQA, and reports answer accuracy alongside stage-wise hallucination rates.
  • Human evaluation: The automatic judge achieves 94.0% agreement and Cohen’s κ of 0.872 for answer correctness, while stage-wise labels reach 89.7–91.7% agreement and κ values of 0.785–0.831.Human-human agreement for answer correctness is 96.2%, with κ of 0.919, supporting the judge for large-scale evaluation.

6 Conclusion

ClinHallu is a benchmark that localizes hallucination sources in medical multimodal reasoning through structured traces spanning three reasoning stages. Experiments show that hallucination bottlenecks vary across datasets, while trace-supervised fine-tuning improves answer accuracy and reduces stage-wise hallucinations.

  • Benchmark contribution: ClinHallu augments medical VQA instances with structured reference traces covering Visual Recognition, Knowledge Recall, and Reasoning Integration.These traces enable hallucination sources to be localized within the reasoning process.
  • Empirical findings: Hallucination bottlenecks vary across datasets, according to experiments on ClinHallu.This highlights the need for stage-wise rather than answer-centric evaluation.
  • Mitigation: Full trace-supervised fine-tuning improves answer accuracy and reduces stage-wise hallucinations.The structured traces therefore provide effective supervision for mitigating reasoning failures.
  • Overall significance: ClinHallu offers a fine-grained testbed for understanding and mitigating hallucinations in medical MLLMs.The benchmark supports development of more reliable medical multimodal systems.

7 Limitations · Appendix of ClinHallu

ClinHallu currently targets medical VQA-style tasks as a controlled initial testbed, rather than long-form report generation or real-world clinical decision support. Future work will extend the benchmark to broader medical reasoning scenarios.

  • 7 Limitations: ClinHallu currently focuses on medical VQA-style tasks.The benchmark uses VQA as its initial setting.
  • 7 Limitations: The benchmark does not cover long-form report generation.This is identified as a current limitation of ClinHallu.
  • 7 Limitations: ClinHallu does not yet cover real-world clinical decision-support scenarios.The paper distinguishes these scenarios from its current VQA-focused scope.
  • 7 Limitations: VQA provides ClinHallu with a controlled testbed for its initial benchmark setting.The authors chose VQA specifically because it enables controlled evaluation.
  • 7 Limitations: The paper identifies the benchmark’s current scope as a limitation.Its present focus excludes broader medical reasoning scenarios.
  • Appendix of ClinHallu: Future work will extend ClinHallu to broader medical reasoning scenarios.This planned expansion addresses the benchmark’s current task coverage limitation.

A Fine-Tuning Configuration … D Use of AI Assistants

The paper fine-tunes Qwen3.5-9B on curated traces, illustrates coupled visual-knowledge failures through stage replacement, documents prompt templates, and states that AI assistance was limited to reviewed English polishing.

  • A Fine-Tuning Configuration: 1,221 VQA-RAD and 10,187 PathVQA samples were curated with faithful structured traces for trace-supervised fine-tuning.Only VQA-RAD and PathVQA provided official training splits.
  • A Fine-Tuning Configuration: Qwen3.5-9B was fine-tuned with LoRA using rank r = 8, scaling factor α = 16, and a 1 × 10−4 initial learning rate.Training used LLaMA-Factory, cosine scheduling, and a warmup ratio of 0.1; other settings followed defaults.
  • B Case Study: The model misidentified an “AIIS avulsion fracture” as a “femoral neck fracture” and consequently answered “gluteus medius” incorrectly.This case study illustrates stage-replacement behavior.
  • B Case Study: Replacing knowledge recall alone was insufficient, whereas replacing both visual recognition and knowledge recall enabled the correct inference “rectus femoris”.The result reveals a coupled visual-knowledge failure hidden by answer-level evaluation alone.
  • C Prompt Templates: Prompt templates were organized around benchmark construction, stage replacement, and automatic judging within the ClinHallu pipeline.The complete templates are provided in Figs. 7–13.
  • C Prompt Templates: The complete prompt templates for the ClinHallu pipeline are provided in Figs. 7–13.The templates cover benchmark construction, stage replacement, and automatic judging.
  • D Use of AI Assistants: AI was used to assist with English writing polish, while the scientific content, experimental design, and conclusions remained solely the authors’ own.No AI-generated text was used without human review and revision.
  • D Use of AI Assistants: No AI-generated text was used without human review and revision.The authors state that AI assistance was restricted to English writing polish.

E Datasets and Licenses

ClinHallu uses four publicly available medical VQA datasets under their respective licenses and follows associated attribution requirements. The benchmark contains no newly collected patient data or directly identifying patient information.

  • Data source: ClinHallu uses VQA-RAD, PathVQA, MedXpertQA, and MedFrameQA, released under CC0 1.0, MIT, MIT, and CC BY 4.0 licenses, respectively.All datasets are used for research purposes, with license terms and attribution requirements followed.
  • Privacy and content screening: The benchmark uses only released images, questions, answers, and metadata, without collecting new patient data or personally identifying information.The curated benchmark does not directly identify individual patients.
  • Ethics review: Human evaluation annotated model outputs and benchmark instances for research validation without collecting sensitive personal information from annotators.The study relies on publicly available benchmark datasets and does not collect new patient data.

F Human Evaluation Details

The human evaluation was conducted independently by two medically trained annotators from the authors’ group, who were compensated through regular research appointments and informed that their annotations would be reported.

  • Annotators: Two annotators with medical backgrounds conducted the human evaluation.Both annotators were recruited from the authors’ group.
  • Compensation: The annotators were compensated as part of their regular research appointments.
  • Evaluation procedure: Each annotator performed the evaluation independently and was informed that the annotations would be reported in the paper.

G Intended Use and Risk Analysis · [REP-VK PROMPT]

ClinHallu is intended for controlled research on medical MLLM evaluation, hallucination diagnosis, and model analysis, not direct patient-care decisions. Its risk analysis addresses over-interpreting benchmark gains as clinical reliability through stage-wise hallucination reporting and failure-source analysis, while the pipeline includes structured reasoning and replacement-prompt templates.

  • G Intended Use and Risk Analysis: ClinHallu is designed to evaluate, diagnose hallucinations, and analyze medical MLLMs under controlled benchmark settings.Its stated use is to measure and compare model behavior in research contexts.
  • G Intended Use and Risk Analysis: ClinHallu is not intended for direct patient-care decision making.
  • G Intended Use and Risk Analysis: To mitigate the risk of overstating clinical reliability, the authors report stage-wise hallucination rates alongside answer accuracy and analyze failure sources.The stated concern is that benchmark improvements may be over-interpreted as clinical reliability.
  • G Intended Use and Risk Analysis: The reference-trace construction prompt instructs an expert medical professional to analyze a medical image or question with structured Chain-of-Thought reasoning.The supplied material identifies this as a reference trace construction prompt for the ClinHallu pipeline.
  • G Intended Use and Risk Analysis: The pipeline uses prompt templates for reference-trace construction and trace validation.The supplied passages identify separate construction and validation prompts, with accompanying prompt-template figures.
  • [REP-VK PROMPT]: The REP-VK prompt continues structured Chain-of-Thought reasoning for a medical question.The subsection also contains figures labeled as prompt templates used in the ClinHallu pipeline.
Loading 2606.14697v1…